Debugging EC2 Termination During Terraform Apply: The Hidden AMI KMS Trap

When working with Terraform to automate AWS infrastructure, you expect things to be smooth until they’re not. Recently, I hit a frustrating issue where an EC2 instance kept getting created and immediately terminated during terraform apply. The error message?

Client.InvalidKMSKey.InvalidState: The KMS key provided is in an incorrect state

It took a bit of digging, but the root cause turned out to be something that’s easy to overlook: the AMI I was using was pre-encrypted with a different KMS key.

Let me walk you through the issue and how I fixed it.

The Setup

I had a VPC deployed using Terraform and was spinning up multiple EC2 instances, all encrypted using a customer-managed KMS key (CMK).

Here’s a simplified version of my EC2 block:

resource "aws_instance" "app3" {
  ami                    = var.app_ami_id
  instance_type          = "t3.micro"
  subnet_id              = aws_subnet.public.id
  iam_instance_profile   = aws_iam_instance_profile.ec2_profile.name

  root_block_device {
    volume_size = 20
    volume_type = "gp3"
    encrypted   = true
    kms_key_id  = var.kms_key_id
  }
}

This same setup worked for two other EC2 machines but the third one (app3) kept failing.

The Error

Terraform would show the instance as being created, but AWS would instantly terminate it. The logs showed:

waiting for EC2 Instance (i-xxxxxxxxxxxxx) create: unexpected state 'terminated', wanted target 'running'
last error: Client.InvalidKMSKey.InvalidState

Confusing, right? Especially because the same KMS key worked just fine on other instances.

The Root Cause

After ruling out IAM issues, VPC networking, and user data scripts, I compared everything between the working and failing instances and found the problem:

The AMI used for the bastion instance was already encrypted with a different KMS key.

In AWS, when you launch an EC2 instance with a root volume that’s based on an encrypted AMI, you can’t override the encryption KMS key unless the source AMI explicitly allows it (which many shared/public AMIs don’t).

So when Terraform tried to create the instance with kms_key_id = var.kms_key_id, AWS rejected the conflicting encryption and the instance terminated.

The Fix

The solution was simple once I identified the issue:

• I replaced the AMI with one that was either:
  • Not encrypted, so I could apply my own KMS key
  • Or encrypted with my KMS key

After updating the ami in my variables, Terraform applied successfully, and the instance stayed alive

Lessons Learned

• Always check whether your AMIs are pre-encrypted, especially when using your own KMS keys in Terraform.
• You can inspect an AMI using:

ec2 describe-images --image-ids ami-xxxxxxxx --query 'Images[*].{Encrypted:BlockDeviceMappings[0].Ebs.Encrypted, KmsKeyId:BlockDeviceMappings[0].Ebs.KmsKeyId}'

• If an AMI is encrypted with a key you don’t control, you likely won’t be able to override its KMS key at launch time.

Pro Tip

To avoid similar issues in the future:

• Stick to standard, unencrypted AMIs and encrypt via your Terraform config
• Or build your own AMIs with Packer so you know exactly what’s inside
• Always review AMI encryption details when working in sensitive environments

Have you run into similar silent-failure issues in Terraform? This one had me scratching my head for longer than I’d like to admit but hopefully, this saves you the time I spent chasing ghosts!