IAM instance profile <name> EntityAlreadyExists, but it actually didn't.

[dpedu]

Terraform Version

v0.12.12

Terraform Configuration Files

resource "aws_vpc" "vpc" {
...
}

resource "aws_iam_instance_profile" "cfnprovision" {
  name = "${var.vpc_name}-CFNInstanceProfile"
  role = aws_iam_role.cfnprovision.id
}

resource "aws_iam_role" "cfnprovision" {
  name               = "${var.vpc_name}-CFNRoleEc2Metadata"
  assume_role_policy = <<EOF
{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Action": "sts:AssumeRole",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Effect": "Allow",
            "Sid": "AllowAssumeRole"
        }
    ]
}
  EOF
}

Debug Output

I suspect this is a timing issue and I was not running Terraform with TF_LOG at the time. Because of changes made to the platform account I was using terraform under, I can't attempt to reproduce this.

Expected Behavior

Terraform successfully persists resources it created to state.

Actual Behavior

Terraform outputs Error creating IAM instance profile [profile name]: EntityAlreadyExists: Instance Profile [profile name] already exists. because it created the resource but did not persist it to its state.

Steps to Reproduce

1. Attempt to apply the above code.
2. Hit a limits error during deployment of the VPC.
3. Attempt to apply again.

Additional Context

I was running this code on 5 separate hosts in parallel with different parameters that would prevent intentional collisions due to naming. The outcome of each is described below. The numbers don't indicate execution order.

Each apply was creating an AWS VPC along with other stuff (roles, permissions, dns, etc). AWS limits accounts to 10 VPCs by default, trying to create more results in an error.

1. Hit a VPC limits error. Raised limits and continued deployment to completion without error.
2. Hit a VPC limits error. After limit raise, plan/apply hit EntityAlreadyExists.
3. Hit a VPC limits error. After limit raise, plan/apply hit EntityAlreadyExists.
4. Hit a VPC limits error. After limit raise, plan/apply hit EntityAlreadyExists.
5. Did not hit VPC limits error. Deployment continued to completion without error.

For each with an EntityAlreadyExists I manually check the state file and AWS. In all cases, the resource had been created in AWS but was missing from the state file.

Looking at an instance that hit problem 2, 3, or 4 above shows the following series of events:

1. terraform plan -out somefile
2. terraform apply somefile
3. Terraform prints messages indicating it is creating the resources printed above. It specifically mentioned the names of resources shown in my snippet above. Here is one such message: module.vpc_common.module.role_provisioning.aws_iam_role.cfnprovision: Creation complete after 1s [id=role-name-censored].
4. Terraform prints an error that a VPC limit was hit and exits with status 1.
5. VPC limit is raised in the target account
6. terraform apply (no plan file this time) hits the EntityAlreadyExists error. Here is the exact error mentioning the role name printed in the previous execution:

Error: Error creating IAM Role role-name-censored: EntityAlreadyExists: Role with name role-name-censored already exists.
	status code: 409, request id: xxxxxxxxxxxxxxxxxxxxxxx

  on ../../modules/role_provisioning/main.tf line 18, in resource "aws_iam_role" "cfnprovision":
  18: resource "aws_iam_role" "cfnprovision" {

This looks and smells like the resource already existed prior to using Terraform. I am confident that this is not the case.

I was unable to workaround this issue by importing the orphaned resource due to hashicorp/terraform-provider-aws#8040.

[ghost]

This issue has been automatically migrated to hashicorp/terraform-provider-aws#12121 because it looks like an issue with that provider. If you believe this is not an issue with the provider, please reply to hashicorp/terraform-provider-aws#12121.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.