IAM instance profile <name> EntityAlreadyExists, but it actually didn't.

[ghost]

This issue was originally opened by @dpedu as hashicorp/terraform#24177. It was migrated here as a result of the provider split. The original body of the issue is below.

---

Terraform Version

v0.12.12

Terraform Configuration Files

resource "aws_vpc" "vpc" {
...
}

resource "aws_iam_instance_profile" "cfnprovision" {
  name = "${var.vpc_name}-CFNInstanceProfile"
  role = aws_iam_role.cfnprovision.id
}

resource "aws_iam_role" "cfnprovision" {
  name               = "${var.vpc_name}-CFNRoleEc2Metadata"
  assume_role_policy = <<EOF
{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Action": "sts:AssumeRole",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Effect": "Allow",
            "Sid": "AllowAssumeRole"
        }
    ]
}
  EOF
}

Debug Output

I suspect this is a timing issue and I was not running Terraform with TF_LOG at the time. Because of changes made to the platform account I was using terraform under, I can't attempt to reproduce this.

Expected Behavior

Terraform successfully persists resources it created to state.

Actual Behavior

Terraform outputs Error creating IAM instance profile [profile name]: EntityAlreadyExists: Instance Profile [profile name] already exists. because it created the resource but did not persist it to its state.

Steps to Reproduce

1. Attempt to apply the above code.
2. Hit a limits error during deployment of the VPC.
3. Attempt to apply again.

Additional Context

I was running this code on 5 separate hosts in parallel with different parameters that would prevent intentional collisions due to naming. The outcome of each is described below. The numbers don't indicate execution order.

Each apply was creating an AWS VPC along with other stuff (roles, permissions, dns, etc). AWS limits accounts to 10 VPCs by default, trying to create more results in an error.

1. Hit a VPC limits error. Raised limits and continued deployment to completion without error.
2. Hit a VPC limits error. After limit raise, plan/apply hit EntityAlreadyExists.
3. Hit a VPC limits error. After limit raise, plan/apply hit EntityAlreadyExists.
4. Hit a VPC limits error. After limit raise, plan/apply hit EntityAlreadyExists.
5. Did not hit VPC limits error. Deployment continued to completion without error.

For each with an EntityAlreadyExists I manually check the state file and AWS. In all cases, the resource had been created in AWS but was missing from the state file.

Looking at an instance that hit problem 2, 3, or 4 above shows the following series of events:

1. terraform plan -out somefile
2. terraform apply somefile
3. Terraform prints messages indicating it is creating the resources printed above. It specifically mentioned the names of resources shown in my snippet above. Here is one such message: module.vpc_common.module.role_provisioning.aws_iam_role.cfnprovision: Creation complete after 1s [id=role-name-censored].
4. Terraform prints an error that a VPC limit was hit and exits with status 1.
5. VPC limit is raised in the target account
6. terraform apply (no plan file this time) hits the EntityAlreadyExists error. Here is the exact error mentioning the role name printed in the previous execution:

Error: Error creating IAM Role role-name-censored: EntityAlreadyExists: Role with name role-name-censored already exists.
	status code: 409, request id: xxxxxxxxxxxxxxxxxxxxxxx

  on ../../modules/role_provisioning/main.tf line 18, in resource "aws_iam_role" "cfnprovision":
  18: resource "aws_iam_role" "cfnprovision" {

This looks and smells like the resource already existed prior to using Terraform. I am confident that this is not the case.

I was unable to workaround this issue by importing the orphaned resource due to #8040.

[dpedu]

I don't think this is a provider issue. Everything that was supposed to happen in aws happened correctly. The problem was with Terraform persisting this data to its state which is NOT handled in the aws provider.

[kookliu]

I encountered the same error. But the actual problem is this.

Terraform v0.12.23
\+ provider.aws v2.53.0

....
aws_iam_instance_profile.k8s-node: Creating...
aws_iam_instance_profile.k8s-master: Creating...

Error: Error creating IAM instance profile k8s-master: EntityAlreadyExists: Instance Profile k8s-master already exists.
status code: 409, request id: 9687303b-fe7c-4754-8700-45bd66063572

on iam.tf line 133, in resource "aws_iam_instance_profile" "k8s-master":
133: resource "aws_iam_instance_profile" "k8s-master" {

Error: Error creating IAM instance profile k8s-node: EntityAlreadyExists: Instance Profile k8s-node already exists.
status code: 409, request id: 78d76002-61b6-49a2-adc6-5917bd870f87

on iam.tf line 138, in resource "aws_iam_instance_profile" "k8s-node":
138: resource "aws_iam_instance_profile" "k8s-node" {

resource "aws_iam_role" "k8s_master_role" {
name = "k8s_master"

assume_role_policy = <<-EOF
...
EOF
tags = {
Name = "k8s_master_role"
}
}

resource "aws_iam_role_policy" "k8s_master" {
name = "k8s-master-policy"
role = aws_iam_role.k8s_master_role.id
policy = <<-EOF
......
EOF
}

resource "aws_iam_instance_profile" "k8s-master" {
name = "k8s-master"
role = aws_iam_role.k8s_master_role.name
}

The same problem occurs in the above configuration.

olny change aws_iam_instance_profile name. As long as it is different from aws_iam_role name. name = "k8s-master" to name = "k8s-master-profile" , the problem resolve.
.......
.......
resource "aws_iam_instance_profile" "k8s-master" {
name = "k8s-master-profile"
role = aws_iam_role.k8s_master_role.name
}

[scott-kausler]

I encountered this problem with aws_iam_role. Seems to be like we most likely hit some sort of timeout. If the timeout is hit, we retry creating the role one last time. I suspect I hit it on that timeout. https://github.com/terraform-providers/terraform-provider-aws/blob/98b5f7be6cd1ebca2bd5749135e0bd0c305418c1/aws/resource_aws_iam_role.go#L170

[justinretzolk]

Hi all 👋 Thank you for taking the time to file this issue, and for the continued discussion! Given that there's been a number of AWS provider releases since this was filed, is anyone still encountering this behavior?

[rymancl]

Hi all 👋 Thank you for taking the time to file this issue, and for the continued discussion! Given that there's been a number of AWS provider releases since this was filed, is anyone still encountering this behavior?

I still run into this as of 3.68.0 with aws_iam_role when making a change that requires re-creation of the role (for example, changing the resource identifier). I had a plan that destroyed a role, policy, and 7 policy attachments in order to re-add them.

╷
│ Error: error creating IAM Role (my-role): EntityAlreadyExists: Role with name my-role already exists.
│ status code: 409, request id: xxxxx
│
│ with module.my-module.aws_iam_role.my-role,
│ on my-module/main.tf line 23, in resource "aws_iam_role" "my-role":
│ 23: resource "aws_iam_role" "my-role" {
│

The role doesn't actually exist.

Running the plan again shows adding all the missing resources. Apply works fine of course.

For me, this is 100% repeatable. It happens every time in my testing.
My initial suspicion was that this was caused by the large number of IAM roles we have in our account (2200+), but I'm not sure.

[maolopez]

I am having this issue with hashicorp/aws v4.8.0. Terrafor version 1.1.7
I have a gitlab runner in the AWS account i plan to deploy.
Terraform plan is successful.
Terraform apply fails with one error. Running terraform apply again fails with multiple errors EntityAlreadyExists.
First Error: error creating Flow Log (vpc-007f21843c93e35ee): InvalidParameter: LogDestination can't be empty if LogGroupName is not provided.
│ status code: 400. (LogGroupName is indeed provided)
Second Error after running terraform apply again: Error: Creating CloudWatch Log Group failed: ResourceAlreadyExistsException: The specified log group already exists:

Error: failed creating IAM Role (vpc-flow-logs-main.us-east-1): EntityAlreadyExists: Role with name vpc-flow-logs-main.us-east-1 already exists.

Error: error creating IAM Policy main_flow_logs_policy.us-east-1: EntityAlreadyExists: A policy called main_flow_logs_policy.us-east-1 already exists. Duplicate names are not allowed.

Please advise how to persisting terraform data into its state?

[tath81]

I'm also running into this issue on version = "4.67.0"
An import of the role and profile, followed by another TF apply works. But this is definitely an issue which i'm also encountering.

[tath81]

What I noticed with the aws_iam_instance_profile resource I'm using, I have the depends_on as shown below which may have caused the issue.

# Create Instance Profile Role
resource "aws_iam_instance_profile" "this" {
  name = var.name
  role = aws_iam_role.this.name

  depends_on = [
    aws_iam_role.this,
    aws_iam_role_policy_attachment.this
  ]
}

After removing this and adding the depends_on to the aws_iam_role_policy_attachment as shown below, this resolved the issue.

# Attach policy to Role
resource "aws_iam_role_policy_attachment" "this" {
  count = length(var.additional_instance_policies)

  role       = aws_iam_role.this.name
  policy_arn = var.additional_instance_policies[count.index]

  depends_on = [
  aws_iam_role.this
  ]

[YakDriver]

As maintainers of the Terraform AWS Provider, we’ve reached a decision to close this longstanding issue. We want to assure you that this decision was made after careful consideration, and we’re committed to transparency in our actions.

At this time, I cannot reproduce the issue which may or may not mean it has been fixed in the interim. We lack clarity on how many users are still affected and the precise nature of the remaining issues. Given these uncertainties and our limited resources, it’s difficult for us to effectively address the problem in its current state.

However, we value community feedback immensely. If you’re still encountering issues, we encourage you to open a new, focused issue outlining the specific problems you’re facing. We especially need a minimal reproduction configuration. We understand the frustration of having to restart the discussion, but the long history of this particular issue necessitates a fresh approach.

While we’ve received reports from community members in the past year, these are no directly related to continuing problems. Moving forward, a new, well-defined problem statement will greatly increase the likelihood of prompt attention from maintainers or fellow community members.

Ultimately, our goal is to ensure that the Terraform AWS Provider remains a dependable tool for realizing your infrastructure goals. Regrettably, this prolonged issue no longer contributes to that objective. By closing it, we aim to clear the path for more effective problem-solving and a smoother experience for all users. We appreciate your understanding and continued support as we work towards a better future for your provider.

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.