Perpetual diffs / Objects have changed outside of Terraform (v3.70.0+)

[YakDriver]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Only reports after v3.70.0

Many of these issues were fixed with a collection of changes shown in #21968, v3.70.0. The purpose of this issue is to separate the older reports that may or may not be fixed from problems seen in v3.70.0+.

Unfortunately, these differences can be caused by a variety of problems. To have any chance of tackling them, we need to organize exactly what's still a problem, and what exactly the problems are. The causes are nuanced so we need specificity.

Main Causes

1. Eventual consistency - AWS reports different things over time. We may need to wait for in certain places for things to settle down.
2. Eqivalent but not equal - AWS responds with different equivalent answers that the provider does not recognize as equivalent.
3. Response disconnect - Various disagreements between AWS and the Provider cause this:
   • Some arguments are not marked computed that should be
   • An AWS nil is interpretted as an empty string or vice versa
4. Defaults - AWS returns default values sometimes. The AWS Provider also has default values. Sometimes these match, sometimes they've changed.

Affected Services and Resources

1. aws_backup_plan - rule without schedule
2. aws_backup_region_settings - not_resources, resources, condition
3. aws_codeartifact_domain_permissions_policy - Principals order
4. aws_codeartifact_repository_permissions_policy - Principals order
5. aws_dms_endpoint - extra_connection_attributes conflicting with s3_settings
6. aws_iam_policy - Empty tags (also)
7. aws_iam_role - Order in assume_role_policy principals
8. aws_iam_role - Empty tags (also)
9. aws_iam_role - Empty inline_policy
10. aws_mq_configuration - data XML argument
11. aws_s3_bucket_policy - principals in policy
12. aws_sqs_queue - order in policy
13. aws_ssoadmin_permission_set_inline_policy - this seems different than many of the others because of actual substantive changes
14. aws_wafv2_web_acl - AWSManagedRulesAmazonIpReputationList

Provider References (Open)

• aws_iam_policy_document orders actions anti-alphabetically #6107
• "Objects changed outside of Terraform" shown wrongly for various resources #19727
• AWS provider 4.10.0 - Order is lost for Bucket policy #24338

Provider References (Closed)

• Order is lost for data aws_iam_policy_document when applied to S3 buckets, iam roles, kms keys, etc #11801
• s3 bucket policy with not_principal always changes due to ordering #13144
• Inconsistent ordering of values in codeartifact_repository_permissions_policy #17024
• Principals in aws_s3_bucket_policy should be ordered for plan #18878
• aws_iam_policy_document Bool condition always shows diff #19245
• aws_ecr_repository_policy tries to enforce order of principals when AWS insists on a different random ordering. #19365
• iam_role inline_policy: Issue importing iam role with inline policies wanting to be deleted and recreated when there is no difference #19444
• Terraform detects spurious bucket policy changes before each apply after upgrading to 0.15.4 #19627
• Unhelpful "changes made outside of terraform" due to AWS vs tf representation in 1.0 #20107
• S3 bucket policy condition block displays endless diff with values array of length 1 #20456
• IAM Policy Principals: Objects have changed outside of Terraform #20665
• Policy documents and other lists causing drift #20781

Other References

• Ignore JSON array order awspolicyequivalence#1
• Convert boolean and numeric values to strings for value comparison awspolicyequivalence#7
• https://github.com/hashicorp/awspolicyequivalence
• Classifying normalization vs. drift terraform-plugin-framework#70
• DiffSuppressFunc should be used for "Objects have changed outside of Terraform" logic terraform-plugin-sdk#801
• Policy elements that are syntactically lists but semantically sets should have their order ignored jen20/awspolicyequivalence#10
• aws_iam_policy_document applies reverse sort on resources list. terraform#22931
• Preserving object key order in jsonencode terraform#27880
• A way to hide certain expected changes from the "refresh" report ("Objects have changed outside of Terraform") terraform#28803
• Random and persistent rearrangements of objects within array of jsonencode function terraform#29685

[ewbankkit]

hashicorp/terraform-plugin-sdk#882, available in the next Terraform Plugin SDK v2 release, should offer a way forward on some of these.

[jhancock93]

I can confirm that I am still seeing issues with diffs on aws_iam_role assume_role_policy principals with provider 3.74.0.

[wuetz]

I can confirm that I am still seeing issues with diffs on aws_iam_role assume_role_policy principals with provider v4.5.0

[ad-m-ss]

I can confirm that I am still seeing issues with diffs on aws_iam_role assume_role_policy principals with provider v4.7.0.

[jcogilvie]

Echoing @jhancock93 with an example:

- Installing hashicorp/aws v3.75.1...

It detects this as drift that happened outside of terraform:

  # module....iam.aws_iam_role.engineer has been changed
!   resource "aws_iam_role" "engineer" {
!       assume_role_policy    = jsonencode(
!           {
!               Statement = [
!                   {
!                       Principal = {
!                           AWS = [
-                               "arn:aws:iam::136...:root",  # this isn't removed; just appears at the bottom of the list now.
                                "arn:aws:iam::119...:root",
+                               "arn:aws:iam::136...:root",
                            ]
                        }
                        # (3 unchanged elements hidden)
                    },
                ]
                # (1 unchanged element hidden)
            }
        )
        id                    = "engineer"
        name                  = "engineer"
        tags                  = {
            "ManagedBy" = "https://github.com/..."
            "Name"      = "engineer"
            "Shared"    = "True"
            "Team"      = "SRE"
        }
        # (8 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

[forty]

Terraform 1.1.7 Provider AWS 4.8.0

We still have order problem for aws aws_iam_role - Order in assume_role_policy

We also have the this one which is not listed:

aws_sqs_queue- Order in policy

[nitrocode]

Also seeing this with

• aws_codeartifact_domain_permissions_policy
• aws_codeartifact_repository_permissions_policy

Empty strings are empty because they are redacted

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_codeartifact_domain_permissions_policy.main[0] will be updated in-place
  ~ resource "aws_codeartifact_domain_permissions_policy" "main" {
        id              = "arn:aws:codeartifact:us-west-2:312851193143:domain/default"
      ~ policy_document = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      ~ Principal = {
                          ~ AWS = [
                              - "",
                              - "",
                              - "",
                              - "",
                              - "",
                              + "",
                                "",
                              - "",
                              + "",
                              + "",
                              + "",
                              + "",
                                "",
                              + "",
                              + "",
                              + "",
                                "",
                              - "",
                              - "",
                            ]

[ad-m-ss]

I have perpetual diff for aws_ssoadmin_permission_set_inline_policy.

Environment:

• Terraform v1.1.7 on darwin_arm64
• registry.terraform.io/hashicorp/aws version = "4.7.0"

Example output:

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_ssoadmin_permission_set_inline_policy.developer["1"] will be updated in-place
  ~ resource "aws_ssoadmin_permission_set_inline_policy" "developer" {
        id                 = "arn:aws:sso:::permissionSet/ssoins-6684bd07964ad0f4/ps-f7fa08c3ff7c1943,arn:aws:sso:::instance/ssoins-6684bd07964ad0f4"
      ~ inline_policy      = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      ~ Action   = [
                          - "logs:StartQuery",
                          - "logs:FilterLogEvents",
                          + "redshift:ViewQueriesFromConsole",
                          + "redshift:ListTables",
                          + "redshift:ListSchemas",
                          + "redshift:ListDatabases",
                          + "redshift:GetClusterCredentials",
                          + "redshift:FetchResults",
                          + "redshift:ExecuteQuery",
                          + "redshift:DescribeTable",
                          + "redshift:DescribeQuery",
                          + "redshift:DescribeClusters",
                          + "redshift:CancelQuery",
                        ]
                      ~ Resource = "arn:aws:logs:us-east-2::log-group:/aws/rds/instance/postgres-test-data/postgresql:*" -> "*"
                        # (2 unchanged elements hidden)
                    },
                  ~ {
                      ~ Action   = [
                          - "sns:List*",
                          - "sns:Get*",
                          + "redshift-data:ListTables",
                          + "redshift-data:ListSchemas",
                          + "redshift-data:ListDatabases",
                          + "redshift-data:ExecuteStatement",
                          + "redshift-data:DescribeTable",
                        ]
                      ~ Resource = "arn:aws:logs:us-east-2::log-group:/aws/rds/instance/postgres-test-data/postgresql:*" -> "*"
                        # (2 unchanged elements hidden)
                    },
                  ~ {
                      ~ Action   = [
                          - "cloudwatch:List*",
                          - "cloudwatch:Get*",
                          - "cloudwatch:Describe*",
                          + "redshift-data:ListStatements",
                          + "redshift-data:GetStatementResult",
                          + "redshift-data:DescribeStatement",
                          + "redshift-data:CancelStatement",
                        ]
                      ~ Resource = "arn:aws:logs:us-east-2::log-group:/aws/rds/instance/postgres-test-data/postgresql:*" -> "*"
                        # (2 unchanged elements hidden)
                    },
                  ~ {
                      ~ Action   = "autoscaling:Describe*" -> [
                          + "s3:ListMultipartUploadParts",
                          + "s3:ListBucketVersions",
                          + "s3:ListBucketMultipartUploads",
                          + "s3:ListBucket",
                          + "s3:GetReplicationConfiguration",
                          + "s3:GetObjectVersionTorrent",
                          + "s3:GetObjectVersionTagging",
                          + "s3:GetObjectVersionForReplication",
                          + "s3:GetObjectVersionAcl",
                          + "s3:GetObjectVersion",
                          + "s3:GetObjectTorrent",
                          + "s3:GetObjectTagging",
                          + "s3:GetObjectRetention",
                          + "s3:GetObjectLegalHold",
                          + "s3:GetObjectAcl",
                          + "s3:GetObject",
                          + "s3:GetMetricsConfiguration",
                          + "s3:GetLifecycleConfiguration",
                          + "s3:GetJobTagging",
                          + "s3:GetInventoryConfiguration",
                          + "s3:GetEncryptionConfiguration",
                          + "s3:GetBucketWebsite",
                          + "s3:GetBucketVersioning",
                          + "s3:GetBucketTagging",
                          + "s3:GetBucketRequestPayment",
                          + "s3:GetBucketPublicAccessBlock",
                          + "s3:GetBucketPolicyStatus",
                          + "s3:GetBucketPolicy",
                          + "s3:GetBucketOwnershipControls",
                          + "s3:GetBucketObjectLockConfiguration",
                          + "s3:GetBucketNotification",
                          + "s3:GetBucketLogging",
                          + "s3:GetBucketLocation",
                          + "s3:GetBucketCORS",
                          + "s3:GetBucketAcl",
                          + "s3:GetAnalyticsConfiguration",
                          + "s3:GetAccessPointPolicyStatus",
                          + "s3:GetAccessPointPolicy",
                          + "s3:GetAccelerateConfiguration",
                          + "s3:DescribeJob",
                        ]
                      ~ Resource = "arn:aws:logs:us-east-2::log-group:/aws/rds/instance/postgres-test-data/postgresql:*" -> [
                          + "arn:aws:s3:::select-star-audit-log-test/*",
                          + "arn:aws:s3:::select-star-audit-log-test",
                        ]
                        # (2 unchanged elements hidden)
                    },
                  - {
                      - Action   = [
                          - "logs:TestMetricFilter",
                          - "logs:StopQuery",
                          - "logs:List*",
                          - "logs:Get*",
                          - "logs:Describe*",
                        ]
                      - Effect   = "Allow"
                      - Resource = "arn:aws:logs:us-east-2:672751098944:log-group:/aws/rds/instance/postgres-test-data/postgresql:*"
                      - Sid      = ""
                    },
                  - {
                      - Action   = "logs:GetQueryResults"
                      - Effect   = "Allow"
                      - Resource = "arn:aws:logs:*:*:log-group::log-stream:"
                      - Sid      = ""
                    },
                ]
                # (1 unchanged element hidden)
            }
        )
        # (2 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

[jhenry82]

I am seeing a similar issue with tags on IAM role and policy resources. Is that the same thing being discussed here or should I open a new issue? Example:

resource "aws_iam_policy" "test" {
  name        = "test"
  description = "test"
}

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply":

# aws_iam_policy.test has changed
  ~ resource "aws_iam_policy" "test" {
        id          = "redacted"
        name        = "test"
      + tags        = {}
        # (6 unchanged attributes hidden)
    }

If I explicitly set tags = {} on the policy, I get the same result.

Provider: v4.21.0
Terraform: v1.2.3

[carltonshen-work]

Seeing related issue with change detection:

  # module.aws_iam_role.some_user_role will be updated in-place
~ resource "aws_iam_role" "some_user_role" {
      ~ assume_role_policy    = jsonencode(
            {
              - Statement = [
                  - {
                      - Action    = "sts:AssumeRoleWithSAML"
                      - Condition = {
                          - IpAddress    = {
                              - aws:SourceIp = [
                                  - "0.0.0.0/0",
                                ]
                            }
                          - StringEquals = {
                              - SAML:aud = "https://signin.aws.amazon.com/saml"
                            }
                        }
                      - Effect    = "Allow"
                      - Principal = {
                          - Federated = "arn:aws:iam::000000000000:saml-provider/provider"
                        }
                      - Sid       = ""
                    },
                ]
              - Version   = "2012-10-17"
            }
        ) -> (known after apply)
        id                    = "administrator"
        name                  = "administrator"
        # (8 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Some information redacted.

I just added a blank line in code and this update plan will be generated among a lot other similar ones, and apply them won't make any actual change. Add another blank line will trigger the same plan.
We did some work to pinpoint the issue on Condition.IpAddress that if we don't put anything there it is fine.

[nestorcolt]

I have the problem with empty + tags = {} from aws_iam_policy and aws_iam_role

  # aws_iam_role.lalala_poweruser has changed
  ~ resource "aws_iam_role" "lalala_poweruser" {
        id                    = "lalala_poweruser"
        name                  = "lalala_poweruser"
      + tags                  = {}
        # (11 unchanged attributes hidden)
    }

  # aws_iam_policy.lalala-admin-role-policy-iam-allow has changed
  ~ resource "aws_iam_policy" "lalala-admin-role-policy-iam-allow" {
        id          = "arn:aws:iam::lalala:policy/cl/sso/admin/lalala-admin-role-policy-iam-allow"
        name        = "lalala-admin-role-policy-iam-allow"
      + tags        = {}
        # (6 unchanged attributes hidden)
    }

All my roles and policies are detecting this false drift with the tags property.

Terraform -v 1.3.2
AWS provider -v 4.34.0

[YakDriver]

We will close this issue soon after doing more work and research. This way we can determine how much of an issue this continues to be.

[YakDriver]

Thank you for your comments!

At this point we have fixed many of these issues. However, with a big issue like this, crossing services, it's difficult to assess what's fixed and what still remains as time passes. Because of the quantity of issues fixed, and tests to back the fixes up, we feel confident closing this issue in order to then be able to focus on what remains.

In other words, if you have perpetual diffs / Objects have changed outside of Terraform going forward, see if there is an existing issue for the specific resource and service you're concerned about. If there isn't one, let us know with a new issue!

[ad-m-ss]

I don't know the project code well, but I wonder why users have to report such things. The project has integration tests, so it seems to me that after each "apply" in the tests, you can introduce a subtest that will check whether the plan is clean. Many of the cases reported here are quite basic use cases.

The tests look like they have a similar structure, so you can either modify some core element from the test framework, or make a bulk refactor and add such a subtest.

[github-actions]

This functionality has been released in v4.50.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[YakDriver]

@ad-m-ss I too would think this was easily tested. Unfortunately, though we run 1000s of tests, and specifically re-apply over and over to check this very thing, we don't run across the same problems. There may be other variables like age of accounts or regions at play but since we have a difficult time reproducing the problem despite heavy testing, we also have difficulty pinpointing a fix.

[ad-m-ss]

@YakDriver thanks for additional context here. I appreciate all efforts to development of AWS terraform provider. Overall very good jobs and very helpful in my job. 🐈

[nitrocode]

Yes, seconded. I've been waiting a long time for this to be resolved. You have shown great fortitude through your hard work and we appreciate it. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.