awscc_s3_bucket with basic configuration re-created when apply are re-deployed

[emnedre]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment
• The resources and data sources in this provider are generated from the CloudFormation schema, so they can only support the actions that the underlying schema supports. For this reason submitted bugs should be limited to defects in the generation and runtime code of the provider. Customizing behavior of the resource, or noting a gap in behavior are not valid bugs and should be submitted as enhancements to AWS via the CloudFormation Open Coverage Roadmap.

Terraform CLI and Terraform AWS Cloud Control Provider Version

Affected Resource(s)

• awscc_s3_bucket

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

resource "awscc_s3_bucket" "example" {
  bucket_name = "exmaple-bucket-123123123123"
}

Debug Output

Panic Output

Expected Behavior

Resource should not be re-deployed every time, seems like there are some changes done on AWS and the provider configuration is not keeping up. Ref. encryption setting from the plan above

Actual Behavior

Re-deployed the same configuration, but the bucket is then deleted and deployed again.

Steps to Reproduce

Apply two times the example in the following documentation: https://registry.terraform.io/providers/hashicorp/awscc/latest/docs/resources/s3_bucket

1. terraform apply

Important Factoids

References

• #0000

[BondAnthony]

I've run into this a few times and it appears to be centered around defaults Cloud Formation populates.

When you create a bucket using the following definition, the Terraform state contains default ownership_controls and bucket_encryption configurations. These controls are not present in a simple awscc_s3_bucket configuration resulting in a difference between the configuration and state.

Users can workaround this problem by populating the ownership_controls and bucket_encryption arguments. The provider should be handling this by default.

Steps to reproduce.

1. Create a simple bucket

resource "awscc_s3_bucket" "example" {
  bucket_name = "example-bucket-958630"
}

2. Review the state information for resource awscc_s3_bucket.example after the first apply. Notice the bucket_encryption and ownership_controls are now populated

# awscc_s3_bucket.example:
resource "awscc_s3_bucket" "example" {
    arn                               = "arn:aws:s3:::example-bucket-958630"
    bucket_encryption                 = {
        server_side_encryption_configuration = [
            {
                bucket_key_enabled                = false
                server_side_encryption_by_default = {
                    sse_algorithm = "AES256"
                }
            },
        ]
    }
    bucket_name                       = "example-bucket-958630"
    domain_name                       = "example-bucket-958630.s3.amazonaws.com"
    dual_stack_domain_name            = "example-bucket-958630.s3.dualstack.us-east-1.amazonaws.com"
    id                                = "example-bucket-958630"
    ownership_controls                = {
        rules = [
            {
                object_ownership = "BucketOwnerEnforced"
            },
        ]
    }
    public_access_block_configuration = {
        block_public_acls       = true
        block_public_policy     = true
        ignore_public_acls      = true
        restrict_public_buckets = true
    }
    regional_domain_name              = "example-bucket-958630.s3.us-east-1.amazonaws.com"
    website_url                       = "http://example-bucket-958630.s3-website-us-east-1.amazonaws.com"
}

3. Compared to the data CloudControl returns in the .ResourceDescription.Properties field.

{
  "PublicAccessBlockConfiguration": {
    "RestrictPublicBuckets": true,
    "BlockPublicPolicy": true,
    "BlockPublicAcls": true,
    "IgnorePublicAcls": true
  },
  "BucketName": "example-bucket-958630",
  "RegionalDomainName": "example-bucket-958630.s3.us-east-1.amazonaws.com",
  "OwnershipControls": {
    "Rules": [
      {
        "ObjectOwnership": "BucketOwnerEnforced"
      }
    ]
  },
  "DomainName": "example-bucket-958630.s3.amazonaws.com",
  "BucketEncryption": {
    "ServerSideEncryptionConfiguration": [
      {
        "BucketKeyEnabled": false,
        "ServerSideEncryptionByDefault": {
          "SSEAlgorithm": "AES256"
        }
      }
    ]
  },
  "WebsiteURL": "http://example-bucket-958630.s3-website-us-east-1.amazonaws.com",
  "DualStackDomainName": "example-bucket-958630.s3.dualstack.us-east-1.amazonaws.com",
  "Arn": "arn:aws:s3:::example-bucket-958630"
}

4. Populate the ownership_controls and bucket_encryption arguments in the HCL to be aligned with CloudControl and the Terraform state file.

resource "awscc_s3_bucket" "example" {
  bucket_name = "example-bucket-958630"
  ownership_controls = {
    rules = [{
      object_ownership = "BucketOwnerEnforced"
    }]
  }
  bucket_encryption = {
    server_side_encryption_configuration = [{
      server_side_encryption_by_default = {
        sse_algorithm = "AES256"
      }
    }]
  }
}

TLDR: I would expect the provider to handle the ownership_controls and default_encryption defaults and not produce a diff. It's unclear to me where these defaults are being defined since the internal resources are generated based on the CloudFormation schemas.

[wellsiau-aws]

got bitten by this again today during my test, which reminded me to get back to this issue.

in my case, the trigger was this particular attribute object_lock_enabled

# awscc_s3_bucket.example must be replaced
-/+ resource "awscc_s3_bucket" "example" {
     . . .
      ~ id                                 = "example-cloudtrail-368288356491" -> (known after apply)
     . . .
      + object_lock_enabled                = (known after apply) # forces replacement

on the Terraform statefile, this attribute was set to null, since I did not declare it on the Terraform config.
subsequently, CCAPI also did not return the value at all:

{
  "PublicAccessBlockConfiguration": {
    "RestrictPublicBuckets": true,
    "BlockPublicPolicy": true,
    "BlockPublicAcls": true,
    "IgnorePublicAcls": true
  },
  "BucketName": "example-cloudtrail-368288356491",
  "RegionalDomainName": "example-cloudtrail-368288356491.s3.us-east-1.amazonaws.com",
  "OwnershipControls": {
    "Rules": [
      {
        "ObjectOwnership": "BucketOwnerEnforced"
      }
    ]
  },
  "DomainName": "example-cloudtrail-368288356491.s3.amazonaws.com",
  "BucketEncryption": {
    "ServerSideEncryptionConfiguration": [
      {
        "BucketKeyEnabled": false,
        "ServerSideEncryptionByDefault": {
          "SSEAlgorithm": "aws:kms",
          "KMSMasterKeyID": "arn:aws:kms:us-east-1:368288356491:key/f84fadd5-d0fd-4abb-9c4b-39a3d378b22b"
        }
      }
    ]
  },
  "WebsiteURL": "http://example-cloudtrail-368288356491.s3-website-us-east-1.amazonaws.com",
  "DualStackDomainName": "example-cloudtrail-368288356491.s3.dualstack.us-east-1.amazonaws.com",
  "Arn": "arn:aws:s3:::example-cloudtrail-368288356491"
}

[wellsiau-aws]

This is known issue, related to #1139

[wellsiau-aws]

#1139 resolves problem when non-mandatory attribute triggers resource replacement because there is no default values for said attribute provided in the schema.

however the lack of default values in the Cfn schema still can trigger drift (without replacement) as shown in this issue.

[wellsiau-aws]

Upon further debug by setting the env var TF_LOG_SDK_PROTO_DATA_DIR to local directory:

PlanResourceChange_Request_PriorState matches what is on the state file:

{
  "accelerate_configuration": null,
  "access_control": null,
  "analytics_configurations": null,
  "arn": "arn:aws:s3:::exmaple-bucket-123123123123",
  "bucket_encryption": {
    "server_side_encryption_configuration": [
      {
        "bucket_key_enabled": false,
        "server_side_encryption_by_default": {
          "kms_master_key_id": null,
          "sse_algorithm": "AES256"
        }
      }
    ]
  },
  "bucket_name": "exmaple-bucket-123123123123",
  "cors_configuration": null,
  "domain_name": "exmaple-bucket-123123123123.s3.amazonaws.com",
  "dual_stack_domain_name": "exmaple-bucket-123123123123.s3.dualstack.us-east-1.amazonaws.com",
  "id": "exmaple-bucket-123123123123",
  "intelligent_tiering_configurations": null,
  "inventory_configurations": null,
  "lifecycle_configuration": null,
  "logging_configuration": null,
  "metrics_configurations": null,
  "notification_configuration": null,
  "object_lock_configuration": null,
  "object_lock_enabled": null,
  "ownership_controls": {
    "rules": [
      {
        "object_ownership": "BucketOwnerEnforced"
      }
    ]
  },
  "public_access_block_configuration": {
    "block_public_acls": true,
    "block_public_policy": true,
    "ignore_public_acls": true,
    "restrict_public_buckets": true
  },
  "regional_domain_name": "exmaple-bucket-123123123123.s3.us-east-1.amazonaws.com",
  "replication_configuration": null,
  "tags": null,
  "versioning_configuration": null,
  "website_configuration": null,
  "website_url": "http://exmaple-bucket-123123123123.s3-website-us-east-1.amazonaws.com"
}

The PlanResourceChange_Request_ProposedNewState set both bucket_encryption and ownership_controls to null.

{
  "accelerate_configuration": null,
  "access_control": null,
  "analytics_configurations": null,
  "arn": "arn:aws:s3:::exmaple-bucket-123123123123",
  "bucket_encryption": null,
  "bucket_name": "exmaple-bucket-123123123123",
  "cors_configuration": null,
  "domain_name": "exmaple-bucket-123123123123.s3.amazonaws.com",
  "dual_stack_domain_name": "exmaple-bucket-123123123123.s3.dualstack.us-east-1.amazonaws.com",
  "id": "exmaple-bucket-123123123123",
  "intelligent_tiering_configurations": null,
  "inventory_configurations": null,
  "lifecycle_configuration": null,
  "logging_configuration": null,
  "metrics_configurations": null,
  "notification_configuration": null,
  "object_lock_configuration": null,
  "object_lock_enabled": null,
  "ownership_controls": null,
  "public_access_block_configuration": {
    "block_public_acls": true,
    "block_public_policy": true,
    "ignore_public_acls": true,
    "restrict_public_buckets": true
  },
  "regional_domain_name": "exmaple-bucket-123123123123.s3.us-east-1.amazonaws.com",
  "replication_configuration": null,
  "tags": null,
  "versioning_configuration": null,
  "website_configuration": null,
  "website_url": "http://exmaple-bucket-123123123123.s3-website-us-east-1.amazonaws.com"
}

Finallly the PlanResourceChange_Response_PlannedState resemble everything again:

{
  "accelerate_configuration": {
    "type": 0,
    "data": {
      "0": 0
    }
  },
  "access_control": {
    "type": 0,
    "data": {
      "0": 0
    }
  },
  "analytics_configurations": {
    "type": 0,
    "data": {
      "0": 0
    }
  },
  "arn": "arn:aws:s3:::exmaple-bucket-123123123123",
  "bucket_encryption": {
    "server_side_encryption_configuration": [
      {
        "bucket_key_enabled": false,
        "server_side_encryption_by_default": {
          "kms_master_key_id": null,
          "sse_algorithm": "AES256"
        }
      }
    ]
  },
  "bucket_name": "exmaple-bucket-123123123123",
  "cors_configuration": {
    "type": 0,
    "data": {
      "0": 0
    }
  },
  "domain_name": "exmaple-bucket-123123123123.s3.amazonaws.com",
  "dual_stack_domain_name": "exmaple-bucket-123123123123.s3.dualstack.us-east-1.amazonaws.com",
  "id": "exmaple-bucket-123123123123",
  "intelligent_tiering_configurations": {
    "type": 0,
    "data": {
      "0": 0
    }
  },
  "inventory_configurations": {
    "type": 0,
    "data": {
      "0": 0
    }
  },
  "lifecycle_configuration": {
    "type": 0,
    "data": {
      "0": 0
    }
  },
  "logging_configuration": {
    "type": 0,
    "data": {
      "0": 0
    }
  },
  "metrics_configurations": {
    "type": 0,
    "data": {
      "0": 0
    }
  },
  "notification_configuration": {
    "type": 0,
    "data": {
      "0": 0
    }
  },
  "object_lock_configuration": {
    "type": 0,
    "data": {
      "0": 0
    }
  },
  "object_lock_enabled": {
    "type": 0,
    "data": {
      "0": 0
    }
  },
  "ownership_controls": {
    "rules": [
      {
        "object_ownership": "BucketOwnerEnforced"
      }
    ]
  },
  "public_access_block_configuration": {
    "block_public_acls": true,
    "block_public_policy": true,
    "ignore_public_acls": true,
    "restrict_public_buckets": true
  },
  "regional_domain_name": "exmaple-bucket-123123123123.s3.us-east-1.amazonaws.com",
  "replication_configuration": {
    "type": 0,
    "data": {
      "0": 0
    }
  },
  "tags": {
    "type": 0,
    "data": {
      "0": 0
    }
  },
  "versioning_configuration": {
    "type": 0,
    "data": {
      "0": 0
    }
  },
  "website_configuration": {
    "type": 0,
    "data": {
      "0": 0
    }
  },
  "website_url": "http://exmaple-bucket-123123123123.s3-website-us-east-1.amazonaws.com"
}

According to this doc:

Terraform Core therefore constructs the proposed new state by taking the attribute value from Configuration if it is non-null, and then using the Prior State as a fallback otherwise, thereby helping a provider to preserve its previously-chosen value for the attribute where appropriate.

As such, I expect that PlanResourceChange_Request_ProposedNewState will not set bucket_encryption and ownership_controls to null.

[wellsiau-aws]

Digging through the framework open issues, I found similar problem reported here: hashicorp/terraform-plugin-framework#898

[wellsiau-aws]

local test confirmed this behavior, after removing the Required : true and set the Computed: true + Optional: true on the suspected attributes

index 99015cb97..55824e56a 100644
--- a/internal/aws/s3/bucket_resource_gen.go
+++ b/internal/aws/s3/bucket_resource_gen.go
@@ -446,7 +446,9 @@ func bucketResource(ctx context.Context) (resource.Resource, error) {
                                                                        // Property: SSEAlgorithm
                                                                        "sse_algorithm": schema.StringAttribute{ /*START ATTRIBUTE*/
                                                                                Description: "Server-side encryption algorithm to use for the default encryption.",
-                                                                               Required:    true,
+                                                                               // Required:    true,
+                                                                               Optional: true,
+                                                                               Computed: true,
                                                                                Validators: []validator.String{ /*START VALIDATORS*/
                                                                                        stringvalidator.OneOf(
                                                                                                "aws:kms",
@@ -466,7 +468,9 @@ func bucketResource(ctx context.Context) (resource.Resource, error) {
                                                }, /*END SCHEMA*/
                                        }, /*END NESTED OBJECT*/
                                        Description: "Specifies the default server-side-encryption configuration.",
-                                       Required:    true,
+                                       // Required:    true,
+                                       Optional: true,
+                                       Computed: true,
                                        Validators: []validator.List{ /*START VALIDATORS*/
                                                listvalidator.UniqueValues(),
                                        }, /*END VALIDATORS*/
@@ -2684,7 +2688,9 @@ func bucketResource(ctx context.Context) (resource.Resource, error) {
                                                }, /*END SCHEMA*/
                                        }, /*END NESTED OBJECT*/
                                        Description: "Specifies the container element for Object Ownership rules.",
-                                       Required:    true,
+                                       // Required:    true,
+                                       Optional: true,
+                                       Computed: true,
                                        Validators: []validator.List{ /*START VALIDATORS*/
                                                listvalidator.UniqueValues(),

Results:

terraform plan
╷
│ Warning: Provider development overrides are in effect
│ 
│ The following provider development overrides are set in the CLI configuration:
│  - hashicorp/awscc in /usr/local/go/bin
│ 
│ The behavior may therefore not match any released version of the provider and applying changes may cause the state to become incompatible with published releases.
╵
awscc_s3_bucket.example: Refreshing state... [id=exmaple-bucket-123123123123]

No changes. Your infrastructure matches the configuration.