Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

r/s3_bucket_lifecycle_configuration: update value set in state for an empty filter argument #23232

Merged
merged 4 commits into from
Feb 17, 2022

Conversation

anGie44
Copy link
Contributor

@anGie44 anGie44 commented Feb 16, 2022

Community Note

  • Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #23228

Description

  • Perhaps we don't need the special handling if the tests still pass? @YakDriver

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccS3BucketLifecycleConfiguration' PKG=s3 ACCTEST_PARALLELISM=5
--- PASS: TestAccS3BucketLifecycleConfiguration_basic (197.31s)
--- PASS: TestAccS3BucketLifecycleConfiguration_EmptyFilter_NonCurrentVersions (197.80s)
--- PASS: TestAccS3BucketLifecycleConfiguration_TransitionZeroDays_intelligentTiering (197.94s)
--- PASS: TestAccS3BucketLifecycleConfiguration_RuleExpiration_emptyBlock (197.94s)
--- PASS: TestAccS3BucketLifecycleConfiguration_TransitionUpdateBetweenDaysAndDate_intelligentTiering (329.40s)
--- PASS: TestAccS3BucketLifecycleConfiguration_TransitionStorageClassOnly_intelligentTiering (194.99s)
--- PASS: TestAccS3BucketLifecycleConfiguration_nonCurrentVersionExpiration (198.93s)
--- PASS: TestAccS3BucketLifecycleConfiguration_ruleAbortIncompleteMultipartUpload (275.39s)
--- PASS: TestAccS3BucketLifecycleConfiguration_RuleExpiration_expireMarkerOnly (279.81s)
--- PASS: TestAccS3BucketLifecycleConfiguration_prefix (192.03s)
--- PASS: TestAccS3BucketLifecycleConfiguration_disappears (55.15s)
--- PASS: TestAccS3BucketLifecycleConfiguration_nonCurrentVersionTransition (192.90s)
--- PASS: TestAccS3BucketLifecycleConfiguration_multipleRules (192.55s)
--- PASS: TestAccS3BucketLifecycleConfiguration_disableRule (326.82s)
--- PASS: TestAccS3BucketLifecycleConfiguration_filterWithPrefix (272.41s)
--- PASS: TestAccS3BucketLifecycleConfiguration_TransitionDate_intelligentTiering (194.79s)
--- PASS: TestAccS3BucketLifecycleConfiguration_TransitionDate_standardIa (203.78s)

@github-actions github-actions bot added service/s3 Issues and PRs that pertain to the s3 service. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. labels Feb 16, 2022
@anGie44 anGie44 added bug Addresses a defect in current functionality. and removed tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. service/s3 Issues and PRs that pertain to the s3 service. labels Feb 16, 2022
@github-actions github-actions bot added the size/M Managed by automation to categorize the size of a PR. label Feb 16, 2022
@anGie44 anGie44 added the service/s3 Issues and PRs that pertain to the s3 service. label Feb 16, 2022
@YakDriver
Copy link
Member

Import is not happy with this right now.

My 3.74 Config

resource "aws_s3_bucket" "example" {
  bucket = "barrameda-994617143"

  lifecycle_rule {
    id      = "log-expiration"
    enabled = true
    prefix  = ""

    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }

    transition {
      days          = 180
      storage_class = "GLACIER"
    }
  }
}

resource "aws_s3_bucket" "example2" {
  bucket = "valladolid-817999296"

  lifecycle_rule {
    id      = "Keep previous version 30 days, then in Glacier another 60"
    enabled = true

    noncurrent_version_transition {
      days          = 30
      storage_class = "GLACIER"
    }

    noncurrent_version_expiration {
      days = 90
    }
  }

  lifecycle_rule {
    id                                     = "Delete old incomplete multi-part uploads"
    enabled                                = true
    abort_incomplete_multipart_upload_days = 7
  }
}

resource "aws_s3_bucket" "example3" {
  bucket = "granada-2433251135"

  lifecycle_rule {
    id      = "log-expiration"
    enabled = true
    prefix = "log/"

    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }

    transition {
      days          = 180
      storage_class = "GLACIER"
    }
  }
}

resource "aws_s3_bucket" "example4" {
  bucket = "marbella-1354962447"

  lifecycle_rule {
    id      = "log-expiration"
    enabled = true

    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }

    transition {
      days          = 180
      storage_class = "GLACIER"
    }
  }
}

Import/v4 Config

resource "aws_s3_bucket" "example" {
  bucket = "barrameda-994617143"
}

resource "aws_s3_bucket_lifecycle_configuration" "example" {
  bucket = aws_s3_bucket.example.id

  rule {
    id     = "log-expiration"
    status = "Enabled"
    prefix = ""

    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }

    transition {
      days          = 180
      storage_class = "GLACIER"
    }
  }
}

resource "aws_s3_bucket" "example2" {
  bucket = "valladolid-817999296"
}

resource "aws_s3_bucket_lifecycle_configuration" "example2" {
  bucket = aws_s3_bucket.example2.id

  rule {
    id     = "Keep previous version 30 days, then in Glacier another 60"
    status = "Enabled"

    noncurrent_version_transition {
      noncurrent_days = 30
      storage_class   = "GLACIER"
    }

    noncurrent_version_expiration {
      noncurrent_days = 90
    }
  }

  rule {
    id     = "Delete old incomplete multi-part uploads"
    status = "Enabled"

    abort_incomplete_multipart_upload {
      days_after_initiation = 7
    }
  }
}

resource "aws_s3_bucket" "example3" {
  bucket = "granada-2433251135"
}

resource "aws_s3_bucket_lifecycle_configuration" "example3" {
  bucket = aws_s3_bucket.example3.id

  rule {
    id     = "log-expiration"
    status = "Enabled"
    
    filter {
        prefix = "log/"
    }

    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }

    transition {
      days          = 180
      storage_class = "GLACIER"
    }
  }
}

resource "aws_s3_bucket" "example4" {
  bucket = "marbella-1354962447"
}

resource "aws_s3_bucket_lifecycle_configuration" "example4" {
  bucket = aws_s3_bucket.example4.id

  rule {
    id     = "log-expiration"
    status = "Enabled"

    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }

    transition {
      days          = 180
      storage_class = "GLACIER"
    }
  }
}

Importing with this PR

aws_s3_bucket.example: Refreshing state... [id=barrameda-994617143]
aws_s3_bucket.example4: Refreshing state... [id=marbella-1354962447]
aws_s3_bucket.example2: Refreshing state... [id=valladolid-817999296]
aws_s3_bucket.example3: Refreshing state... [id=granada-2433251135]
aws_s3_bucket_lifecycle_configuration.example2: Refreshing state... [id=valladolid-817999296]
aws_s3_bucket_lifecycle_configuration.example: Refreshing state... [id=barrameda-994617143]
aws_s3_bucket_lifecycle_configuration.example3: Refreshing state... [id=granada-2433251135]
aws_s3_bucket_lifecycle_configuration.example4: Refreshing state... [id=marbella-1354962447]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following
symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_s3_bucket_lifecycle_configuration.example will be updated in-place
  ~ resource "aws_s3_bucket_lifecycle_configuration" "example" {
        id     = "barrameda-994617143"
        # (1 unchanged attribute hidden)

      ~ rule {
            id     = "log-expiration"
            # (1 unchanged attribute hidden)

          - filter {
            }

            # (2 unchanged blocks hidden)
        }
    }

  # aws_s3_bucket_lifecycle_configuration.example2 will be updated in-place
  ~ resource "aws_s3_bucket_lifecycle_configuration" "example2" {
        id     = "valladolid-817999296"
        # (1 unchanged attribute hidden)

      ~ rule {
            id     = "Keep previous version 30 days, then in Glacier another 60"
            # (1 unchanged attribute hidden)

          - filter {
            }


            # (2 unchanged blocks hidden)
        }
      ~ rule {
            id     = "Delete old incomplete multi-part uploads"
            # (1 unchanged attribute hidden)


          - filter {
            }
            # (1 unchanged block hidden)
        }
    }

  # aws_s3_bucket_lifecycle_configuration.example4 will be updated in-place
  ~ resource "aws_s3_bucket_lifecycle_configuration" "example4" {
        id     = "marbella-1354962447"
        # (1 unchanged attribute hidden)

      ~ rule {
            id     = "log-expiration"
            # (1 unchanged attribute hidden)

          - filter {
            }

            # (2 unchanged blocks hidden)
        }
    }

Plan: 0 to add, 3 to change, 0 to destroy.

Importing with v4.1.0

aws_s3_bucket.example4: Refreshing state... [id=marbella-1354962447]
aws_s3_bucket.example2: Refreshing state... [id=valladolid-817999296]
aws_s3_bucket.example3: Refreshing state... [id=granada-2433251135]
aws_s3_bucket.example: Refreshing state... [id=barrameda-994617143]
aws_s3_bucket_lifecycle_configuration.example3: Refreshing state... [id=granada-2433251135]
aws_s3_bucket_lifecycle_configuration.example2: Refreshing state... [id=valladolid-817999296]
aws_s3_bucket_lifecycle_configuration.example: Refreshing state... [id=barrameda-994617143]
aws_s3_bucket_lifecycle_configuration.example4: Refreshing state... [id=marbella-1354962447]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

@anGie44
Copy link
Contributor Author

anGie44 commented Feb 17, 2022

My current observations:

If we apply aws_s3_bucket.example using provider v3.74.1, when we run aws s3api get-bucket-lifecycle-configuration barrameda-994617143 it returns:

{
    "Rules": [
        {
            "ID": "example",
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled",
            "Transitions": [
                {
                    "Days": 30,
                    "StorageClass": "STANDARD_IA"
                },
                {
                    "Days": 180,
                    "StorageClass": "GLACIER"
                }
            ]
        }
    ]
}

We can see prefix is housed within filter.
Thus to translate this into v4.0 aws_s3_bucket_lifecycle_configuration, we include filter:

resource "aws_s3_bucket_lifecycle_configuration" "example" {
  bucket = aws_s3_bucket.example.id

  rule {
    id = "example"
    filter {
      prefix = ""
    }

   # ... etc. ...
}

Similarly if we take example2 where prefix is never specified, the AWS CLI returns:

{
    "Rules": [
        {
            "ID": "Keep previous version 30 days, then in Glacier another 60",
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled",
            "NoncurrentVersionTransitions": [
                {
                    "NoncurrentDays": 30,
                    "StorageClass": "GLACIER"
                }
            ],
            "NoncurrentVersionExpiration": {
                "NoncurrentDays": 90
            }
        },
        {
            "ID": "Delete old incomplete multi-part uploads",
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled",
            "AbortIncompleteMultipartUpload": {
                "DaysAfterInitiation": 7
            }
        }
    ]
}

Here again we see prefix inside filter, so the translation can look like:

resource "aws_s3_bucket_lifecycle_configuration" "example2" {
  bucket = aws_s3_bucket.example2.id

  rule {
    id     = "Keep previous version 30 days, then in Glacier another 60"
    status = "Enabled"

    filter {
       prefix = ""
    }
  
    noncurrent_version_transition {
      noncurrent_days = 30
      storage_class   = "GLACIER"
    }

    noncurrent_version_expiration {
      noncurrent_days = 90
    }
  }

  rule {
    id     = "Delete old incomplete multi-part uploads"
    status = "Enabled"

    filter {
      prefix = ""
    }

    abort_incomplete_multipart_upload {
      days_after_initiation = 7
    }
  }
}

This is all to say that I believe (a) v4.0 was doing the right thing with how filter was getting set back in state; I just didn't communicate it well in the upgrade guide how to consider this translation of prefix to filter {} (b) we need to reinstate setting the filter block with an empty prefix string as suggested in the flex.go code change.

Perhaps now the question is, if buckets are old, and the lifecycle configuration was configured in V1 format, does the s3api still return V2 format as seen above OR is V1 returned? something to ask the OP.

@YakDriver
Copy link
Member

YakDriver commented Feb 17, 2022

Not a fan of having to set anything to empty or an empty string to make it work. Seems messy. I'd prefer a diff suppress. But, if needs must.

@anGie44
Copy link
Contributor Author

anGie44 commented Feb 17, 2022

@YakDriver one last experiment to show the change IMO represents reality.

Let's say i create a bucket in the console, and the aws s3api get-bucket-lifecycle-configuration --bucket its-me-bb-bucket returns:

{
    "Rules": [
        {
            "ID": "test",
            "Filter": {},
            "Status": "Enabled",
            "NoncurrentVersionTransitions": [
                {
                    "NoncurrentDays": 30,
                    "StorageClass": "GLACIER_IR"
                }
            ]
        }
    ]
}

and then the AWS Go SDK returns <Filter/>:

<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Rule><ID>test</ID><Filter/><Status>Enabled</Status><NoncurrentVersionTransition><NoncurrentDays>30</NoncurrentDays><StorageClass>GLACIER_IR</StorageClass></NoncurrentVersionTransition></Rule></LifecycleConfiguration>

so for terraform to be representative of what exists in S3, the resource would look like:

resource "aws_s3_bucket_lifecycle_configuration" "example_bb" {
  bucket = "its-me-bb-bucket"
  rule {
    id = "test"
    status = "Enabled"

    filter {}

    noncurrent_version_transition {
      noncurrent_days = 30
      storage_class = "GLACIER_IR"
    }
  }
}

and we should thus expect terraform to recognize the empty filter{} as valid and set it in state, result in

$ terraform show
# aws_s3_bucket_lifecycle_configuration.example_bb:
resource "aws_s3_bucket_lifecycle_configuration" "example_bb" {
    bucket = "its-me-bb-bucket"
    id     = "its-me-bb-bucket"

    rule {
        id     = "test"
        status = "Enabled"

        filter {
        }

        noncurrent_version_transition {
            newer_noncurrent_versions = 0
            noncurrent_days           = 30
            storage_class             = "GLACIER_IR"
        }
    }
}

@github-actions github-actions bot added the tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. label Feb 17, 2022
@YakDriver
Copy link
Member

Ah, I see the difference in opinion: viewing what AWS does as a source of correctness vs. not. I like a layer of correctness on top of the API to smooth out the various inconsistencies. But, there's something to be said for adhereing to "the way."

@anGie44
Copy link
Contributor Author

anGie44 commented Feb 17, 2022

Ahh yes that's a good way to put it 👍 I think in this case it's important contextual data to keep in terraform state but we can keep discussing 😃 . I'm gonna mark this as ready in the meantime.

@anGie44 anGie44 marked this pull request as ready for review February 17, 2022 18:40
@anGie44
Copy link
Contributor Author

anGie44 commented Feb 17, 2022

Documentation may need an edit but i'll double check with @justinretzolk if that's already in the works since we somewhat strayed from the original plan in Slack thread

@justinretzolk
Copy link
Member

I've not started on documentation edits as of yet @anGie44 - happy to do so, but wanted to keep an eye on what the final outcome was first.

@YakDriver YakDriver self-assigned this Feb 17, 2022
Copy link
Member

@YakDriver YakDriver left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MPMB! 🎉 I especially like the branch "fitler!"

As long as I add in the fitler {}, er, filter {} my import tests pass.

Output from acceptance tests (us-west-2):

% make testacc TESTS=TestAccS3BucketLifecycleConfig PKG=s3
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/s3/... -v -count 1 -parallel 20 -run='TestAccS3BucketLifecycleConfig'  -timeout 180m
--- PASS: TestAccS3BucketLifecycleConfiguration_disappears (62.22s)
--- PASS: TestAccS3BucketLifecycleConfiguration_basic (201.36s)
--- PASS: TestAccS3BucketLifecycleConfiguration_prefix (206.86s)
--- PASS: TestAccS3BucketLifecycleConfiguration_nonCurrentVersionExpiration (206.87s)
--- PASS: TestAccS3BucketLifecycleConfiguration_TransitionDate_standardIa (206.87s)
--- PASS: TestAccS3BucketLifecycleConfiguration_RuleExpiration_emptyBlock (206.95s)
--- PASS: TestAccS3BucketLifecycleConfiguration_TransitionDate_intelligentTiering (207.01s)
--- PASS: TestAccS3BucketLifecycleConfiguration_TransitionStorageClassOnly_intelligentTiering (207.01s)
--- PASS: TestAccS3BucketLifecycleConfiguration_multipleRules (207.04s)
--- PASS: TestAccS3BucketLifecycleConfiguration_TransitionZeroDays_intelligentTiering (207.10s)
--- PASS: TestAccS3BucketLifecycleConfiguration_nonCurrentVersionTransition (207.11s)
--- PASS: TestAccS3BucketLifecycleConfiguration_EmptyFilter_NonCurrentVersions (208.66s)
--- PASS: TestAccS3BucketLifecycleConfiguration_RuleExpiration_expireMarkerOnly (281.83s)
--- PASS: TestAccS3BucketLifecycleConfiguration_filterWithPrefix (286.26s)
--- PASS: TestAccS3BucketLifecycleConfiguration_ruleAbortIncompleteMultipartUpload (286.55s)
--- PASS: TestAccS3BucketLifecycleConfiguration_disableRule (339.27s)
--- PASS: TestAccS3BucketLifecycleConfiguration_TransitionUpdateBetweenDaysAndDate_intelligentTiering (339.45s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/s3	341.317s

@github-actions github-actions bot added size/L Managed by automation to categorize the size of a PR. documentation Introduces or discusses updates to documentation. and removed size/M Managed by automation to categorize the size of a PR. labels Feb 17, 2022
…ilter in rule block; add more examples in upgrade guide
@anGie44 anGie44 force-pushed the b-s3-bucket-lifecycle-configuration-empty-fitler branch from a68ea14 to 481687f Compare February 17, 2022 20:59
@anGie44 anGie44 added this to the v4.2.0 milestone Feb 17, 2022
@anGie44 anGie44 merged commit 5e0ac8d into main Feb 17, 2022
@anGie44 anGie44 deleted the b-s3-bucket-lifecycle-configuration-empty-fitler branch February 17, 2022 22:31
github-actions bot pushed a commit that referenced this pull request Feb 17, 2022
@github-actions
Copy link

This functionality has been released in v4.2.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

@github-actions
Copy link

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 14, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Addresses a defect in current functionality. documentation Introduces or discusses updates to documentation. service/s3 Issues and PRs that pertain to the s3 service. size/L Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Using certain features when creating s3 bucket lifecycle rules on all files throws errors
3 participants