S3 target iam policy not following user guide

[raphavr]

Description

The S3 target policy configured by the module is not following the AWS doc, hence leading to permission errors.

• ✋ I have searched the open/closed issues and my issue is not listed.

Versions

• Module version [Required]:

• Terraform version:
  1.5.7

• Provider version(s):
  5.48.0

Reproduction Code [Required]

s3_endpoints = {
    s3-target = {
        endpoint_id   = "cdc-test-target"
        endpoint_type = "target"
        engine_name   = "s3"

        bucket_folder               = "source=dms"
        bucket_name                 = "raw.cdc.test.raphavr.tools"
        compression_type            = "GZIP"
        data_format                 = "parquet"
        date_partition_enabled      = true
        date_partition_delimiter    = "none"
        ssl_mode                    = "none"
        tags                        = {}
    }
}

access_target_s3_bucket_arns = ["arn:aws:s3:::raw.cdc.test.raphavr.tools"]

Expected behavior

A policy as below should be created:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:PutObjectTagging",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::raw.cdc.test.raphavr.tools/*",
            "Sid": "S3ObjectsTarget"
        },
        {
            "Action": [
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::raw.cdc.test.raphavr.tools",
            "Sid": "S3BucketTarget"
        }
    ]
}

Actual behavior

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:PutObjectTagging",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::raw.cdc.test.raphavr.tools",
            "Sid": "S3Target"
        }
    ]
}

Additional context

I've opened a PR following the user guide

[bryantbiggs]

can you show the error message you are getting?

[sigfigsteve]

I'm experiencing the same issue. The first error I get is in cloudwatch logs when I start with a full load and it tries to truncate existing data from the target bucket. It shows up in cloudwatch logs:
2024-06-10T17:19:14 [TARGET_LOAD ]E: Failed to delete s3 object schema/table/LOAD00000001.parquet [1021807] (file_imp.c:2758)
This happens despite the endpoint test from the AWS console succeeding. AWS docs at https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.S3.html show the correct permissions that @raphavr is setting. When I add a /* resource identifier for my bucket policy, it works for me too.

[github-actions]

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

[github-actions]

This issue was automatically closed because of stale in 10 days