@aws-cdk/aws-s3: CloudFormation stack deployment fails intermittently when adding a bucket policy and EventBridge notifications

[ShanikaEdiriweera]

Describe the bug

CloudFormation stack deployment fails intermittently when adding a bucket policy (PutBucketPolicy) and EventBridge notifications (PutBucketNotification) to a S3 bucket due to race condition.

Received response status [FAILED] from custom resource. Message returned: Error: An error occurred (OperationAborted) when calling the PutBucketNotificationConfiguration operation: A conflicting conditional operation is currently in progress against this resource. Please try again.. See the details in CloudWatch Log Stream: 2023/10/17/[$LATEST]9f69597966xxxxa8449646270045 (RequestId: 2xx08c-74ad-4317-8a30-83xxxf2dc9)

Below is my CDK code of using S3 Bucket construct with eventBridgeEnabled: true and adding the policy after creating the bucket.

export class S3Bucket extends s3.Bucket {
  constructor(scope: Construct, id: string, props: S3BucketProperties) {
    super(scope, id, {
      ...props,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      encryption: s3.BucketEncryption.KMS,
      encryptionKey: props.encryptionKey,
      eventBridgeEnabled: true,
    });
  }
}
---------------------------------------------
this.s3Bucket = new S3Bucket(this, 'bucket', {
      encryptionKey,
    });
    this.s3Bucket.addToResourcePolicy(
      createS3BucketSSLRequestsOnlyPolicyStatement(
        this.s3Bucket.bucketArn
      )
    );

Similar issue #16811

Expected Behavior

S3 Bucket policy (PutBucketPolicy) and bucket notifications (PutBucketNotification/PutBucketNotificationConfiguration) happen without failing.

Current Behavior

Stack deployment fails intermittently

Reproduction Steps

Below is my CDK code of using S3 Bucket construct with eventBridgeEnabled: true and adding the policy after creating the bucket.

export class S3Bucket extends s3.Bucket {
  constructor(scope: Construct, id: string, props: S3BucketProperties) {
    super(scope, id, {
      ...props,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      encryption: s3.BucketEncryption.KMS,
      encryptionKey: props.encryptionKey,
      eventBridgeEnabled: true,
    });
  }
}
---------------------------------------------
this.s3Bucket = new S3Bucket(this, 'bucket', {
      encryptionKey,
    });
    this.s3Bucket.addToResourcePolicy(
      createS3BucketSSLRequestsOnlyPolicyStatement(
        this.s3Bucket.bucketArn
      )
    );

Possible Solution

(AWS Support Recommended) Implement retry with incremental back off into the custom resource code. This includes retrying operations with an exponential back off time to cater for any issues.

aws-cdk/packages/@aws-cdk/aws-s3/lib/notifications-resource/lambda/index.py

Line 26 in ecb94d7

put_bucket_notification_configuration(bucket, config)

Additional Information/Context

No response

CDK CLI Version

2.99.1

Framework Version

No response

Node.js Version

18

OS

MacOS

Language

TypeScript

Language Version

No response

Other information

No response

[indrora]

Thank you for your report.

[zhaoyi0113]

I am facing the same issue, is there any update on this one?

[yerzhan7]

It looks like PutBucketPolicy and PutBucketNotification API calls are happening at the same time and causing race condition because S3 does not allow parallel bucket edits.

One workaround is to add BucketPolicy as dependency to BucketNotification custom resource. This way Bucket Policy would be created before PutBucketNotification API call is made.

For example:

yourBucket.node.findChild('Notifications').node.addDependency(yourBucket.policy!);

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.