(logs): Policy Limit Reached for LogGroup ResourcePolicies

[badaldavda8]

Describe the bug

After resolution of #17544 A new issue has turned up if you tend to create multiple ECS tasks referring multiple log groups.

Resource handler returned message: "Resource limit exceeded. (Service: CloudWatchLogs, Status Code: 400, Request ID: 25bec134-657e-43c3-ae85-810a0ce56fa0)" (RequestToken: 948dab8b-fac6-2903-695d-f9d825eaea90, HandlerErrorCode: ServiceLimitExceeded)
This is because Default quota for resource policies

Resource policies	Up to 10 CloudWatch Logs resource policies per Region per account. This quota can't be changed.

Expected Behavior

No error after 10th ECS Task/service

Current Behavior

Each ECS task creates a new log group finally exhausting this limit.

Reproduction Steps

Create 10 log groups for ecs and you will start to face this.

Possible Solution

Let us avoid creating not create Resource Policies for CW Logs until this issue is resolved. I understand this defeats the purpose of lowest privilege, but causes issues.

Additional Information/Context

WorkAround

separate logGroup if created within taskDefinition and add following in the code for now.

logGroup.node.tryRemoveChild('Policy')

CDK CLI Version

2.24.0

Framework Version

No response

Node.js Version

v17.9.0

OS

macOS

Language

Typescript

Language Version

No response

Other information

No response

[badaldavda8]

Hi Any update on this?

[badaldavda8]

Any update on this?

[brignolij]

Hi,
Any update on this? i'm facing the same issue.

[tt-fozail]

Same problem please take a look at this @comcalvi

[aaa-khoa]

im facing the same issue, any update on this? I've tried using L1 constructs and it still creates a resource policy whenever a log group is attached to a pipeline project, is there a workaround for this?

[xmatusmic]

Same issue!

[jaredcnance]

LogGroups do not support ResourcePolicies being attached directly. The only use case for ResourcePolicies today is to enable vended data (see this). I don't think CDK should be creating these policies at all unless the customer is trying to enable vended data, which is not the case for ECS because ECS uses a role in the customer's account to write data.

[kevinswarner]

I believe this issue is causing problems for our attempts to deploy Fargate ECS tasks. I have not seen a resolution yet and am wondering if anyone has advice. We are simply defining our Fargate tasks like this...

taskDefinition.addContainer("SampleContainer", {
  image: aws_ecs.RepositoryImage.fromEcrRepository(repository, commitHash),
  essential: true,
  logging: aws_ecs.LogDriver.awsLogs({
    streamPrefix: "sample-container-prefix",
  }),
});

We have several (probably more than 10 or 15) tasks spread across different Fargate instances. We are now getting the following CDK error when attempting to deploy...

indexing-gc-dev-error-worker | 1/7 | 4:48:36 PM | CREATE_FAILED |
AWS::Logs::ResourcePolicy | ErrorWorkerTaskDefinition/ErrorWorkerContainer/
LogGroup/Policy/ResourcePolicy (ErrorWorkerTaskDefinitionErrorWorkerContainerLog
GroupPolicyResourcePolicy402AEAF5) Resource handler returned message: "Resource 
limit exceeded. (Service: CloudWatchLogs, Status Code: 400, Request ID: 
333b2ee5-001e-4a15-9ea55a7b726a4cde)" (RequestToken: 
faeb374a-d7a0-1850-e977-6cc73706e858, HandlerErrorCode: ServiceLimitExceeded)

If I remove the "logging" property from the config object, we are able to deploy. But, I was under the impression that adding the logging is necessary to get logs to be delivered to CloudWatch.

I appreciate any help with this.

[xmatusmic]

I believe this issue is causing problems for our attempts to deploy Fargate ECS tasks. I have not seen a resolution yet and am wondering if anyone has advice. We are simply defining our Fargate tasks like this...

taskDefinition.addContainer("SampleContainer", {
  image: aws_ecs.RepositoryImage.fromEcrRepository(repository, commitHash),
  essential: true,
  logging: aws_ecs.LogDriver.awsLogs({
    streamPrefix: "sample-container-prefix",
  }),
});

We have several (probably more than 10 or 15) tasks spread across different Fargate instances. We are now getting the following CDK error when attempting to deploy...

indexing-gc-dev-error-worker | 1/7 | 4:48:36 PM | CREATE_FAILED |
AWS::Logs::ResourcePolicy | ErrorWorkerTaskDefinition/ErrorWorkerContainer/
LogGroup/Policy/ResourcePolicy (ErrorWorkerTaskDefinitionErrorWorkerContainerLog
GroupPolicyResourcePolicy402AEAF5) Resource handler returned message: "Resource 
limit exceeded. (Service: CloudWatchLogs, Status Code: 400, Request ID: 
333b2ee5-001e-4a15-9ea55a7b726a4cde)" (RequestToken: 
faeb374a-d7a0-1850-e977-6cc73706e858, HandlerErrorCode: ServiceLimitExceeded)

If I remove the "logging" property from the config object, we are able to deploy. But, I was under the impression that adding the logging is necessary to get logs to be delivered to CloudWatch.

I appreciate any help with this.

try this
Hi, what we do and it works is:

log_group = logs.LogGroup(
          self,
          f'log-group-name',
          log_group_name=f'/ecs/log-group-name',
          removal_policy = RemovalPolicy.DESTROY)

log_group.node.try_remove_child("Policy")

[kevinswarner]

@xmatusmic Thanks for the suggestion. I tried what I think you suggested, but received the same error. Here is my full code example for the stack...

import {
  App,
  Stack,
  StackProps,
  RemovalPolicy,
  aws_ecr,
  aws_iam,
  aws_ecs,
  aws_ec2,
  aws_elasticache,
  aws_logs,
} from "aws-cdk-lib";

interface ParseWorkerStackProps extends StackProps {
  environment: string;
  commitHash: string;
  isGCEnvironment: boolean;
  vpc: aws_ec2.IVpc;
  executionRole: aws_iam.IRole;
  taskRole: aws_iam.IRole;
  repository: aws_ecr.IRepository;
  redis: aws_elasticache.CfnReplicationGroup;
  cluster: aws_ecs.ICluster;
}

class ParseWorkerStack extends Stack {
  constructor(
    scope: App,
    id: string,
    {
      commitHash,
      isGCEnvironment,
      vpc,
      executionRole,
      taskRole,
      repository,
      redis,
      cluster,
      ...props
    }: ParseWorkerStackProps
  ) {
    super(scope, id, props);

    const executionRoleReference = aws_iam.Role.fromRoleArn(
      this,
      "ExecutionRole",
      executionRole.roleArn
    );

    const taskRoleReference = aws_iam.Role.fromRoleArn(
      this,
      "TaskRole",
      taskRole.roleArn
    );

    const logGroup = new aws_logs.LogGroup(this, "ParseWorkerLogGroup", {
      removalPolicy: RemovalPolicy.DESTROY,
    });
    logGroup.node.tryRemoveChild("Policy");

    const taskDefinition = new aws_ecs.FargateTaskDefinition(
      this,
      "ParseWorkerTaskDefinition",
      {
        executionRole: executionRoleReference,
        taskRole: taskRoleReference,
        memoryLimitMiB: 16384,
        cpu: 8192,
      }
    );

    taskDefinition.addContainer("ParseWorkerContainer", {
      image: aws_ecs.RepositoryImage.fromEcrRepository(repository, commitHash),
      essential: true,
      logging: new aws_ecs.AwsLogDriver({
        logGroup,
        streamPrefix: "indexing-parse-worker-",
      }),
    });

    const securityGroup = new aws_ec2.SecurityGroup(this, "SecurityGroup", {
      vpc,
      allowAllOutbound: true,
    });

    new aws_ecs.FargateService(this, "ParseWorkerFargateService", {
      cluster,
      taskDefinition,
      desiredCount: isGCEnvironment ? 1 : 3,
      maxHealthyPercent: 200,
      minHealthyPercent: 100,
      securityGroups: [securityGroup],
      vpcSubnets: {
        subnetGroupName: "application_layer",
      },
      circuitBreaker: {
        rollback: !isGCEnvironment,
      },
    });
  }
}

export default ParseWorkerStack;

This seems like a fairly significant limitation, but I may be misunderstanding things. We have plans to stand up many Fargate ECS tasks and services, but it seems that this problem will not allow us to do this IF we want logging to CloudWatch. IF I remove the "logging" option on the addContainer, the service and tasks deploy fine, but we do not get any logging at all in CloudWatch.

[xmatusmic]

logGroup.node.tryRemoveChild("Policy");
it will work, move this line to the bottom of your code

[kevinswarner]

Thanks @xmatusmic That seems to work. I think the reason this works (but I am guessing a little here) is that the "addContainer" call with the logging property is adding the policy, so you have to place the "tryRemoveChild" call after the "addContainer" call. Would be great for someone from the CDK team to provide even more insight into this. But, I at least can now deploy multiple Fargate tasks / services.

[xmatusmic]

Thanks @xmatusmic That seems to work. I think the reason this works (but I am guessing a little here) is that the "addContainer" call with the logging property is adding the policy, so you have to place the "tryRemoveChild" call after the "addContainer" call. Would be great for someone from the CDK team to provide even more insight into this. But, I at least can now deploy multiple Fargate tasks / services.

good :) yeah, its exactly what you said, it has to be done after you add log to the container.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.