Cannot open s3 fluentbit config file when using init container

[cadence-mathieu-beliveau]

Describe the question/issue

Hi! I am systematically getting a file not found error when trying to read my fluentbit config file from s3.

Configuration

ECS task def:

 {
            "name": "solution-service-1-firelens-logrouter",
            "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:init-debug-2.32.2.20240516",
            "cpu": 0,
            "portMappings": [],
            "essential": true,
            "environment": [
                {
                    "name": "aws_fluent_bit_init_file_1",
                    "value": "arn:aws:s3:::my-bucket-name/fluent-bit.conf"
                }
            ],
            "mountPoints": [],
            "volumesFrom": [],
            "user": "0",
            "dockerLabels": {},
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "ServiceMeshStack-solutionservice1firelenslogrouterLogGroup0EEA1EA6-IcjZ4M4lXkNG",
                    "awslogs-region": "us-east-1",
                    "awslogs-stream-prefix": "solution-service-1-firelens-logrouter"
                }
            },
            "systemControls": [],
            "firelensConfiguration": {
                "type": "fluentbit",
                "options": {}
            }
        },

To be sure, I added full access to S3 to both the task role and task execution role.

Fluent Bit Log Output

AWS Cloudwatch logs for the firelens container:

time="2024-07-02T00:39:33Z" level=error msg="open arn:aws:s3:::my-bucket-name/fluent-bit.conf: no such file or directory"
time="2024-07-02T00:39:33Z" level=fatal msg="[FluentBit Init Process] Cannot open file: arn:aws:s3:::my-bucket-name/fluent-bit.conf\n"

Consider enabling debug logging by setting env var FLB_LOG_LEVEL=debug

Fluent Bit Version Info

Like it can be seen in the above config tried aws images tagged with: debug-2.32.2.20240516 and the regular 2.32.2.20240516

Had the issue with both version.

Cluster Details

• what is the networking setup?
• do you use App Mesh or a service mesh? YES
• do you use VPC endpoints in a network restricted VPC? NO
• ECS or EKS: ECS
• Fargate or EC2: Fargate launch mode
• Daemon or Sidecar deployment for Fluent Bit: Sidecar deployment

Application Details

I want a custom fluentbit config for proper multiline parsing and consuming events from tcp socket, not stdout.

Steps to reproduce issue

My intuition is that there is a permission issue for reaching the s3 config file but given that both task role and task exec role have the full access to s3 through the aws managed policy, I don't know what else can be needed.