consul-dataplane 1.23 and 1.3.1 keep crashing with "Too many open files" on EKS

[remiphilippe]

Hello,
We use EKS 1.24 (containerd) configured with:

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity

But our consul-dataplane containers keep dying with the following error:

2023-12-27T15:42:31.825Z+00:00 [critical] envoy.assert(23) assert failure: SOCKET_VALID(result.return_value_). Details: socket(2) failed, got error: Too many open files
2023-12-27T15:42:31.825Z+00:00 [critical] envoy.backtrace(23) Caught Aborted, suspect faulting address 0x176b00000010
2023-12-27T15:42:31.825Z+00:00 [critical] envoy.backtrace(23) Backtrace (use tools/stack_decode.py to get line numbers):
2023-12-27T15:42:31.825Z+00:00 [critical] envoy.backtrace(23) Envoy version: c2919e90b0e63ad78602122d6c2c3e2c0df1e0fc/1.26.6/Clean/RELEASE/BoringSSL
[symbolize_elf.inc : 1000] RAW: /proc/self/task/16/maps: errno=24

we can't find any reasons for this, even our elastics work on these nodes (which are usually the fd problem child). Any pointers would be great!

Thanks

[remiphilippe]

probably going to self reply, but looks a lot like this: awslabs/amazon-eks-ami#1551

[remiphilippe]

confirmed that was the issue