Set containerd LimitNOFILE to recommended value

[cartermckinnon]

Description of changes:

The service unit in AL2 currently sets LimitNOFILE=infinity on containerd. This is known to cause issues, and the recommendation is to set it explicitly to LimitNOFILE=1024:524288 on systems using systemd older that v240.

More info: containerd/containerd#8924

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.to replicate. -->

See this guide for recommended testing for PRs. Some tests may not apply. Completing tests and providing additional validation steps are not required, but it is recommended and may reduce review time and time to merge.

[cartermckinnon]

/ci

[github-actions]

@cartermckinnon roger that! I've dispatched a workflow. 👍

[github-actions]

@cartermckinnon the workflow that you requested has completed. 🎉

Kubernetes version	Build	Launch	Test
1.23	success ✅	success ✅	success ✅
1.24	success ✅	success ✅	success ✅
1.25	success ✅	success ✅	success ✅
1.26	success ✅	success ✅	success ✅
1.27	success ✅	success ✅	success ✅
1.28	success ✅	success ✅	success ✅

[dims]

LGTM

[mmerkes]

LGTM

[atishpatel]

FYI, this AMI release caused issues in production for my eks cluster.

[2023-12-22 03:31:18.128][198][critical][assert] [external/envoy/source/common/network/socket_interface_impl.cc:72] assert failure: SOCKET_VALID(result.return_value_). Details: socket(2) failed, got error: Too many open files

Can we revert this PR? And is there a higher number or setting we can use for the default soft limit?
opened case with AWS, case ID 170322283601444

---

edit:
Adding this to our bootstrap in userData for now to 10x the soft limit.

- /usr/bin/sed -i 's/^LimitNOFILE.*$/LimitNOFILE=102400:524288/' /etc/systemd/system/containerd.service.d/20-limitnofile.conf || true

[lnattrass]

This also caused issues in production for our EKS cluster running envoy.
Very unlucky this was released the day before the christmas break.

[illiasokolov]

+1 Same for my clusters

Is there a workaround to change that using Terraform?

[jpedrobf]

+1 here. The karpenter controller updated to the new ami and we crashed.

Failed to resolve '<url>': No file descriptors available (after 0ms in state CONNECT, 2 identical error(s) suppressed)

[erikkrieg]

Yes, this resulted in a major production outage :(

[pkoraca]

Same same, Merry Xmas 🎅

[shawnce]

The limit basically went from "infinity" to 1,024 with "infinity" looking to be 1,048,576 (aka 1024*1024) ... at least from what we saw. This seems a rather risky change to make without forewarning.

[polarathene]

The limit basically went from "infinity" to 1,024

The soft limit. This is standard outside of containers (where it's been incorrectly set to infinity). Software that requires anything more than the soft limit is meant to request to raise that (with the hard limit as the ceiling).

This seems a rather risky change to make without forewarning.

All this change has done has exposed that software is incorrectly relying on bad settings in the first place. It would be better to identify what software that is and to have it correctly fix the problem at their end.

infinity had resulted in many bug reports over the years, and a very slow process in getting the change to fix it approved/upstreamed. I extensively researched the topic to justify the change as the correct approach. The only part of my contribution that was dismissed was keeping an explicit config with contextual comment for awareness as a form of documentation, with removal 12-24 months later.

---

Context

In the meantime, you would be advised to workaround that by setting a higher soft limit, but I would highly discourage infinity. infinity can resolve to approx 1 million (2^20), or worse over a billion (2^30) depending on the host system.

Notably with the 2^30 scenario is when other software that expects a sane soft limit (such as daemons), regress heavily in performance (a task that would complete in less than a second, can stall for 10 minutes.. or more than an hour).

• Fedora's DNF package manager was a good example of this IIRC.
• rsyslog, postsrsd, fail2ban are all examples of software that was negatively impacted by infinity.
• Some databases have had many reports of consuming excessive amounts of memory during init triggering OOM.

The soft limit is meant to be low, only raised for processes that are aware they need it and explicitly request to raise it. Please respect that.

Github Actions has a hard limit of 65,536 (2^16) on their CI runners for example. The limits apply per-process, 2^16 may be too low of a hard-limit for larger production deployments (with databases), but it's more likely you're encountering software that hits the 1024 soft limit and doesn't attempt to raise the hard-limit at all, which is an easy bottleneck to hit (implies the software is only ever really deployed within a containerized environment, otherwise that'd have been fixed promptly).

---

Workaround

If you must raise it as a workaround, do not use infinity. 524288 (2^19) should easily satisfy the majority of software (enough to run 65k busybox containers with 200GiB+ memory).

The cases where that is insufficient are generally documented by the software and workload dependent.

For Envoy, there is this 2019 issue regarding limits. That is a demonstration of where you'll need a sufficient hard limit. This 2022 issue also requests for Envoy to raise the soft limit, but the maintainers dismiss the request.

I've opened an feature request at Envoy to convey the importance of their software raising the limit internally.

[jordanlewis]

All this change has done has exposed that software is incorrectly relying on bad settings in the first place. It would be better to identify what software that is and to have it correctly fix the problem at their end.

I disagree with your assessment here. This may be true for the soft limit change, but this recommendation also changed the hard limit, which caused applications that legitimately request a higher ulimit than the new hard cap to fail. That is the failure mode that our workload encountered as a result of the AMI update.

Raising the hard limit therefore isn't just a "workaround" as there are plenty of legitimate use cases where software may need more than 500k FDs. It would be better to recommend a higher hard cap - sure infinity is not ok at this stage, but why recommend 2^19 instead of 2^20 for example?

[polarathene]

TL;DR:

• My contribution to prefer the defaults 1024:524288 still seems valid, but I can understand it not being appropriate with the deployments you work with.
• You should have more flexibility with the hard-limit, most of the regressions that infinity caused were due to the soft limit not being 1024 AFAIK. But software will raise the soft-limit to the hard-limit implicitly, any regression will scale with that increase.
• 524288 seemed appropriate for workstations and CI.
  • I don't have expertise in large deployments to advise an appropriate hard-limit.
    • Try 2^24? (16.77 million, only a 32x increase much less than 2048x)
  • Prior to infinity the limit was set to 2^20 (only because it was more than enough) to removing a few hundred containers (probably due to an implicit 1024 soft limit), not for any specific process needs within a container. The context for raising to infinity was lacking and seemed misunderstood.

---

In the collapsed section below, I would like to highlight an interest in knowing what use-cases are demanding more than 500k FDs per process.

• If the process can spawn child processes that can delegate the workloads creating all those FDs, they'd each have their own limit which does not contribute to parent process usage.
• What is the software doing where that requires all FDs managed by a single process? Especially at scale where I thought load balancing multiple instances was a common practice?
• Could you share what sort of peak FD usage you're seeing for a process?
• If these are open-source projects like Envoy, could you cite them please?

---

Original response (verbose)

which caused applications that legitimately request a higher ulimit than the new hard cap to fail. That is the failure mode that our workload encountered as a result of the AMI update.

Yes, sorry about that.

I was not aware that you were unable to set a limit higher than the AMI LimitNOFILE hard limit, that's not applicable from my contribution which did the opposite (removed LimitNOFILE from service configs, where you can configure a higher hard-limit like it was before via a systemd drop-in override).

I don't have access to such environments, only what I could find online in discussions / reports and software docs.

• I don't recall finding anything that needed more than systemd default hard-limit of 524288 / 2^19, and had extensive research coverage documented for anything I was aware of.
• With Envoy, there isn't any documentation on the subject only a few issues where the Debian hard-limit of 2^20 was not sufficient and maintainers advice to set the soft limit to max 🤷‍♂️

Out of curiosity, is the software that raises the soft-limit to the hard-limit public? Does it document that requirement or is it specific to the scale of your deployment needs?

Does your software actually use more than 2^19 FDs per process, or was the problem that you had an explicit hard-limit set like 2^20 where only that failure is the issue and it could have functioned fine with 2^19? (I only say this since sometimes configuration is parroted, especially when the values are sourced as "works for me")

---

It would be better to recommend a higher hard cap - sure infinity is not ok at this stage, but why recommend 2^19 instead of 2^20 for example?

The other main motivation behind choosing 524288 was for consistency outside of containerized environments.

You'd have to raise the hard-limit there, so the assumption was businesses would have the talent with expertise to more easily identify this configuration need and handle it (vs the many individuals affected by the bugs infinity caused).

It's a bit problematic when software that runs fine out of a container behaves in weird unexpected ways in a container environment due to defaults biased to enterprise.

Doubling this hard limit did not seem worthwhile to match the traditional hard-limit of 2^20 (that existed prior to systemd v240 normalizing one, IIRC it was not as consistent prior, but 2^20 was a bound set by the kernel but not the default hard-limit)

It's been years with 524288 being adequate (it was not chosen arbitrarily).

• If it were a more common issue encountered, there would be a larger push to raise it? (Over a year since the change, the author notes how there was little pushback received from the reduced hard limit)
• The only time that doesn't seem to be the case is with enterprise deployments, and 2^20 is not necessarily enough in that context (according to an Envoy report) nor yours it seems? Aligning with systemds hard-limit seemed like an appropriate decision?

---

Raising the hard limit therefore isn't just a "workaround" as there are plenty of legitimate use cases where software may need more than 500k FDs.

I'd appreciate you sharing some of those use cases?

• The limits are per process (child processes have their own limit, they don't contribute to the parents FD limit usage), thus software that can scale horizontally via multiple processes to split the FD usage is better than a monolithic process amassing FDs.

My "workaround" advice refers to raising the soft-limit to 2^19 as an example for software like Envoy that won't raise it.

• There is no workaround suggestion regarding the hard-limit, which I state is possible but should be obvious when you need it.
• The "workaround" phrasing was more oriented towards reverting the soft limit. Since the diff from this PR is only additive, I don't know what the limit was prior? 🤷‍♂️ (looks like it's eagerly applying the change arriving in containerd v2?)
• Thus in future, the AMI may add a similar config override but instead prefer LimitNOFILE=42 (where 42 is whatever soft limit with implicit hard limit is deemed appropriate).
  • Otherwise the 1024 soft limit is still likely to cause some problems it seems until affected processes raise it internally.
  • Meanwhile the more you increment the soft limit from 2^19 to 2^30 the more that regression scales on affected software, additionally anything using select() implicitly will be fun to troubleshoot.

I have no real opinion with the AMI configuration here, other than discouraging infinity.

• Those contributing such a change here should hopefully understand the context of what limits would be appropriate. It's not a fun topic, so I can understand why the change to Docker / containerd may be followed.
• This is not an area I work in, but those that do could help contribute what the highest FD usage for a process they observe is?
• Some software implicitly raises the soft-limit to the hard-limit, and it has been observed with MySQL and JDK allocating memory equivalent to the limit (524288 4MB vs 2^30 8GB).

MongoDB documents 64k, Kafka documents 100k (as a minimum).

• Since I doubt anyone wants to guess a sufficient value that turns out to be too low once deployed, I would assume many would set a value that seems high enough (with typical advice online being 2^20 or infinity).
• Tuning the value down could be done, but if raising it high enough doesn't cause any problems I don't really see anyone investing the time (if there is no clear benefits from doing so).

The approach systemd took involved finding what an appropriate hard limit would be with a variety of software and feedback cycles IIRC. Initially it was going to be 2^18, possibly less. I recall Google Earth needed around 300k, and was one of the reasons the limit was raised to 2^19 before official release of v240.

For Docker and containerd, since they no longer set LimitNOFILE they just go with what is configured as default by systemd.

• When that's not sufficient and you need a higher hard-limit, I believe you could still set that beyond 524288 per service as an override.
  • The change with the AMI here apparently sets a ceiling, which prevented your own config to exceed that.
  • On Docker and Containerd, I think I recall this not being the case and that I could have a container run with a higher limit than the one I set for dockerd / containerd processes.
• The system-wide default could also be adjusted. I have very little AWS experience, but assume somewhere it would be possible to support your own override for the hard-limit when you need that, which shouldn't be blocked like you experienced? (if not perhaps that's a relevant feature request)