You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion): Any version using containerd
Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version): Any version using containerd
AMI Version: Any version using containerd
Kernel (e.g. uname -a): NA
Release information (run cat /etc/eks/release on a node):
BASE_AMI_ID="ami-044dcf122cb58da69"
BUILD_TIME="Mon Jul 3 03:32:16 UTC 2023"
BUILD_KERNEL="5.4.247-162.350.amzn2.x86_64"
ARCH="x86_64"
Issue:
The issue here is that the configuration used for containerd can result in significantly increased disk usage because containerd will store both the packed and unpacked layers which in the case where the image does not compress well as a tarball can result in increased utilization of the root volume. To see this in action you can used these commands [1] to make a dockerfile as the random data will not compress well.
The workaround is to create your own containerd config file then add discard_unpacked_layers = true to the plugins."io.containerd.grpc.v1.cri".containerd section. Then pass this to the bootstrap command using the --containerd-config-file after putting the file on the instance, usually by downloading from S3 although a custom AMI could also be used. However, this seems like a lot of work just to bring containerd in line with the space usage of docker. Which bring us to the question, should the containerd-config.toml file have discard_unpacked_layers set to true by default?
I think it makes sense to enable this by default. My understanding is you'd only need the unpacked layers if you plan to push the image later on; which shouldn't be a common case on a Kubernetes node. I'll run this by the container runtime team for a sanity check 👍
Environment:
aws eks describe-cluster --name <name> --query cluster.platformVersion
): Any version using containerdaws eks describe-cluster --name <name> --query cluster.version
): Any version using containerduname -a
): NAcat /etc/eks/release
on a node):Issue:
The issue here is that the configuration used for containerd can result in significantly increased disk usage because containerd will store both the packed and unpacked layers which in the case where the image does not compress well as a tarball can result in increased utilization of the root volume. To see this in action you can used these commands [1] to make a dockerfile as the random data will not compress well.
The workaround is to create your own containerd config file then add
discard_unpacked_layers = true
to theplugins."io.containerd.grpc.v1.cri".containerd section
. Then pass this to the bootstrap command using the --containerd-config-file after putting the file on the instance, usually by downloading from S3 although a custom AMI could also be used. However, this seems like a lot of work just to bring containerd in line with the space usage of docker. Which bring us to the question, should the containerd-config.toml file have discard_unpacked_layers set to true by default?[1]
The text was updated successfully, but these errors were encountered: