Enable discard_unpacked_layers by default

[cartermckinnon]

Issue #, if available:

Fixes #1359.

Description of changes:

Sets discard_unpacked_layers = true in the containerd config file. This is a disk space optimization.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Testing

Verified the config key:

> containerd --version
containerd github.com/containerd/containerd 1.6.19 1e1ea6e986c6c86565bc33d52e34b81b3e2bc71f

> containerd config default | grep -A3 -B3 discard_unpacked_layers
    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"
      disable_snapshot_annotations = true
      discard_unpacked_layers = false
      ignore_rdt_not_enabled_errors = false
      no_pivot = false
      snapshotter = "overlayfs"

[bwagner5]

nice! LGTM

[dineshp-09]

@cartermckinnon After this change we are having issue with image export in containerd:
containerd/containerd#9277
containerd/nerdctl#2588
when we disable this flag manually, image export is successful.
Would you have any thoughts on this?

[cartermckinnon]

Why do you need to export images from the node’s container runtime?

[dineshp-09]

@cartermckinnon we need to create an image tar and do some processing in that tar.
basically, the image export functionality should work.

[cartermckinnon]

You could do that within a pod without interacting with the node's container runtime, with a tool like crane: https://github.com/google/go-containerregistry/tree/main/cmd/crane

In general, the container runtime used by the kubelet should be treated as a part of Kubernetes, it shouldn't be used by pods. Enabling discard_unpacked_layers is the right default for a Kubernetes node, because the kubelet never needs to push or export an image once it's pulled -- and it's a significant disk space optimization.

[dineshp-09]

@cartermckinnon got your point. Thanks for the crane tool suggestion.

[hellt]

👋 @cartermckinnon
I wonder if you know a workaround for a challenge we get with this setting being the new default.

In our case we need to export an image from the k8s worker node to the pod. The reason we need to it via the node is that we want k8s users to leverage Pull Secrets and get the image on the nodes using their credentials that our pod has no idea about.

Once the image is pulled, we wanted to export it from the node's image store to the POD by bind mounting the containerd.sock, but it seems that it is not possible anymore, because of the missing layer.

Would appreciate if anyone found a workaround for this.

[cartermckinnon]

In general, I would recommend a tool like crane for working with container images inside a pod. Mounting the node's container runtime socket into a pod is risky because 1) it gives the pod the ability to mutate any other pod on the node and 2) it lets the pod circumvent its resource limits by consuming CPU/memory/disk that it has not been allocated (which can interfere with kubelet).

That being said, you know your workload better than I do. 😄 We're working on #1630, which will allow you to disable this setting in the containerd config.

[hellt]

Thanks for your comments, @cartermckinnon

We looked at crane and skopeo, but we couldn't still export the image that was pulled by a kubelet and extensive googling failed me as well.

We are not concerned about the security limitations or resource shunting as, indeed, the clusters we create are ephemeral, not multi-tenanted and specifically are made for CI.

The only reason we need the export capability is to export the image pulled by the k8s and import it in the docker-in-docker pod we are running. This works great when discard_unpacked_layers is not set.

Would be amazing to have this setting exposed tot he EKS users, but also I would be super super curious to know why a simple layer packing can't be done when layers are missing, since the image is run-worthy, why can't one pack it up?

[zaidmnsr]

You can disable it via setting it to false using below strategy.

I updated the config.toml using below command as eks bootstrap script is using this file as the base file.

sed -i 's/discard_unpacked_layers = true/discard_unpacked_layers = false/g' /etc/eks/containerd/containerd-config.toml

Once you update this, you can verify this using below command from your worker node.

containerd config dump

[hellt]

Hi @zaidmnsr
would you mind sharing more details about this approach? At what step you change the config.toml?

[zaidmnsr]

Hi @hellt

You can use this command in userdata script. You can modify userdata script in launch template by creating a new version of launch template.