-
Notifications
You must be signed in to change notification settings - Fork 511
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Configure image registry credentials in containerd #1227
Comments
Hi @Niksko, thanks for the feature request and outlining your alternative approaches in contrast. We'll look into how to model and support registry credentials through our settings API. |
I'm happy to submit a PR for this, it doesn't seem like a huge change. A few questions:
A flat array of objects is easier for mustache to iterate over. Using the |
I like your approach, but you nailed the big concern: I'm not comfortable with handling secrets with the mechanisms we have today. Userdata is not encrypted; settings are in plaintext when they're stored on disk, returned by the API, and rendered into templates. I expect we'd want containerd to support per-registry credential helpers or plugins first, as discussed in containerd/containerd#6637 - then it would just be a matter of shipping those helpers in the base image, or offering a way to load them at runtime. |
Would bootstrap containers work? |
@gabegorelick yes, they probably would, good suggestion! |
Bootstrap containers would be blocked by the SELinux policy from modifying configuration files directly, but they would help work around the security problems of passing secrets in user data. For EC2, the instance could have role credentials that allowed pulling a bootstrap container and reading secrets from SSM or Secrets Manager, then apply them to the API using Settings are no longer readable by unprivileged containers after ea35f1b, so the remaining hurdle is to secure the configuration files after they're generated from templates. That could be done by either:
|
Yeah I kind of assumed that when mentioning bootstrap containers. But this leads to another issue: how would it work if you wanted to pull these containers from a registry that can't be accessed via instance role? E.g. if your bootstrap container is in Docker Hub you can't expect it to set your Docker Hub credentials. |
This is a blocker to the adoption of Bottlerocket as this can currently be easily done via userdata. An alternative view on this is that the credentials being provided here should only be R/O so the MVP would be to be able to specify these as plaintext with warnings and then to iterate on a "better" solution. |
Seems like the alpha Kubelet credential provider plugins feature may help us solve this. The kubelet arguments and configuration seem like they would be straightforward to implement, but how would we facilitate providing the binaries? Perhaps via a bootstrap container? |
@Niksko I hadn't seen that feature - it looks pretty neat and seems like it could help, at least at first glance. You hit the nail on the head though, making sure the correct credential provider binaries are provided is indeed part of the problem with it. We'll look into this feature more as it matures - hopefully it can help! |
Has there been any progress on this? With an AL2 node we write the credentials out to Could the API not be updated to support this as @Niksko suggested which would allow the end user to decide if they want to add credentials to userdata or to use a bootstrap container to fetch them dynamically and set them with the API? |
For your interest, I implemented this feature through the kubelet credential provider. I developed a token server and a simple kubelet credential provider, and inject the provider using the bootstrap container. Here is a company tech blog article - https://hyperconnect.github.io/2022/02/21/no-more-image-pull-secrets.html (unfortunately, it is Korean, so you may need a translator...) |
What I'd like:
A way to configure image registry credentials in containerd, for pulling images from private registries (or in the case of Dockerhub, with authentication to avoid rate limiting).
This is a common alternative to defining image pull secrets on a per pod basis, and is recommended by the Kubernetes docs if you want particular credentials to apply to all pods on a node.
Having this exposed as a setting would be ideal. Containerd seems to support configuration of registry authentication via toml file as per this pr. As a first pass, could we expose a setting that passes through to this section of the containerd configuration?
Any alternatives you've considered:
ImagePullPolicy
is set correctly, the node will just use the pre-pulled image. We could likely orchestrate pulling all essential images shortly after startup, but again, it would be difficult to do.The text was updated successfully, but these errors were encountered: