VolumeSource: OCI Artifact and/or Image

[sallyom]

Enhancement Description

• One-line enhancement description (can be used as a release note): VolumeSource: OCI Artifact or OCI Image
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/4639-oci-volume-source
• Discussion Link:
• Primary contact (assignee): sgrunert@redhat.com, somalley@redhat.com
• Responsible SIGs: sig-node, sig-storage
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.31
  • Beta release target (x.y): 1.32
  • Stable release target (x.y): 1.33
• Alpha
  • KEP (k/enhancements) update PR(s): KEP-4639: adding OCI VolumeSource #4642
  • Code (k/k) update PR(s):
    • [KEP-4639] Add OCI VolumeSource CRI API kubernetes#125659
      • [KEP-4639] Update CRI API and workflow #4751
    • [KEP-4639] Add ImageVolumeSource API kubernetes#125660
    • [KEP-4639] Add ImageVolumeSource implementation kubernetes#125663
    • Nice to have:
      • [KEP-4639] Mention that fsGroupChangePolicy has no effect kubernetes#126281
      • Fix runtime panic in imagevolume CanSupport method kubernetes#126323
      • [KEP-4639] Add ImageVolumeSource node e2e tests kubernetes#126220
  • Out of tree update PR(s):
    • Add OCI Volume Source support to crictl [create|run] --with-pull kubernetes-sigs/cri-tools#1464
    • Add OCI/Image Volume Source support cri-o/cri-o#8317
    • Make ImageVolume garbage collection work cri-o/cri-o#8408
    • CRI-O: add pull-kubernetes-node-crio-cgrpv2-imagevolume-e2e test job test-infra#33071
    • Fix imagevolume test focus test-infra#33088
  • Docs (k/website) update PR(s):
    • Document OCI volume sources / KEP-4639 website#46946
    • Blog post about OCI volume sources / KEP-4639 website#46960

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[sallyom]

I've had informal discussions about this - there's enough interest IMO to open a KEP & I will present this issue at upcoming sig-node, sig-storage mtgs with a KEP draft

/sig node
/sig storage

[xing-yang]

Can you use the Volume Populator? It allows you to create a PVC from an extenal data source. https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1495-volume-populators

[saschagrunert]

Happy to support here from a SIG node perspective.

cc @kubernetes/sig-node-proposals

[mikebrow]

+1 happy to help from the SIG-Node/CRI and OCI image/distribution spec perspectives..

[SergeyKanzhelev]

/label lead-opted-in
/milestone v1.31

as discussed at SIG Node meeting this week, we will try and see if this can make it to 1.31

[SergeyKanzhelev]

/stage alpha

[rhuss]

For reference, in KServe a workaround for directly accessing files within an OCI image is currently implemented and available via a sidecar approach ("modelcar") by leveraging root FS system access via the /proc filesystem when shareProcessNamespace: true is set on the Pod. You can find details in the KServe documentation and in the Design Document. It actually implements the desired behavior with current means, but of course is more or less just a workaround for the OCI volume type (as discussed here and raised already a long time ago in kubernetes/kubernetes#831).

So KServe would be more than happy to leverage such a volume type, and we are happy to support any efforts in this direction.

[sreeram-venkitesh]

/stage alpha

[sreeram-venkitesh]

Hello @sallyom 👋, v1.31 Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 14th June 2024 / 19:00 PDT Thursday 13th June 2024.

This enhancement is targeting for stage alpha for v1.31 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.31. KEPs targeting stable will need to be marked as implemented after code PRs are merged and the feature gates are removed.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline (6th June) so that the PRR team has enough time to review your KEP before the enhancements freeze.

For this KEP, most of the above items are taken care of in #4642. We'd need to do the following:

• Update the status from provisional to implementable in the kep.yaml file here
• Create a prod-readiness yaml file as shown here.
• Update the graduation criteria in the KEP readme file.
• Make sure that the PRR questionnaire is filled.

The status of this enhancement is marked as At risk for enhancements freeze. Once the above tasks are done, I can mark it as tracked.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Let me know if you have any questions! Thank you!

[sreeram-venkitesh]

@sallyom Pinging once again as a slight reminder that we're approaching the enhancements freeze deadline on 14th June, this Friday!

[dipesh-rawat]

Hi @sallyom @SergeyKanzhelev 👋, 1.31 Enhancements team here,

Just a quick friendly reminder as we approach the enhancements freeze in few hours, at 02:00 UTC Friday 14th June 2024 / 19:00 PDT Thursday 13th June 2024.

The current status of this enhancement is marked as at risk for enhancement freeze. There are a few requirements mentioned in the comment #4639 (comment) that are addressed as part of PR #4642 which still needs to be merged.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

[dipesh-rawat]

Hello @sallyom @SergeyKanzhelev 👋, 1.31 Enhancements team here.

Unfortunately, this enhancement did not meet requirements for enhancements freeze.

If you still wish to progress this enhancement in v1.31, please file an exception request as soon as possible, within three days. If you have any questions, you can reach out in the #release-enhancements channel on Slack and we'll be happy to help. Thanks!

[sreeram-venkitesh]

/milestone clear

[bitoku]

/assign

[sreeram-venkitesh]

Marked as tracked for enhancements freeze! Thanks everyone!

[Princesso]

Hello @bitoku, @sallyom 👋, 1.31 Docs Lead here.
Does this enhancement work planned for 1.31 require any new docs or modifications to existing docs?
If so, please follow the steps here to open a PR against dev-1.31 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday June 27, 2024 18:00 PDT.
Also, take a look at Documenting for a release to get yourself familiarised with the docs requirement for the release.
Thank you!

[hailkomputer]

Hi @saschagrunert , @sallyom

👋 from the v1.31 Communications Team! We'd love for you to opt in to write a feature blog about your enhancement!
Some reasons why you might want to write a blog for this feature include (but are not limited to) if this introduces breaking changes, is important to our users, or has been in progress for a long time and is graduating.

To opt in, let us know and open a Feature Blog placeholder PR against the website repository by 3rd July, 2024. For more information about writing a blog see the blog contribution guidelines.

Note: In your placeholder PR, use XX characters for the blog date in the front matter and file name. We will work with you on updating the PR with the publication date once we have a final number of feature blogs for this release.

[saschagrunert]

@hailkomputer thank you for the hint, I added a placeholder PR in kubernetes/website#46960

[utam0k]

@saschagrunert Do you know where the containerd side is implemented?
containerd has a special behavior when pulling images following their registry configuration. I would like to know if this behavior is the same when retrieving from OCI Artifact, so I would like to see the implementation.
https://github.com/containerd/containerd/blob/main/docs/cri/config.md#registry-configuration

[saschagrunert]

@saschagrunert Do you know where the containerd side is implemented? containerd has a special behavior when pulling images following their registry configuration. I would like to know if this behavior is the same when retrieving from OCI Artifact, so I would like to see the implementation. https://github.com/containerd/containerd/blob/main/docs/cri/config.md#registry-configuration

I would assume that we have no implementation right now for containerd. @mikebrow may provide more details on that.

[sreeram-venkitesh]

Hey again @saschagrunert @sallyom 👋 v1.31 Enhancements team here,

Just checking in as we approach code freeze at 02:00 UTC Wednesday 24th July 2024 / 19:00 PDT Tuesday 23rd July 2024.

Here's where this enhancement currently stands:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PR/s are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

For this enhancement, it looks like the following PRs are open and need to be merged before code freeze (and we need to update the Issue description to include all the related PRs of this KEP):

• [KEP-4639] Add OCI VolumeSource CRI API kubernetes#125659
• [KEP-4639] Update CRI API and workflow #4751
• [KEP-4639] Add ImageVolumeSource API kubernetes#125660
• [KEP-4639] Add ImageVolumeSource implementation kubernetes#125663

Out of tree:

• Add OCI/Image Volume Source support cri-o/cri-o#8317
• Add OCI Volume Source support to crictl [create|run] --with-pull kubernetes-sigs/cri-tools#1464

If you anticipate missing code freeze, you can file an exception request in advance.

I'm marking this KEP as At risk for code freeze for now since all the associated PRs haven't been approved/lgtm'd. But everything looks good here, thanks a lot for keeping the issue description up to date with all the PRs! I can update the KEP to tracked as soon as the code PRs are approved. Also, please let me know if there are other PRs in k/k we should be tracking for this KEP. As always, we are here to help if any questions come up. Thanks!

[sreeram-venkitesh]

/milestone v1.31

[sreeram-venkitesh]

Hi again @sallyom @saschagrunert! We are one week away from code freeze and I wanted to ping you here as a friendly reminder for making sure the code PRs are merged in time before the deadline. Also, please let me know if there are any more PRs that needs to be tracked other than the ones linked in #4639 (comment). Thanks!

[sreeram-venkitesh]

@sallyom @saschagrunert All the PRs mentioned in #4639 (comment) are merged. Are we waiting for cri-o/cri-o#8408 and kubernetes/kubernetes#126220 to be merged as part of this KEP too? Please let me know!

[saschagrunert]

@sreeram-venkitesh we're now actively working on the docs, the implementation is done and everything which is still open is nice to have. Out of tree PR's are not a requirement for this KEP.

[sreeram-venkitesh]

Thanks @saschagrunert! Marking this KEP as tracked for code freeze 🎉

[sreeram-venkitesh]

The PR kubernetes/kubernetes#126220 which adds tests is still open. Please make sure to get it merged by the test freeze deadline (01:00 UTC Wednesday 31st July 2024 / 19:00 PDT Tuesday 30th July 2024).

@saschagrunert @sallyom

[utam0k]

@saschagrunert FYI: This is the issue for containerd to support this KEP.
containerd/containerd#10496

[saschagrunert]

@utam0k thank you for tracking the enhancement into containerd. Let me know if you need anything as support!

[asviel]

Hi everyone. Are there plans to support subPath or (better) projected volumes for OCI VolumeSource? I have multiple (10+) ML models and would like to be able to mount them in one place:

apiVersion: v1
kind: Pod
metadata:
  name: image-volume
spec:
  containers:
    - name: shell
      command: ["sleep", "infinity"]
      image: debian
      volumeMounts:
        - name: models
          mountPath: /models
  volumes:
    - name: models
      projected:
        sources:
          - image:
              reference: quay.io/crio/artifact-1:v1
              pullPolicy: IfNotPresent
          - image:
              reference: quay.io/crio/artifact-2:v1
              pullPolicy: IfNotPresent

[saschagrunert]

@asviel maybe in some later graduation of the feature. Thank you for the input here!

[deepforu47]

Hello Everyone,
Anyone able to test image volume using Minikube with pod man and ImageVolume featureGate is enabled.

I have tried and getting below error-
Error: failed to generate container "cc992779b80f62ff8a8db99d005ab1caa6b282e18f2a6eb162cc5b2d882eb283" spec: failed to generate spec: failed to mkdir "": mkdir : no such file or directory

pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: image-volume
spec:
  containers:
  - name: shell
    command: ["sleep", "infinity"]
    image: busybox
    volumeMounts:
    - name: volume
      mountPath: /volume
  volumes:
  - name: volume
    image:
      reference: quay.io/crio/alpine:3.9
      pullPolicy: Always

[saschagrunert]

@deepforu47 please open an issue in https://github.com/kubernetes/kubernetes. I assume you're using containerd? Right now only CRI-O main supports it.

[deepforu47]

Sure @saschagrunert will open new issue.
On your comment yes I was using containers, then tried with CRI-O as well. I saw documentation that it will work with CRI-O > 1.31(which I can't see http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/).

New Error -

Warning Failed 2s kubelet Error: mount.HostPath is empty

[mikebrow]

@deepforu47 @saschagrunert FYI for tracking wip pr for containerd v2 release candidate: containerd/containerd#10579