Working Group Proposal for Reference Types #934

lachie83 · 2022-08-09T18:43:40Z

This is a culmination of many people's work over the last few months to introduce reference types to OCI as part of the OCI Reference Types WG.

This PR contains the changes from our chosen proposal, Proposal E, relevant to the image-spec. We're looking forward to hearing feedback from image-spec maintainers on these changes!

Associated distribution-spec PR - opencontainers/distribution-spec#335

vbatts · 2022-08-09T23:03:50Z

OHMAN

mounchin · 2022-08-10T19:29:29Z

artifact.md

+
+    Also, see [Pre-Defined Annotation Keys](annotations.md#pre-defined-annotation-keys).
+
+  User defined annotations MAY be used to filter various artifact types, e.g. signature public key hash, attestation type, and SBOM schema.


As the spec allows user defined annotations, do we want to cap the number of annotations stored or returned as part of referrers API ?

The current proposal is that all annotations would be returned. If we were to trim down the list, then we need to define the criteria. For example, only the first 10 annotations would be returned. That still doesn't cap the size of the payload since a single key could actually be large enough.

I think it would be good to raise this issue in the distribution spec since the referrers response payload and registry implementations needs to now ensure that data from all manifests are returned. opencontainers/distribution-spec#335 /cc @michaelb990

The index we return is rather minimal, so having enough annotations to cause this to exceed the recommended 4MB limit is likely to also exceed that limit in the referrer's manifest too. I think any limits we impose should be set on the original manifest, and not this API, to minimize complexity.

Lets say worst case, there are like 20 artifacts in the referrers API reponse and each had say about ~3MB of annotations, then the API reponse payload would be very large. In these cases what should the API return? May be capping the reponse size per artifact?

At that point, the response would trigger pagination, with one descriptor per page. It also wouldn't be supported to include more than one of those descriptors in the tag schema fallback. My hope is that manifests like that result in a bad enough UX, that bug reports get opened on the source and alternatives are used.

The downside of capping what is returned in the API response, is that any partial descriptor would then need to be pulled as a full manifest to determine if the client side filters match (the filtered annotation could be one that was trimmed). We would also need to provide an indicate that the descriptor is a partial response so clients know when they need to fall back to pulling the full manifest directly. When thinking of client implementations, that would probably be one pull for the API, one of the manifest by the artifact listing method, and then another pull of the manifest by the artifact get method, unless the manifest is cached between those methods.

sudo-bmitch · 2022-08-11T17:49:35Z

@opencontainers/image-spec-maintainers given the size of this, we are giving this 2 weeks for any objections before merging. It was presented on the OCI call today and hopefully a recording will be posted soon. After merging it still won't be a tagged release, so if there are any issues raised, we can always open another PR.

lachie83 · 2022-08-11T18:34:31Z

We would kindly like to ask for reviews and/or approval from maintainers by August 25th, 2022. Please let us know if you don't think this is attainable.

lachie83 · 2022-08-11T18:35:47Z

Adding link to similar comment on distribution-spec PR. opencontainers/distribution-spec#335 (comment)

descriptor.md

mikebrow

can we, should we add something over in image-index.md, such as another example, for using the image.index manifest to hold artifact manifests in the list of manifests? See example provided in distribution api..

mikebrow · 2022-08-12T18:20:21Z

manifest.md

@@ -65,6 +65,11 @@ Unlike the [image index](image-index.md), which contains information about a set

        Entries in this field will frequently use the `+gzip` types.

+- **`refers`** *[descriptor](descriptor.md)*
+
+    This OPTIONAL property specifies a [descriptor](descriptor.md) of another image or [artifact](artifact.md).


can we clarify if there is a restriction or not for media type of the refers descriptor.. image manifest and image index manifest for example..

We shouldn't have a restriction here. An artifact shipped with an Artifact manifest or this Image manifest can refer to any other manifest, including an Index. (I.e. You can attach a signature on a multi-platform manifest.)

before suggesting text changes here to include that clarification.. Do we have to worry about/restrict dependency loops here? E.g. an image manifest with a refers to it's own image.index.. an image manifest with a refers to an artifact that refers to this image manifest... a manifest of any type with a refers to itself?

For loops, I'm looking at whether the refers descriptor can point to a normal child of the manifest. If all children are blobs (as with Image and Artifact manifests), and not manifests (like we have with an Index), you can't point to a manifest that's a child.

It's a reverse pointer, so we aren't worried about parents (both pointers would be going the same direction). And you can't know the digest of the parent in advance without a hash collision/prediction.

there may be an opportunity in platform image index.. where an artifact manifest included in the platform image index could be to an artifact that does not have a refers.. and in that case the artifact could be implied to be for that index (refer) by inclusion... thus allowing an index to have an artifact that refers to itself without incurring the loop restriction.. thinking out loud.

artifact.md

vbatts

This looks good. Nothing breaking and will provide clarity on a tremendous pattern for clients and platforms to provide.

There are a couple nits that need to be cleaned up, but they can be follow-up PRs as they don't affect behavior/functionality.

One unfortunate aspect of merging this from the oci-playground branch is that a number of the commits are noisy as they get shadowed or cleaned up later... That isn't ideal and the only benefit I see in leaving it as is would be so that all the authors preserve their involvement in the commit history.

lachie83 · 2022-08-25T17:01:51Z

artifact.md

+
+- **`refers`** *[descriptor](descriptor.md)*
+
+  This OPTIONAL property specifies a [descriptor](descriptor.md) of another manifest.


cc @vbatts @sudo-bmitch - Does this provide more clarity in response to https://github.com/opencontainers/image-spec/pull/934/files/5240cdaf5cfb2f62f1b6b8860e5561332437270e#r955050333

Suggested change

This OPTIONAL property specifies a [descriptor](descriptor.md) of another manifest.

This OPTIONAL property specifies a [descriptor](descriptor.md) of another manifest. If defined, the artifact `refers` to, or has a reference to the specified manifest

I think so. (please put that sentence on its own newline)

vbatts · 2022-08-25T17:53:15Z

during discussion on the call, this has enough features and fields that it merits a point release, but not a major version change.

estesp · 2022-08-25T18:14:12Z

Thanks to all for the work on this in the WG and getting all this together (and dealing with all the feedback) and ready! IANAM but LGTM! 🎉

This is a culmination of many people's work over the last few months to introduce reference types to OCI as part of the [OCI Reference Types WG](oci-ref-type-wg). This PR contains the changes from our chosen proposal, [Proposal E][proposal-e]. [oci-ref-type-wg]: https://github.com/opencontainers/tob/blob/main/proposals/wg-reference-types.md [proposal-e]: https://github.com/opencontainers/wg-reference-types/blob/main/docs/proposals/PROPOSAL_E.md Co-authored-by: nisha <nisha@ctlfsh.tech> Co-authored-by: Josh Dolitsky <393494+jdolitsky@users.noreply.github.com> Co-authored-by: Sajay Antony <sajaya@microsoft.com> Co-authored-by: Lachlan Evenson <lachlan.evenson@microsoft.com> Co-authored-by: Michael Brown <brownxmi@amazon.com> Signed-off-by: Brandon Mitchell <git@bmitch.net>

jdolitsky force-pushed the pr branch from bc2252f to 1c4ad9a Compare August 9, 2022 19:20

sajayantony force-pushed the pr branch from 80d4234 to 94ef2a1 Compare August 9, 2022 20:09

lachie83 marked this pull request as ready for review August 9, 2022 20:19

lachie83 requested review from jonjohnsonjr, jonboulle, stevvooe, vbatts and cyphar as code owners August 9, 2022 20:19

michaelb990 mentioned this pull request Aug 10, 2022

Working Group Proposal for Reference Types opencontainers/distribution-spec#335

Merged

mounchin reviewed Aug 10, 2022

View reviewed changes

This was referenced Aug 11, 2022

Support OCI Artifacts and reference types oras-project/oras-go#267

Closed

Support OCI Artifacts and referrers API oras-project/oras-go#268

Closed

shizhMSFT reviewed Aug 12, 2022

View reviewed changes

descriptor.md Show resolved Hide resolved

shizhMSFT mentioned this pull request Aug 12, 2022

feat: push an ORAS artifact manifest when no blob is specified oras-project/oras#486

Closed

lachie83 mentioned this pull request Aug 12, 2022

Remove duplicate artifactType entry oci-playground/image-spec#14

Merged

mikebrow reviewed Aug 12, 2022

View reviewed changes

artifact.md Outdated Show resolved Hide resolved

wmesard reviewed Aug 15, 2022

View reviewed changes

artifact.md Show resolved Hide resolved

This was referenced Aug 15, 2022

Add image index example with mixed media types oci-playground/image-spec#15

Merged

Allow null values in blobs array oras-project/artifacts-spec#114

Closed

imjasonh mentioned this pull request Aug 16, 2022

FR: Support OCI reference types google/go-containerregistry#1434

Closed

sajayantony force-pushed the pr branch from 8f7509c to be11dc5 Compare August 16, 2022 19:27

This was referenced Aug 16, 2022

Use annotations to map a zTOC to its corresponding layer awslabs/soci-snapshotter#12

Closed

Remove null blobs from Soci Index awslabs/soci-snapshotter#23

Merged

michaelb990 force-pushed the pr branch from 133cff7 to 1a89a51 Compare August 17, 2022 21:35

sudo-bmitch reviewed Aug 17, 2022

View reviewed changes

artifact.md Outdated Show resolved Hide resolved

vbatts previously approved these changes Aug 25, 2022

View reviewed changes

sudo-bmitch dismissed stale reviews from vbatts and themself via 91c0539 August 25, 2022 16:31

lachie83 commented Aug 25, 2022

View reviewed changes

vbatts added this to the v1.1 milestone Aug 25, 2022

sudo-bmitch force-pushed the pr branch from 91c0539 to 7303b74 Compare August 25, 2022 18:09

sudo-bmitch previously approved these changes Aug 25, 2022

View reviewed changes

sajayantony previously approved these changes Aug 25, 2022

View reviewed changes

sudo-bmitch dismissed stale reviews from sajayantony and themself via b04b320 August 25, 2022 18:17

sudo-bmitch force-pushed the pr branch from 7303b74 to b04b320 Compare August 25, 2022 18:17

sudo-bmitch approved these changes Aug 25, 2022

View reviewed changes

sajayantony approved these changes Aug 25, 2022

View reviewed changes

sudo-bmitch mentioned this pull request Aug 25, 2022

Reference Types Working Group remaining tasks #940

Closed

4 tasks

sudo-bmitch merged commit ff02571 into opencontainers:main Aug 25, 2022

This was referenced Aug 28, 2022

Add reference field #828

Closed

Proposal: Add References #827

Closed

lengrongfu mentioned this pull request Aug 29, 2022

Support OCI reference types distribution/distribution#3719

Closed

imjasonh mentioned this pull request Sep 12, 2022

Why are the signatures OCI Images rather than OCI Artifacts? sigstore/cosign#913

Closed

sudo-bmitch mentioned this pull request Sep 25, 2022

Update 1.0 branch for 1.0.3 release #955

Closed

SteveLasker mentioned this pull request Oct 18, 2022

add a new blog post for ORAS v0.15 oras-project/oras-www#61

Merged

maxgio92 mentioned this pull request Oct 28, 2022

falcoctl cosign signature verification falcosecurity/falcoctl#174

Closed

bcaton85 mentioned this pull request Nov 16, 2022

Reconciling with the new OCI Artifacts spec oras-project/artifacts-spec#117

Open

This was referenced Dec 12, 2022

Update cosign to v2.0.2 fluxcd/source-controller#979

Closed

[Feature] Tracking Issue for Referrers API in Cosign kyverno/kyverno#5655

Open

itaysk mentioned this pull request Mar 1, 2023

discover sbom in OCI registry aquasecurity/trivy#3735

Closed

sudo-bmitch deleted the pr branch July 8, 2024 13:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Working Group Proposal for Reference Types #934

Working Group Proposal for Reference Types #934

lachie83 commented Aug 9, 2022

vbatts commented Aug 9, 2022

mounchin Aug 10, 2022

sajayantony Aug 10, 2022 •

edited

Loading

sudo-bmitch Aug 11, 2022

mounchin Aug 11, 2022

sudo-bmitch Aug 12, 2022

sudo-bmitch commented Aug 11, 2022 •

edited

Loading

lachie83 commented Aug 11, 2022

lachie83 commented Aug 11, 2022

mikebrow left a comment

mikebrow Aug 12, 2022

sudo-bmitch Aug 15, 2022

mikebrow Aug 16, 2022 •

edited

Loading

sudo-bmitch Aug 16, 2022

mikebrow Aug 17, 2022

vbatts left a comment •

edited

Loading

lachie83 Aug 25, 2022

vbatts Aug 25, 2022

vbatts commented Aug 25, 2022

estesp commented Aug 25, 2022


		Also, see [Pre-Defined Annotation Keys](annotations.md#pre-defined-annotation-keys).

		User defined annotations MAY be used to filter various artifact types, e.g. signature public key hash, attestation type, and SBOM schema.


		- `refers` [descriptor](descriptor.md)

		This OPTIONAL property specifies a [descriptor](descriptor.md) of another manifest.

Working Group Proposal for Reference Types #934

Working Group Proposal for Reference Types #934

Conversation

lachie83 commented Aug 9, 2022

vbatts commented Aug 9, 2022

Choose a reason for hiding this comment

sajayantony Aug 10, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sudo-bmitch commented Aug 11, 2022 • edited Loading

lachie83 commented Aug 11, 2022

lachie83 commented Aug 11, 2022

mikebrow left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikebrow Aug 16, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vbatts left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vbatts commented Aug 25, 2022

estesp commented Aug 25, 2022

sajayantony Aug 10, 2022 •

edited

Loading

sudo-bmitch commented Aug 11, 2022 •

edited

Loading

mikebrow Aug 16, 2022 •

edited

Loading

vbatts left a comment •

edited

Loading