Proposal: Define the "data" field

[jonjohnsonjr]

As discussed on the call last week, we should define this field:

https://github.com/opencontainers/image-spec/blob/master/descriptor.md#reserved

[SteveLasker]

The basic question is this: What defines a spec? Should specs be updated with discipline and vigor?

If a spec is about creating a set of expectations for behavior, whereby others adopt the spec, what expectations should be made to it's stability?

The industry has adopted the distribution-spec and image-spec, and implemented products, services and community projects around these specs. OCI Artifacts was created to state how to use the existing specs with the previously defined extensibility, because we were unable to make changes without bumping the image-spec version.

We need a way to add capabilities, we all agree. The question isn't should we add more capabilities, rather how.

Specs have versions, so that implementors of a spec can declare their conformance to a spec.

The proposal here is to add two additional optional properties. While it may sound fairly trivial, as they're optional, lets consider the implications of these.

references - There's general consensus we wish to add linkages between things in a registry. The two proposals focus on references between manifests or blobs (aka layers). Where those references are added is a separate discussion. The concerns is adding references implies registries track these references. This is a new index to be maintained. Valuable, but a substantial change. And, with references comes the question of how to cleanup deleted or orphaned objects in a registry. This is otherwise referred to as garbage collection. If users can start pushing new manifests, not previously known to a registry, what is the expected behavior? Can a user, or paying customer of one of the products and/or services claim the registry must conform to behavior included in a spec? How does a registry state it's contractual obligations to customers for SLAs? Can a user or customer simply say, it's in the ___ spec, so you should support it? When they can't reduce the size of their registry, because referenced objects become orphaned, or content is deleted, can a user open a support ticket? When users want to use this capability, how do they get referenced objects back out? There's an implication of an API for retrieving these referenced objects.

data field - Users continually ping registries with requests for updates to a tag, determining if something was updated and the artifact should be pulled again. To mitigate what could be viewed as a DOS attack, registries cache manifests. Adding an unbound data field means the manifest can exponentially expand in size, changing the cache-hit ratio.

The position isn't that we shouldn't enable change. The position is a spec declares a version, where consumers of that spec can adopt that standard and support it with full and consistent expectations.

Updates will take years to adopt

The position has been it will take too long if we create a new version or a new spec. The premise is we can insert new behavior and force implementors to adopt the change. This degrades the entire intent of a versioned spec. If extensibility is defined in a versioned spec, implementors build around that expectation.

I'd suggest time doesn't drive adoption, it's value that drives adoption. If we can set clear expectations, and value, consumers will adopt the change quickly. The question is how do we opt-in adoption that doesn't destabilize the ecosystem we're looking to enable.

[waynr]

As discussed on the call last week, we should define this field:

Not everyone who lands on this issue was on that call, so at the very least I would expect some clarification around why this is considered necessary and to @SteveLasker's point, why it should be done in violation of

This key is RESERVED for future versions of the specification.

Which explicitly reserves the data key for future versions of the specification, not the current version.

[jonjohnsonjr]

Which explicitly reserves the data key for future versions of the specification, not the current version.

Merging a PR doesn't retroactively affect previous versions of the specification, which are tagged: https://github.com/opencontainers/image-spec/releases

Once we tag e.g. v1.0.2 or v1.1.0, it will be a future version from the perspective of the current version.

[waynr]

Once we tag e.g. v1.0.2 or v1.1.0, it will be a future version from the perspective of the current version.

Oh right, duh. I'm currently suffering from some postprandial drowsiness so I probably shouln't be commenting on issues 😅