Add stream metadata #477

cgwalters · 2020-12-18T22:07:33Z

Moving a subthread from coreos/fedora-coreos-tracker#98 (comment) (part of openshift/enhancements#201 )here.

Basically let's add https://mirror.openshift.com/dependencies/rhcos/stream-4.8.json

(Note that stream metadata contains multiple architectures by design, so it's not under the existing arch-dependent location).

This could then also be used by coreos-installer download -o rhcos -s rhcos-4.8, and we'd also update all the UPI documentation to refer to it instead of e.g. hardcoding AMIs in the documentation.

Now, we have the option to move this data out of openshift/installer entirely; in connected installs we just download it dynamically, in disconnected installs we require the user to mirror it. I'm on the fence about this; perhaps too big of a change for 4.8.

Proposal: mirror.openshift.com path

Introduce github.com/openshift/rhel-coreos-bootimage that contains a single stream.json file with multiple branches.
Add an initially-manual flow to submit a PR there to bump it, that works just like bumping openshift/installer would - e.g. we have CI hooked up
Add an ART process that scrapes this JSON from git and syncs it to mirror.openshift.com
Initially also do a manual PR to update openshift/installer

The major disadvantage of this is that we would have bootimages in two places, and the potential for divergence between IPI and UPI. But, I am hopeful that if we streamline CI for rhel-coreos-bootimage we can just wave through sync PRs to update openshift/installer.

It also slightly breaks the "hermetic" nature of the current (openshift-install, release image) tuple because there's another input which can change over time not versioned with those. Perhaps a middle ground compromise is that we add versioned stream metadata, i.e. https://mirror.openshift.com/dependencies/rhcos/stream-4.8-20200325.json and openshift-install has a pinned copy of it.

Proposal: Embed in release image path

Introduce github.com/openshift/rhel-coreos-bootimage that contains a single stream.json file with multiple branches.
Add a Dockerfile which puts this JSON in a container image
Add that container image to the release image
Change openshift/installer to pull and extract that piece of the release image

Advantages: Makes it more obvious how to get this data "in cluster" for eventual in-cluster updates. Versioned with the release image naturally.
Disadvantages: Nontrivial change to openshift/installer because it has to pull the release image client side (but maybe openshift-installer can just require oc to be present and fork off oc image extract?)

The text was updated successfully, but these errors were encountered:

cgwalters · 2020-12-18T22:09:35Z

For the implementation of this, I think the simplest is probably to directly go from the existing RHCOS cosa metadata and every time we do a build that generates bootimages, update the stream immediately at bootimages.svc. The mirror.openshift.com version would derive from the installer-pinned version.

bgilbert · 2020-12-18T22:21:20Z

I think the simplest is probably to directly go from the existing RHCOS cosa metadata

To be clear, is the proposal that the stream metadata would be exactly the meta.json? I'd be in favor of using an explicitly designed schema, either the FCOS stream metadata schema or a new one.

cgwalters · 2020-12-19T14:07:54Z

Oh no the idea is it looks exactly the same as the existing FCOS stream metadata; why would it differ? I'm just saying that we don't create intermediate release.json, i.e. the translation is just cosa meta.json ➡️ stream, right?

bgilbert · 2020-12-21T23:37:55Z

Ah, okay. As an implementation detail, it might make sense to internally generate an intermediate release.json so we can reuse existing tooling.

Part of implementing openshift/os#477 For now I decided to go directly from cosa to a stream because: - The cosa2release code is in a separate repo in Python - The RHCOS pipeline currently generates "one big build" anyways

This is a straight up import of almost-exactly the contents of https://github.com/coreos/fedora-coreos-releng-automation/blob/master/coreos-meta-translator/trans.py Prep for adding more stream tooling into coreos-assembler as part of openshift/os#477

cgwalters · 2021-01-15T16:19:45Z

That said, openshift/enhancements#201 calls for having this as part of the release image. Though...hmm, perhaps the simplest thing is that the installer uploads it as a configmap into the cluster. Now if we decide that this will ultimately be the MCO's job, perhaps it becomes oc -n machine-config-operator configmap/bootimages? Or it could go in oc -n openshift configmap/coreos-bootimages?

But a problem is that doesn't make it easy to get from oc for UPI installs in a way that reliably picks up the same installer-pinned version...which we might want for the "simulate UPI in CI" case. But some UPI installs are going to want to access the mirror.openshift.com data (particularly bare metal), so we could say even e.g. AWS UPI installs should get the AMIs from there, and we do the same for our CI.

But if we do want to implement the oc adm release info --bootimages or whatever...:

"oc adm release new scrapes openshift-install"

First, something like openshift/installer#4102
Then next, we change oc adm release new to explicitly scrape the data out of the installer image and add it to the image it generates (so it's convenient to download). Or we could even try to compress it into container image metadata for the image.

LorbusChris · 2021-01-15T17:56:56Z

@cgwalters one reason it seemed enticing to me to store the bootimage refs in the payload as opposed to the installer repo was that they wouldn't have to be maintained in the installer repo's rhcos.json (and fcos.json for OKD) in the future.

I nonetheless think having a configmap for the bootimages does make a lot of sense, but why not scrape that data off the payload?

Prep for further work on openshift/os#477

Part of implementing openshift/os#477 I don't think for RHCOS (at least initially) we will go through the whole "release.json" middle ground, so let's add a command to directly turn 1 or more cosa build(s) into a stream JSON. Because the "cosa build" -> "release" bits are in Python, we semi-hackily fork it off as a subprocess. But, it works.

cgwalters · 2021-01-15T22:43:33Z

@cgwalters one reason it seemed enticing to me to store the bootimage refs in the payload as opposed to the installer repo was that they wouldn't have to be maintained in the installer repo's rhcos.json (and fcos.json for OKD) in the future.

But the core problem there is the bootstrap node. Unless we change the installer to pivot in all cases, and we decide to accept the potential skew with the "in cluster bootimages" then the installer needs to pin.

Honestly long term the way I think things should work is basically that if you run openshift-install outside of the target cloud, it can spin up a traditional RHEL cloud image (if available, or RHCOS, or FCOS, or heck even anything that can run podman) and use that to start as a "bootstrap bootstrap node" that then e.g. downloads the release image and (short term) spins up e.g an RHCOS bootstrap node and proceeds from there. Longer term, if we made some of the bootstrap logic run inside a containerized kubelet, then we'd only have one bootstrap node.

Prep for further work on openshift/os#477

Part of implementing openshift/os#477 I don't think for RHCOS (at least initially) we will go through the whole "release.json" middle ground, so let's add a command to directly turn 1 or more cosa build(s) into a stream JSON. Because the "cosa build" -> "release" bits are in Python, we semi-hackily fork it off as a subprocess. But, it works.

Prep for further work on openshift/os#477

Part of implementing openshift/os#477 I don't think for RHCOS (at least initially) we will go through the whole "release.json" middle ground, so let's add a command to directly turn 1 or more cosa build(s) into a stream JSON. Because the "cosa build" -> "release" bits are in Python, we semi-hackily fork it off as a subprocess. But, it works.

Prep for further work on openshift/os#477

Part of implementing openshift/os#477 I don't think for RHCOS (at least initially) we will go through the whole "release.json" middle ground, so let's add a command to directly turn 1 or more cosa build(s) into a stream JSON. Because the "cosa build" -> "release" bits are in Python, we semi-hackily fork it off as a subprocess. But, it works.

Prep for openshift/os#477 Clarify internally that this sha256 is uncompressed; the stream will also have the compressed sha256 which we should support verifying too (as well as the size). Since we're now parsing the URL better, use the uncompressed filename as the target name. This makes it much *much* more obvious to users how to manually clean up old images in the cache. We can do this because for both FCOS/RHCOS the filenames contain versions and are immutable - we will never go back and change an image.

cgwalters · 2021-01-21T15:44:26Z

openshift/installer#4576

See openshift/os#477 TODO: - Azure - Multiple architectures

This implements part of the plan from: openshift/os#477 When we originally added the pinned RHCOS metadata `rhcos.json` to the installer, we also changed the coreos-assembler `meta.json` format into an arbitrary new format in the name of some cleanups. In retrospect, this was a big mistake because we now have two formats. Then Fedora CoreOS appeared and added streams JSON as a public API. We decided to unify on streams metadata; there's now a published Go library for it: https://github.com/coreos/stream-metadata-go Among other benefits, it is a single file that supports multiple architectures. UPI installs should now use stream metadata, particularly to find public cloud images. This is an important preparatory step for exposing this via `oc` as well as having something in the cluster update to it.

This is part of adapting RHCOS to use the same stream metadata JSON format that FCOS uses: openshift/os#477 This change specifically will pair with a pending change in: openshift/installer#4582 to embed CoreOS stream metadata in the installer git instead of the undocumented ad-hoc JSON format that ships there now.

This implements part of the plan from: openshift/os#477 When we originally added the pinned RHCOS metadata `rhcos.json` to the installer, we also changed the coreos-assembler `meta.json` format into an arbitrary new format in the name of some cleanups. In retrospect, this was a big mistake because we now have two formats. Then Fedora CoreOS appeared and added streams JSON as a public API. We decided to unify on streams metadata; there's now a published Go library for it: https://github.com/coreos/stream-metadata-go Among other benefits, it is a single file that supports multiple architectures. UPI installs should now use stream metadata, particularly to find public cloud images. This is an important preparatory step for exposing this via `oc` as well as having something in the cluster update to it.

cgwalters · 2021-03-10T18:18:02Z

This is now openshift/enhancements#679

This implements part of the plan from: openshift/os#477 When we originally added the pinned RHCOS metadata `rhcos.json` to the installer, we also changed the coreos-assembler `meta.json` format into an arbitrary new format in the name of some cleanups. In retrospect, this was a big mistake because we now have two formats. Then Fedora CoreOS appeared and added streams JSON as a public API. We decided to unify on streams metadata; there's now a published Go library for it: https://github.com/coreos/stream-metadata-go Among other benefits, it is a single file that supports multiple architectures. UPI installs should now use stream metadata, particularly to find public cloud images. This is an important preparatory step for exposing this via `oc` as well as having something in the cluster update to it. HOWEVER as a (really hopefully temporary) hack, we *duplicate* the metadata so that IPI installs use the new stream format, and UPI CI jobs can still use the old format (with different RHCOS versions). We will port the UPI docs and CI jobs after this merges.

This implements part of the plan from: openshift/os#477 When we originally added the pinned RHCOS metadata `rhcos.json` to the installer, we also changed the coreos-assembler `meta.json` format into an arbitrary new format in the name of some cleanups. In retrospect, this was a big mistake because we now have two formats. Then Fedora CoreOS appeared and added streams JSON as a public API. We decided to unify on streams metadata; there's now a published Go library for it: https://github.com/coreos/stream-metadata-go Among other benefits, it is a single file that supports multiple architectures. UPI installs should now use stream metadata, particularly to find public cloud images. This is exposed via a new `openshift-install coreos print-stream-json` command. This is an important preparatory step for exposing this via `oc` as well as having something in the cluster update to it. HOWEVER as a (really hopefully temporary) hack, we *duplicate* the metadata so that IPI installs use the new stream format, and UPI CI jobs can still use the old format (with different RHCOS versions). We will port the UPI docs and CI jobs after this merges.

This implements part of the plan from: openshift/os#477 When we originally added the pinned RHCOS metadata `rhcos.json` to the installer, we also changed the coreos-assembler `meta.json` format into an arbitrary new format in the name of some cleanups. In retrospect, this was a big mistake because we now have two formats. Then Fedora CoreOS appeared and added streams JSON as a public API. We decided to unify on streams metadata; there's now a published Go library for it: https://github.com/coreos/stream-metadata-go Among other benefits, it is a single file that supports multiple architectures. UPI installs should now use stream metadata, particularly to find public cloud images. This is exposed via a new `openshift-install coreos print-stream-json` command. This is an important preparatory step for exposing this via `oc` as well as having something in the cluster update to it. HOWEVER as a (really hopefully temporary) hack, we *duplicate* the metadata so that IPI installs use the new stream format, and UPI CI jobs can still use the old format (with different RHCOS versions). We will port the UPI docs and CI jobs after this merges. Co-authored-by: Matthew Staebler <staebler@redhat.com>

Prep for openshift/os#477 Clarify internally that this sha256 is uncompressed; the stream will also have the compressed sha256 which we should support verifying too (as well as the size). Since we're now parsing the URL better, use the uncompressed filename as the target name. This makes it much *much* more obvious to users how to manually clean up old images in the cache. We can do this because for both FCOS/RHCOS the filenames contain versions and are immutable - we will never go back and change an image.

This implements part of the plan from: openshift/os#477 When we originally added the pinned RHCOS metadata `rhcos.json` to the installer, we also changed the coreos-assembler `meta.json` format into an arbitrary new format in the name of some cleanups. In retrospect, this was a big mistake because we now have two formats. Then Fedora CoreOS appeared and added streams JSON as a public API. We decided to unify on streams metadata; there's now a published Go library for it: https://github.com/coreos/stream-metadata-go Among other benefits, it is a single file that supports multiple architectures. UPI installs should now use stream metadata, particularly to find public cloud images. This is exposed via a new `openshift-install coreos print-stream-json` command. This is an important preparatory step for exposing this via `oc` as well as having something in the cluster update to it. HOWEVER as a (really hopefully temporary) hack, we *duplicate* the metadata so that IPI installs use the new stream format, and UPI CI jobs can still use the old format (with different RHCOS versions). We will port the UPI docs and CI jobs after this merges. Co-authored-by: Matthew Staebler <staebler@redhat.com>

cgwalters mentioned this issue Dec 18, 2020

Stream and release metadata coreos/fedora-coreos-tracker#98

Open

cgwalters mentioned this issue Jan 13, 2021

plume: add cosa2stream coreos/coreos-assembler#2000

Merged

cgwalters mentioned this issue Jan 14, 2021

Import FCOS trans.py that converts meta.json → release.json coreos/coreos-assembler#2006

Merged

cgwalters added a commit to cgwalters/coreos-assembler that referenced this issue Jan 15, 2021

mantle: bump coreos/stream-metadata-go

c5a6031

Prep for further work on openshift/os#477

cgwalters added a commit to cgwalters/coreos-assembler that referenced this issue Jan 19, 2021

mantle: bump coreos/stream-metadata-go

5a178a5

Prep for further work on openshift/os#477

cgwalters added a commit to cgwalters/coreos-assembler that referenced this issue Jan 19, 2021

mantle: bump coreos/stream-metadata-go

df8cf14

Prep for further work on openshift/os#477

openshift-merge-robot pushed a commit to coreos/coreos-assembler that referenced this issue Jan 20, 2021

mantle: bump coreos/stream-metadata-go

3226193

Prep for further work on openshift/os#477

cgwalters mentioned this issue Jan 21, 2021

Stream metadata prep openshift/installer#4576

Merged

cgwalters added a commit to cgwalters/installer that referenced this issue Jan 21, 2021

WIP: use stream metadata

a36299c

See openshift/os#477 TODO: - Azure - Multiple architectures

cgwalters mentioned this issue Jan 21, 2021

Add and use stream metadata for RHCOS, add openshift-install coreos print-stream-json openshift/installer#4582

Merged

cgwalters mentioned this issue Mar 1, 2021

tree-wide: Support pending change to use stream metadata for CoreOS openshift/release#16355

Closed

cgwalters closed this as completed Mar 10, 2021

cgwalters mentioned this issue Apr 26, 2021

productize redirector service #542

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add stream metadata #477

Add stream metadata #477

cgwalters commented Dec 18, 2020 •

edited

Loading

cgwalters commented Dec 18, 2020

bgilbert commented Dec 18, 2020

cgwalters commented Dec 19, 2020

bgilbert commented Dec 21, 2020

cgwalters commented Jan 15, 2021 •

edited

Loading

LorbusChris commented Jan 15, 2021

cgwalters commented Jan 15, 2021

cgwalters commented Jan 21, 2021

cgwalters commented Mar 10, 2021

Add stream metadata #477

Add stream metadata #477

Comments

cgwalters commented Dec 18, 2020 • edited Loading

Proposal: mirror.openshift.com path

Proposal: Embed in release image path

cgwalters commented Dec 18, 2020

bgilbert commented Dec 18, 2020

cgwalters commented Dec 19, 2020

bgilbert commented Dec 21, 2020

cgwalters commented Jan 15, 2021 • edited Loading

"oc adm release new scrapes openshift-install"

LorbusChris commented Jan 15, 2021

cgwalters commented Jan 15, 2021

cgwalters commented Jan 21, 2021

cgwalters commented Mar 10, 2021

cgwalters commented Dec 18, 2020 •

edited

Loading

cgwalters commented Jan 15, 2021 •

edited

Loading