Add stream metadata for RHCOS (bump x86_64 to 48.83.202102230316-0)

This implements part of the plan from:
openshift/os#477

When we originally added the pinned RHCOS metadata `rhcos.json`
to the installer, we also changed the coreos-assembler `meta.json`
format into an arbitrary new format in the name of some cleanups.
In retrospect, this was a big mistake because we now have two
formats.

Then Fedora CoreOS appeared and added streams JSON as a public API.

We decided to unify on streams metadata; there's now a published
Go library for it: https://github.com/coreos/stream-metadata-go

Among other benefits, it is a single file that supports multiple
architectures.

UPI installs should now use stream metadata, particularly
to find public cloud images.

This is an important preparatory step for exposing this via
`oc` as well as having something in the cluster update to
it.

HOWEVER as a (really hopefully temporary) hack, we *duplicate*
the metadata so that IPI installs use the new stream format,
and UPI CI jobs can still use the old format (with different RHCOS versions).

We will port the UPI docs and CI jobs after this merges.

docs/user/aws/install_upi.md

@@ -21,8 +21,8 @@ $ openshift-install create install-config
 ### Optional: Create Encrypted AMIs
 
 The IPI-based installer creates an encrypted AMI by default. If you wish to have an encrypted AMI for UPI-based
-installs, you will need to create it directly. You can find a list of the appropriate base AMIs 
-[here](../../../data/data/rhcos.json).
+installs, you will need to create it directly. You can find a list of the appropriate base AMIs
+as part of the CoreOS stream metadata [here](../../../data/data/rhcos-4.8.json).
 
 You will make an encrypted copy of the AMI according to the [AWS documentation][encrypted-copy].
 

docs/user/metal/customization_ipi.md

@@ -24,12 +24,12 @@
 
 When doing a disconnected installation, the baremetal platform has the
 additional requirement that we have locations to download the RHCOS
-images. The installer downloads these from a location described in
-[/data/data/rhcos.json](/data/data/rhcos.json), but they can be
+images. The installer downloads these from a CoreOS stream metadata
+embedded in the installer code, but they can be
 overridden to point to a local mirror.
 
 The SHA256 parameter in the URLs are required, and should match the
-uncompressed SHA256 from rhcos.json.
+uncompressed SHA256 from the embedded file e.g. `rhcos-4.8.json`.
 
 
 * `bootstrapOSImage` (optional string): Override the image used for the

docs/user/openstack/customization.md

@@ -112,13 +112,13 @@ sshKey: ssh-ed25519 AAAA...
 
 ## Image Overrides
 
-Normally the installer downloads the RHCOS image from a predetermined location described in [data/data/rhcos.json](/data/data/rhcos.json)). But the download URL can be overridden, notably for disconnected installations.
+The OpenShift installer currently pins the version of RHEL CoreOS.  Since OpenShift 4.8, this is done via a "stream metadata" file embedded at [data/data/rhcos-4.8.json](/data/data/rhcos-4.8.json).  Normally the installer downloads the RHCOS image from that location.  But the download URL can be overridden, notably for disconnected installations.
 
 To do so and upload binary data from a custom location the user may set `clusterOSImage` parameter in the install config that points to that location, and then start the installation. In all other respects the process will be consistent with the default.
 
 **NOTE:** For this to work, the parameter value must be a valid http(s) URL.
 
-**NOTE:** The optional `sha256` query parameter can be attached to the URL, which will force the installer to check the image file checksum before uploading it into Glance.
+**NOTE:** The optional `sha256` query parameter can be attached to the URL, which will force the installer to check the uncompressed image file checksum before uploading it into Glance.
 
 Example:
 

hack/update-rhcos-bootimage.py

@@ -1,4 +1,23 @@
 #!/usr/bin/env python3
+# As of 4.8 we are aiming to switch to stream metadata:
+# https://github.com/openshift/enhancements/pull/679
+# That transition hasn't yet fully completed; there are two copies of the
+# RHCOS metadata:
+# 
+# - data/data/rhcos-4.8.json (stream format, 4.8+)
+# - data/data/rhcos-$arch.json (openshift/installer specific, 4.7 and below)
+# 
+# See https://github.com/coreos/coreos-assembler/pull/2000 in particular.
+# 
+# The initial file data/data/rhcos-4.8 was generated this way:
+# 
+# $ plume cosa2stream --name rhcos-4.8 --distro rhcos  x86_64=48.83.202102230316-0 s390x=47.83.202102090311-0 ppc64le=47.83.202102091015-0 > data/data/rhcos-4.8.json
+# 
+# To update the bootimage for one or more architectures, use e.g.
+# 
+# $ plume cosa2stream --target data/data/rhcos-4.8.json --distro rhcos  x86_64=48.83.202102230316-0 s390x=47.83.202102090311-0 ppc64le=47.83.202102091015-0
+#
+# To update the legacy metadata, use:
 # Usage: ./hack/update-rhcos-bootimage.py https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.6/46.82.202008260918-0/x86_64/meta.json amd64
 import codecs,os,sys,json,argparse
 import urllib.parse

images/baremetal/Dockerfile.ci

@@ -12,6 +12,7 @@ RUN TAGS="libvirt baremetal" hack/build.sh
 FROM registry.ci.openshift.org/ocp/4.8:base
 COPY --from=builder /go/src/github.com/openshift/installer/bin/openshift-install /bin/openshift-install
 COPY --from=builder /go/src/github.com/openshift/installer/data/data/rhcos.json /var/cache/
+COPY --from=builder /go/src/github.com/openshift/installer/data/data/rhcos-stream.json /var/cache/
 
 RUN yum update -y && \
     yum install --setopt=tsflags=nodocs -y \

images/installer/Dockerfile.upi.ci

@@ -15,6 +15,9 @@ COPY --from=cli /usr/bin/oc /bin/oc
 COPY --from=builder /go/src/github.com/openshift/installer/bin/openshift-install /bin/openshift-install
 COPY --from=builder /go/src/github.com/openshift/installer/upi /var/lib/openshift-install/upi
 COPY --from=builder /go/src/github.com/openshift/installer/data/data/rhcos.json /var/lib/openshift-install/rhcos.json
+# See https://github.com/openshift/release/pull/16355
+# and https://github.com/openshift/enhancements/pull/679
+COPY --from=builder /go/src/github.com/openshift/installer/data/data/rhcos-4.8.json /var/lib/openshift-install/coreos-stream.json
 
 RUN rpm --import https://packages.microsoft.com/keys/microsoft.asc
 RUN sh -c 'echo -e "[azure-cli]\nname=Azure CLI\nbaseurl=https://packages.microsoft.com/yumrepos/azure-cli\nenabled=1\ngpgcheck=1\ngpgkey=https://packages.microsoft.com/keys/microsoft.asc" >/etc/yum.repos.d/azure-cli.repo'

pkg/asset/cluster/tfvars.go

@@ -9,6 +9,7 @@ import (
 	"strings"
 
 	igntypes "github.com/coreos/ignition/v2/config/v3_2/types"
+	coreosarch "github.com/coreos/stream-metadata-go/arch"
 	gcpprovider "github.com/openshift/cluster-api-provider-gcp/pkg/apis/gcpprovider/v1beta1"
 	kubevirtprovider "github.com/openshift/cluster-api-provider-kubevirt/pkg/apis/kubevirtprovider/v1alpha1"
 	kubevirtutils "github.com/openshift/cluster-api-provider-kubevirt/pkg/utils"
@@ -338,16 +339,30 @@ func (t *TerraformVariables) Generate(parents asset.Parents) error {
 		}
 		preexistingnetwork := installConfig.Config.GCP.Network != ""
 
-		imageRaw, err := rhcospkg.GCPRaw(ctx, installConfig.Config.ControlPlane.Architecture)
+		archName := coreosarch.RpmArch(string(installConfig.Config.ControlPlane.Architecture))
+		st, err := rhcospkg.FetchCoreOSBuild(ctx)
 		if err != nil {
-			return errors.Wrap(err, "failed to find Raw GCP image URL")
+			return err
+		}
+		streamArch, err := st.GetArchitecture(archName)
+		if err != nil {
+			return err
+		}
+
+		img := streamArch.Images.Gcp
+		if img == nil {
+			return fmt.Errorf("%s: No GCP build found", st.FormatPrefix(archName))
 		}
+		// For backwards compatibility, we generate this URL to the image (only applies to RHCOS, not FCOS/OKD)
+		// right now.  It will only be used if nested virt or other licenses are enabled, which we
+		// really should deprecate and remove - xref https://github.com/openshift/installer/pull/4696
+		imageURL := fmt.Sprintf("https://storage.googleapis.com/rhcos/rhcos/%s.tar.gz", img.Name)
 		data, err := gcptfvars.TFVars(
 			gcptfvars.TFVarsSources{
 				Auth:               auth,
 				MasterConfigs:      masterConfigs,
 				WorkerConfigs:      workerConfigs,
-				ImageURI:           imageRaw,
+				ImageURI:           imageURL,
 				ImageLicenses:      installConfig.Config.GCP.Licenses,
 				PublicZoneName:     publicZoneName,
 				PublishStrategy:    installConfig.Config.Publish,

pkg/asset/rhcos/bootstrap_image.go

@@ -3,8 +3,11 @@ package rhcos
 
 import (
 	"context"
+	"fmt"
 	"time"
 
+	"github.com/coreos/stream-metadata-go/arch"
+
 	"github.com/openshift/installer/pkg/asset"
 	"github.com/openshift/installer/pkg/asset/installconfig"
 	"github.com/openshift/installer/pkg/rhcos"
@@ -37,28 +40,44 @@ func (i *BootstrapImage) Generate(p asset.Parents) error {
 	p.Get(ic)
 	config := ic.Config
 
-	var osimage string
-	var err error
 	ctx, cancel := context.WithTimeout(context.TODO(), 30*time.Second)
 	defer cancel()
+
 	switch config.Platform.Name() {
 	case baremetal.Name:
-		// Check for RHCOS image URL override
-		if boi := config.Platform.BareMetal.BootstrapOSImage; boi != "" {
-			osimage = boi
-			break
+		archName := arch.RpmArch(fmt.Sprintf("%s", config.ControlPlane.Architecture))
+		st, err := rhcos.FetchCoreOSBuild(ctx)
+		if err != nil {
+			return err
+		}
+		streamArch, err := st.GetArchitecture(archName)
+		if err != nil {
+			return err
 		}
 
+		// Check for CoreOS image URL override
+		if boi := config.Platform.BareMetal.BootstrapOSImage; boi != "" {
+			*i = BootstrapImage(boi)
+			return nil
+		}
 		// Baremetal IPI launches a local VM for the bootstrap node
 		// Hence requires the QEMU image to use the libvirt backend
-		osimage, err = rhcos.QEMU(ctx, config.ControlPlane.Architecture)
+		if a, ok := streamArch.Artifacts["qemu"]; ok {
+			u, err := rhcos.FindArtifactURL(a)
+			if err != nil {
+				return err
+			}
+			*i = BootstrapImage(u)
+			return nil
+		}
+		return fmt.Errorf("%s: No qemu build found", st.FormatPrefix(archName))
 	default:
 		// other platforms use the same image for all nodes
-		osimage, err = osImage(config)
-	}
-	if err != nil {
-		return err
+		u, err := osImage(config)
+		if err != nil {
+			return err
+		}
+		*i = BootstrapImage(u)
+		return nil
 	}
-	*i = BootstrapImage(osimage)
-	return nil
 }

pkg/asset/rhcos/image.go

@@ -7,7 +7,7 @@ import (
 	"os"
 	"time"
 
-	"github.com/pkg/errors"
+	"github.com/coreos/stream-metadata-go/arch"
 	"github.com/sirupsen/logrus"
 
 	"github.com/openshift/installer/pkg/asset"
@@ -66,67 +66,99 @@ func (i *Image) Generate(p asset.Parents) error {
 }
 
 func osImage(config *types.InstallConfig) (string, error) {
-	arch := config.ControlPlane.Architecture
-
-	var osimage string
-	var err error
 	ctx, cancel := context.WithTimeout(context.TODO(), 30*time.Second)
 	defer cancel()
+
+	archName := arch.RpmArch(fmt.Sprintf("%s", config.ControlPlane.Architecture))
+
+	st, err := rhcos.FetchCoreOSBuild(ctx)
+	if err != nil {
+		return "", err
+	}
+	streamArch, err := st.GetArchitecture(archName)
+	if err != nil {
+		return "", err
+	}
 	switch config.Platform.Name() {
 	case aws.Name:
 		if len(config.Platform.AWS.AMIID) > 0 {
-			osimage = config.Platform.AWS.AMIID
-			break
+			return config.Platform.AWS.AMIID, nil
 		}
 		region := config.Platform.AWS.Region
 		if !configaws.IsKnownRegion(config.Platform.AWS.Region) {
 			region = "us-east-1"
 		}
-		osimage, err = rhcos.AMI(ctx, arch, region)
+		osimage, err := st.GetAMI(archName, region)
+		if err != nil {
+			return "", err
+		}
 		if region != config.Platform.AWS.Region {
 			osimage = fmt.Sprintf("%s,%s", osimage, region)
 		}
+		return osimage, nil
 	case gcp.Name:
-		osimage, err = rhcos.GCP(ctx, arch)
+		if streamArch.Images.Gcp != nil {
+			img := streamArch.Images.Gcp
+			return fmt.Sprintf("projects/%s/global/images/%s", img.Project, img.Name), nil
+		}
+		return "", fmt.Errorf("%s: No GCP build found", st.FormatPrefix(archName))
 	case libvirt.Name:
-		osimage, err = rhcos.QEMU(ctx, arch)
-	case openstack.Name:
-		if oi := config.Platform.OpenStack.ClusterOSImage; oi != "" {
-			osimage = oi
-			break
+		// 𝅘𝅥𝅮 Everything's going to be a-ok 𝅘𝅥𝅮
+		if a, ok := streamArch.Artifacts["qemu"]; ok {
+			return rhcos.FindArtifactURL(a)
 		}
-		osimage, err = rhcos.OpenStack(ctx, arch)
+		return "", fmt.Errorf("%s: No qemu build found", st.FormatPrefix(archName))
 	case ovirt.Name:
-		osimage, err = rhcos.OpenStack(ctx, arch)
+		fallthrough
 	case kubevirt.Name:
-		osimage, err = rhcos.OpenStack(ctx, arch)
+		fallthrough
+	case openstack.Name:
+		op := config.Platform.OpenStack
+		if op != nil {
+			if oi := op.ClusterOSImage; oi != "" {
+				return oi, nil
+			}
+		}
+		if a, ok := streamArch.Artifacts["openstack"]; ok {
+			return rhcos.FindArtifactURL(a)
+		}
+		return "", fmt.Errorf("%s: No openstack build found", st.FormatPrefix(archName))
 	case azure.Name:
-		osimage, err = rhcos.VHD(ctx, arch)
+		ext := streamArch.RHELCoreOSExtensions
+		if ext == nil {
+			return "", fmt.Errorf("%s: No azure build found", st.FormatPrefix(archName))
+		}
+		azd := ext.AzureDisk
+		if azd == nil {
+			return "", fmt.Errorf("%s: No azure build found", st.FormatPrefix(archName))
+		}
+		return azd.URL, nil
 	case baremetal.Name:
-		// Check for RHCOS image URL override
+		// Check for image URL override
 		if oi := config.Platform.BareMetal.ClusterOSImage; oi != "" {
-			osimage = oi
-			break
+			return oi, nil
 		}
 
 		// Note that baremetal IPI currently uses the OpenStack image
 		// because this contains the necessary ironic config drive
 		// ignition support, which isn't enabled in the UPI BM images
-		osimage, err = rhcos.OpenStack(ctx, arch)
+		if a, ok := streamArch.Artifacts["openstack"]; ok {
+			return rhcos.FindArtifactURL(a)
+		}
+		return "", fmt.Errorf("%s: No openstack build found", st.FormatPrefix(archName))
 	case vsphere.Name:
-		// Check for RHCOS image URL override
+		// Check for image URL override
 		if config.Platform.VSphere.ClusterOSImage != "" {
-			osimage = config.Platform.VSphere.ClusterOSImage
-			break
+			return config.Platform.VSphere.ClusterOSImage, nil
 		}
 
-		osimage, err = rhcos.VMware(ctx, arch)
+		if a, ok := streamArch.Artifacts["vmware"]; ok {
+			return rhcos.FindArtifactURL(a)
+		}
+		return "", fmt.Errorf("%s: No vmware build found", st.FormatPrefix(archName))
 	case none.Name:
+		return "", nil
 	default:
-		return "", errors.New("invalid Platform")
-	}
-	if err != nil {
-		return "", err
+		return "", fmt.Errorf("invalid platform %v", config.Platform.Name())
 	}
-	return osimage, nil
 }