BUILD-250 pull in multi-arch friendly version of roots2i and simples2i, add simples2i to mirroring test infra

[alicerum]

Replaced registry.svc.ci.openshift.org/ocp/4.7:test-build-roots2i in Origin build tests with quay.io/multi-arch/test-build-roots2i:ubi8.
Latest image is multiarch - has multiple references of different architectures in its manifest.

Since quay.io/multi-arch/test-build-roots2i:ubi8 is a new image, it is not present in the quay.io/openshift/community-e2e-images - needs to be mirrored there as described in the util/image README.md.

[yselkowitz]

quay.io/multi-arch is not intended as the permanent home of anything, but as a place for proof of concepts, and supplementary images for testing purposes. The proper path forward is to arrange to get multi-arch builds in the canonical location.
/hold

[gabemontero]

quay.io/multi-arch is not intended as the permanent home of anything, but as a place for proof of concepts, and supplementary images for testing purposes. The proper path forward is to arrange to get multi-arch builds in the canonical location.
/hold

Thanks for the clarification @yselkowitz

That said, I should clarify that the use of the quay.io/multi-arch pull specs is an intermediate step wrt the openshift CI support for image usage in a disconnected env process.

Ultimately, once we prove whatever new roots2i and simples2i images work in the e2e's, those images will be pushed to CI's internal registry, and references will be changed back to registry.ci.openshift.org/ocp/4.7:test-build-roots2i and registry.ci.openshift.org/ocp/4.7:test-build-simples2i IN THIS PR

So, with all that, a simple question: when those 2 images were created and pushed to quay.io/multi-arch, were they built using the changes from openshift/build-test-images#7?

EDITOR NOTE: ... I removed misleading info that followed in the remainder of this comment. See @yselkowitz 's response to my question followed but my subsequent responses as I corrected myself and remembered how the @#$% all this stuff worked :-)

[yselkowitz]

The latest tags were built with openshift/build-test-images#7; the ubi8 tags were built with a similar patch which uses FROM registry.access.redhat.com/ubi8-minimal:latest instead. (The forthcoming ubi8-micro images might be a good candidate too once 8.4 ships.)

[gabemontero]

The latest tags were built with openshift/build-test-images#7; the ubi8 tags were built with a similar patch which uses FROM registry.access.redhat.com/ubi8-minimal:latest instead. (The forthcoming ubi8-micro images might be a good candidate too once 8.4 ships.)

OK thanks for the confirm. Let's move the discussion of which FROM we want to go with to that PR.

[gabemontero]

@alicerum - ICYMI the e2e-aws-builds run failed trying to use the new root s2i image.

It because you went too far with the changes. We need to back some of them off temporarily until we verify that the multi arch image works. Then we merge openshift/build-test-images#7 with the right from image. Then we move on with some of the changes you've made.

Ultimately you are getting the error:

error: unable to locate any images in image streams, local docker images with name "quay.io/openshift/community-e2e-images:e2e-quay-io-multi-arch-test-build-roots2i-ubi8-R6dAQ2y0GtQGLBWh"

I'll elaborate on what I think you need to do with review comments in-line.

There will be some changes to Then we update https://github.com/openshift/release/blob/master/ci-operator/config/openshift/build-test-images/openshift-build-test-images-master.yaml to also do simples2i as well. I'll get into that in a bit.

@adambkaplan FYI

[alicerum]

@gabemontero i have removed the image.LocationFor wrap for the roots2i image, so we can test it works here.
I also replaced simples2i image with multiarch, within a different commit, as suggested by @adambkaplan
Can squash before merge if needed. When we see it works, I will return the image.LocationFor wrap for the roots2i image so that it is used from the mirror registry.

[gabemontero]

Bottom line @alicerum I forgot important details during planning for this work. Also, the process is "non-trivial".

Assuming

• WIP: temp use of quay.io/multi-arch/test-build-roots2i:ubi8 to prove… #26150 passes for roots2i, then you do a similar change for simples2i using quay.io/multi-arch/test-build-simples2i:ubi8 for the simples2i image usages as I noted in use registry.access.redhat.com/ubi8-minimal for multi-arch support build-test-images#7 (comment) and those pass
• we can merge a new form of use registry.access.redhat.com/ubi8-minimal for multi-arch support build-test-images#7 where it is changed to use FROM registry.access.redhat.com/ubi8-minimal:latest instead of docker.io/busybox
• the roots2i stuff should just be picked up since we already have an entry for that in test/extended/util/image/image.go
• but you'll need to create a similar entry for simples2i, but after updating https://github.com/openshift/release/blob/master/ci-operator/config/openshift/build-test-images/openshift-build-test-images-master.yaml for simples2i

hopefully @adambkaplan you can help with questions @alicerum may have while I'm out

otherwise, we pick back up when I return

[gabemontero]

Yeah remove this change to quay.io.... Instead, we want it to change to registry.ci.openshift.org/ocp/4.8:test-build-roots2i

Then, once we merge openshift/build-test-images#7 with the right from image, then
https://github.com/openshift/release/blob/master/ci-operator/config/openshift/build-test-images/openshift-build-test-images-master.yaml#L12-L15 will take care of the rest for us.

[gabemontero]

Also, we will want to add

"registry.ci.openshift.org/ocp/4.8:test-build-simples2i": -1,

once we merge openshift/build-test-images#7 and update https://github.com/openshift/release/blob/master/ci-operator/config/openshift/build-test-images/openshift-build-test-images-master.yaml

to add under images:

- context_dir: simples2i
  dockerfile_path: Dockerfile
  from: base
  to: test-build-simples2i

I think, like with #25745 we'll need to engage with Clayton or Maciej to set up the CI mirrors as well.

[gabemontero]

you'll want to re-run make update to reset this.

[gabemontero]

OK, with a hold or WIP flag in place, you can try changing this to

firstArgString := fmt.Sprintf("%s~https://github.com/sclorg/nodejs-ex", "quay.io/multi-arch/test-build-roots2i:ubi8"))

trying to do image.LocationFor on this public image isn't working.

Anyway, we just want to prove that with the iamge quay.io/multi-arch/test-build-roots2i:ubi8 works with the test.

Now, it may be that they have checks in CI that will prevent you from using these images directly. Don't remember exactly.

To help things along, I've moved this image test to #26150

If CI prevents us doing that, then we have to run the tests manually against a test cluster to confirm
quay.io/multi-arch/test-build-roots2i:ubi8 is OK.

[gabemontero]

Yeah revert this change as well. We'll use https://github.com/openshift/origin/pull/26150/files to test out the multi arch image.

[gabemontero]

wait you switched things while I was working on my review ... this change is the same as https://github.com/openshift/origin/pull/26150/files :-)

that said, I hope based on my other explanation that if this passes, we then move forward with openshift/build-test-images#7 this will change to

image.LocationFor("registry.ci.openshift.org/ocp/4.8:test-build-roots2i"))

to pick up the (4.8 only changes) from openshift/build-test-images#7

[gabemontero]

Same ... revert

[gabemontero]

the roots2i test passed in the failed e2e-gcp-builds run

[alicerum]

I have put WIP on this PR. meanwhile I have updated images yet again, to point towards registry.ci.openshift.org.
When images get rebuilt, we can test these changes and see how they work.

[gabemontero]

OK ... spent most of today @alicerum (re)learning this test image mirror process so I can try to explain it.

Gorry details embedded in line.

And of course we can review etc. at the next team video conf.

But the overall intent of this process as I (now re)understand it is:

• we build test images from https://github.com/openshift repos with our CI
• we do NOT include those CI built images in the payload
• while we can reference the CI registry for images, per a manual mirroring process the hi level CI admins perform, those non-payload images are mapped and mirrored to quay.io/openshift/community-e2e-images where the image tag is specific to the image (vs. a a tag used for versioning a single image)
• and this quay.io/openshift/community-e2e-images is wired into the mirroring needed when e2e's run in a disconnected cluster

If I finally have it right this time, hopefully with you going through this and other team members observing, we have a better chance of remembering it all from the start next time.

[gabemontero]

We'll minimally need to add

"registry.ci.openshift.org/ocp/4.8:test-build-simples2i": -1,

as well

[gabemontero]

I believe quay.io/openshift/community-e2e-images:e2e-registry-ci-openshift-org-ocp-4-8-test-build-roots2i-FmW7zebvtGKkls4B will get set up when we reach out to Maciej / Clayton to do the mirroring.

[gabemontero]

This is getting closer @alicerum but this complicated process (and my having only gone through it once before a few months ago) will require some more changes. And in another case or two, those changes still might be experiments. Also, to be fair, even with my attempt to provide a detailed explanation in my last set of comments before PTO, looks like there were still some blind spots where I assumed you knew some things that in hindsight are new to you.

It looks like you changed /test/extended/testdata/bindata.go directly @alicerum ... if so, apologies for the confusion, but you do not change these directly.

With the comment I pointed you to earlier, openshift/build-test-images#7 (comment), you would change:

./testdata/templates/templateinstance_readiness.yaml:          name: quay.io/redhat-developer/test-build-simples2i:latest
./testdata/builds/build-timing/test-s2i-build.json:          "name": "quay.io/redhat-developer/test-build-simples2i:latest"
./testdata/builds/test-s2i-no-outputname.json:          "name": "quay.io/redhat-developer/test-build-simples2i:latest"
./testdata/builds/test-build-cluster-config.yaml:          name: quay.io/redhat-developer/test-build-simples2i:latest
./testdata/builds/test-build-cluster-config.yaml:          name: quay.io/redhat-developer/test-build-simples2i:latest
./testdata/builds/test-auth-build.yaml:          name: quay.io/redhat-developer/test-build-simples2i:latest
./testdata/builds/test-build-proxy.yaml:          name: quay.io/redhat-developer/test-build-simples2i:latest
./testdata/builds/test-build-proxy.yaml:          name: quay.io/redhat-developer/test-build-simples2i:latest
./testdata/builds/test-s2i-build-quota.json:          "name":"quay.io/redhat-developer/test-build-simples2i:latest"
./testdata/builds/statusfail-pushtoregistry.yaml:        name: quay.io/redhat-developer/test-build-simples2i:latest
./testdata/builds/statusfail-failedassemble.yaml:        name: quay.io/redhat-developer/test-build-simples2i:latest
./testdata/builds/test-s2i-build.json:          "name": "quay.io/redhat-developer/test-build-simples2i:latest"
./testdata/builds/valuefrom/failed-sti-build-value-from-config.yaml:        name: quay.io/redhat-developer/test-build-simples2i:latest
./testdata/builds/valuefrom/successful-sti-build-value-from-config.yaml:        name: quay.io/redhat-developer/test-build-simples2i:latest
./testdata/builds/build-postcommit/sti.yaml:          name: quay.io/redhat-developer/test-build-simples2i:latest
./testdata/builds/test-build-revision.json:              "name": "quay.io/redhat-developer/test-build-simples2i:latest"
./testdata/builds/test-nosrc-build.json:              "name": "quay.io/redhat-developer/test-build-simples2i:latest"
./testdata/builds/statusfail-postcommithook.yaml:        name: quay.io/redhat-developer/test-build-simples2i:latest

then in your local clone of the openshift/origin branch for this PR, you run make update. That will then change files like test/extended/testdata/bindata.go

You then commit the results from make update in a separate commit.

Now, as the last e2e build run showed, the simples2i image is not available.

helpers.go:115] error: build error: After retrying 2 times, Pull image still failed due to error: Error initializing source docker://registry.ci.openshift.org/ocp/4.8:test-build-simples2i: Error reading manifest test-build-simples2i in registry.ci.openshift.org/ocp/4.8: manifest unknown: manifest unknown

And for the roots2i change you made:

May 13 10:21:22.311: INFO: Error running /usr/bin/oc --namespace=e2e-test-s2i-build-root-5bqn4 --kubeconfig=/tmp/configfile520207212 new-build quay.io/openshift/community-e2e-images:e2e-registry-ci-openshift-org-ocp-4-8-test-build-roots2i-FmW7zebvtGKkls4B~https://github.com/sclorg/nodejs-ex --name nodejspass:
StdOut>
error: unable to locate any images in image streams, local docker images with name "quay.io/openshift/community-e2e-images:e2e-registry-ci-openshift-org-ocp-4-8-test-build-roots2i-FmW7zebvtGKkls4B"

But that is expected for both, though for different reasons.

First, for simples2i, openshift/release#18523 has to merge. In particular https://github.com/openshift/release/pull/18523/files#diff-b088fb9b067aa48fb733563295dbc3e6033d35467fabbfe786bf07aacfcb4b53R16-R19

Then, if you want to prove it is there, you log into the CI cluster and CI registry. For the CI cluster, you log into its console https://console-openshift-console.apps.ci.l2s4.p1.openshiftapps.com/ with your github creds and then clidk the
copy login command after clicking the triangle by your name. After you login into the cluster. You login into the registry via the instructions at https://docs.ci.openshift.org/docs/how-tos/use-registries-in-build-farm/#how-do-i-log-in-to-pull-images-that-require-authentication ... there is also a --to option on oc registry login that sends the creds to a separte config.json file. I usually do that one. The you can:

docker pull registry.ci.openshift.org/ocp/4.8:test-build-simples2i --authfile=/home/gmontero/pwds/cluster-via-api-ci-pull-secret.json

To get the existing roots2i image.

docker pull registry.ci.openshift.org/ocp/4.8:test-build-roots2i --authfile=/home/gmontero/pwds/cluster-via-api-ci-pull-secret.json

So that image exists. But we got the error because we pass it as a pararmeter into image.LocationFor. Since we have not reached out to Clayton or Maciej yet, our image has not been pushed to quay.io/openshift/community-e2e-images:[roots2i or simples2i tag]. So image.LocationFor` can't get the mapping.
I believe this maps to what happened last time with #25745 (comment) and #25745 (comment)

But this is where my recollection is still fuzzy. But I verified things with PR #26161. To make sure registry.ci.openshift.org/ocp/4.8:test-build-roots2i or registry.ci.openshift.org/ocp/4.8:test-build-simples2i work we do a temporary test run with referencing them directly.
No image.LocationFor usage.

If they are OK, we add the image.LocationFor usage back.

So, when "construction of the image ref" is entirely in a golang file, the change will look similar to the s2i_root.go change in this PR, but

image.LocationFor("registry.ci.openshift.org/ocp/4.8:test-build-simples2i")

The s2i_root.go change failed because we have not reached out to Clayton or Maciej to mirror registry.ci.openshift.org/ocp/4.8:test-build-roots2i to quay.io/openshift/community-e2e-images yet.

Now, for the references to registry.ci.openshift.org/ocp/4.8:test-build-simples2i in the json/yaml files, our temporary sniff tests with direct references to registry.ci.openshift.org/ocp/4.8:test-build-simples2i will verify that image is OK.

But we then want to move onto, after getting Clayton or Maciej to do the quay.io/openshift/community-e2e-images:[roots2i or simples2i tag] mirroring, we have to incorporate use of image.LocationFor

The complication there is IIR why I did not bother with simples2i last time :-)

I think we'll need update the the objects created from those files, modifying the image fields with the results from
image.LocationFor("registry.ci.openshift.org/ocp/4.8:test-build-simples2i") before using/creating any objects from those json/yamls.

But what happens with that image.LocationFor call is that the registry.ci image gets mapped to a quay.io/openshift/community-e2e-images one.

Quick fyi ... after the mirroring, if you look today for the roots2i one today, you see it at:

./test/extended/util/image/zz_generated.txt:registry.ci.openshift.org/ocp/4.7:test-build-roots2i quay.io/openshift/community-e2e-images:e2e-registry-ci-openshift-org-ocp-4-7-test-build-roots2i-tG08qb6aSmDhxBBy

there will be an update when we go from 4.7 to 4.8 for roots2i, and a new entry for simples2i after we mirror that.

So those are the gorry details of how

• we build test images from https://github.com/openshift repos with our CI
• we do NOT include those CI built images in the payload
• while we can reference the CI registry for images, per a manual mirroring process the hi level CI admins perform, those non-payload images are mapped and mirrored to quay.io/openshift/community-e2e-images where the image tag is specific to the image (vs. a a tag used for versioning a single image)
• and this quay.io/openshift/community-e2e-images is wired into the mirroring needed when e2e's run in a disconnected cluster

[alicerum]

no, i didn't change bindata directly. it has a big comment in it that states this file is generated. i changed yaml files and ran make update.

[alicerum]

i'm already logged into the ci cluster, and that's how i checked that manifests there don't seem to be multi-arch, so this is where my confusion comes from.

[alicerum]

registry.ci.openshift.org/ocp/4.8:test-build-simples2i is never referenced by image.LocationFor, and that is why it doesn't seem to need to be added to images.go. Previous test-build-simples2i was not there either.

[gabemontero]

we need to add references to registry.ci.openshift.org/ocp/4.8:test-build-simples2i from new image.LocationFor calls that need to be added to existing tests.

And yep I missed the json updates in the PR. My bad. That said, again, we are going to have to add a layer where we replace that direct use of registry.ci.openshift.org/ocp/4.8:test-build-simples2i in the image ref's in those json files with a the output of image.LocationFor, where you pass in registry.ci.openshift.org/ocp/4.8:test-build-simples2i as the parameter.

Any TEMPORARY use of registry.ci.openshift.org/ocp/4.8:test-build-simples2i in the PR to make sure the new image didn't break anything is just that. A sanity check that the image build from CI is OK.

But yes, ultimately we'll be mapping registry.ci.openshift.org/ocp/4.8:test-build-simples2i to quay.io/openshift/community-e2e-images:[some tag] .... it is just that some tag is too dynamic that we cannot hard code it in our test files.

[alicerum]

ohhhhhh it makes sense. let's discuss that after scrum and make sure we are on the same page please.

[gabemontero]

So distilling https://github.com/openshift/origin/pull/26149/files#r632694287 for this line.

1. temporary step, change this to firstArgString := fmt.Sprintf("%s~https://github.com/sclorg/nodejs-ex", "registry.ci.openshift.org/ocp/4.8:test-build-roots2i")

2. if tests from this file are OK, we reach out to @soltysh or @smarterclayton and have them mirroregistry.ci.openshift.org/ocp/4.8:test-build-roots2i to quay.io/openshift/community-e2e-images

3. then we revert this line back to the way you have it now

[gabemontero]

So distilling https://github.com/openshift/origin/pull/26149/files#r632694287 for this line.

1. temporary step, change this to firstArgString := fmt.Sprintf("%s~https://github.com/sclorg/nodejs-ex", "registry.ci.openshift.org/ocp/4.8:test-build-roots2i")

2. if tests from this file are OK, we reach out to @soltysh or @smarterclayton and have them mirroregistry.ci.openshift.org/ocp/4.8:test-build-roots2i to quay.io/openshift/community-e2e-images

3. then we revert this line back to the way you have it now

[gabemontero]

So looking at the recent e2e-gcp-build failures, the simples2i image is now found, but the system is not happy about it @alicerum and @adambkaplan

		Generating dockerfile with builder image registry.ci.openshift.org/ocp/4.8:test-build-simples2i
		error: build error: image "0" must specify a user that is numeric and within the range of allowed users

[gabemontero]

Conversely the test with this image passed:

`

: [sig-builds][Feature:Builds] s2i build with a root user image should create a root build and pass with a privileged SCC [Skipped:Disconnected] [Suite:openshift/conformance/parallel]	1m14s

`

It is safe to ask the admins to mirror this one, and then we can add the image.LocationFor back in, but use 4.8 instead of 4.7

[gabemontero]

reminder @alicerum you still need to add the image.LocationFor back in.

just keep the update from the 4.7 to 4.8 ref

[alicerum]

@gabemontero apparently, mechanism for replacing images in bindata already exists, yaml and json assets are written to disk with images replaced for any images added to the mirrorlist in image.go
they are replaced here

So I added the simples2i image to this list, and manually checked that image is replaced correctly with it's mirrored counterpart.

[gabemontero]

@gabemontero apparently, mechanism for replacing images in bindata already exists, yaml and json assets are written to disk with images replaced for any images added to the mirrorlist in image.go
they are replaced here

So I added the simples2i image to this list, and manually checked that image is replaced correctly with it's mirrored counterpart.

Good find @alicerum .... the wiring of ReplaceImage into FixturePath makes our job easier here.

[gabemontero]

OK let's temporarily remove this @alicerum so we can prove that the USER 1001 fix to registry.ci.openshift.org/ocp/4.8:test-build-simples2i worked.

registry.ci.openshift.org/ocp/4.8:test-build-simples2i has been updated with your recently merged build-test-images PR

[gabemontero]

fyi @alicerum I think it is worthwhile to capture a build log where we see buildah's attempt to pull the image from quay.io/openshift/community-e2e-test and show it to @nalind to see if he thinks the output produced when it is pulling the image, and that it takes so long, is indicative of an issue that needs to be looked at on the buildah side (vs. quay.io just being slow).

You could run the test manually and use --include-success along with perhaps some short term changes to the test so it prints the build log even when it succeeds.

@adambkaplan FYI

[gabemontero]

/hold cancel

[alicerum]

@gabemontero I increased the CPU quota in the quota test to 400m. Should be enough to process bigger images. Tests pass locally, so if they also do here, i think we can unhold and merge.

[gabemontero]

per team discussion in scrum .... this new value is still significantly less than what we would proscribe to a customer ... to quote @adambkaplan, proscribing an entire core for builds could be considered "best practice"

and certainly our iterating over this has proven the value is getting properly applied to the build object, which is the real intent of the s2i quota test :-)

[alicerum]

60m still works, too, but takes a lot of time to complete.

[gabemontero]

yes fair clarification

the current test timeout of 10min was bumped to 15min to allow for that, but we are going with this approach instead

[gabemontero]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alicerum, gabemontero, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• test/extended/OWNERS [soltysh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[gabemontero]

I'll go glass half full and expect what @alicerum saw in her local testing will translate to CI, and not wait for the e2e-gcp-build run to land before applying lgtm

For the superstitious, hopefully I did not jinx it :-)

[gabemontero]

I'll go glass half full and expect what @alicerum saw in her local testing will translate to CI, and not wait for the e2e-gcp-build run to land before applying lgtm

For the superstitious, hopefully I did not jinx it :-)

e2e-gcp and e2e-gcp-builds have passed with https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/26149/pull-ci-openshift-origin-master-e2e-gcp/1397920921167073280 and https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/26149/pull-ci-openshift-origin-master-e2e-gcp-builds/1397920921200627712

should be on to navigating non-related flakes and whatever additional labels might be needed to merge this

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.