Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kola/kernel-replace: adapt for RHCOS and SCOS #3048

Merged
merged 1 commit into from
Jul 16, 2024

Conversation

c4rt0
Copy link
Member

@c4rt0 c4rt0 commented Jul 2, 2024

There were instances where kola/switch-kernel test would fail.
It was recommended to remove it (PR) and generalize this test, so that it can work with FCOS, RHCOS and SCOS.

@c4rt0 c4rt0 requested a review from jbtrystram July 2, 2024 11:28
Copy link
Contributor

@jbtrystram jbtrystram left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shellcheck will complain other than this it looks good.
Could you run the test locally on RHCOS ?

tests/kola/rpm-ostree/kernel-replace Outdated Show resolved Hide resolved
tests/kola/rpm-ostree/kernel-replace Outdated Show resolved Hide resolved
@c4rt0 c4rt0 force-pushed the PR/kernel branch 2 times, most recently from d95cceb to 18d9d86 Compare July 5, 2024 18:37
@c4rt0
Copy link
Member Author

c4rt0 commented Jul 5, 2024

Could you run the test locally on RHCOS ?

Just tried it now, it's failing. I'm now investigating:

=== RUN   ext.config.shared.rpm-ostree.kernel-replace
systemctl status kola-runext.service:
× kola-runext.service
     Loaded: loaded (/etc/systemd/system/kola-runext.service; static)
     Active: failed (Result: exit-code) since Fri 2024-07-05 18:34:49 UTC; 399ms ago
   Duration: 36.717s
    Process: 1266 ExecStart=/usr/local/bin/kola-runext-kernel-replace (code=exited, status=1/FAILURE)
   Main PID: 1266 (code=exited, status=1/FAILURE)
        CPU: 33.418s

Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1270]: ++ grep '^ID=' /etc/os-release
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + OS_ID=rhcos
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + image_dir=/var/tmp/coreos
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + image=oci:/var/tmp/coreos
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + image_pull=ostree-unverified-image:oci:/var/tmp/coreos
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + tmp_imagedir=/var/tmp/coreos-tmp
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1273]: ++ arch
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + arch=x86_64
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + kver=6.2.9-300.fc38.x86_64
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + case "${AUTOPKGTEST_REBOOT_MARK:-}" in
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1266]: + skopeo copy oci:/var/tmp/coreos containers-storage:localhost/rhcos
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1274]: time="2024-07-05T18:34:13Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1274]: Getting image source signatures
Jul 05 18:34:13 qemu0 kola-runext-kernel-replace[1274]: Copying blob sha256:5d033c3086bc3b8a75fe007c50fa48d18cdea48560bd6d6b2b359d79397bb167
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1274]: Copying config sha256:5b8ccce2c709b5c14f75df3761da4eccfcef89d39165711f14c71196df75b28c
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1274]: Writing manifest to image destination
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + rm /var/tmp/coreos -rf
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1307]: ++ mktemp -d
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + td=/tmp/tmp.q2EfPgCVdW
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + cd /tmp/tmp.q2EfPgCVdW
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1310]: ++ grep Version
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1309]: ++ rpm-ostree --version
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + version=' Version: '\''2024.3'\'''
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + cat
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + touched_resolv_conf=0
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + test '!' -f /etc/resolv.conf
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1266]: + podman build --net=host -t localhost/rhcos-derived --squash .
Jul 05 18:34:48 qemu0 podman[1313]: 2024-07-05 18:34:48.253248775 +0000 UTC m=+0.050040248 system refresh
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1313]: STEP 1/3: FROM localhost/rhcos
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1313]: STEP 2/3: RUN rpm-ostree cliwrap install-to-root /
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1345]: Successfully enabled cliwrap for /
Jul 05 18:34:48 qemu0 kola-runext-kernel-replace[1313]: STEP 3/3: RUN rpm-ostree override replace     https://koji.fedoraproject.org/koji/buildinfo?buildID=2178613 &&     ostree container commit
Jul 05 18:34:49 qemu0 kola-runext-kernel-replace[1397]: Downloading https://koji.fedoraproject.org/koji/buildinfo?buildID=2178613...done
Jul 05 18:34:49 qemu0 kola-runext-kernel-replace[1397]: (rpm-ostree override replace:2): libdnf-WARNING **: 18:34:49.701: failed to read RPM: /tmp/.tmpysF0mq/13.rpm: not a rpm, skipping
Jul 05 18:34:49 qemu0 kola-runext-kernel-replace[1397]: error: Invalid RPM file: /tmp/.tmpysF0mq/13.rpm
Jul 05 18:34:49 qemu0 podman[1313]: 2024-07-05 18:34:49.787637333 +0000 UTC m=+1.584428810 image build  
Jul 05 18:34:49 qemu0 kola-runext-kernel-replace[1313]: Error: building at STEP "RUN rpm-ostree override replace     https://koji.fedoraproject.org/koji/buildinfo?buildID=2178613 &&     ostree container commit": while running runtime: exit status 1
Jul 05 18:34:49 qemu0 systemd[1]: kola-runext.service: Main process exited, code=exited, status=1/FAILURE
Jul 05 18:34:49 qemu0 systemd[1]: kola-runext.service: Failed with result 'exit-code'.
Jul 05 18:34:49 qemu0 systemd[1]: kola-runext.service: Consumed 33.418s CPU time.
--- FAIL: ext.config.shared.rpm-ostree.kernel-replace (171.99s)
        cluster.go:162: Error: Unit kola-runext.service exited with code 1
        cluster.go:162: 2024-07-05T18:34:50Z cli: Unit kola-runext.service exited with code 1
        harness.go:1265: kolet failed: : kolet run-test-unit failed: Process exited with status 1
FAIL, output in tmp/kola
Error: harness: test suite failed

@jlebon
Copy link
Member

jlebon commented Jul 5, 2024

Trying to replace the RHCOS kernel with a Fedora one in theory should work, but I think we can get closer to what users are likely to do by replacing with a CentOS Stream kernel instead. So basically:

  • FCOS: replace with Fedora kernel
  • RHCOS and SCOS: replace with CentOS Stream kernel

For Fedora, the test currently pins to a specific Koji build, which is stale now. We should bump it. Once we have dnf in all the streams, we should probably change that actually to just dynamically always take the latest kernel that's != to the kernel embedded in the container image (similarly for RHCOS/SCOS too).

@c4rt0 c4rt0 force-pushed the PR/kernel branch 2 times, most recently from 1899c9f to 3ec17ba Compare July 9, 2024 16:08
@c4rt0
Copy link
Member Author

c4rt0 commented Jul 9, 2024

I ran this on both rhcos (4.17) and the latest fcos build. Test passes as expected.

@c4rt0
Copy link
Member Author

c4rt0 commented Jul 9, 2024

Trying to replace the RHCOS kernel with a Fedora one in theory should work, but I think we can get closer to what users are likely to do by replacing with a CentOS Stream kernel instead. So basically:

* FCOS: replace with Fedora kernel

* RHCOS and SCOS: replace with CentOS Stream kernel

That's now done.

For Fedora, the test currently pins to a specific Koji build, which is stale now. We should bump it.

That's also done.

Once we have dnf in all the streams, we should probably change that actually to just dynamically always take the latest kernel that's != to the kernel embedded in the container image (similarly for RHCOS/SCOS too).

I can look into this in the future.

@c4rt0 c4rt0 requested review from jbtrystram and jlebon July 9, 2024 16:20
tests/kola/rpm-ostree/kernel-replace Outdated Show resolved Hide resolved
tests/kola/rpm-ostree/kernel-replace Outdated Show resolved Hide resolved
tests/kola/rpm-ostree/kernel-replace Outdated Show resolved Hide resolved
tests/kola/rpm-ostree/kernel-replace Outdated Show resolved Hide resolved
tests/kola/rpm-ostree/kernel-replace Outdated Show resolved Hide resolved
@c4rt0
Copy link
Member Author

c4rt0 commented Jul 10, 2024

Again, latest changes successfully passed test both on fcos and rhcos. Thank you for all of the suggestions!

@c4rt0 c4rt0 marked this pull request as ready for review July 10, 2024 12:11
@c4rt0 c4rt0 force-pushed the PR/kernel branch 2 times, most recently from e99b318 to e4c36a4 Compare July 10, 2024 12:24
@c4rt0 c4rt0 requested a review from jlebon July 10, 2024 15:36
Copy link
Member

@jlebon jlebon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested commit title:

kola/kernel-replace: adapt for RHCOS and SCOS

For the commit message, I think in general we should avoid just linking to some other URL that has all the context. Let's inline that context, and provide the URL only as an additional reference in the footer.

tests/kola/rpm-ostree/kernel-replace Outdated Show resolved Hide resolved
tests/kola/rpm-ostree/kernel-replace Outdated Show resolved Hide resolved
tests/kola/rpm-ostree/kernel-replace Outdated Show resolved Hide resolved
Copy link
Member

@jlebon jlebon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall. Thoughts on suggested commit title in #3048 (review) ? I think mentioning OS_ID there isn't really capturing it.

tests/kola/rpm-ostree/kernel-replace Outdated Show resolved Hide resolved
@c4rt0 c4rt0 changed the title kola/kernel-replace: Add OS_ID recogntion kola/kernel-replace: adapt for RHCOS and SCOS Jul 15, 2024
jbtrystram
jbtrystram previously approved these changes Jul 16, 2024
Copy link
Contributor

@jbtrystram jbtrystram left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall. I am a bit concerned by the hard-coded kernel package URLs, won't they go stale eventually, breaking the test?

I don't have a good alternative to suggest though, except adding an item to the rebase checklist..

tests/kola/rpm-ostree/kernel-replace Show resolved Hide resolved
@c4rt0
Copy link
Member Author

c4rt0 commented Jul 16, 2024

LGTM overall. I am a bit concerned by the hard-coded kernel package URLs, won't they go stale eventually, breaking the test?

I don't have a good alternative to suggest though, except adding an item to the rebase checklist..

I agree, @jlebon also talked about it here: #3048 (comment). Let's wait with that until we have DNF in all of the streams.

In one of the older issues, kola switch-kernel test was failing. In the discussion @jlebon suggested to remove named test and make it external, so that it can be utilized by FCOS, and also SCOS/RHCOS.
Additionally, the kernel version for FCOS was updated in this PR to the latest stable version `6.9.8-200.fc40`.

See: coreos/coreos-assembler#1245
@c4rt0 c4rt0 enabled auto-merge (rebase) July 16, 2024 11:56
@c4rt0 c4rt0 merged commit 721394b into coreos:testing-devel Jul 16, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants