bootc switch error: ostree-tar: Processing deferred hardlink

[oglok]

[root@agx ~]# bootc switch quay.io/redhat-et/rhel-9.4-jetpack:36.3.1-20240516220919
layers already present: 24; layers needed: 53 (1.1 GB)
 124.94 MiB [████████████████████] (0s) Fetching layer sha256:a8051b9da1d2 (131.0 MB)
ERROR Switching: Pulling: Importing: Parsing layer blob sha256:a8051b9da1d2659b9c3b35cf885a7cdbb4ffb308a9da42827ae033951dd90f5a: error: ostree-tar: Processing deferred hardlink etc/sos/sos.conf: Failed to find object: No such file or directory: etc: Processing tar: Failed to commit tar: ExitStatus(unix_wait_status(256))

The Containerfile looks like:

ARG BASE_IMAGE
FROM ${BASE_IMAGE} as builder

WORKDIR /app
RUN dnf update -y
RUN dnf install -y make g++ rpmdevtools git-lfs yum-utils python3-devel python3-rpm-macros kernel-devel createrepo
COPY . .

# Build Jetpack drivers using the Bootc Image Kernel version
RUN make kmod-rpm
RUN make jetpack-rpms

RUN createrepo /app/rpmbuild/RPMS/

FROM ${BASE_IMAGE}

RUN dnf update -y
COPY --from=builder /app/rpmbuild/RPMS/ /opt
COPY scripts/bootc/nvidia-local.repo /etc/yum.repos.d/
COPY nvidia-container-toolkit.repo /etc/yum.repos.d/

# Bootc does not support proper installation of kernel modules. We need to use rpm-ostree experimental features for this.
RUN rpm-ostree cliwrap install-to-root /
RUN rpm-ostree install nvidia-jetpack-all nvidia-jetpack-kmod nvidia-container-toolkit-base && ostree container commit

Being BASE_IMAGE the bootc RHEL 9.4 image.

[cgwalters]

The Containerfile looks like:

Except...

COPY . .

is an important thing here to be able to reproduce and we're missing that. Please either:

• Give the full context (git repository link)
• Minimize a reproducer

(Or of course both!)

[cgwalters]

Prior issues/PRs in this area are:

• handle modified hardlinks to /sysroot ostreedev/ostree-rs-ext#405
• tar: Fix multiple hardlinks ostreedev/ostree-rs-ext#558

Based on the symptoms, it might be that something in your script is doing e.g.

RUN sed -i -e 's,...something...,' /etc/dnf/dnf.conf

If so, it may be a workaround to first copy the file via e.g.

RUN cp /etc/dnf/dnf.conf{,.tmp} && mv /etc/dnf/dnf.conf{.tmp,}

or so

[javipolo]

I hit this, or something very similar while building a qcow2 image from a bootc container

To reproduce, create a container with the following Containerfile:

FROM registry.redhat.io/rhel9/rhel-bootc:9.4
RUN mv /etc/selinux /etc/selinux.tmp
RUN mv /etc/selinux.tmp /etc/selinux

And then try to generate an image out of it:

  -v /var/lib/containers/storage:/var/lib/containers/storage \
  --privileged \
  -v ./output:/output \
  -v ./store:/store \
  quay.io/centos-bootc/bootc-image-builder \
    --type qcow2 \
    --local \
    quay.io/jpolo/tmp:bootc

It fails with:

ERROR Installing to filesystem: Creating ostree deployment: Performing deployment: Importing: Parsing layer blob sha256:7f3625126b6801e73e70c4df27d8ffd17e06efd98452c0b76801791ba25d70ab: error: ostree-tar: Processing deferred hardlink etc/selinux.tmp/targeted/active/modules/100/abrt/lang_ext: Failed to find object: No such file or directory: etc: Processing tar: Failed to commit tar: ExitStatus(unix_wait_status(256))
Traceback (most recent call last):
  File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 53, in <module>
    r = main(args["options"], args["inputs"], args["paths"])
  File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 48, in main
    subprocess.run(pargs, env=env, check=True)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['bootc', 'install', 'to-filesystem', '--source-imgref', 'containers-storage:[overlay@/run/osbuild/containers/storage+/run/containers/storage]516e1ce3d29ea2db729c85893311f0328a36683748bb29d8ae07bcd93a3ba89f', '--skip-fetch-check', '--generic-image', '--karg', 'rw', '--karg', 'console=tty0', '--karg', 'console=ttyS0', '--target-imgref', 'quay.io/jpolo/tmp:bootc', '/run/osbuild/mounts']' returned non-zero exit status 1.

[cgwalters]

ostreedev/ostree-rs-ext#648 fixes this.

[cgwalters]

Fixed by #666