Build of functions stalls when using podman on macOS

[fc7]

This is an issue I could reproduce on two different machines, so I believe it is not just a local setup matter.

On a fresh macOS (without Docker):

brew install podman
sudo podman-mac-helper install             # for Docker compatibility
brew tap knative-sandbox/kn-plugins
brew install func
podman machine init
podman machine start
func create -l python hello
cd hello
func build -v

The output will look like this (with a lot left out in-between:

Building function image
Pulling image gcr.io/paketo-buildpacks/builder:base
<snip>
Creating builder with the following buildpacks:
<snip>
===> ANALYZING
<snip>
===> DETECTING
<snip>
===> RESTORING
<snip>
===> BUILDING
<snip>
===> EXPORTING
<snip>

Then the end of the output will look like this:

Adding cache layer 'paketo-buildpacks/pip-install:cache'
Layer 'paketo-buildpacks/pip-install:cache' SHA: sha256:10b6b706349d2716d7d1cef642cde0555dafb160ff66f092e28fb8fb4ca8c02d
Reusing tarball for layer "paketo-buildpacks/pip-install:packages" with SHA: sha256:66c8492a08678ef827c60b61c48d9d05490cda08d799afcbc707709af1ece300
Adding cache layer 'paketo-buildpacks/pip-install:packages'
Layer 'paketo-buildpacks/pip-install:packages' SHA: sha256:66c8492a08678ef827c60b61c48d9d05490cda08d799afcbc707709af1ece300
Reading buildpack directory: /layers/paketo-buildpacks_python-start
Reading buildpack directory item: launch.toml
Reading buildpack directory: /layers/paketo-buildpacks_procfile
Reading buildpack directory item: launch.toml
Found SBOM of type cache for at /layers/sbom/cache
Reusing tarball for layer "buildpacksio/lifecycle:cache.sbom" with SHA: sha256:6a217876fc8f7bfc184ec28cb1dd29fcfe185a88a3c1c724fdc570c00e963239
Adding cache layer 'buildpacksio/lifecycle:cache.sbom'
Layer 'buildpacksio/lifecycle:cache.sbom' SHA: sha256:6a217876fc8f7bfc184ec28cb1dd29fcfe185a88a3c1c724fdc570c00e963239

and the process stalls there forever, until I interrupt it with CTRL-C.
As I said, I have tested this on two different Macbooks.

I have also tried with a rootful podman machine but the result is the same.

Any idea how to debug this?

[matejvasek]

@fc7 what version of podman do you use?

[matejvasek]

@lance @zroubalik could you please try to reproduce this?

[fc7]

@matejvasek v4.4.1

[matejvasek]

@fc7 could you please get syslogs from the VM? What's there?

[fc7]

Here it is (only the relevant portion) syslog.log

[fc7]

Here the output from func build -v func_build.log
(2nd run, which is why you see it reuses caches, but the result was the same, after the line about cache.sbom layer, it stalls)

[matejvasek]

@lance @zroubalik nevermind, I already figured out what is the issue.

[matejvasek]

This has been fixed in containers/podman#16781
however VM images (fedora-coreos) do not contain the fix yet.

[fc7]

All current qemu images of Fedora CoreOS indeed come with podman 4.3. Let's hope the next release with podman 4.4 happens soon enough.

Side-note: a workaround is to use s2i instead of buildpacks; unfortunately func by default is not aware of any s2i builder image for Python (or Rust for that matter), which is why I was using pack instead. But this is a topic for a separate issue...

[lance]

Side-note: a workaround is to use s2i instead of buildpacks; unfortunately func by default is not aware of any s2i builder image for Python (or Rust for that matter), which is why I was using pack instead. But this is a topic for a separate issue...

Indeed - we should see this in the next release: #1562