Error creating cluster on Fedora 34 / Podman.

[simon-geard]

Steps to reproduce the issue:

On a reasonably stock Fedora Workstation 34 installation, installed Minikube from RPM according to the instructions here:

• https://minikube.sigs.k8s.io/docs/start/

...and started simply with "minikube start", again from the instructions above.

After much time spent downloading images, it's reported a failure to start the minikube container, and I suspect has failed to create it. The network and volume of the same name have both been created, but there's no evidence that the container itself has ever existed.

I'm also seeing what seems to be identical behaviour on a RHEL8 system at work - I'm raising this after trying on my personal machine - but happy to focus on the Fedora case for now.

Run minikube logs --file=logs.txt and drag and drop the log file into this issue

Full output of failed command if not minikube start:

😄 minikube v1.23.2 on Fedora 34
✨ Automatically selected the podman driver
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...
💾 Downloading Kubernetes v1.22.2 preload ...
> gcr.io/k8s-minikube/kicbase: 46.41 MiB / 355.40 MiB 13.06% 336.55 KiB p > preloaded-images-k8s-v13-v1...: 37.75 MiB / 511.84 MiB 7.38% 187.99 KiB > gcr.io/k8s-minikube/kicbase: 46.50 MiB / 355.40 MiB
minikube_logs_628fcb9b3bdfce71b7c231f977786c07a9b93749_0.log
13.08% 344.07 KiB p > preloaded-images-k8s-v13-v1...: 37.78 MiB / 511.84 MiB 7.38% 187.99 KiB > gcr.io/k8s-minikube/kicbase: 46.66 MiB / 355.40 MiB 13.13% 344.07 KiB p > preloaded-images-k8s-v13-v1...: 37.83 MiB / 511.84 MiB 7.39% 187.99 KiB > gcr.io/k8s-minikube/kicbase: 46.77 MiB / 355.40 MiB 13.16% 344.07 KiB p > preloaded-images-k8s-v13-v1...: 37.87 MiB / 511.84 MiB 7.40% 189.63 KiB > gcr.io/k8s-minikube/kicbase: 46.89 MiB / 355.40 MiB 13.19% 364.87 KiB p > preloaded-images-k8s-v13-v1...: 37.91 MiB / 511.84 MiB 7.41% 189.63 KiB > gcr.io/k8s-minikube/kicbase: 46.91 MiB / 355.40 MiB 13.20% 364.87 KiB p > preloaded-images-k8s-v13-v1...: 37.91 MiB / 511.84 MiB 7.41% 189.63 KiB > gcr.io/k8s-minikube/kicbase: 46.91 MiB / 355.40 MiB 13.20% 364.87 KiB p > preloaded-images-k8s-v13-v1...: 37.91 MiB / 511.84 MiB 7.41% 180.83 KiB > gcr.io/k8s-minikube/kicbase: 46.91 MiB / 355.40 MiB 13.20% 343.05 KiB p > preloaded-images-k8s-v13-v1...: 37.95 MiB / 511.84 MiB 7.41% 180.83 KiB > gcr.io/k8s-minikube/kicbase: 47.08 MiB / 355.40 MiB 13.25% 343.05 KiB p > preloaded-images-k8s-v13-v1...: 37.97 MiB / 511.84 MiB 7.42% 180.83 KiB > gcr.io/k8s-minikube/kicbase: 47.08 MiB / 355.40 MiB 13.25% 343.05 KiB p > preloaded-images-k8s-v13-v1...: 37.97 MiB / 511.84 MiB 7.42% 176.05 KiB > gcr.io/k8s-minikube/kicbase: 47.17 MiB / 355.40 MiB 13.27% 350.15 KiB p > preloaded-images-k8s-v13-v1...: 38.00 MiB / 511.84 MiB 7.42% 176.05 KiB > gcr.io/k8s-minikube/kicbase: 47.19 MiB / 355.40 MiB 13.28% 350.15 KiB p > preloaded-images-k8s-v13-v1...: 38.00 MiB / 511.84 MiB 7.42% 176.05 KiB > gcr.io/k8s-minikube/kicbase: 47.22 MiB / 355.40 MiB 13.29% 350.15 KiB p > preloaded-images-k8s-v13-v1...: 38.01 MiB / 511.84 MiB 7.43% 169.86 KiB > gcr.io/k8s-minikube/kicbase: 47.23 MiB / 355.40 MiB 13.29% 334.45 KiB p > preloaded-images-k8s-v13-v1...: 38.01 MiB / 511.84 MiB 7.43% 169.86 KiB > gcr.io/k8s-minikube/kicbase: 47.23 MiB / 355.40 MiB 13.29% 334.45 KiB p > preloaded-images-k8s-v13-v1...: 38.05 MiB / 511.84 MiB 7.43% 169.86 KiB > gcr.io/k8s-minikube/kicbase: 47.36 MiB / 355.40 MiB 13.33% 334.45 KiB p > preloaded-images-k8s-v13-v1...: 38.08 MiB / 511.84 MiB 7.44% 165.78 KiB > gcr.io/k8s-minikube/kicbase: 47.47 MiB / 355.40 MiB 13.36% 338.69 KiB p > preloaded-images-k8s-v13-v1...: 38.09 MiB / 511.84 MiB 7.44% 165.78 KiB > gcr.io/k8s-minikube/kicbase: 47.50 MiB / 355.40 MiB 13.37% 338.69 KiB p > preloaded-images-k8s-v13-v1...: 38.14 MiB / 511.84 MiB 7.45% 165.78 KiB > gcr.io/k8s-minikube/kicbase: 47.59 MiB / 355.40 MiB 13.39% 338.69 KiB p > preloaded-images-k8s-v13-v1...: 38.17 MiB / 511.84 MiB 7.46% 165.41 KiB > gcr.io/k8s-minikube/kicbase: 47.77 MiB / 355.40 MiB 13.44% 349.48 KiB p > preloaded-images-k8s-v13-v1...: 38.20 MiB / 511.84 MiB 7.46% 165.41 KiB > gcr.io/k8s-minikube/kicbase: 47.89 MiB / 355.40 MiB 13.48% 349.48 KiB p > preloaded-images-k8s-v13-v1...: 38.25 MiB / 511.84 MiB 7.47% 165.41 KiB > gcr.io/k8s-minikube/kicbase: 47.92 MiB / 355.40 MiB 13.48% 349.48 KiB p > preloaded-images-k8s-v13-v1...: 38.25 MiB / 511.84 MiB 7.47% 163.34 KiB > gcr.io/k8s-minikube/kicbase: 48.03 MiB / 355.40 MiB 13.51% 356.19 KiB p > preloaded-images-k8s-v13-v1...: 38.30 MiB / 511.84 MiB 7.48% 163.34 KiB > gcr.io/k8s-minikube/kicbase: 48.11 MiB / 355.40 MiB 13.54% 356.19 KiB p > preloaded-images-k8s-v13-v1...: 38.33 MiB / 51 > preloaded-images > gcr.io/k8s-min > preloaded-images-k8s-v13-v1...: 511.84 MiB / 511.84 MiB 100.00% 502.15 K / 355.40 MiB 13.59% 359.03 KiB p/oaded-images-k8s-v13-v1...: 38.37 MiB / 511.84 MiB 7.50% 164.84 KiB 4 KiB > gcr.io/k8s-minikube/kicbase: 355.39 MiB / 355.40 MiB 100.00% 337.50 KiB
E1008 22:05:24.369718 5243 cache.go:201] Error downloading kic artifacts: not yet implemented, see issue #8426
🔥 Creating podman container (CPUs=2, Memory=3900MB) ...
🤦 StartHost failed, but will try again: creating host: create host timed out in 360.000000 seconds
🔄 Restarting existing podman container for "minikube" ...
😿 Failed to start podman container. Running "minikube delete" may fix it: podman inspect ip minikube: sudo -n podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container minikube

❌ Exiting due to GUEST_PROVISION: Failed to start host: podman inspect ip minikube: sudo -n podman container inspect -f minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container minikube

╭───────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ 😿 If the above advice does not help, please let us know: │
│ 👉 https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ Please run minikube logs --file=logs.txt and attach logs.txt to the GitHub issue. │
│ │
╰──────────────────────────────────────────────────────────────────────────────────

[afbjorklund]

No logs here, unfortunately

[simon-geard]

Is that better? I thought I'd attached them, but must have made an error (not used to GitHub)

minikube_logs_628fcb9b3bdfce71b7c231f977786c07a9b93749_0.log

[afbjorklund]

Thanks for the logs, seems network inspect fails.

network_create.go:258] error running [sudo -n podman network inspect minikube]: sudo -n podman network inspect minikube: exit status 125

It is supposed to create the private network if it doesn't exist already, instead of exiting with an error like that.

The shown error was misleading, since it only says "container".

podman inspect ip minikube: sudo -n podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125

[simon-geard]

That may be the case, but the network has been created successfully. If I simply inspect it without the format part, here's the full output:

sudo -n podman network inspect minikube 
[
    {
        "cniVersion": "0.4.0",
        "name": "minikube",
        "plugins": [
            {
                "bridge": "cni-podman1",
                "hairpinMode": true,
                "ipMasq": true,
                "ipam": {
                    "ranges": [
                        [
                            {
                                "gateway": "192.168.49.1",
                                "subnet": "192.168.49.0/24"
                            }
                        ]
                    ],
                    "routes": [
                        {
                            "dst": "0.0.0.0/0"
                        }
                    ],
                    "type": "host-local"
                },
                "isGateway": true,
                "type": "bridge"
            },
            {
                "capabilities": {
                    "portMappings": true
                },
                "type": "portmap"
            },
            {
                "backend": "",
                "type": "firewall"
            },
            {
                "type": "tuning"
            },
            {
                "capabilities": {
                    "aliases": true
                },
                "domainName": "dns.podman",
                "type": "dnsname"
            }
        ]
    }
]

So it looks to me that the "podman network inspect" commands have failed with status 125, indicating that the named network doesn't exist - at which point it creates it a few lines later:

I1008 22:05:24.848563    5243 network_create.go:106] attempt to create podman network minikube 192.168.49.0/24 with gateway 192.168.49.1 and MTU of 0 ...
I1008 22:05:24.848594    5243 cli_runner.go:115] Run: sudo -n podman network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 minikube
I1008 22:05:24.976642    5243 network_create.go:90] podman network minikube 192.168.49.0/24 created
I1008 22:05:24.976652    5243 kic.go:106] calculated static IP "192.168.49.2" for the "minikube" container

[simon-geard]

This is the part of the logs that concerns me:

I1008 22:05:24.848594    5243 cli_runner.go:115] Run: sudo -n podman network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 minikube
I1008 22:05:24.976642    5243 network_create.go:90] podman network minikube 192.168.49.0/24 created
I1008 22:05:24.976652    5243 kic.go:106] calculated static IP "192.168.49.2" for the "minikube" container
I1008 22:05:24.976688    5243 cli_runner.go:115] Run: sudo -n podman ps -a --format {{.Names}}
I1008 22:05:25.103826    5243 cli_runner.go:115] Run: sudo -n podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true
I1008 22:05:25.232227    5243 oci.go:102] Successfully created a podman volume minikube
I1008 22:05:25.232268    5243 cli_runner.go:115] Run: sudo -n podman run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.27 -d /var/lib
I1008 22:11:24.377299    5243 ssh_runner.go:152] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I1008 22:11:24.377335    5243 cli_runner.go:115] Run: sudo -n podman version --format {{.Version}}
I1008 22:11:24.508301    5243 cli_runner.go:115] Run: sudo -n podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
W1008 22:11:24.544640    5243 cli_runner.go:162] sudo -n podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube returned with exit code 125

I see it successfully creating the network, and I see it successfully creating the volume. And then I see it start a container called "minikube-preload-sidecar", before querying for a running container called "minikube".

Assuming this logging is showing all podman commands, I don't see anywhere in the logs where a "minikube" container is created or started.