Can not install Sysbox into K0s cluster (Kubelet binary not identified.)

[NiklasRosenstein]

Hello 👋 I have a K0s cluster (v1.23.13+k0s) that I want to install Sysbox into, but unfortunately the installation fails.

$ kubectl logs -n kube-system sysbox-deploy-k8s-mnm9h
Detected Kubernetes version v1.23
Deploying Kubelet config agent on the host ...
Running Kubelet config agent on the host (will restart Kubelet and temporary bring down all pods on this node for ~1 min) ...
Job for kubelet-config-helper.service failed because the control process exited with error code. See "systemctl status kubelet-config-helper.service" and "journalctl -xe" for details.

I find the error in the kubelet-config-helper logs:

$ sudo journalctl -eu kubelet-config-helper
...
Nov 13 23:26:22 my-host sh[2889612]: Detected systemd-managed host-based kubelet deployment on host.
Nov 13 23:26:22 my-host sh[2889655]: ++ get_kubelet_bin
Nov 13 23:26:22 my-host sh[2889657]: +++ systemctl show kubelet
Nov 13 23:26:22 my-host sh[2889658]: +++ grep ExecStart=
Nov 13 23:26:22 my-host sh[2889659]: +++ cut -d ';' -f1
Nov 13 23:26:22 my-host sh[2889655]: ++ local tmp=
Nov 13 23:26:22 my-host sh[2889655]: ++ tmp=
Nov 13 23:26:22 my-host sh[2889660]: ++ echo ''
Nov 13 23:26:22 my-host sh[2889661]: ++ xargs
Nov 13 23:26:22 my-host sh[2889612]: + kubelet_bin=
Nov 13 23:26:22 my-host sh[2889612]: + '[' -z '' ']'
Nov 13 23:26:22 my-host sh[2889612]: + die 'Kubelet binary not identified.'
Nov 13 23:26:22 my-host sh[2889612]: + msg='Kubelet binary not identified.'
Nov 13 23:26:22 my-host sh[2889612]: + echo 'ERROR: Kubelet binary not identified.'
Nov 13 23:26:22 my-host sh[2889612]: ERROR: Kubelet binary not identified.
Nov 13 23:26:22 my-host sh[2889612]: + exit 1
Nov 13 23:26:22 my-host systemd[1]: kubelet-config-helper.service: Main process exited, code=exited, status=1/FAILURE
Nov 13 23:26:22 my-host systemd[1]: kubelet-config-helper.service: Failed with result 'exit-code'.
Nov 13 23:26:22 my-host systemd[1]: Failed to start Kubelet config service.

It appears that this helper script is unable to determine the correct Kubelet path. I first tried to symlink /usr/bin/kubelet -> /var/lib/k0s/bin/kubelet until I realized this is actually trying to obtain the Kubelet path from systemctl.

However, the Kubelet path is not to be found in systemctl show kubelet (kubelet.zip).

Is there a config option I can use to explicitly point the Sysbox installer to the right Kubelet binary?

[rodnymolina]

@NiklasRosenstein, thanks for reporting this issue.

Your analysis is correct: the failing script is attempting to identify the kubelet binary through systemctl output. This is not always the case, as we support other k8s distros that don't rely on systemd for kubelet's execution (e.g., .rke, rke2, locomotive, etc). However, in K0's case, as we haven't properly tested it yet, we are failing to catch its peculiarities, which explains why our code handles the installation through the common (default) scenario, which is the one where kubelet is managed through systemd.

We will need to make some adjustments to our daemonset-based installer for K0s to be properly supported.

[NiklasRosenstein]

Thanks for the confirmation, @rodnymolina. Is there anything I can do to help? Or is there a workaround I can apply?

I've not been able to locate the source of this helper on GitHub thus far.

[lsoica]

Hit this same issue when installing sysbox into a K3S cluster.

$ systemctl status kubelet-config-helper.service
+ kubelet_bin=
+ '[' -z '' ']'
+ die 'Kubelet binary not identified.'
+ msg='Kubelet binary not identified.'
+ echo 'ERROR: Kubelet binary not identified.'
ERROR: Kubelet binary not identified.
+ exit 1
kubelet-config-helper.service: Main process exited, code=exited, status=1/FAILURE
kubelet-config-helper.service: Failed with result 'exit-code'.
Failed to start Kubelet config service.

With k3s there's no kubelet binary.

[ctalledo]

Thanks @lsoica.

With k3s there's no kubelet binary.

That's correct, and it makes it very difficult for sysbox-deploy-k8s to work with K3s since it needs to reconfigure the kubelet to switch it from containerd -> CRI-O.

[Sharpz7]

#626

I created a discussion to talk about this more widely - support for k0s and k3s would be great.