[apiclient] Created API client, waiting for the control plane to become ready stuck

[jainmnsh]

Is this a request for help?

If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.

What keywords did you search in kubeadm issues before filing this one?

[apiclient] Created API client, waiting for the control plane to become ready stuck, I search through all open /close issues on this subject but I couldnt find on running cluster on 1.7.4 version.

If you have found any duplicates, you should instead reply there and close this page.

If you have not found any duplicates, delete this section and continue on.

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.3", GitCommit:"2c2fe6e8278a5db2d15a013987b53968c743f2a1", GitTreeState:"clean", BuildDate:"2017-08-03T06:43:48Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.3", GitCommit:"2c2fe6e8278a5db2d15a013987b53968c743f2a1", GitTreeState:"clean", BuildDate:"2017-08-03T07:00:21Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
  The connection to the server localhost:8080 was refused - did you specify the right host or port?

• Cloud provider or hardware configuration:

• OS (e.g. from /etc/os-release):
  kubeadm version: &version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.3", GitCommit:"2c2fe6e8278a5db2d15a013987b53968c743f2a1", GitTreeState:"clean", BuildDate:"2017-08-03T06:43:48Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
  [root@node01 ~]# kubectl version
  Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.3", GitCommit:"2c2fe6e8278a5db2d15a013987b53968c743f2a1", GitTreeState:"clean", BuildDate:"2017-08-03T07:00:21Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
  The connection to the server localhost:8080 was refused - did you specify the right host or port?
  [root@node01 ~]# more /etc/os-release
  NAME="CentOS Linux"
  VERSION="7 (Core)"
  ID="centos"
  ID_LIKE="rhel fedora"
  VERSION_ID="7"
  PRETTY_NAME="CentOS Linux 7 (Core)"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:centos:centos:7"
  HOME_URL="https://www.centos.org/"
  BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Kernel (e.g. uname -a):
  Linux node01 3.10.0-514.26.2.el7.x86_64 kubeadm join on slave node fails preflight checks #1 SMP Tue Jul 4 15:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
• Others:

What happened?

when I am running kubeadm init the process stuck at [apiclient] Created API client, waiting for the control plane to become ready

What you expected to happen?

I would like my kubeadm init to initialize the ready as per documentation on below link
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/

How to reproduce it (as minimally and precisely as possible)?

I am following the instruction as per and setup kubeadm, kubectl and updated but having issues
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/

Anything else we need to know?

[rajatchopra]

Can you paste the output of 'kubeadm init' ?
Also check with 'netstat -tulpn | grep 8080' to confirm if anything is really bound to 8080 or not.

[jainmnsh]

Here you go. nothing running on port 8080.

[root@node01 ~]# kubeadm init
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.7.4
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks
[preflight] WARNING: docker version is greater than the most recently validated version. Docker version: 17.07.0-ce-rc3. Max validated version: 1.12
[preflight] WARNING: hostname "node01" could not be reached
[preflight] WARNING: hostname "node01" lookup node01 on 127.0.0.1:53: no such host
[preflight] Starting the kubelet service
[kubeadm] WARNING: starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use --token-ttl 0)
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [node01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.0.0.4]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[apiclient] Created API client, waiting for the control plane to become ready
^C
[root@node01 ~]# netstat -tulpn | grep 8080
[root@node01 ~]# netstat -tulpn | grep 8080
[root@node01 ~]#

[ParagDoke]

I see the same on Ubuntu 16.04 also. Pasting command outputs (hoping to help debug).

# kubeadmversion
kubeadm version: &version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.3", GitCommit:"2c2fe6e8278a5db2d15a013987b53968c743f2a1", GitTreeState:"clean", BuildDate:"2017-08-03T06:43:48Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

# kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.3", GitCommit:"2c2fe6e8278a5db2d15a013987b53968c743f2a1", GitTreeState:"clean", BuildDate:"2017-08-03T07:00:21Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?

# cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.1 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.1 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

# uname -a
Linux kubemaster 4.4.0-57-generic #78-Ubuntu SMP Fri Dec 9 23:50:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

# docker info
Containers: 8
 Running: 8
 Paused: 0
 Stopped: 0
Images: 5
Server Version: 1.12.6
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-57-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.859 GiB
Name: kubemaster
ID: GCI7:ZRR6:UJCF:YKET:XHN2:7B7X:VWZL:PBYL:IXXX:PKSY:DZTW:DRVC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

Additionally, docker containers for the control plane are up.

# docker ps | grep -v pause
CONTAINER ID        IMAGE                                                                                                                            COMMAND                  CREATED             STATUS              PORTS               NAMES
1348b47bee64        gcr.io/google_containers/etcd-amd64@sha256:d83d3545e06fb035db8512e33bd44afb55dea007a3abd7b17742d3ac6d235940                      "etcd --listen-client"   17 hours ago        Up 17 hours                             k8s_etcd_etcd-kubemaster_kube-system_9ef6d25e21bb4befeabe4d0e4f72d1ca_0
784965ba8991        gcr.io/google_containers/kube-controller-manager-amd64@sha256:7e31b7f71a8c1904c8b38c1681666ef551d0598fbbb4142522b05074ae0c9fd1   "kube-controller-mana"   17 hours ago        Up 17 hours                             k8s_kube-controller-manager_kube-controller-manager-kubemaster_kube-system_59b43503aa8bb44e3a50fe29043a342c_0
e246800aa037        gcr.io/google_containers/kube-apiserver-amd64@sha256:f880371b4cee1a810d7caf4c8a2c0b8fa169879545b06a537e4ea6bcdfbbe1f6            "kube-apiserver --req"   17 hours ago        Up 17 hours                             k8s_kube-apiserver_kube-apiserver-kubemaster_kube-system_97c5e455be011e8f2b29e39a0cf9ddbc_0
409084ce6c0f        gcr.io/google_containers/kube-scheduler-amd64@sha256:3712116f370e21938e6a55e0f73dc02a4a6f4830a33304127105ed89451ee527            "kube-scheduler --add"   17 hours ago        Up 17 hours                             k8s_kube-scheduler_kube-scheduler-kubemaster_kube-system_b7014587ed2aff55c7240b2b6a9eb2e8_0

Kubelet shows (the same?) problem:

# journalctl -u kubelet
Aug 22 05:18:51 kubemaster kubelet[6438]: E0822 05:18:51.477831    6438 kubelet_node_status.go:106] Unable to register node "kubemaster" with API server: nodes "kubemaster" is forbidden: node master cannot modify node kubemaster
Aug 22 05:18:54 kubemaster kubelet[6438]: W0822 05:18:54.890864    6438 cni.go:189] Unable to update cni config: No networks found in /etc/cni/net.d
Aug 22 05:18:54 kubemaster kubelet[6438]: E0822 05:18:54.891524    6438 kubelet.go:2136] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Related to kubernetes/kubernetes#43815 ?

[ParagDoke]

CC @mikedanese (who suggested a patch) and @luxas (who summarized the problem at the end).

[jiangpengcheng]

I met the same error on CentOS 7,2.1511 before

the error log looks like below:

# journalctl -eu kubelet
Aug 22 03:42:31 hyperd02 kubelet[26831]: error: failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgoupfs"

just change the cgroup driver in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf, and kubeadm is ok.

hope this will be helpful

[ParagDoke]

Thanks @jiangpengcheng. Maybe a different issue on CentOS then.
For me (on Ubuntu 16.04), using the unstable package worked (I got kubeadm 1.7.4).

[selvik]

@ParagDoke Can you share the steps you used to install kubeadm 1.7.4? I am facing the same issue as you did. Thanks!

[jainmnsh]

@jiangpengcheng my /etc/systemd/system/kubelet.service.d/10-kubeadm.conf file has following entry for cgroup, can you let me know what you change for this value?

Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"

[thomas-oo]

@jainmnsh run docker info | grep Cgroup. That'll tell you what cgroup driver your docker is using, replace that value with systemd in the line Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"

Anyways, I have the same issue but it is not a cgroup driver issue, I suspect it is because I am running behind a proxy. I can still pull the docker images and run them but when I curl to :6443/api/v1/....... I get a 403. The containers are trying to get at the api using the external ip and some proxy settings are not screwing it up?

[jainmnsh]

@thomas-oo thanks for responding. my cgroup is "cgroupfs", I have updated conf with changes, but still can't bypass it. I am not even running behind the proxy. just simple azure VM installed centOS. I didn't built any docker images yet I am just trying to run "kubeadm init" to initiate master machine.

I am new to Kubernetes and can't move forward. appreciate your help in progressing further.

[jiangpengcheng]

@jainmnsh would you please show your output of journalctl -eu kubelet? It may give the detail error messages

[thomas-oo]

@jainmnsh after you've replaced it, I forgot to mention to run systemctl daemon-reload and try to run kubeadm init after. If all goes well, your journalctl -eu kubelet should not be showing the error error: failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgoupfs"

[thomas-oo]

Update on my issue: It was indeed a proxy issue, if anyone else was having the same scenario as myself:

If your kubeadm init is hanging but when you run docker ps and the containers are running.
Check to see that you can access your apiserver at localhost:6443 if so, try at <public ip>:6443.

If the public ip fails, check the logs of your controller-manager by doing docker logs <id of controller-manager>. You should see that all the api requests are to your public ip and they are failing.

In my case, this was a proxy problem because when I tried to curl my apiserver at <public ip>:6443, I'd get a 403 from proxy after connect error.

How I fixed it:
Basically the controller-manager container has to have its NO_PROXY/no_proxy env var to include your public ip.
From a previous project, I had to set some HTTP and HTTPS_PROXY env vars in my docker daemon and I had overlooked them, now I had to add NO_PROXY env variable for my docker daemon by going to /etc/systemd/system/docker.service.d and creating a file http-proxy.conf in there. Then I put
[Service] <MY PREVIOUS HTTP AND HTTPS PROXY SETTINGS> Environment="NO_PROXY=localhost,127.0.0.1,<YOUR PUBLIC IP>"

then do systemctl daemon-reload, and systemctl restart docker. Do docker info and look at No Proxy to confirm that your machine's ip shows up there. Then stop kubeadm init if you still had it running, do kubeadm reset, then do kubeadm init again. Hopefully it works!

BTW: I'm not the best at configuring these proxy settings, if there is a better way to do this, please let me know

[ParagDoke]

@selvik

# cat /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial-unstable main

Notice presence of "-unstable".

Others, sorry for responding with Ubuntu specific steps on a CentOS related thread.

[jamiehannaford]

@jainmnsh After changing docker's cgroup driver to systemd, did you restart both docker and kubelet? If it's still not working, can you paste the apiserver logs? You can see them by looking for the docker container: docker ps -a | grep api

[jainmnsh]

@jamiehannaford it worked after changed cgroup driver and restart both docker and Kubelet. thank you for your help.

[jamiehannaford]

@jainmnsh Awesome. I've created a doc for this requirement, so I'm going to go ahead and close this. If you run into any more problems, please comment and I'll reopen this thread.