Support upgrading an external, static-pod etcd cluster using kubeadm upgrade

[johnharris85]

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: FEATURE REQUEST

Kubeadm currently supports upgrading Kubernetes clusters with either local (stacked) or external etcd clusters. In the latter case, kubeadm detects that etcd is external and the onus is on the operator to pre-upgrade etcd. In the former case, kubeadm exercises specific etcd upgrade logic (involving the backup of etcd data, and some sanity checking).

There is a third use-case that kubeadm upgrade doesn't currently handle (afaics), and that's if etcd is external to the control-plane, but was setup using the local etcd functionality of kubeadm. In this case, the kubeadm upgrade (run on those etcd nodes) fails because of some erroneous assumptions:

• That the node has access / knows about a cluster / APIServer (not necessary in this standalone scenario where kubelet doesn't register to a cluster and just being used as a runner for static pods)
• That the etcd endpoints are the same as the APIServer endpoints (in this case, they're not as it's standalone)
• That other component manifests will be present (checking during pre-flight)

This use case is currently supported by simply re-running kubeadm init phase etcd local (with the new kubeadm version) on the standalone etcd nodes as this will produce a new static manifest (with the new version) but still mount the existing data and certs directories of the previous version. This flow however does not exercise the specific upgrade / rollback logic that kubeadm contains for stacked etcd upgrades.

This request is for kubeadm upgrade to recognize and support this scenario so that the same code path / functionality can be used, and to make the upgrade in this case more intuitive.

Since the goal of kubeadm is to make the instantiation and upgrade of kubernetes simpler. Any of the commands like kubeadm upgrade node --etcd-upgrade should make no assumptions about an existing kubernetes control plane. etcd is generally managed before the control plane is started.

[neolit123]

hi, external etcd upgrades are out of scope for kubeadm, because there are a number of ways to provision them. some use kubelet + static pods, some use systemd, some don't even use an init system.

there is currently a tool in the works called http://github.com/kubernetes-sigs/etcdadm and it has per node etcd upgrades in scope. but to my knowledge even etcdadm will not cover all possibilities.

one day kubeadm will start using etcdadm as a backend but external etcd cluster upgrades will still be out of scope as far as i can see it.

i'm not going to close this issue, but leave it for @fabriziopandini and @dlipovetsky to comment.

[johnharris85]

Thanks @neolit123, broadly upgrading external etcd being out of scope makes sense for sure, but in this case where it's kubeadm that been used to stand up the etcd cluster as static pods, it should be able to upgrade it also. I chatted briefly with the etcdadm folks on Monday but right now that tool doesn't support etcd as static pods. I have opened an issue on that over there also.

I agree / think long term all of this should live in etcdadm, but in my opinion kubeadm should be able to upgrade an etcd cluster it created (regardless of whether it's stacked or separate).

Edit: to clarify, I'm talking about the case where kubeadm upgrade would be run on the etcd nodes, so etcd is local there, rather than external, but has no knowledge of a cluster.

[neolit123]

This request is for kubeadm upgrade to recognize and support this scenario so that the same code path / functionality can be used, and to make the upgrade in this case more intuitive.

this might be possible once kubeadm upgrade apply phases are supported.
#1318

This flow however does not exercise the specific upgrade / rollback logic that kubeadm contains for stacked etcd upgrades.

we might have to tolerate this as a trade off until etcdadm/kubeadm-upgrade-apply-phases are ready.

[fabriziopandini]

@johnharris85
AFAIK, for SCL the target solution for etcd management (including upgrades) should be etcdadmin, not kubeadm. So:

• I gave +1 to Support etcd lifecycle (deploy / upgrade) as static pods kubernetes-retired/etcdadm#117 (thanks for opening this issue @johnharris85); I will try to sync to the etcdadmin team as well
• I updated kubeadm upgrade apply phases #1318 to take track of the "allow atomic upgrade of etcd" requirement, which is something I consider

@neolit123 I'm +1 for closing this issue considering all the actionable items tracked, but let's way till the next backlog grooming

[neolit123]

missed the /close here.
we should delegate to etcdadm in the long term and not use the kubeadm upgrade --config pattern.

[shahaf600]

Any way to upgrade etcd with this situation? anyway? because me and my team are in it too

[neolit123]

@shahaf600 if you have external etcd where the server instances are managed by kubelet as static pods, you could go on each etcd host and:

• (optional) call kubeadm alpha certs renew to renew the etcd certificates (those expire after one year)
  this should work as long as you have /etc/kubernetes/pki/etcd/ca.{crt|key} on the nodes.
• modify the static pod manifest /etc/kubernetes/manifests/etcd.yaml to point to a new version of the etcd image.

this will result in the etcd server instance on the etcd host being restarted, make sure you do this one at a time and before doing it backup /var/lib/etcd.

[shahaf600]

@neolit123 Works like a charm! thank you