Kubeadm for Windows KEP #994

ksubrmnn · 2019-04-24T18:04:08Z

KEP PR for Issue #995

ksubrmnn · 2019-04-24T18:08:59Z

/assign @timothysc

neolit123 · 2019-04-24T18:22:25Z

@kubernetes/sig-cluster-lifecycle-pr-reviews

neolit123 · 2019-04-24T18:23:27Z

/assign @michmike @fabriziopandini @rosti

rosti

Thanks for this proposal!

The KEP in its shape and form is looking like imposing a wrapper script and Flannel in the long run. These are both unacceptable as a long term solution, but I can see the point of their use in a few releases.

I think, that the long term goal of this KEP should be to have kubeadm join and reset be implemented on Windows with as similar and as simple UX as on Linux.
For that matter we'll have to employ privileged containers on Windows, get kube-proxy & CNI plugins to run in pods again, and ideally, get rid of the wrapper script.

Of course, this rosy dream of mine relies on getting privileged Windows containers.
@PatrickLang is there any possibility of getting privileged containers on Windows in the future?

keps/sig-cluster-lifecycle/kubeadm/20190424-kubeadm-for-windows.md

benmoss · 2019-04-25T13:37:28Z

Of course, this rosy dream of mine relies on getting privileged Windows containers.

I have a questionably acceptable idea to workaround this, but basically a RPC server running on the host that gets mounted into the container via a named pipe. It obviously is still less ideal than if privileged containers could be added to Windows, but assuming we can get a general-enough RPC API, we could then be in the position of distributing/maintaining the provisioning code via pods.

neolit123 · 2019-04-25T13:53:55Z

@benmoss

Of course, this rosy dream of mine relies on getting privileged Windows containers.

I have a questionably acceptable idea to workaround this, but basically a RPC server running on the host that gets mounted into the container via a named pipe. It obviously is still less ideal than if privileged containers could be added to Windows, but assuming we can get a general-enough RPC API, we could then be in the position of distributing/maintaining the provisioning code via pods.

we need to evaluate the roadmap of privilege containers on Windows.
the question here is when are they coming, and if hey are not coming anytime soon we can start thinking about alternatives.

the RPC idea seems like something that can be proposed in a document, but workarounds may still block the Beta. for this KEP it seems that we might want to hold onto the Windows services idea.

keps/sig-cluster-lifecycle/kubeadm/20190424-kubeadm-for-windows.md

timothysc

So I stopped reviewing part way through b/c imo this effort should span releases. I don't want to force this through given other higher priority efforts. I'd much rather us refactor properly.

keps/sig-cluster-lifecycle/kubeadm/20190424-kubeadm-for-windows.md

timothysc · 2019-04-26T14:18:24Z

keps/sig-cluster-lifecycle/kubeadm/20190424-kubeadm-for-windows.md

+
+To support Windows specific flags for the kubelet, there is a requirement to split kubeadm’s app/phases/kubelet/flags.go files into two:
+* app/phases/kubelet/flags_windows.go
+* app/phases/kubelet/flags_linux.go


I'd like work from the componentconfig folks.

could you elaborate on your ideas?
this could end up being something quite isolated to kubeadm -> kubelet.
if os == "windows" { use flagset A } else { use flag set B }

You could build meta-data in apis ~= +omitempty

At the current state of kubelet's component config (which is at v1beta1), there are a few command line flags that don't have a corresponding field in the config. Those are:

dockershim related flags

container-runtime && container-runtime-endpoint flags

register-with-taints flag

hostname-override

Only the resolv-conf flag has a representation in the component config. However, we would have to patch it, after fetching the config from the config map, for the local machine setting to take place.

timothysc · 2019-04-26T14:31:30Z

keps/sig-cluster-lifecycle/kubeadm/20190424-kubeadm-for-windows.md

+
+Kubeadm makes a number of non-portable assumptions about paths. E.g. “/etc/kubernetes” is a hardcoded path in kubeadm.
+
+We need to evaluate the kubeadm codebase for such instances of non-portable paths - CRI sockets, Cert paths, etc. Such paths need to be defaulted properly in the kubeadm configuration API.


IMO this should span several releases and be done judiciously, I want us todo this wisely and not in a rush.

timothysc · 2019-04-26T16:41:38Z

/hold
Please come to the next kubeadm office hours.

ksubrmnn · 2019-04-26T17:36:46Z

Of course, this rosy dream of mine relies on getting privileged Windows containers.

I have a questionably acceptable idea to workaround this, but basically a RPC server running on the host that gets mounted into the container via a named pipe. It obviously is still less ideal than if privileged containers could be added to Windows, but assuming we can get a general-enough RPC API, we could then be in the position of distributing/maintaining the provisioning code via pods.

@benmoss @neolit123

This is a feasible idea, and I completed a POC for this a month ago. It works. I think we can propose both ideas in the doc, and I can advocate for both. This is a more immediate option than waiting on privileged containers, but I think the community should discuss.

PatrickLang · 2019-04-29T22:51:22Z

There's a lot of circling around "privileged containers on Windows". There are some Windows engineers looking into this, but I don't have a clear answer on whether some form of privileged containers could be made to work on Windows Server 2019, or if it would take a new OS version. I'm trying to get them to write up a public proposal but don't expect anything for at least a few weeks.

rosti · 2019-04-30T09:24:07Z

Thanks for the clarification @PatrickLang !
With or without privileged containers on Windows, there is a bunch of stuff that needs to be done either way to get kubeadm running on Windows - CRI detection, kubelet drop-in params, correct handling of Windows paths, etc.
Therefore, I think, that the focus of the initial effort should be on those grounds.

keps/sig-cluster-lifecycle/kubeadm/20190424-kubeadm-for-windows.md

neolit123 · 2019-05-02T10:45:10Z

keps/sig-cluster-lifecycle/kubeadm/20190424-kubeadm-for-windows.md

+
+To support Windows specific flags for the kubelet, there is a requirement to split kubeadm’s app/phases/kubelet/flags.go files into two:
+* app/phases/kubelet/flags_windows.go
+* app/phases/kubelet/flags_linux.go


could you elaborate on your ideas?
this could end up being something quite isolated to kubeadm -> kubelet.
if os == "windows" { use flagset A } else { use flag set B }

keps/sig-cluster-lifecycle/kubeadm/20190424-kubeadm-for-windows.md

timothysc · 2019-05-02T14:46:45Z

poke me on slack after it's been updated to include the content changes from yesterdays review.

timothysc

/approve
/hold

Please address the comments and I'll let @neolit123 do the final lgtm.

keps/sig-cluster-lifecycle/kubeadm/20190424-kubeadm-for-windows.md

timothysc · 2019-05-02T20:28:42Z

keps/sig-cluster-lifecycle/kubeadm/20190424-kubeadm-for-windows.md

+
+To support Windows specific flags for the kubelet, there is a requirement to split kubeadm’s app/phases/kubelet/flags.go files into two:
+* app/phases/kubelet/flags_windows.go
+* app/phases/kubelet/flags_linux.go


You could build meta-data in apis ~= +omitempty

keps/sig-cluster-lifecycle/kubeadm/20190424-kubeadm-for-windows.md

timothysc · 2019-05-02T20:31:49Z

keps/sig-cluster-lifecycle/kubeadm/20190424-kubeadm-for-windows.md

+
+This proposal plans for FlannelD as the default option. Currently, FlannelD has to be started before the kube-proxy Windows service is started. FlannelD creates an HNS network on the Windows host, and kube-proxy will crash if it cannot find the network. This should be fixed in the scope of this project so that kube-proxy will wait until the network comes up. Therefore, kube proxy can be started at any time. 
+
+However, if FlannelD is deployed in VXLAN (Overlay) mode, then we need to rewrite the KubeProxyConfiguration with the correct Overlay specific values, and kube-proxy will need to read this config again.


You're going to need to specify the default path settings that flow through the kubelet config as well here.

fabriziopandini

Few minors,
At first sight, this requires a considerable amount of work, so let's get this started!

keps/sig-cluster-lifecycle/kubeadm/20190424-kubeadm-for-windows.md

rosti

I am fine with how things are ATM. It's clear that this is still provisional and would require a lot of work done before things are set in stone.
However, let's be mindful, that we should update the KEP when the actual implementation deviates from initial designs here.

keps/sig-cluster-lifecycle/kubeadm/20190424-kubeadm-for-windows.md

rosti · 2019-05-03T09:16:45Z

keps/sig-cluster-lifecycle/kubeadm/20190424-kubeadm-for-windows.md

+
+To support Windows specific flags for the kubelet, there is a requirement to split kubeadm’s app/phases/kubelet/flags.go files into two:
+* app/phases/kubelet/flags_windows.go
+* app/phases/kubelet/flags_linux.go


At the current state of kubelet's component config (which is at v1beta1), there are a few command line flags that don't have a corresponding field in the config. Those are:

dockershim related flags

container-runtime && container-runtime-endpoint flags

register-with-taints flag

hostname-override

Only the resolv-conf flag has a representation in the component config. However, we would have to patch it, after fetching the config from the config map, for the local machine setting to take place.

keps/sig-cluster-lifecycle/kubeadm/20190424-kubeadm-for-windows.md

rosti · 2019-05-03T09:42:23Z

keps/sig-cluster-lifecycle/kubeadm/20190424-kubeadm-for-windows.md

+* Installing the Container Runtime (e.g. Docker or containerd)
+* Implement kubeadm init for Windows
+* Implement kubeadm join --control-plane for Windows (at this time)
+* Supporting upgrades using kubeadm upgrade for Windows (to be revisited for Beta)


Going through the code, I think, that this would be quite easy to do for a worker node. The only kubeadm portion here that need revising is kubeadm upgrade node config. It has a tiny bit of OS specific code, that is shared with join and, thus, it would have been taken care of with the kubeadm join porting effort.

rosti

Thanks for the updates and squash @ksubrmnn !
Let's get this in.
/lgtm
/hold cancel

k8s-ci-robot · 2019-05-03T19:14:36Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ksubrmnn, rosti, timothysc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~keps/sig-cluster-lifecycle/OWNERS~~ [timothysc]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Apr 24, 2019

k8s-ci-robot requested review from luxas and roberthbailey April 24, 2019 18:04

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. labels Apr 24, 2019

k8s-ci-robot assigned timothysc Apr 24, 2019

k8s-ci-robot assigned fabriziopandini, michmike and rosti Apr 24, 2019

ksubrmnn mentioned this pull request Apr 24, 2019

Kubeadm for Windows #995

Open

neolit123 mentioned this pull request Apr 24, 2019

tracking issue for Windows support kubernetes/kubeadm#1393

Open

17 tasks

rosti reviewed Apr 25, 2019

View reviewed changes

benmoss reviewed Apr 25, 2019

View reviewed changes

keps/sig-cluster-lifecycle/kubeadm/20190424-kubeadm-for-windows.md Show resolved Hide resolved

benmoss reviewed Apr 25, 2019

View reviewed changes

keps/sig-cluster-lifecycle/kubeadm/20190424-kubeadm-for-windows.md Outdated Show resolved Hide resolved

timothysc reviewed Apr 26, 2019

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 26, 2019

neolit123 reviewed May 2, 2019

View reviewed changes

timothysc approved these changes May 2, 2019

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 2, 2019

fabriziopandini reviewed May 2, 2019

View reviewed changes

rosti reviewed May 3, 2019

View reviewed changes

Kubeadm for Windows KEP

bd3236e

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels May 3, 2019

rosti reviewed May 3, 2019

View reviewed changes

k8s-ci-robot merged commit 557352e into kubernetes:master May 3, 2019

daschott mentioned this pull request May 13, 2019

Support windows kubernetes-sigs/kubespray#2889

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kubeadm for Windows KEP #994

Kubeadm for Windows KEP #994

ksubrmnn commented Apr 24, 2019 •

edited

Loading

ksubrmnn commented Apr 24, 2019

neolit123 commented Apr 24, 2019

neolit123 commented Apr 24, 2019

rosti left a comment

benmoss commented Apr 25, 2019 •

edited

Loading

neolit123 commented Apr 25, 2019

timothysc left a comment

timothysc Apr 26, 2019

neolit123 May 2, 2019

timothysc May 2, 2019

rosti May 3, 2019

timothysc Apr 26, 2019

timothysc commented Apr 26, 2019 •

edited

Loading

ksubrmnn commented Apr 26, 2019

PatrickLang commented Apr 29, 2019

rosti commented Apr 30, 2019

neolit123 May 2, 2019

timothysc commented May 2, 2019

timothysc left a comment

timothysc May 2, 2019

timothysc May 2, 2019

fabriziopandini left a comment

rosti left a comment

rosti May 3, 2019

rosti May 3, 2019

rosti left a comment

k8s-ci-robot commented May 3, 2019


		Kubeadm makes a number of non-portable assumptions about paths. E.g. “/etc/kubernetes” is a hardcoded path in kubeadm.

		We need to evaluate the kubeadm codebase for such instances of non-portable paths - CRI sockets, Cert paths, etc. Such paths need to be defaulted properly in the kubeadm configuration API.


		This proposal plans for FlannelD as the default option. Currently, FlannelD has to be started before the kube-proxy Windows service is started. FlannelD creates an HNS network on the Windows host, and kube-proxy will crash if it cannot find the network. This should be fixed in the scope of this project so that kube-proxy will wait until the network comes up. Therefore, kube proxy can be started at any time.

		However, if FlannelD is deployed in VXLAN (Overlay) mode, then we need to rewrite the KubeProxyConfiguration with the correct Overlay specific values, and kube-proxy will need to read this config again.

Kubeadm for Windows KEP #994

Kubeadm for Windows KEP #994

Conversation

ksubrmnn commented Apr 24, 2019 • edited Loading

ksubrmnn commented Apr 24, 2019

neolit123 commented Apr 24, 2019

neolit123 commented Apr 24, 2019

rosti left a comment

Choose a reason for hiding this comment

benmoss commented Apr 25, 2019 • edited Loading

neolit123 commented Apr 25, 2019

timothysc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timothysc commented Apr 26, 2019 • edited Loading

ksubrmnn commented Apr 26, 2019

PatrickLang commented Apr 29, 2019

rosti commented Apr 30, 2019

Choose a reason for hiding this comment

timothysc commented May 2, 2019

timothysc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fabriziopandini left a comment

Choose a reason for hiding this comment

rosti left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rosti left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented May 3, 2019

ksubrmnn commented Apr 24, 2019 •

edited

Loading

benmoss commented Apr 25, 2019 •

edited

Loading

timothysc commented Apr 26, 2019 •

edited

Loading