Bump kubectl and packages to 1.21

[squaremo]

Kubernetes 1.19 is out --> bump to 1.18. (the packages may .. take some doing)

[kingdonb]

I'd like to understand when this will become a burning issue, where we are behind the 8-ball, so I can be sure this is completed before it is urgent. According to the version skew policy and support calendar, Kubernetes 1.18, 1.19, and 1.20 are currently in support. This leads me to believe this can be really urgent, and should be prioritized.

But I am not sure what degree this version disconnect can actually lead to breaking behavior, I have been upgrading my Kubernetes clusters to the latest version available and I have not yet noticed any degree of issues running the v1 maintenance series of Flux on the later supported versions, with several-versions-earlier client libraries like 1.17 that we still have.

[kingdonb]

So, I've done some research and I understand why this issue is still behind the 8-ball. I'm removing it from the 1.22.0 milestone after understanding that some of our backwards compatibility guarantees may prevent us from upgrading again.

Kubernetes 1.18 client-go was a breaking change that introduced a context.Context argument for all functions. This is innocent enough by itself, but there is a web of dependencies that makes it a problem. One maintainer explained to me that this whole situation was one of the driving reasons to start a rewrite, (since support for Helm v2 was dropped upstream, no possible future had an upgrade path for Flux v1 and Helm Operator that could preserve backwards compatibility forever.)

It all starts with, Helm v2 does not offer support for Kubernetes client newer than 1.16.x – Helm version skew policy describes the earliest and latest Kubernetes version that is formally supported by each Helm version. The first Helm version with support for Kubernetes 1.18.x+ is Helm 3.2.x. There is no Helm 2 support for context.Context. Helm does not offer "forward compatibility" guarantees – eg. in order to receive support for a later version of the Kubernetes API, one must usually upgrade to the next major or minor release of Helm that adds support for it.

To put this in context of Flux and Helm Operator backwards compatibility guarantees: since Flux v1 is in maintenance mode, it is not breaking compatibility with anything as there are no breaking releases allowed in the maintenance mode for Flux v1. Helm Operator supports Helm v2 and v3, as well as documenting a migration path for Helm v2to3 conversion.

Helm Operator (also in maintenance mode) will continue to support both Helm v2 and v3 (at least according to my reading, this is not a new guarantee) until it is sunset. After then, support can be provided only for Helm Controller (which is by then, I would anticipate, a stable release.)

Flux (v1) depends on Helm Operator's codebase because the HelmRelease CRD is on the image automation feature's supported list, eg. Deployment, DaemonSet, StatefulSet, CronJob, HelmRelease can all receive image updates through the fluxcd.io/automated annotation (though other CRDs cannot be updated by Flux v1, HelmRelease can be, and as of now this has been resolved in Flux v2, arbitrary CRDs or K8s API objects can be supported without reaching through and depending on their API or codebase. This is a major breakthrough in Flux v2 that can enable maintainers to offer stronger compatibility guarantees and provide better support into the future.)

In concrete terms, this is what blocks the update: when applying clientgofix changes to Flux v1, everything was going fine until I hit this compiler error.

$ make all
go install ./cmd/fluxctl
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o build/fluxd  -ldflags "-X main.version=master-2dc2680d-wip" ./cmd/fluxd
# github.com/fluxcd/helm-operator/pkg/client/clientset/versioned/typed/helm.fluxcd.io/v1
../../../go/pkg/mod/github.com/fluxcd/helm-operator@v1.2.0/pkg/client/clientset/versioned/typed/helm.fluxcd.io/v1/helmrelease.go:74:5: not enough arguments in call to c.client.Get().Namespace(c.ns).Resource("helmreleases").Name(name).VersionedParams(&options, scheme.ParameterCodec).Do
	have ()
	want (context.Context)
../../../go/pkg/mod/github.com/fluxcd/helm-operator@v1.2.0/pkg/client/clientset/versioned/typed/helm.fluxcd.io/v1/helmrelease.go:91:5: not enough arguments in call to c.client.Get().Namespace(c.ns).Resource("helmreleases").VersionedParams(&opts, scheme.ParameterCodec).Timeout(timeout).Do
	have ()
	want (context.Context)
../../../go/pkg/mod/github.com/fluxcd/helm-operator@v1.2.0/pkg/client/clientset/versioned/typed/helm.fluxcd.io/v1/helmrelease.go:108:8: not enough arguments in call to c.client.Get().Namespace(c.ns).Resource("helmreleases").VersionedParams(&opts, scheme.ParameterCodec).Timeout(timeout).Watch
	have ()
	want (context.Context)
../../../go/pkg/mod/github.com/fluxcd/helm-operator@v1.2.0/pkg/client/clientset/versioned/typed/helm.fluxcd.io/v1/helmrelease.go:118:5: not enough arguments in call to c.client.Post().Namespace(c.ns).Resource("helmreleases").Body(helmRelease).Do
	have ()
	want (context.Context)
../../../go/pkg/mod/github.com/fluxcd/helm-operator@v1.2.0/pkg/client/clientset/versioned/typed/helm.fluxcd.io/v1/helmrelease.go:131:5: not enough arguments in call to c.client.Put().Namespace(c.ns).Resource("helmreleases").Name(helmRelease.ObjectMeta.Name).Body(helmRelease).Do
	have ()
	want (context.Context)
../../../go/pkg/mod/github.com/fluxcd/helm-operator@v1.2.0/pkg/client/clientset/versioned/typed/helm.fluxcd.io/v1/helmrelease.go:147:5: not enough arguments in call to c.client.Put().Namespace(c.ns).Resource("helmreleases").Name(helmRelease.ObjectMeta.Name).SubResource("status").Body(helmRelease).Do
	have ()
	want (context.Context)
../../../go/pkg/mod/github.com/fluxcd/helm-operator@v1.2.0/pkg/client/clientset/versioned/typed/helm.fluxcd.io/v1/helmrelease.go:159:5: not enough arguments in call to c.client.Delete().Namespace(c.ns).Resource("helmreleases").Name(name).Body(options).Do
	have ()
	want (context.Context)
../../../go/pkg/mod/github.com/fluxcd/helm-operator@v1.2.0/pkg/client/clientset/versioned/typed/helm.fluxcd.io/v1/helmrelease.go:175:5: not enough arguments in call to c.client.Delete().Namespace(c.ns).Resource("helmreleases").VersionedParams(&listOptions, scheme.ParameterCodec).Timeout(timeout).Body(options).Do
	have ()
	want (context.Context)
../../../go/pkg/mod/github.com/fluxcd/helm-operator@v1.2.0/pkg/client/clientset/versioned/typed/helm.fluxcd.io/v1/helmrelease.go:188:5: not enough arguments in call to c.client.Patch(pt).Namespace(c.ns).Resource("helmreleases").SubResource(subresources...).Name(name).Body(data).Do
	have ()
	want (context.Context)

That is an error that evidently can't be resolved, since Helm Operator's client-go version 0.17.x doesn't provide the methods with the context.Context parameter, and Helm v2 can't be upgraded past this version of the client library.

Since Helm v2 cannot support K8s client-go 0.18.x+ at all, this apparently can not resolve without a breaking change. So, Flux v1 cannot upgrade past Helm Operator's transitive dependency on K8s client 0.17.x. This will unfortunately be a blocker for supporting new Kubernetes API features in Flux v1. We can not provide better compatibility guarantees than our upstreams.

The GitOps Toolkit's replacement for Helm Operator, "Helm Controller," only supports Helm v3, simplifying things drastically into the future. The unfortunate thing for our support guarantees is that... since Kubernetes 1.17.x is already out of support, if you are on this version of Kubernetes, you are already out of maintenance releases from Kubernetes upstream. The K8s ecosystem moves quickly and staying on an older version of Kubernetes that cannot receive support in the form of patches or security updates, is not advisable.

The good news is, if I understand this README (kubernetes/client-go) then, unlike Helm v2, the Kubernetes client actually does provide some guarantees about compatibility both forward and backward. New releases of Kubernetes will take instructions from old versions of the client, at least within the range described here (at this time the support matrix shows +- back to 1.15.x, forward to 1.20.x)

It explains:

Since Kubernetes is backwards compatible with clients, older client-go versions will work with many different Kubernetes cluster versions.

We will backport bugfixes--but not new features--into older versions of client-go.

So, the support status of Helm Operator with Helm v2 is that, it can no longer be supported on any currently in-support series of Kubernetes release based on upstream (Helm) support. It would be necessary to start dropping features from Flux in order to upgrade past this point. At some future date it may become necessary to drop HelmRelease image automation from Flux v1 in order to maintain support for newer Kubernetes releases; I don't think that will happen, Flux v1 committed to maintaining backwards compatibility until Flux v1 maintenance is over.

That EOL for Flux v1 will happen on a future date that has yet to be announced. The guideline I've heard thrown around is "at least 6 months after Flux v2 goes GA, not less than."

I will leave the issue open since it represents a potential obstacle to providing support for Flux in the future, and as a record of what was learned by trying to make this upgrade. (As I mentioned, I removed this from the milestone 1.22.0 that is planned to release soon, with envsubst support in Flux v1.)

[kingdonb]

Related: fluxcd/helm-operator#599

[kingdonb]

I think this can probably be resolved now, since apimachinery has been upgraded so far already in the Flux 1.23.0 release, and Helm Operator 1.4.0.

I see a handful of places where code changes are needed to make the upgrade. I opened #3515 to give this a try!

[kingdonb]

#3537 should merge soon (instead of #3515), and we will be current on all Kubernetes client versions as of K8s 1.22.0 released just two weeks ago, as well as many other module updates.

Closing this, as the issue is being tracked elsewhere 👍