Add envsubst package to docker image

[stevehipwell]

Describe the feature
It would be really useful to have envsubst available when generating manifests.

What would the new user story look like?
The ability to call envsubst from .flux.yaml.

Expected behavior
Easy to generate manifests with environment expansion.

[stevehipwell]

I'm happy to add a PR for this as I think all that would be required would be to install the envsubst package during the Docker build?

[stevehipwell]

Could someone let me know if you're accepting PRs?

[zemaj]

I think it would be incredibly useful for the base flux image to support envsubst. It's the missing piece for generating manifests in multi stage repositories.

kustomize is specifically designed to not support arbitrary variable expansion because it's typically done after the build. https://kubernetes-sigs.github.io/kustomize/faq/eschewedfeatures/#unstructured-edits

Unless envsubst is included in the flux image you'd have to create and maintain a new base image just to include envsubst. When using eksctl to create clusters, this isn't an option.

Spent a day trying to figure out a way around this, the only option I could come up with is using sed and manually entering all the variables to replace.

version: 1
patchUpdated:
  generators:
    - command: source .env && (kustomize build . | sed -e "s/\${VARIABLE_1}/$VARIABLE_1/g" | sed -e "s/\${VARIABLE_2}/$VARIABLE_2/g" | sed -e "s/\${VARIABLE_3}/$VARIABLE_3/g")
  patchFile: flux-patch.yaml

[stefanprodan]

@zemaj can you please comment on this RFC for Flux2, I think it should cover your use-case fluxcd/kustomize-controller#253

[kingdonb]

@stevehipwell Is this still needed in Flux v1, it looks like there will be support for this feature added soon/already in Flux v2

[stevehipwell]

@kingdonb we'll still be using v1 for a while so this would be great if it could be merged

[kingdonb]

Alrighty, let's try and get it ready for the next milestone then!

Since it is a new feature it may get pushed to 1.22.0 (or we may skip straight to that minor release, if it turns out to be ready in time.) I'm not expecting to get another release out for close to a month, sorry for the tremendous delay, but there is a large backlog of issues I've been working through and I want to have made a serious dent before promoting too many releases.

It is a pretty straightforward and small change, and this is a lot of ceremony, I hope you haven't been blocked from using this because it isn't merged. I can imagine you've built this for yourself and are already using it (hope you're not waiting on us?)

[stevehipwell]

@kingdonb we assumed this would take no time to get approved and built so we did a limited sed workaround and lived with the pain. I'm going to be afk for a while so if there are any rebase changes required on my PR could one of the maintainers do this?

[kingdonb]

I can handle rebasing now no problem, I have been given write access. Now that I know the release process myself I can probably expedite this and get a new release out sooner as well. Sorry for the delays. It is a simple change and shouldn't be any reason it takes a whole month to release. I can't approve it by myself but I can get it in front of the maintainers this coming week, and we'll have a better idea if it can be released quickly after that. It would be good to have some assurance from you first that it actually solves the issue for you, is why I ask if you have been able to use (and test) it successfully on your end.

[stevehipwell]

@kingdonb I can confirm that it would indeed solve our issue. I've tested it locally but didn't want to publish and maintain an image for something this simple.

[kingdonb]

We are going to skip straight ahead to 1.22.0 since this is a new feature, and that means a minor version bump according to semver. This is a priority feature for users of Flux v1 who are trying to use envsubst to replace the old manifest generation feature for lightweight uses that can be covered by envsubst, to be able to migrate to Flux v2 and provide continuity.

The only other priority for the release is #3378 at this point. Since we are several minor versions behind Kubernetes upstream packages, catching up with that release stream is also a priority now. (Everything else in the 1.21.3 milestone today, will get left behind/booted to a later release, unless between now and next release something else turns out to be priority too...)