Backend kubernetes and terraform version 1.1.0 and above

[kubovdm]

Hi. I have container with terraform version 1.0.11 and setuped terraform backend kubernetes. I use ENV vars KUBE_HOST,KUBE_CLUSTER_CA_CERT_DATA,KUBE_TOKEN inside containers. If i try upgrade terraform version i have error Error: Failed to initialize kubernetes configuration: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
Such a feeling backend ignore ENVs.

Terraform Version

Terraform v1.0.11
on linux_amd64

Terraform Configuration Files

provider "kubernetes" {
  host                   = data.vault_generic_secret.k8s_auth.data.HOST
  cluster_ca_certificate = data.vault_generic_secret.k8s_auth.data.CLUSTER_CA_CERTIFICATE
  token                  = data.vault_generic_secret.k8s_auth.data.k8s_sa_token
  load_config_file       = false
}

terraform {
  # The configuration for this backend will be filled in by Terragrunt
  backend "kubernetes" {
    secret_suffix    = "suffix"
    namespace = "tekton-pipelines"
    load_config_file = false
  }
    flux = {
      source  = "registry.terraform.io/fluxcd/flux"
      version = ">= 0.0.13"
    }
  }
}

Debug Output

2022-06-19T14:25:44.487Z [INFO]  Terraform version: 1.1.2
2022-06-19T14:25:44.488Z [INFO]  Go runtime version: go1.17.2
2022-06-19T14:25:44.489Z [INFO]  CLI args: []string{"terraform", "plan", "-var-file=/home/runner/workdir/k8s/config.tfvars.json"}
2022-06-19T14:25:44.489Z [TRACE] Stdout is not a terminal
2022-06-19T14:25:44.489Z [TRACE] Stderr is not a terminal
2022-06-19T14:25:44.489Z [TRACE] Stdin is a terminal
2022-06-19T14:25:44.490Z [DEBUG] Attempting to open CLI config file: /home/fake/.terraformrc
2022-06-19T14:25:44.490Z [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2022-06-19T14:25:44.494Z [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2022-06-19T14:25:44.494Z [DEBUG] ignoring non-existing provider search directory /home/fake/.terraform.d/plugins
2022-06-19T14:25:44.494Z [DEBUG] ignoring non-existing provider search directory /home/fake/.local/share/terraform/plugins
2022-06-19T14:25:44.494Z [DEBUG] ignoring non-existing provider search directory /usr/local/share/terraform/plugins
2022-06-19T14:25:44.494Z [DEBUG] ignoring non-existing provider search directory /usr/share/terraform/plugins
2022-06-19T14:25:44.495Z [INFO]  CLI command args: []string{"plan", "-var-file=/home/runner/workdir/k8s/config.tfvars.json"}
2022-06-19T14:25:44.539Z [TRACE] Meta.Backend: built configuration for "kubernetes" backend with hash value 2084429565
2022-06-19T14:25:44.547Z [TRACE] Preserving existing state lineage "df078657-3ab2-457f-ef26-136224aea6e6"
2022-06-19T14:25:44.549Z [TRACE] Preserving existing state lineage "df078657-3ab2-457f-ef26-136224aea6e6"
2022-06-19T14:25:44.552Z [TRACE] Meta.Backend: working directory was previously initialized for "kubernetes" backend
2022-06-19T14:25:44.554Z [TRACE] backendConfigNeedsMigration: given configuration matches cached configuration, so no migration is required
2022-06-19T14:25:44.554Z [TRACE] Meta.Backend: using already-initialized "kubernetes" backend configuration
╷
│ Error: Failed to initialize kubernetes configuration: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
│ 
│ 
╵
ERRO[0000] 1 error occurred:
        * exit status 1

[jbardin]

HI @kubovdm,

Your comment suggests that you are using Terragrunt to complete the backend configuration, but the error suggest that is not being done correctly. There is no config attribute being set in the backend config in your example, nor is there a -backend-config flag being set on the command line. Can you show an example of what configuration is being passed to the backend?

Thanks!

[kubovdm]

Hi @jbardin,
You are right. I use terragrunt to generate providers.tf and this part of code is from this file, and it is work on terraform version 1.0.11, and stop working after 1.1.0. In the example configuration https://www.terraform.io/language/settings/backends/kubernetes i don't need use config part, or something was changed?

[jbardin]

I don't see any changes in the Terraform backend for v1.1. It's possible that the supporting libraries made an update which affects the parsing of the configuration. Have you tried what the error suggests by setting KUBERNETES_MASTER?

[kubovdm]

Add KUBERNETES_MASTER with copy value from KUBE_HOST and have same error. Such feeling that env are ignored.

[kubovdm]

I think backend ignore env and try to search .kube/config, but why...

[kubovdm]

Find this in documentation:
Note: In Terraform versions prior to 1.1.0, backends were also classified as being 'standard' or 'enhanced', where the latter term referred to the ability of the remote backend to store state and perform Terraform operations. This classification has been removed, clarifying the primary purpose of backends. Refer to Using Terraform Cloud for details about how to store state, execute remote operations, and use Terraform Cloud directly from Terraform.
And try to understand what i need to do.

[redzioch]

It looks like same bug: #31272

Data source values are not accessible during apply.

[carhartl]

I'm running into the same issue, attempting to set up the backend with a service account instead of kubeconfig. Neither attempting to rely on the respective env vars to be sourced, nor providing explicit values work, resulting in the "Failed to initialize kubernetes configuration: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable" error.

terraform {
  backend "kubernetes" {
    secret_suffix = "foo"
    host          = "..."
    token         = "..."
    insecure      = true
  }
}

[carhartl]

I was able to work around it using a dummy kubeconfig:

# ./kubeconfig
kind: Config
apiVersion: v1
clusters:
  - cluster:
      server: https://localhost
    name: dummy
contexts:
  - context:
      cluster: dummy
    name: dummy
current-context: dummy
preferences: {}
users: []

terraform {
  backend "kubernetes" {
    config_path = "./kubeconfig"
  }
}

[sarg3nt]

Thanks @carhartl , that worked for me too.
Kind of a bummer it has to be done but at least it works!

[sarg3nt]

On a related note, the docs say.

For most use cases either in_cluster_config, config_path, or config_paths will need to be set. If all flags are set the configuration at config_path will be used.

So I tried using in_cluster_config but it then throws
Error: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined
Which are two more env variables that are not in the docs.
It's like we are working with a new version of the backend the docs don't apply to. Or it's just super super broken.

[eszense]

I think this is due to a bug here, when the required config e.g. token/client_certificate is only provided as parameter to the backend with no kubeconfig file given.

configure() calls getInitialConfig() before applying the token, client_certificate etc
But getInitialConfig() includes validation of the current config by calling tryLoadingConfigFile() and clientcmd.NewNonInteractiveDeferredLoadingClientConfig() which throws error on empty config.

Compare this with configure.go in the kubernetes provider, where the token is provided as part to overrides argument of clientcmd.NewNonInteractiveDeferredLoadingClientConfig(), allowing it to validate as non-empty config.

terraform/internal/backend/remote-state/kubernetes/backend.go

Lines 228 to 267 in 2864351

	func (b *Backend) configure(ctx context.Context) error {
	if b.config != nil {
	return nil
	}
	// Grab the resource data
	data := schema.FromContextBackendConfig(ctx)
	cfg, err := getInitialConfig(data)
	if err != nil {
	return err
	}
	// Overriding with static configuration
	cfg.UserAgent = fmt.Sprintf("HashiCorp/1.0 Terraform/%s", version.String())
	if v, ok := data.GetOk("host"); ok {
	cfg.Host = v.(string)
	}
	if v, ok := data.GetOk("username"); ok {
	cfg.Username = v.(string)
	}
	if v, ok := data.GetOk("password"); ok {
	cfg.Password = v.(string)
	}
	if v, ok := data.GetOk("insecure"); ok {
	cfg.Insecure = v.(bool)
	}
	if v, ok := data.GetOk("cluster_ca_certificate"); ok {
	cfg.CAData = bytes.NewBufferString(v.(string)).Bytes()
	}
	if v, ok := data.GetOk("client_certificate"); ok {
	cfg.CertData = bytes.NewBufferString(v.(string)).Bytes()
	}
	if v, ok := data.GetOk("client_key"); ok {
	cfg.KeyData = bytes.NewBufferString(v.(string)).Bytes()
	}
	if v, ok := data.GetOk("token"); ok {
	cfg.BearerToken = v.(string)
	}