terrakube ignores provider configuration block during terraform execution

[requiming]

Bug description 🐞

After configuration of first project and state uploading, i started terrakube job in created workspace, added necesarry ENV variables from site24x7 except one which was included in provider configuration block, and after JOB failed i found such error below, unfortunaly not able fix by changing terraform code.

Steps to reproduce

1. Create new workspace
2. Configure git, ssh_key, workspace subfolder, terraform version
3. Add environment variables for site24x7 provider
4. Run job
5. I see error in the plan

Expected behavior

locally terraform plan works without error

Example repository

terraform.tf

terraform {
  required_version = "~> 1.3.9"
  required_providers {
    site24x7 = {
      source  = "site24x7/site24x7"
    }
  }
  cloud {
    hostname = "terrakube-api.domain.lan"
    organization = "infrastructure"
    workspaces {
      name = "site24x7"
    }
  }
}

providers.tf

provider "site24x7" {
  data_center = "US"
}

Anything else?

Downloading archive from url in main page of workspace shows content of providers.tf the same as in repository or locally
output from terrakube

---

Initializing Terrakube Job 18 Step cf092d77-a0a4-4164-a518-1e0f97b8c193
Running Terraform 1.3.9

---

Running Terraform Init:

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...

• Finding latest version of site24x7/site24x7...
• Installing site24x7/site24x7 v1.0.77...
• Installed site24x7/site24x7 v1.0.77 (signed by a HashiCorp partner, key ID DACE8029C0095571)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

---

Running Terraform PLAN

---

���
��� Error: Missing required argument
���
��� The argument "data_center" is required, but was not set.
���

[alfespa17]

Hello @requiming

I tried your example and I dont see any issue:

main.tf.

terraform {
  required_version = "~> 1.3.9"
  required_providers {
    site24x7 = {
      source  = "site24x7/site24x7"
    }
  }
  cloud {
    hostname = "terrakube-api.minikube.net"
    organization = "simple"
    workspaces {
      name = "site24x7"
    }
  }
}

providers.tf

provider "site24x7" {
  data_center = "US"
}

Remote logs:

[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Downloading terraform version 1.3.9 architecture amd64 Type Linux
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Downloading: https://releases.hashicorp.com/terraform/1.3.9/terraform_1.3.9_linux_amd64.zip
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Unzip: /home/cnb/.terraform-spring-boot/terraform/1.3.9/terraform
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Terraform setExecutable successful
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.terraform.TerraformExecutorServiceImpl - 
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.terraform.TerraformExecutorServiceImpl - Initializing the backend...
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.terraform.TerraformExecutorServiceImpl - 
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.terraform.TerraformExecutorServiceImpl - Successfully configured the backend "s3"! Terraform will automatically
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.terraform.TerraformExecutorServiceImpl - use this backend unless the backend configuration changes.
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.terraform.TerraformExecutorServiceImpl - 
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.terraform.TerraformExecutorServiceImpl - Initializing provider plugins...
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.terraform.TerraformExecutorServiceImpl - - Reusing previous version of site24x7/site24x7 from the dependency lock file
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.terraform.TerraformExecutorServiceImpl - - Installing site24x7/site24x7 v1.0.77...
[threadPoolTaskExecutor-1] WARN org.terrakube.executor.service.terraform.TerraformExecutorServiceImpl - No commands to run before terraform operation Job 7
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - ***************************************
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Running Terraform PLAN
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.logs.LogsConsumer - ***************************************
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.plugin.tfstate.aws.AwsTerraformStateImpl - Generating backend override file for terraform 1.3.9
[threadPoolTaskExecutor-1] WARN org.terrakube.executor.service.terraform.TerraformExecutorServiceImpl - Not using any SSH key to download modules
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformClient - Creating terraform downloader using custom terraform release URL: https://releases.hashicorp.com/terraform/index.json
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Initialize TerraformDownloader using custom URL
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - User Home Directory: /home/cnb
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Validate/Create download temp directory: /home/cnb/.terraform-spring-boot/download/
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Validate/Create terraform directory: /home/cnb/.terraform-spring-boot/terraform/
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - User Home Directory for tofu download: /home/cnb
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Validate/Create tofu download temp directory: /home/cnb/.terraform-spring-boot/download/tofu/
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Validate/Create tofu directory: /home/cnb/.terraform-spring-boot/tofu/
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Downloading terraform releases list
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Deleting Temp /home/cnb/.terraform-spring-boot/terraform-9563087335461414682-release
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Found 309 terraform releases
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - Downloading terraform version 1.3.9 architecture amd64 Type Linux
[threadPoolTaskExecutor-1] INFO org.terrakube.terraform.TerraformDownloader - terraform_1.3.9_linux_amd64.zip already exists
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - 
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - No changes. Your infrastructure matches the configuration.
[threadPoolTaskExecutor-1] WARN org.terrakube.executor.service.terraform.TerraformExecutorServiceImpl - Terraform plan Executed Successfully: true
[threadPoolTaskExecutor-1] WARN org.terrakube.executor.service.terraform.TerraformExecutorServiceImpl - No commands to run after terraform operation Job true
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - 
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - Terraform has compared your real infrastructure against your configuration
[ForkJoinPool-1-worker-1] INFO org.terrakube.executor.service.logs.LogsConsumer - and found no differences, so no changes are needed.
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.plugin.tfstate.aws.AwsTerraformStateImpl - terraformStateFile: tfstate/d9b58bd3-f3fc-4056-a026-1163297e80a8/f11d5101-29c4-431e-bc58-b082b6c37cc6/7/04f0d6ea-3328-4722-baba-e5b8bc3e6f91/terraformLibrary.tfPlan
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.plugin.tfstate.aws.AwsTerraformStateImpl - terraformStateFile Path: /home/cnb/.terraform-spring-boot/executor/d9b58bd3-f3fc-4056-a026-1163297e80a8/f11d5101-29c4-431e-bc58-b082b6c37cc6/terraformLibrary.tfPlan true
[threadPoolTaskExecutor-1] WARN com.amazonaws.util.Base64 - JAXB is unavailable. Will fallback to SDK implementation which may be less performant.If you are using Java 9+, you will need to include javax.xml.bind:jaxb-api as a dependency.
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.plugin.tfoutput.aws.AwsTerraformOutputImpl - blobKey: tfoutput/d9b58bd3-f3fc-4056-a026-1163297e80a8/7/04f0d6ea-3328-4722-baba-e5b8bc3e6f91.tfoutput
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.status.UpdateJobStatusImpl - Job 7 is still active
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.status.UpdateJobStatusImpl - StepId: 04f0d6ea-3328-4722-baba-e5b8bc3e6f91
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.status.UpdateJobStatusImpl - output: 362
[threadPoolTaskExecutor-1] INFO org.terrakube.executor.service.status.UpdateJobStatusImpl - outputError: 0

[requiming]

it seems it doesn't work at all, do you have configured credentials for site24x7? it will not run without them

[requiming]

i think it's another bug, i had same output when initially configured repository, like empty, having empty state and auto.tfvars in repository, and plan wasn't showing and changes/trying to request site24x7 api, after i pushed state it started to do something, but failed with this error before.

[alfespa17]

it seems it doesn't work at all, do you have configured credentials for site24x7? it will not run without them

I just used the files that you have provided to run a plan, I have no idea about that provider or how to use it

[alfespa17]

What Terrakube version are you using?

Are you using some folder structure with modules in your repository?

[requiming]

docker.io/azbuilder/api-server:2.19.2
repository_root/module_main/submodules
repository_root/module_main/variables.auto.tfvars
repository_root/module_main/*.tf

[requiming]

[requiming]

[requiming]

main.tf (part of file)

#
# Tags
#

resource "site24x7_tag" "terraform" {
  tag_name      = "terraform"
  tag_value     = "true"
  tag_color     = "#4895A8" # blue
}

resource "site24x7_tag" "prod" {
  tag_name      = "prod"
  tag_value     = "true"
  tag_color     = "#4895A8" # blue
}

resource "site24x7_tag" "non_prod" {
  tag_name      = "prod"
  tag_value     = "false"
  tag_color     = "#4895A8" # blue
}

resource "site24x7_tag" "infra" {
  tag_name      = "infra"
  tag_value     = "true"
  tag_color     = "#4895A8" # blue
}

#
# Tenant's targets
#

module "tenant" {
    source                  = "./submodules/tenant"

    count                   = length(var.tenants)

    tenant_name             = var.tenants[count.index].name
    use_subdomain           = var.tenants[count.index].use_subdomain
    base_domain             = var.tenants[count.index].base_domain
    tag_value_override      = var.tenants[count.index].tag_value_override
    notification_profile_id = var.tenants[count.index].prod_tag ? site24x7_notification_profile.prod.id : site24x7_notification_profile.non_prod.id
    tag_ids                 = [
        site24x7_tag.terraform.id,
        var.tenants[count.index].prod_tag ? site24x7_tag.prod.id : site24x7_tag.non_prod.id
    ]
}

[alfespa17]

I was checking your information and I am not really sure why you are having that issue because in the end if you have a VCS workspace and you run a "speculative plan" using the CLI driven workflow it will basically take all the files that are include in your directory and simple run a terraform plan for one particular folder.

There was one issue where some files were not copy to run the remote plan as you can see here, not sure if that could be related for your configuration, maybe you can take a look there

#661 (comment)