contributor permission recreated

[gevraud]

Is there an existing issue for this?

• I have searched the existing issues

Greenfield/Brownfield provisioning

greenfield

Terraform Version

1.8.2

Module Version

8.0.0

AzureRM Provider Version

3.101

Affected Resource(s)/Data Source(s)

azurerm_resource_group_template_deployment

Terraform Configuration Files

module "aks" {
  source               = "Azure/aks/azurerm"
  version              = "8.0.0"
  tracing_tags_enabled = true
  tracing_tags_prefix  = "aks_module_"
  tags       = var.tags

  cluster_name                 = var.kubernetes_cluster_name
  resource_group_name          = azurerm_resource_group.this.name
  location                     = azurerm_resource_group.this.location
  kubernetes_version           = "1.29"
  automatic_channel_upgrade    = "patch"
  sku_tier                     = "Standard"
  identity_ids                 = [azurerm_user_assigned_identity.this.id]
  identity_type                = "UserAssigned"
  microsoft_defender_enabled   = true 
  node_resource_group          = "node-${azurerm_resource_group.this.name}"
  node_os_channel_upgrade      = "NodeImage"
  // Monitoring
  log_analytics_solution = {
    id = azurerm_log_analytics_solution.this.id
  }
  log_analytics_workspace_enabled = true
  log_analytics_workspace = {
    id   = azurerm_log_analytics_workspace.this.id
    name = azurerm_log_analytics_workspace.this.name
  }
  msi_auth_for_monitoring_enabled = true
  // RBAC
  rbac_aad                          = true
  rbac_aad_managed                  = true
  role_based_access_control_enabled = true
  rbac_aad_admin_group_object_ids = [
    data.azuread_group.cluster_admins.object_id
  ]

  // Network
  vnet_subnet_id = module.vnet_aks_dev.subnets["${local.subnets_names[0]}"].id
  private_dns_zone_id = "/subscriptions/xxxx/resourceGroups/yyyy/providers/Microsoft.Network/privateDnsZones/aks.xxx.azmk8s.io"
  prefix = "aks-dev" 
  network_contributor_role_assigned_subnet_ids = {
    subnet1 = module.vnet_aks_dev.subnets["${local.subnets_names[0]}"].id
    subnet2 = module.vnet_aks_dev.subnets["${local.subnets_names[1]}"].id
    subnet3 = module.vnet_aks_dev.subnets["${local.subnets_names[2]}"].id
    subnet4 = module.vnet_aks_dev.subnets["${local.subnets_names[3]}"].id
  }
  private_cluster_enabled             = true
  private_cluster_public_fqdn_enabled = false
  azure_policy_enabled      = true
  net_profile_outbound_type = "userDefinedRouting"
  network_plugin            = "azure"
  network_plugin_mode       = "overlay"
  network_policy            = "calico"   
  // Node Pools
  agents_availability_zones = ["1"]
  agents_count              = null // because autoscalling enabled
  agents_pool_name          = "default"
  agents_labels = {
    type = "system"
  }
  enable_auto_scaling = true
  agents_max_count = 10 // for all
  agents_min_count = 1
  agents_max_pods  = 50
  agents_type      = "VirtualMachineScaleSets"
  temporary_name_for_rotation = "defaulttemp"
  
  node_pools = {
    worker1 = {
      name                = "worker1"
      vm_size             = "Standard_D2s_v3"
      node_count          = 1
      max_count = 10
      vnet_subnet_id      = module.vnet_aks_dev.subnets["${local.subnets_names[1]}"].id
      enable_auto_scaling = true
      os_disk_size_gb     = 128
      os_sku              = "Ubuntu"
      create_before_destroy = true
    }

  }
  maintenance_window = {
    allowed = [
      {
        day   = "Sunday",
        hours = [20, 23]
      },
    ]
    not_allowed = [
      {
        start = "2035-01-01T20:00:00Z",
        end   = "2035-01-01T21:00:00Z"
      },
    ]
  }
  maintenance_window_node_os = {
    frequency  = "Daily"
    interval   = 1
    start_time = "07:00"
    utc_offset = "+01:00"
    duration   = 16
  }

  depends_on = [
    module.vnet_aks_dev
  ]
}

tfvars variables values

rg_name                 = "rg-xxx-001"
kubernetes_cluster_name = "aks-xxx-dev-001"
kubernetes_dns_prefix   = "dev"
dns_servers             = ["x.x.x.x"]

Debug Output/Panic Output

# module.aks.azurerm_role_assignment.network_contributor_on_subnet["subnet1"] must be replaced
-/+ resource "azurerm_role_assignment" "network_contributor_on_subnet" {
      ~ id                                     = "/subscriptions/xxxxb/resourceGroups/rg-aks-dev-001/providers/Microsoft.Network/virtualNetworks/vnet-aks-dev-001/subnets/snet-aks-dev-001/providers/Microsoft.Authorization/roleAssignments/zzzz" -> (known after apply)
      ~ name                                   = "zzzz" -> (known after apply)
      ~ principal_id                           = "aaaa" # forces replacement -> (known after apply) # forces replacement
      ~ principal_type                         = "ServicePrincipal" -> (known after apply)
      ~ role_definition_id                     = "/subscriptions/xxxx/providers/Microsoft.Authorization/roleDefinitions/aaaa" -> (known after apply)
      + skip_service_principal_aad_check       = (known after apply)
        # (6 unchanged attributes hidden)
    }

  # module.aks.azurerm_role_assignment.network_contributor_on_subnet["subnet2"] must be replaced
-/+ resource "azurerm_role_assignment" "network_contributor_on_subnet" {
      ~ id                                     = "/subscriptions/4xxx/resourceGroups/rg-aks-dev-001/providers/Microsoft.Network/virtualNetworks/vnet-aks-dev-001/subnets/snet-aks-dev-002/providers/Microsoft.Authorization/roleAssignments/aaaa" -> (known after apply)
      ~ name                                   = "aaaa" -> (known after apply)
      ~ principal_id                           = "bbbb" # forces replacement -> (known after apply) # forces replacement
      ~ principal_type                         = "ServicePrincipal" -> (known after apply)
      ~ role_definition_id                     = "/subscriptions/xxxx/providers/Microsoft.Authorization/roleDefinitions/aaaa" -> (known after apply)
      + skip_service_principal_aad_check       = (known after apply)
        # (6 unchanged attributes hidden)
    }

  # module.aks.azurerm_role_assignment.network_contributor_on_subnet["subnet3"] must be replaced
-/+ resource "azurerm_role_assignment" "network_contributor_on_subnet" {
      ~ id                                     = "/subscriptions/xxxx/resourceGroups/rg-aks-dev-001/providers/Microsoft.Network/virtualNetworks/vnet-aks-dev-001/subnets/snet-aks-dev-003/providers/Microsoft.Authorization/roleAssignments/aaaa" -> (known after apply)
      ~ name                                   = "aaaa" -> (known after apply)
      ~ principal_id                           = "bbbb" # forces replacement -> (known after apply) # forces replacement
      ~ principal_type                         = "ServicePrincipal" -> (known after apply)
      ~ role_definition_id                     = "/subscriptions/xxxx/providers/Microsoft.Authorization/roleDefinitions/aaaa" -> (known after apply)
      + skip_service_principal_aad_check       = (known after apply)
        # (6 unchanged attributes hidden)
    }

  # module.aks.azurerm_role_assignment.network_contributor_on_subnet["subnet4"] must be replaced
-/+ resource "azurerm_role_assignment" "network_contributor_on_subnet" {
      ~ id                                     = "/subscriptions/xxxx/resourceGroups/rg-aks-dev-001/providers/Microsoft.Network/virtualNetworks/vnet-aks-dev-001/subnets/snet-aks-dev-004/providers/Microsoft.Authorization/roleAssignments/aaaa" -> (known after apply)
      ~ name                                   = "aaaa" -> (known after apply)
      ~ principal_id                           = "bbbb" # forces replacement -> (known after apply) # forces replacement
      ~ principal_type                         = "ServicePrincipal" -> (known after apply)
      ~ role_definition_id                     = "/subscriptions/xxxx/providers/Microsoft.Authorization/roleDefinitions/aaaa" -> (known after apply)
      + skip_service_principal_aad_check       = (known after apply)
        # (6 unchanged attributes hidden)
    }

Expected Behaviour

no recreation should happen

Actual Behaviour

it recreates the permission each time Terraform is running

Steps to Reproduce

No response

Important Factoids

No response

References

No response

[oscarwest]

We get the same thing every time we plan/apply.

module.aks.azurerm_role_assignment.application_gateway_resource_group_reader
and
module.aks.azurerm_role_assignment.application_gateway_existing_vnet_network_contributor

will be re-created every time (known after apply)

[zioproto]

For some reason the value of the principal_id is changing at every Terraform plan.

terraform-azurerm-aks/role_assignments.tf

Line 42 in 4729aee

principal_id = coalesce(try(data.azurerm_user_assigned_identity.cluster_identity[0].principal_id, azurerm_kubernetes_cluster.main.identity[0].principal_id), var.client_id)

@gevraud in your actual plan what do you see instead of "bbbb" ? Does the value actually change?

[zioproto]

I confirm I was able to reproduce the problem on 8.0.0 but not on 9.0.0 or 9.1.0.

[zioproto]

It is working properly in 9.0.0 and newer versions because since the version 9.0.0 there is this change: 6abee9e from PR #554

AKS had a change in the default behaviour that caused a state drift. When using 8.0.0 you should have also the following on top of the proposed plan:

Terraform detected the following changes made outside of Terraform since the last "terraform apply" which may have affected this plan:

  # module.aks.azurerm_kubernetes_cluster.main has changed
  ~ resource "azurerm_kubernetes_cluster" "main" {
        id                                  = "/subscriptions/xxxx/resourceGroups/<rg>/providers/Microsoft.ContainerService/managedClusters/ccccc"
        name                                = "cccccc"
        # (39 unchanged attributes hidden)

      ~ identity {
          + identity_ids = []
            # (3 unchanged attributes hidden)
        }

        # (7 unchanged blocks hidden)
    }

Those identity ids change propagate into the problem described in this issue.

Could you please try to reproduce on version 9.1.0 and confirm the problem is fixed ?

thanks

[gevraud]

For some reason the value of the principal_id is changing at every Terraform plan.

terraform-azurerm-aks/role_assignments.tf

Line 42 in 4729aee

principal_id = coalesce(try(data.azurerm_user_assigned_identity.cluster_identity[0].principal_id, azurerm_kubernetes_cluster.main.identity[0].principal_id), var.client_id)

@gevraud in your actual plan what do you see instead of "bbbb" ? Does the value actually change?
Hello,

I created permission without the module. I don't use the module permission anymore.

Regards