refactor: remove existing vpc endpoint configurations from base modul…

…e and move into sub-module (terraform-aws-modules#635)

.pre-commit-config.yaml

@@ -1,6 +1,6 @@
 repos:
   - repo: git://github.com/antonbabenko/pre-commit-terraform
-    rev: v1.48.0
+    rev: v1.50.0
     hooks:
       - id: terraform_fmt
       - id: terraform_validate

UPGRADE-3.0.md

@@ -0,0 +1,52 @@
+# Upgrade from v2.x to v3.x
+
+If you have any questions regarding this upgrade process, please consult the `examples` directory:
+
+- [Complete-VPC](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/complete-vpc)
+
+If you find a bug, please open an issue with supporting configuration to reproduce.
+
+## List of backwards incompatible changes
+
+Previously, VPC endpoints were configured as standalone resources with their own set of variables and attributes. Now, this functionality is provided via a module which loops over a map of maps using `for_each` to generate the desired VPC endpoints. Therefore, to maintain the existing set of functionality while upgrading, you will need to perform the following changes:
+
+1. Move the endpoint resource from the main module to the sub-module. The example state move below is valid for all endpoints you might have configured (reference [`complete-vpc`](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/complete-vpc) example for reference), where `ssmmessages` should be updated for and state move performed for each endpoint configured:
+
+```
+terraform state mv 'module.vpc.aws_vpc_endpoint.ssm[0]' 'module.vpc_endpoints.aws_vpc_endpoint.this["ssm"]'
+terraform state mv 'module.vpc.aws_vpc_endpoint.ssmmessages[0]' 'module.vpc_endpoints.aws_vpc_endpoint.this["ssmmessages"]'
+terraform state mv 'module.vpc.aws_vpc_endpoint.ec2[0]' 'module.vpc_endpoints.aws_vpc_endpoint.this["ec2"]'
+...
+```
+
+2. Remove the gateway endpoint route table association separate resources. The route table associations are now managed in the VPC endpoint resource itself via the map of maps provided to the VPC endpoint sub-module. Perform the necessary removals for each route table association and for S3 and/or DynamoDB depending on your configuration:
+
+```
+terraform state rm 'module.vpc.aws_vpc_endpoint_route_table_association.intra_dynamodb[0]'
+terraform state rm 'module.vpc.aws_vpc_endpoint_route_table_association.private_dynamodb[0]'
+terraform state rm 'module.vpc.aws_vpc_endpoint_route_table_association.public_dynamodb[0]'
+...
+```
+
+### Variable and output changes
+
+1. Removed variables:
+
+   - `enable_*_endpoint`
+   - `*_endpoint_type`
+   - `*_endpoint_security_group_ids`
+   - `*_endpoint_subnet_ids`
+   - `*_endpoint_private_dns_enabled`
+   - `*_endpoint_policy`
+
+2. Renamed variables:
+
+See the [VPC endpoint sub-module](modules/vpc-endpoints) for the more information on the variables to utilize for VPC endpoints
+
+3. Removed outputs:
+
+   - `vpc_endpoint_*`
+
+4. Renamed outputs:
+
+VPC endpoint outputs are now provided via the VPC endpoint sub-module and can be accessed via lookups. See [`complete-vpc`](https://github.com/terraform-aws-modules/terraform-aws-vpc/tree/master/examples/complete-vpc) for further examples of how to access VPC endpoint attributes from outputs

examples/complete-vpc/README.md

@@ -21,20 +21,22 @@ Note that this example may create resources which can cost money (AWS Elastic IP
 
 | Name | Version |
 |------|---------|
-| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.21 |
-| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.10 |
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.12.26 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.15 |
 
 ## Providers
 
 | Name | Version |
 |------|---------|
-| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.10 |
+| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.15 |
 
 ## Modules
 
 | Name | Source | Version |
 |------|--------|---------|
 | <a name="module_vpc"></a> [vpc](#module\_vpc) | ../../ |  |
+| <a name="module_vpc_endpoints"></a> [vpc\_endpoints](#module\_vpc\_endpoints) | ../../modules/vpc-endpoints |  |
+| <a name="module_vpc_endpoints_nocreate"></a> [vpc\_endpoints\_nocreate](#module\_vpc\_endpoints\_nocreate) | ../../modules/vpc-endpoints |  |
 
 ## Resources
 
@@ -43,7 +45,7 @@ Note that this example may create resources which can cost money (AWS Elastic IP
 | [aws_iam_policy_document.dynamodb_endpoint_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
 | [aws_iam_policy_document.generic_endpoint_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
 | [aws_security_group.default](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/security_group) | data source |
-| [aws_vpc_endpoint.dynamodb](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/vpc_endpoint) | data source |
+| [aws_vpc_endpoint_service.dynamodb](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/vpc_endpoint_service) | data source |
 
 ## Inputs
 

examples/complete-vpc/main.tf

@@ -2,19 +2,27 @@ provider "aws" {
   region = "eu-west-1"
 }
 
-data "aws_security_group" "default" {
-  name   = "default"
-  vpc_id = module.vpc.vpc_id
+locals {
+  name   = "complete-example"
+  region = "eu-west-1"
+  tags = {
+    Owner       = "user"
+    Environment = "staging"
+    Name        = "complete"
+  }
 }
 
+################################################################################
+# VPC Module
+################################################################################
+
 module "vpc" {
   source = "../../"
 
-  name = "complete-example"
-
+  name = local.name
   cidr = "20.10.0.0/16" # 10.0.0.0/8 is reserved for EC2-Classic
 
-  azs                 = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
+  azs                 = ["${local.region}a", "${local.region}b", "${local.region}c"]
   private_subnets     = ["20.10.1.0/24", "20.10.2.0/24", "20.10.3.0/24"]
   public_subnets      = ["20.10.11.0/24", "20.10.12.0/24", "20.10.13.0/24"]
   database_subnets    = ["20.10.21.0/24", "20.10.22.0/24", "20.10.23.0/24"]
@@ -53,80 +61,6 @@ module "vpc" {
   dhcp_options_domain_name         = "service.consul"
   dhcp_options_domain_name_servers = ["127.0.0.1", "10.10.0.2"]
 
-  # VPC endpoint for S3
-  # Note - S3 Interface type support is only available on AWS provider 3.10 and later
-  enable_s3_endpoint              = true
-  s3_endpoint_type                = "Interface"
-  s3_endpoint_private_dns_enabled = false
-  s3_endpoint_security_group_ids  = [data.aws_security_group.default.id]
-
-  # VPC endpoint for DynamoDB
-  enable_dynamodb_endpoint = true
-  dynamodb_endpoint_policy = data.aws_iam_policy_document.dynamodb_endpoint_policy.json
-
-  # VPC endpoint for SSM
-  enable_ssm_endpoint              = true
-  ssm_endpoint_private_dns_enabled = true
-  ssm_endpoint_security_group_ids  = [data.aws_security_group.default.id]
-
-  # VPC endpoint for Lambda
-  enable_lambda_endpoint              = true
-  lambda_endpoint_private_dns_enabled = true
-  lambda_endpoint_security_group_ids  = [data.aws_security_group.default.id]
-
-  # VPC endpoint for SSMMESSAGES
-  enable_ssmmessages_endpoint              = true
-  ssmmessages_endpoint_private_dns_enabled = true
-  ssmmessages_endpoint_security_group_ids  = [data.aws_security_group.default.id]
-
-  # VPC Endpoint for EC2
-  enable_ec2_endpoint              = true
-  ec2_endpoint_policy              = data.aws_iam_policy_document.generic_endpoint_policy.json
-  ec2_endpoint_private_dns_enabled = true
-  ec2_endpoint_security_group_ids  = [data.aws_security_group.default.id]
-
-  # VPC Endpoint for EC2MESSAGES
-  enable_ec2messages_endpoint              = true
-  ec2messages_endpoint_private_dns_enabled = true
-  ec2messages_endpoint_security_group_ids  = [data.aws_security_group.default.id]
-
-  # VPC Endpoint for ECR API
-  enable_ecr_api_endpoint              = true
-  ecr_api_endpoint_policy              = data.aws_iam_policy_document.generic_endpoint_policy.json
-  ecr_api_endpoint_private_dns_enabled = true
-  ecr_api_endpoint_security_group_ids  = [data.aws_security_group.default.id]
-
-  # VPC Endpoint for ECR DKR
-  enable_ecr_dkr_endpoint              = true
-  ecr_dkr_endpoint_policy              = data.aws_iam_policy_document.generic_endpoint_policy.json
-  ecr_dkr_endpoint_private_dns_enabled = true
-  ecr_dkr_endpoint_security_group_ids  = [data.aws_security_group.default.id]
-
-  # VPC endpoint for KMS
-  enable_kms_endpoint              = true
-  kms_endpoint_private_dns_enabled = true
-  kms_endpoint_security_group_ids  = [data.aws_security_group.default.id]
-
-  # VPC endpoint for ECS
-  enable_ecs_endpoint              = true
-  ecs_endpoint_private_dns_enabled = true
-  ecs_endpoint_security_group_ids  = [data.aws_security_group.default.id]
-
-  # VPC endpoint for ECS telemetry
-  enable_ecs_telemetry_endpoint              = true
-  ecs_telemetry_endpoint_private_dns_enabled = true
-  ecs_telemetry_endpoint_security_group_ids  = [data.aws_security_group.default.id]
-
-  # VPC endpoint for CodeDeploy
-  enable_codedeploy_endpoint              = true
-  codedeploy_endpoint_private_dns_enabled = true
-  codedeploy_endpoint_security_group_ids  = [data.aws_security_group.default.id]
-
-  # VPC endpoint for CodeDeploy Commands Secure
-  enable_codedeploy_commands_secure_endpoint              = true
-  codedeploy_commands_secure_endpoint_private_dns_enabled = true
-  codedeploy_commands_secure_endpoint_security_group_ids  = [data.aws_security_group.default.id]
-
   # Default security group - ingress/egress rules cleared to deny all
   manage_default_security_group  = true
   default_security_group_ingress = []
@@ -138,22 +72,124 @@ module "vpc" {
   create_flow_log_cloudwatch_iam_role  = true
   flow_log_max_aggregation_interval    = 60
 
-  tags = {
-    Owner       = "user"
-    Environment = "staging"
-    Name        = "complete"
+  tags = local.tags
+}
+
+################################################################################
+# VPC Endpoints Module
+################################################################################
+
+module "vpc_endpoints" {
+  source = "../../modules/vpc-endpoints"
+
+  vpc_id             = module.vpc.vpc_id
+  security_group_ids = [data.aws_security_group.default.id]
+
+  endpoints = {
+    s3 = {
+      service = "s3"
+      tags    = { Name = "s3-vpc-endpoint" }
+    },
+    dynamodb = {
+      service         = "dynamodb"
+      service_type    = "Gateway"
+      route_table_ids = flatten([module.vpc.intra_route_table_ids, module.vpc.private_route_table_ids, module.vpc.public_route_table_ids])
+      policy          = data.aws_iam_policy_document.dynamodb_endpoint_policy.json
+      tags            = { Name = "dynamodb-vpc-endpoint" }
+    },
+    ssm = {
+      service             = "ssm"
+      private_dns_enabled = true
+      subnet_ids          = module.vpc.private_subnets
+    },
+    ssmmessages = {
+      service             = "ssmmessages"
+      private_dns_enabled = true
+      subnet_ids          = module.vpc.private_subnets
+    },
+    lambda = {
+      service             = "lambda"
+      private_dns_enabled = true
+      subnet_ids          = module.vpc.private_subnets
+    },
+    ecs = {
+      service             = "ecs"
+      private_dns_enabled = true
+      subnet_ids          = module.vpc.private_subnets
+    },
+    ecs_telemetry = {
+      service             = "ecs-telemetry"
+      private_dns_enabled = true
+      subnet_ids          = module.vpc.private_subnets
+    },
+    ec2 = {
+      service             = "ec2"
+      private_dns_enabled = true
+      subnet_ids          = module.vpc.private_subnets
+    },
+    ec2messages = {
+      service             = "ec2messages"
+      private_dns_enabled = true
+      subnet_ids          = module.vpc.private_subnets
+    },
+    ecr_api = {
+      service             = "ecr.api"
+      private_dns_enabled = true
+      subnet_ids          = module.vpc.private_subnets
+      policy              = data.aws_iam_policy_document.generic_endpoint_policy.json
+    },
+    ecr_dkr = {
+      service             = "ecr.dkr"
+      private_dns_enabled = true
+      subnet_ids          = module.vpc.private_subnets
+      policy              = data.aws_iam_policy_document.generic_endpoint_policy.json
+    },
+    kms = {
+      service             = "kms"
+      private_dns_enabled = true
+      subnet_ids          = module.vpc.private_subnets
+    },
+    codedeploy = {
+      service             = "codedeploy"
+      private_dns_enabled = true
+      subnet_ids          = module.vpc.private_subnets
+    },
+    codedeploy_commands_secure = {
+      service             = "codedeploy-commands-secure"
+      private_dns_enabled = true
+      subnet_ids          = module.vpc.private_subnets
+    },
   }
 
-  vpc_endpoint_tags = {
+  tags = merge(local.tags, {
     Project  = "Secret"
     Endpoint = "true"
-  }
+  })
+}
+
+module "vpc_endpoints_nocreate" {
+  source = "../../modules/vpc-endpoints"
+
+  create = false
+}
+
+################################################################################
+# Supporting Resources
+################################################################################
+
+data "aws_security_group" "default" {
+  name   = "default"
+  vpc_id = module.vpc.vpc_id
 }
 
 # Data source used to avoid race condition
-data "aws_vpc_endpoint" "dynamodb" {
-  vpc_id       = module.vpc.vpc_id
-  service_name = "com.amazonaws.eu-west-1.dynamodb"
+data "aws_vpc_endpoint_service" "dynamodb" {
+  service = "dynamodb"
+
+  filter {
+    name   = "service-type"
+    values = ["Gateway"]
+  }
 }
 
 data "aws_iam_policy_document" "dynamodb_endpoint_policy" {
@@ -171,7 +207,7 @@ data "aws_iam_policy_document" "dynamodb_endpoint_policy" {
       test     = "StringNotEquals"
       variable = "aws:sourceVpce"
 
-      values = [data.aws_vpc_endpoint.dynamodb.id]
+      values = [data.aws_vpc_endpoint_service.dynamodb.id]
     }
   }
 }
@@ -191,7 +227,7 @@ data "aws_iam_policy_document" "generic_endpoint_policy" {
       test     = "StringNotEquals"
       variable = "aws:sourceVpce"
 
-      values = [data.aws_vpc_endpoint.dynamodb.id]
+      values = [data.aws_vpc_endpoint_service.dynamodb.id]
     }
   }
 }