DependencyViolation when deleting security group associated to lambda and vpc_endpoint

[ghost]

This issue was originally opened by @pnandita as hashicorp/terraform#26892. It was migrated here as a result of the provider split. The original body of the issue is below.

---

Description:
Create 2 lambdas in terraform. Each lambda will have its own security group. Also create a VPC endpoint with a security group of it’s own. Add the lambda security groups as an ingress into vpc_endpoint’s security group. Now, delete one of the lambda it’s associated security group, and also remove the reference from the ingress rules from vpc_endpoint. This results in a dependency violation, and the lambda security group never ends up getting deleted and fails with the error:
aws_security_group.lambda_sg2: Still destroying... [id=sg-xxxxx, 19m41s elapsed]
aws_security_group.lambda_sg2: Still destroying... [id=sg-xxxxx, 19m51s elapsed]

Error: Error deleting security group: DependencyViolation: resource sg-xxxx has a dependent object
status code: 400, request id: xxxx-xxx-xx-xx-xxxxx

Terraform Version

Terraform v0.13.5
+ provider registry.terraform.io/hashicorp/aws v3.14.1

Terraform Configuration Files

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

variable "vpc_id" {}
variable "role" {}
variable "subnet_id" {}
variable "lambda_filename" {}
variable "lambda_runtime" {}
variable "lambda_handler" {}


resource "aws_security_group" "lambda_sg1" {
  name   = "lambda_sg1"
  vpc_id = var.vpc_id
}

resource "aws_lambda_function" "test_lambda1" {
  filename      = var.lambda_filename
  function_name = "test_lambda1"
  role          = var.role
  handler       = var.lambda_handler

  source_code_hash = filebase64sha256(var.lambda_filename)

  runtime = var.lambda_runtime
  vpc_config {
    subnet_ids         = [var.subnet_id]
    security_group_ids = [aws_security_group.lambda_sg1.id]
  }
}

resource "aws_security_group" "lambda_sg2" {
  name   = "lambda_sg2"
  vpc_id = var.vpc_id
}

resource "aws_lambda_function" "test_lambda2" {
  filename      = var.lambda_filename
  function_name = "test_lambda2"
  role          = var.role
  handler       = var.lambda_handler

  source_code_hash = filebase64sha256(var.lambda_filename)

  runtime = var.lambda_runtime
  vpc_config {
    subnet_ids         = [var.subnet_id]
    security_group_ids = [aws_security_group.lambda_sg2.id]
  }
}

resource "aws_security_group" "vpc_endpoint_sg" {
  name   = "vpc_endpoint_sg"
  vpc_id = var.vpc_id

  ingress {
    from_port       = 443
    to_port         = 443
    protocol        = "tcp"
    security_groups = [aws_security_group.lambda_sg1.id, aws_security_group.lambda_sg2.id]
  }
}

resource "aws_vpc_endpoint" "vpc_endpoint" {
  vpc_id              = var.vpc_id
  subnet_ids          = [var.subnet_id]
  service_name        = "com.amazonaws.us-east-1.execute-api"
  vpc_endpoint_type   = "Interface"
  private_dns_enabled = false
  security_group_ids  = [aws_security_group.vpc_endpoint_sg.id]
}

Debug Output

aws_security_group.lambda_sg2: Still destroying... [id=sg-xxxxx, 19m41s elapsed]
aws_security_group.lambda_sg2: Still destroying... [id=sg-xxxxx, 19m51s elapsed]

Error: Error deleting security group: DependencyViolation: resource sg-xxxx has a dependent object
	status code: 400, request id: xxxx-xxx-xx-xx-xxxxx

Crash Output

2020-11-11T20:16:41.259-0600 [DEBUG] plugin.terraform-provider-aws_v3.14.1_x5: created by google.golang.org/grpc.(*Server).serveStreams.func1
2020-11-11T20:16:41.259-0600 [DEBUG] plugin.terraform-provider-aws_v3.14.1_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:857 +0x204
2020-11-11T20:16:41.269-0600 [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2020/11/11 20:16:41 [DEBUG] aws_security_group.lambda_sg2: apply errored, but we're indicating that via the Error pointer rather than returning it: rpc error: code = Unavailable desc = transport is closing
2020/11/11 20:16:41 [TRACE] eval: *terraform.EvalWriteState
2020/11/11 20:16:41 [TRACE] EvalWriteState: writing current state object for aws_security_group.lambda_sg2
2020/11/11 20:16:41 [TRACE] eval: *terraform.EvalApplyPost
2020/11/11 20:16:41 [WARN] Early exit triggered by hook: *terraform.stopHook
2020/11/11 20:16:41 [TRACE] eval: *terraform.EvalApplyPost, early exit err: early exit
2020/11/11 20:16:41 [TRACE] eval: *terraform.EvalSequence, early exit err: early exit
2020/11/11 20:16:41 [TRACE] eval: *terraform.EvalOpFilter, early exit err: early exit
2020/11/11 20:16:41 [TRACE] [walkApply] Exiting eval tree: aws_security_group.lambda_sg2 (destroy)
2020/11/11 20:16:41 [TRACE] vertex "aws_security_group.lambda_sg2 (destroy)": visit complete
2020/11/11 20:16:41 [TRACE] dag/walk: visiting "aws_security_group.vpc_endpoint_sg"
2020/11/11 20:16:41 [TRACE] vertex "aws_security_group.vpc_endpoint_sg": starting visit (*terraform.NodeApplyableResourceInstance)
2020/11/11 20:16:41 [TRACE] vertex "aws_security_group.vpc_endpoint_sg": evaluating
2020/11/11 20:16:41 [TRACE] [walkApply] Entering eval tree: aws_security_group.vpc_endpoint_sg
2020/11/11 20:16:41 [TRACE] eval: *terraform.EvalSequence
2020/11/11 20:16:41 [TRACE] eval: *terraform.EvalGetProvider
2020/11/11 20:16:41 [TRACE] eval: *terraform.EvalReadDiff
2020/11/11 20:16:41 [TRACE] EvalReadDiff: Read Update change from plan for aws_security_group.vpc_endpoint_sg
2020/11/11 20:16:41 [TRACE] eval: *terraform.EvalIf
2020/11/11 20:16:41 [TRACE] eval: terraform.EvalNoop
2020/11/11 20:16:41 [TRACE] eval: *terraform.EvalIf
2020/11/11 20:16:41 [TRACE] eval: *terraform.EvalReadState
2020/11/11 20:16:41 [TRACE] EvalReadState: reading state for aws_security_group.vpc_endpoint_sg
2020/11/11 20:16:41 [TRACE] UpgradeResourceState: schema version of aws_security_group.vpc_endpoint_sg is still 1; calling provider "aws" for any other minor fixups
2020/11/11 20:16:41 [TRACE] GRPCProvider: UpgradeResourceState
2020/11/11 20:16:41 [ERROR] eval: *terraform.EvalReadState, err: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: EOF"
2020/11/11 20:16:41 [ERROR] eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: EOF"
2020/11/11 20:16:41 [TRACE] [walkApply] Exiting eval tree: aws_security_group.vpc_endpoint_sg
2020/11/11 20:16:41 [TRACE] vertex "aws_security_group.vpc_endpoint_sg": visit complete
2020/11/11 20:16:41 [TRACE] dag/walk: upstream of "meta.count-boundary (EachMode fixup)" errored, so skipping
2020/11/11 20:16:41 [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/hashicorp/aws\"] (close)" errored, so skipping
2020/11/11 20:16:41 [TRACE] dag/walk: upstream of "root" errored, so skipping
2020/11/11 20:16:41 [WARN] terraform: stop complete
2020/11/11 20:16:41 [TRACE] backend/local: graceful stop has completed
2020/11/11 20:16:41 [TRACE] statemgr.Filesystem: preparing to manage state snapshots at terraform.tfstate
2020-11-11T20:16:41.270-0600 [DEBUG] plugin: plugin process exited: path=.terraform/plugins/registry.terraform.io/hashicorp/aws/3.14.1/darwin_amd64/terraform-provider-aws_v3.14.1_x5 pid=47234 error="exit status 2"
2020/11/11 20:16:41 [TRACE] statemgr.Filesystem: existing snapshot has lineage "eab5ade0-ee72-7fb9-35d9-020b0f21eda2" serial 9
2020/11/11 20:16:41 [TRACE] statemgr.Filesystem: not making a backup, because the new snapshot is identical to the old
2020/11/11 20:16:41 [TRACE] statemgr.Filesystem: no state changes since last snapshot
2020/11/11 20:16:41 [TRACE] statemgr.Filesystem: writing snapshot at terraform.tfstate
2020-11-11T20:16:41.278-0600 [DEBUG] plugin: plugin exited



!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Expected Behavior

Should be able to delete the lambda and security group when removed from terraform.

Actual Behavior

The security group never gets deleted due to a Dependency violation:

aws_security_group.lambda_sg2: Still destroying... [id=sg-xxxxx, 19m41s elapsed]
aws_security_group.lambda_sg2: Still destroying... [id=sg-xxxxx, 19m51s elapsed]

Error: Error deleting security group: DependencyViolation: resource sg-xxxx has a dependent object
	status code: 400, request id: xxxx-xxx-xx-xx-xxxxx

Steps to Reproduce

1. Run terraform init && terraform apply -auto-approve
2. Remove lambda_sg2, test_lambda2, and vpc_endpoint_sg's reference to aws_security_group.lambda_sg2.id
3. Re-run terraform apply -auto-approve

[github-actions]

This functionality has been released in v4.29.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.