ENI is not deleted when sagemaker notebook instance is destroyed and a security group is attached to it

[DavideAG]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

• Terraform v1.1.9
• provider registry.terraform.io/hashicorp/aws v4.24.0

Affected Resource(s)

• aws_sagemaker_notebook_instance
• aws_security_group

Terraform Configuration Files

Actually I'm using a simple configuration with two different modules.
The idea is to create an AWS sagemaker notebook instance in a subnet and then attach a SG to it.

module "sagemaker_notebook_instance" {
  source                   = "../../../modules/sagemaker-notebookinstance"
  instance_type            = "ml.t2.medium"
  name                     = "test-sg"
  instance_subnet_id       = "subnet-0751a8c66XXXXXXXX" # hidden
  instance_security_groups = [module.security-group.id]
}

module "security-group" {
  source      = "../../../modules/security-group"
  name        = "sagemaker-test-sg"
  description = "sagemaker-test security group"
  vpc_id      = "vpc-06c413a2eXXXXXXXX" # hidden
}

Some detail about the two modules

resource "aws_sagemaker_notebook_instance" "sagemaker_instance" {
  for_each                = var.instance_ids
  instance_type           = var.instance_type
  name                    = join("-", [module.resource_naming.composed_resource_name, each.value])
  role_arn                = aws_iam_role.default.arn
  lifecycle_config_name   = var.create_lifecycle ? aws_sagemaker_notebook_instance_lifecycle_configuration.sagemaker_lifecycle_conf[0].name : ""
  default_code_repository = var.create_git_config ? aws_sagemaker_code_repository.repo[0].code_repository_name : ""

  subnet_id              = var.instance_subnet_id
  security_groups        = var.instance_security_groups
  kms_key_id             = var.instance_kms_key_id
  direct_internet_access = var.instance_direct_internet_access

  lifecycle {
    create_before_destroy = true
    ignore_changes        = []
  }
}

resource "aws_security_group" "self_sg" {
  name_prefix = module.resource_naming.composed_resource_name
  description = var.description
  vpc_id      = var.vpc_id

  ingress {
    description = "Allow inbound traffic from network interfaces (and their associated instances) that are assigned to the same security group."
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    self        = true
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  lifecycle {
    create_before_destroy = true
  }
}

Expected Behavior

ENI should be removed when terraform tries to destroy our sagemaker notebook instance, in order to avoid dependency violations when security group is deleted. Actually, this creates a deadlock and our security group cannot be deleted.

The basic idea is that we need to delete ENI resource before trying to delete the attached SG. In the following AWS provider doc is described the process to use in order to avoid stucks during SG config changes, but is not mentioned how to overcome a fully deletion problem

• https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/security_group#change-of-name-or-name-prefix-value

Actual Behavior

After a bit, terraform destroy fails because ENI still exists and security group cannot be deleted.

...
module.security-group.aws_security_group.self_sg: Still destroying... [id=sg-08cb8e3c478XXXXXX, 14m30s elapsed]
module.security-group.aws_security_group.self_sg: Still destroying... [id=sg-08cb8e3c478XXXXXX, 14m40s elapsed]
module.security-group.aws_security_group.self_sg: Still destroying... [id=sg-08cb8e3c478XXXXXX, 14m50s elapsed]
╷
│ Error: deleting Security Group (sg-08cb8e3c478XXXXXX): DependencyViolation: resource sg-08cb8e3c478XXXXXX has a dependent object
│       status code: 400, request id: 96f38922-3170-4dec-93c7-92c39XXXXXX
│
│
╵

Steps to Reproduce

1. terraform apply
2. terraform destroy

References

Probably this issue is similar to:

• Lambda Associated EC2 Subnet and Security Group Deletion Issues and Improvements #10329
• provider/aws: Cleanup the Lambda ENI deletion fails on destroy (after update Terraform to v0.7.11) terraform#10272

[github-actions]

This functionality has been released in v4.29.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.