Security Group /security Group-rule recreation error

[jensDideriksen]

Hello.

I have problems recreating securitygrouprules. Seem to be related to multiply old cases, that seem resolved and closed.
would be great to have some feedback or workaround.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.12.20

• provider.aws v2.27.0 (also tried 2.53.0)
• provider.dns v2.2.0
• provider.template v2.1.2

Affected Resource(s)

aws_security_group

aws_security_group_rule

Terraform Configuration Files

resource "aws_security_group" "security_group" {
  count = var.additional_listener_port == [] ? 0 :1
  name        = "${var.env}-${var.app_name}-securityGroup"
  description = "Securitygroup for ${var.env}-${var.app_name}"
  vpc_id      = module.network.vpc_id
}
resource "aws_security_group_rule" "allow_inbound_rules" {
  for_each = { for v in var.additional_listener_port : v => v }
  type              = "ingress"
  to_port           = tonumber(each.value)
  protocol          = "TCP"
  cidr_blocks       = sort(module.network.app_layer_cidrs)
  from_port         = tonumber(each.value)
  security_group_id = aws_security_group.beanstalk_specific_sg[0].id
}

Debug Output

Error revoking security group sg-087e80ad801d081f2 rules: InvalidPermission.NotFound: The specified rule does not exist in this security group.
status code: 400, request id: f0d73cde-e0e9-4bd3-8dd7-1fa617edee66

Expected Behavior

Recreating the security group rule when changing protocol from "-1" to "TCP"

Actual Behavior

Error revoking security group sg-087e80ad801d081f2 rules: InvalidPermission.NotFound: The specified rule does not exist in this security group.
status code: 400, request id: f0d73cde-e0e9-4bd3-8dd7-1fa617edee66

Steps to Reproduce

1. create securitygroup
2. create securitygrouprule
3. change protocol in securitygrouprule and apply

• #0000

[Sk8rMarc]

I have a similar issue, it seems. I have an AWS MSK stack where I have a cluster security group where I have the option to add and remove client security groups by group name or name tag. This stack also has an optional utility instance it will stand up, with a empty ingress security group, but instead of adding this to the cluster security group directly, I'm using an a conditional aws_security_group_rule that adds the the utility instance's security group to the cluster's. When I create the stack with the utility instance, things seem to work. If I instantiate the stack without the utility instance, and later decide to stand up the utility instance, that works too. BUT when I go to remove the utility instance, I run into problems:

---

aws_iam_role_policy_attachment.utility_ec2[0]: Destroying... [id=ts-message-bus-dev-green-ref20-msk-utility-us-east-1-20200410160845019500000001]
aws_security_group_rule.msk_utility[0]: Destroying... [id=sgrule-2872173746]
aws_instance.msk_utility[0]: Destroying... [id=i-064f99abe5b8d9866]
aws_security_group.msk_cluster: Modifying... [id=sg-08070a71c0adbb244]
aws_iam_role_policy_attachment.utility_ec2[0]: Destruction complete after 0s
aws_security_group.msk_cluster: Modifications complete after 1s [id=sg-08070a71c0adbb244]
aws_instance.msk_utility[0]: Still destroying... [id=i-064f99abe5b8d9866, 10s elapsed]
aws_instance.msk_utility[0]: Still destroying... [id=i-064f99abe5b8d9866, 20s elapsed]
aws_instance.msk_utility[0]: Still destroying... [id=i-064f99abe5b8d9866, 30s elapsed]
aws_instance.msk_utility[0]: Still destroying... [id=i-064f99abe5b8d9866, 40s elapsed]
aws_instance.msk_utility[0]: Still destroying... [id=i-064f99abe5b8d9866, 50s elapsed]
aws_instance.msk_utility[0]: Destruction complete after 51s
aws_iam_instance_profile.msk_utility[0]: Destroying... [id=ts-message-bus-dev-green-ref20-msk-utility-us-east-1]
aws_iam_instance_profile.msk_utility[0]: Destruction complete after 0s
aws_iam_role.utility_instance_profile[0]: Destroying... [id=ts-message-bus-dev-green-ref20-msk-utility-us-east-1]
aws_iam_role.utility_instance_profile[0]: Destruction complete after 1s

Error: Error revoking security group sg-08070a71c0adbb244 rules: InvalidPermission.NotFound: The specified rule does not exist in this security group.
status code: 400, request id: bfad67ca-3e0f-4fde-91b2-ae2c747e5c9f

Also, when some change causes the utility instance to be bounced, the stack neglects to re-apply the aws_security_group_rule!. I would have to re-apply, to get the rule created.

[Gooygeek]

I have been facing this issue also, although going between UDP and TCP protocols.
After a deeper dig here are my findings:
When changing the protocols, terraform will destroy the old rule and create a new one.
The issue seems to be that when terraform is destroying the old rule it uses the protocol of the rule it is trying to create, rather than the rule it is destroying.
When this happens, typically the new rule won't exist yet and thus it will be trying to find a rule that does not exist.
This leads to the error.

[nmarchini]

I am getting the same issues with this and it's blocking us from developing some code as the workaround is manual deletion of the rules and then apply again, can this get some focus please.
Terraform v0.12.28

• provider.aws v2.70.0

[dinvlad]

Same issue here, we had to manually delete the rule from security group and TF state as a workaround.

[gotojeffray]

same issue here, not able to change the protocol from tcp to udp.

[bbe64]

Same issue with versions:

Terraform v0.13.4
+ provider registry.terraform.io/-/aws v3.2.0
+ provider registry.terraform.io/hashicorp/aws v3.2.0

[emalihin]

Just caught the same bug changing SG Rule protocol from TCP to UDP

Terraform v0.13.5
hashicorp/aws v3.17.0

[blortuga]

we hit this, and only deleting the group rule in AWS console was enough to work around this error. (we did not have to delete it from state). The only change in the rule was updating protocol from "TCP" to "-1".

[Xercoy]

Hit this error as well. My change was the same as @blortuga, I updated an ingress rule's protocol from TCP to -1. Removing the rule in the console so that terraform perceived the change as a whole new change rather than a recreation worked.

[skitamura7446]

I found a workaround for this.
Add the following settings in "aws_security_group_rule".

  lifecycle {
    create_before_destroy = true
  }

In my environment, I was able to achieve the change from "tcp" to "udp" by adding this setting.

Terraform v0.14.5
hashicorp/aws v3.27.0

result...

  # module.security_group_rule.aws_security_group_rule.security_group_rule["rule_1"] must be replaced
+/- resource "aws_security_group_rule" "security_group_rule" {
      ~ id                       = "sgrule-181227XXXX" -> (known after apply)
      - ipv6_cidr_blocks         = [] -> null
      - prefix_list_ids          = [] -> null
      ~ protocol                 = "tcp" -> "udp" # forces replacement
      + source_security_group_id = (known after apply)
        # (7 unchanged attributes hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.security_group_rule.aws_security_group_rule.security_group_rule["rule_1"]: Creating...
module.security_group_rule.aws_security_group_rule.security_group_rule["rule_1"]: Creation complete after 0s [id=sgrule-271816XXXX]
module.security_group_rule.aws_security_group_rule.security_group_rule["rule_1"]: Destroying... [id=sgrule-181227XXXX]
module.security_group_rule.aws_security_group_rule.security_group_rule["rule_1"]: Destruction complete after 0s

Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

[edmundcraske-bjss]

#8769 appears to refer to the same issue.

[YakDriver]

I am not able to repro this problem, even without create_before_destroy. I'm adding a test in #26553 that changes protocols and show no errors changing from UDP -> TCP -> UDP. As a result, I'm going to close this issue with #26553. Let us know if the test case fails to capture the problem.

[github-actions]

This functionality has been released in v4.29.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.