aws data provider Error: multiple VPC Endpoint Services matched

[jonathancolby-olx]

versions:

vpc module (this module): 2.33.0
terraform version: v0.12.21
terraform providers: v2.62.0

background info:

we use this TF codebase in 2 different accounts, and 2 different regions. the error only appears in one of the accounts in eu-west-1. in this account, we do have 2 custom vpc endpoints (PrivateLink services). Just a theory, but perhaps the aws_vpc_endpoint_service is incorrectly returning those endpoints, even though they are not S3.

creating this issue to see if anyone else is having the issue.

Error: multiple VPC Endpoint Services matched; use additional constraints to reduce matches to a single VPC Endpoint Service

  on .terraform/modules/vpc/terraform-aws-vpc-2.33.0/vpc-endpoints.tf line 4, in data "aws_vpc_endpoint_service" "s3":
   4: data "aws_vpc_endpoint_service" "s3" {

[bryantbiggs]

hi @jonathancolby-olx - this error message looks like there are are 2 s3 vpc endpoints and the data filter is unable to return just a single result (its returning multiple matches and the data source does not know how to handle multiple matches, it needs to converge on a single match). currently the filter criteria is just to match on the s3 service

terraform-aws-vpc/vpc-endpoints.tf

Line 7 in 26bb020

service = "s3"

- if needed, we could look at providing additional filtering functionality but I'm not sure that would solve this exact problem. I could see cases where multiple s3 endpoints are provisioned, but you can only have one s3 endpoint per subnet, so without knowing more its hard to tell what your exact scenario is. do you already have an s3 endpoint deployed?

[bryantbiggs]

hi @jonathancolby-olx - were you able to check if an s3 endpoint already existed in the subnet(s) you are provisioning into?

[jonathancolby-olx]

@bryantbiggs - This issue is resolved.

The problem was not directly related to the terraform aws provider. However it might appear again for other users in the same situation.

Our aws account was enrolled in the beta testing of the new "Interface Endpoint for S3". The interface endpoint was activated alongside the Gateway Endpoint.

The terraform aws provider of course did not know how to distinguish between the two. Indeed it is expected that only one aws S3 endpoint is returned.

Anyone in this beta testing program might run into the same issue. It's not clear whether both endpoints will be allow to be activated at the same time when the s3 interface endpoint feature is launched to production. This would determine whether the aws provider needs to filter the desired endpoint.

[tony-kerz]

@jonathancolby-olx did you ever figure out how to filter this? i'm getting the same thing, and having issues attempting to filter the ghost endpoint...

update:

eventually i landed on the following which worked:

data "aws_vpc_endpoint_service" "s3" {
  count   = local.s3.switch
  service = "s3"
  filter {
    name   = "vpc_id"
    values = [aws_vpc.this.id]
  }
}

i swear there were no other s3 endpoints in the entire account tho 🤷

[joncolby]

@tony-kerz - see my last comment above. I recommend checking the vpc endpoints using the aws-cli. It may not be visible in the aws web console. check the "type" in the output. Make sure you do not have an endpoint with "interface". Maybe aws rolled this out to your account? Maybe post some error output. It's hard to say for sure what's going on in your situation.

Our aws account was enrolled in the beta testing of the new "Interface Endpoint for S3". The interface endpoint was activated alongside the Gateway Endpoint.

[yermulnik]

Seems like the S3 Privatelink was rolled out earlier today and hence this issue became actual for others
https://aws.amazon.com/ru/blogs/aws/aws-privatelink-for-amazon-s3-now-available/

[tombasche]

Confirmed with AWS Support several minutes ago; the rollout of S3 Privatelink is the culprit and the solution seems to be to unenroll from it.

[dimay7]

How do you unenroll?

[tombasche]

In our specific case, the S3 service team were the ones that created the PrivateLink's, so only they can remove it by raising a support ticket. Otherwise there is apparently an option to unenroll in the VPC console

Although perhaps it's quicker to simply update to the latest Terraform VPC module, as that appears to have been released now

[bryantbiggs]

yes, if you can @tombasche that is probably the easiest route and more future proof. note though, the Terraform AWS provider min version has been bumped now as part of this change - so if you can go up to v3.10.0 of the AWS provider then upgrading the VPC module would be the ideal route

[KA-ROM]

If you're not using the Terraform VPC module (like us who are using our own), it was enough of a fix to use AWS provider version 3.0 or higher, and add service_type to the endpoint's configuration.

data "aws_vpc_endpoint_service" "s3" {
  service      = "s3"
  service_type = "Gateway"
}

[shabir61]

how do we do this in terraform 11 - which goes only provider.aws: version = "~> 2.59"

[dgonzalez]

You can't. You need to upgrade to Terraform 0.13 at least.

[mijho]

@shabir61 @dgonzalez you can pin to version = "~> 3.10" in terraform 0.12 too, if you are making the jump from 0.11 it'd be a good idea to update fully but 0.12 is compatible at least.

[dgonzalez]

I am trying that but will need to create yet another provider as we have an RDS cluster that the plan for newer versions of the provider tries to recreate (I know, right?). I am planning to upgrade to 0.14 soon but this just hit us in the floating line: we cannot modify our infrastructure using Terraform at the moment.

[DJRH]

Adding to @KA-ROM 's comment, we were looping over different services with a for_each, so the following did the trick for us:

data "aws_vpc_endpoint_service" "service" {
  for_each = local.services
  service  = each.key
  service_type = each.key == "s3" ? "Gateway" : null
}

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.