Error when using a template as the policy for an s3 bucket

[dansteen]

When I try to use a template as the policy for my s3 bucket:

resource "template_file" "woofound-policy" {
    filename = "${path.module}/s3-policy.tpl"

    vars {
        env = "${var.env}"
        iam = "${var.iam}"
    }
}

resource "aws_s3_bucket" "woofound" {
  bucket = "woofound-images-${var.env}"
  policy = "${template_file.woofound-policy.rendered}"
}

I get the following message (not an error) when I run "terraform plan --module-depth=1":

+ module.woofound_stag.aws_s3_bucket.woofound
    acl:              "" => "private"
    bucket:           "" => "woofound-images-stag"
    force_destroy:    "" => "0"
    hosted_zone_id:   "" => "<computed>"
    policy:           "" => "Error parsing JSON: invalid character '$' looking for beginning of value"
    region:           "" => "<computed>"
    website_domain:   "" => "<computed>"
    website_endpoint: "" => "<computed>"

However, when I run "terraform apply" everything works fine, and the policy is correctly set on S3.

[toddmichael]

Just ran into same issue. Been racking my brain trying to get plan to work until finding this bug. Ran apply and it worked. Also doing aws_s3_bucket resource. Terraform v0.6.3. Thanks for saving my evening @dansteen.

[jen20]

👍 Just ran into this as well. I'll take a look to see if I can spot anything obviously wrong here.

[ColinHebert]

Just a wild guess, is it due to the StateFunc (here normalize json) being applied too early?

In

terraform/helper/schema/schema.go

Line 922 in 8fdffb6

if schema.StateFunc != nil {

[dwradcliffe]

@radeksimko @catsby Any ideas here?

[mrwacky42]

A partial workaround is to first render the template to your tfstate with something like:

terraform apply --target template_file.woofound-policy

Then you can terraform plan to see the rest of the plan and it will not bellyache.

[chiefy]

👍 just ran into this, made me think my JSON was bad, head scratched for a while.

[mrwilby]

Also observed the above problem today.

As an aside, not sure if it's related, but if I:

plan
apply -> creates s3 bucket for the first time
plan

The second plan results in a spurious looking "change":

policy: "{\"Statement\":[{\"Action\":[\"s3:ListBucket\",\"s3:GetBucketLocation\"],\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"arn:aws:iam::yyyy:root\"},\"Resource\":\"arn:aws:s3:::xxx\",\"Sid\":\"x\"}],\"Version\":\"2012-10-17\"}" => "{\"Statement\":[{\"Action\":[\"s3:GetBucketLocation\",\"s3:ListBucket\"],\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"arn:aws:iam::yyyy:root\"},\"Resource\":[\"arn:aws:s3:::xxx\"],\"Sid\":\"x\"}],\"Version\":\"2012-10-17\"}"

The only difference is the bracketing around "Resource"

[jszwedko]

@mrwilby we've seen that as well, but I think it is unrelated to this issue. You can workaround your issue by formatting the JSON in the exact way that AWS formats it in its response (i.e. you should use brackets around your Resource too). It looks like Terraform just does a string comparison so rather than semantically comparing the JSON (which would admittedly be pretty difficult).

[mrwilby]

@jszwedko Thanks. I actually have the JSON with the brackets. Something inside the terraform codebase (I assume) is doing the conversion with/without brackets.

[jszwedko]

Oops! I reversed what you had above, it looks like AWS is storing it sans-brackets -- you can verify by doing a aws s3api get-bucket-policy.

[scheffield]

Any updates on this issue?

[dansteen]

So, I did a compare on what amazon thinks is the policy, and the policy that terraform is pushing, and I got the following:

Terraform:
{
  "Version": "2008-10-17",
  "Id": "<id>",
  "Statement": [
    {
      "Sid": "Stmt1403128717600",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::xxxxx:role/prod-iam"
      },
      "Action": [
        "s3:PutBucketAcl",
        "s3:GetBucketAcl",
        "s3:PutBucketPolicy",
        "s3:GetObjectAcl",
        "s3:PutBucketCORS",
        "s3:RestoreObject",
        "s3:ListBucket",
        "s3:DeleteObject",
        "s3:PutObjectAcl",
        "s3:GetObject",
        "s3:ListBucketMultipartUploads",
        "s3:ListMultipartUploadParts",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket>",
        "arn:aws:s3:::<bucket>/*"
      ]
    }
  ]
}

and

AWS:
{
  "Version": "2008-10-17",
  "Id": "<id>",
  "Statement": [
    {
      "Sid": "Stmt1403128717600",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::xxxxxxxx:role/prod-iam"
      },
      "Action": [
        "s3:PutBucketAcl",
        "s3:GetBucketAcl",
        "s3:PutBucketPolicy",
        "s3:GetObjectAcl",
        "s3:PutBucketCORS",
        "s3:RestoreObject",
        "s3:ListBucket",
        "s3:DeleteObject",
        "s3:PutObjectAcl",
        "s3:GetObject",
        "s3:ListBucketMultipartUploads",
        "s3:ListMultipartUploadParts",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket>/*",
        "arn:aws:s3:::<bucket>"
      ]
    }
  ]
}

Notice that the Resource elements are in the reverse order! Once I adjusted the ordering I no longer have this issue.

I do wish we could have a more semantic compare though...

v0.6.13

EDIT: The issue with the above fix is that the ordering returned by AWS is not always the same. So in cases where you are using modules, and have multiple versions, it may not be possible to specify the correct ordering for each instance of the module.

[radeksimko]

@dansteen I think your last comment is unrelated to the original bug report with Error parsing JSON.
What you described in the last comment is tracked under #4948

I have a feeling the original report is a core bug that affects other resources using template_file too: #5462

Hopefully will be fixed in #4961

[eedwardsdisco]

getting similar issue with using a template file's rendered output for the policy field for a kms key (I even tried adding the depends_on to see if it would help, it didn't):

resource "aws_kms_key" "kms_key" {
    description = "${var.kms_key_description}"
    key_usage = "${var.kms_key_usage}"
    policy = "${template_file.key_policy.rendered}"
    deletion_window_in_days = "${var.kms_key_deletion_window_in_days}"
    is_enabled = "${var.kms_key_is_enabled}"
    enable_key_rotation = "${var.kms_key_enable_key_rotation}"

    depends_on = ["template_file.key_policy"]
}

+ aws_kms_key.kms_key
    arn:                     "" => "<computed>"
    deletion_window_in_days: "" => "30"
    description:             "" => "KMS Key for ..."
    enable_key_rotation:     "" => "0"
    is_enabled:              "" => "1"
    key_id:                  "" => "<computed>"
    key_usage:               "" => "ENCRYPT_DECRYPT"
    policy:                  "" => "Error parsing JSON: invalid character '$' looking for beginning of value"

[dansteen]

@radeksimko Sounds reasonable. Looking forward to #4961!

[ahaynssen]

I'm still hitting this error on 0.6.16. I'm trying to pass my "Resource" value to a template file and hitting the "Error parsing JSON: invalid character '$' looking for beginning of value"

[meyerbro]

I'm also getting this error on v0.6.16, it always ask to change but there's no change on the policy.

[meyerbro]

The solution at the moment was to split each Action in one statement and keep the "Sid": "", on each... Also, removed all unnecessary brackets.

[kyl191]

@ahaynssen Workaround I found was to add the template_file first, run terraform apply, then use the .rendered wherever I needed to.

[mitchellh]

This is fixed in #4961.

[jszwedko]

@mitchellh I'm probably missing something, but it looks like the PR you linked to (#4961) as well as the PR that that one links to as superseding (#6430) were both closed without merging. Were these rebased onto master as other commits?

I also see some commits above by @phinze referencing this issue, but these appear to not be yet merged (#7204).

[jszwedko]

To close the loop on this, it is no longer a problem if you use the template data source, e.g.:

# main.tf
provider "aws" {
  region = "us-west-1"
}

data "template_file" "init" {
  template = "${file("s3-policy.tpl")}"
}

resource "aws_s3_bucket" "woofound" {
  bucket = "12345-soechu-images"
  policy = "${data.template_file.init.rendered}"
}

echo '{}' > s3-policy.tpl

...
+ aws_s3_bucket.woofound
    acceleration_status: "<computed>"
    acl:                 "private"
    arn:                 "<computed>"
    bucket:              "12345-soechu-images"
    force_destroy:       "false"
    hosted_zone_id:      "<computed>"
    policy:              "{}"
    region:              "<computed>"
    request_payer:       "<computed>"
    website_domain:      "<computed>"
    website_endpoint:    "<computed>"
...

It is still a problem with the normal template provider, but that has been deprecated so I would consider not an issue.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.