Error: Cannot get key for artifact location, because it is invalid

[mprzygrodzki]

Hello,

I'm trying to build workflow based on Terraform Automation from official templates. The main thing i've changed it's support to use Terragrunt instead Terraform but that doesn't matters i guess. Anyway workflow is failing due to following error:

time="2021-09-02T11:11:54.228Z" level=info msg="Main container completed"
time="2021-09-02T11:11:54.228Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2021-09-02T11:11:54.228Z" level=info msg="No output parameters"
time="2021-09-02T11:11:54.228Z" level=info msg="Saving output artifacts"
time="2021-09-02T11:11:54.228Z" level=info msg="Staging artifact: terraform-plan"
time="2021-09-02T11:11:54.228Z" level=info msg="Staging /home/terraform/terraform/staging/vpc from mirrored volume mount /mainctrfs/home/terraform/terraform/staging/vpc"
time="2021-09-02T11:11:54.228Z" level=info msg="No compression strategy needed. Staging skipped"
time="2021-09-02T11:11:54.228Z" level=error msg="executor error: key unsupported: cannot get key for artifact location, because it is invalid"
time="2021-09-02T11:11:54.228Z" level=info msg="Annotating pod with output"
time="2021-09-02T11:11:54.275Z" level=info msg="Patch pods 200"
time="2021-09-02T11:11:54.293Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label \"io.kubernetes.container.name\"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=apps --filter=label=io.kubernetes.pod.nam
e=terragrunt-vpc-staging-2298416500"
time="2021-09-02T11:11:54.318Z" level=info msg="listed containers" containers="map[init:{09db411f2ed77cecbeb33a32ab0d5e424f59c4269a977185be30e2c3ea6ba17d Exited {0 63766177884 <nil>}} main:{a39b0e23fbeed78e270197c5ff42e9172bea5d38cef566f04cb401b14
8cef6c8 Exited {0 63766177887 <nil>}} wait:{4b48729b32568db053526c67f8f623aaf22b2cea8213099133946073ede7d70a Up {0 63766177887 <nil>}}]"
time="2021-09-02T11:11:54.318Z" level=info msg="Killing sidecars [\"init\"]"
time="2021-09-02T11:11:54.337Z" level=info msg="Get pods 200"
time="2021-09-02T11:11:54.337Z" level=info msg="docker kill --signal TERM 09db411f2ed77cecbeb33a32ab0d5e424f59c4269a977185be30e2c3ea6ba17d"
time="2021-09-02T11:11:54.358Z" level=error msg="`docker kill --signal TERM 09db411f2ed77cecbeb33a32ab0d5e424f59c4269a977185be30e2c3ea6ba17d` failed: Error response from daemon: Cannot kill container: 09db411f2ed77cecbeb33a32ab0d5e424f59c4269a9771
85be30e2c3ea6ba17d: Container 09db411f2ed77cecbeb33a32ab0d5e424f59c4269a977185be30e2c3ea6ba17d is not running\n"
time="2021-09-02T11:11:54.359Z" level=warning msg="Ignored error from 'docker kill --signal TERM': Error response from daemon: Cannot kill container: 09db411f2ed77cecbeb33a32ab0d5e424f59c4269a977185be30e2c3ea6ba17d: Container 09db411f2ed77cecbeb33
a32ab0d5e424f59c4269a977185be30e2c3ea6ba17d is not running"
time="2021-09-02T11:11:54.359Z" level=info msg="[docker wait 09db411f2ed77cecbeb33a32ab0d5e424f59c4269a977185be30e2c3ea6ba17d]"

Here's how the template looks like:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  namespace: apps
  annotations:
    workflows.argoproj.io/description: This workflow run a basic Terraform plan.
    workflows.argoproj.io/maintainer: '@descrepes'
    workflows.argoproj.io/tags: terraform
    workflows.argoproj.io/version: '>= 2.9.0'
  name: terragrunt-vpc-staging
spec:
  entrypoint: update
  arguments:
    parameters:
    - name: terraform-docker-image
      value: "xxxxxxxxxx.dkr.ecr.eu-central-1.amazonaws.com/terraform:1.0.15"
    - name: terraform-script-repository
      value: "git@bitbucket.org:testcloud/test-infra.git"
  templates:
    - name: plan
      inputs:
        artifacts:
          - name: terraform
            path: /home/terraform
            git:
              repo: "{{workflow.parameters.terraform-script-repository}}"
              sshPrivateKeySecret:
                name: github-creds
                key: ssh-private-key
              insecureIgnoreHostKey: true
              depth: 1
      script:
        imagePullPolicy: "Always"
        image: "{{workflow.parameters.terraform-docker-image}}"
        command:
          - sh
          - -c
          - |-
            ls -la
            mkdir -p /home/terraform/output
            cd terraform/staging/vpc
            terragrunt init -input=false
            terragrunt plan -parallelism=2 -input=false -no-color -out=/home/terraform/output/tfclientsplan >> /tmp/terraform-change.log
            cat /tmp/terraform-change.log
      outputs:
        artifacts:
          - name: terraform-plan
            path: /home/terraform/terraform/staging/vpc
            archive:
              none: {}
          - name: terraform-log
            path: /tmp/terraform-change.log
            archive:
              none: {}

    - name: apply
      inputs:
        artifacts:
        - name: terraform-plan
          path: /home/terraform
      serviceAccountName: argo          
      script:
        imagePullPolicy: "Always"
        image: "{{workflow.parameters.terraform-docker-image}}"
        command: ["sh"]
        source: |
          cd terraform/staging/vpc
          terragrunt apply -input=false -parallelism=2 -no-color /home/terraform/output/tfclientsplan

    - name: approve
      suspend: {}

    - name: update
      dag:
        tasks:
          - name: plan
            template: plan
            outputs:
              artifacts:
                - name: terraform-plan
                  path: /home/terraform/terraform/staging/vpc
          - name: approve
            dependencies: [plan]
            template: approve
          - name: apply
            template: apply
            dependencies: [plan, approve]
            arguments:
              artifacts:
              - name: terraform-plan
                from: "{{tasks.plan.outputs.artifacts.terraform-plan}}"

Application is deployed via Helm Chart on K8s, and artifacts are enabled:

artifactRepository:
  # archiveLogs will archive the main container logs as an artifact
  archiveLogs: false
  s3:
    # Note the `key` attribute is not the actual secret, it's the PATH to
    # the contents in the associated secret, as defined by the `name` attribute.
    accessKeySecret:
      name: artifactory-s3
      key: accesskey
    secretKeySecret:
      name: artifactory-s3
      key: secretkey
    insecure: true
    bucket: testcloud-argo-workflows
    endpoint: s3.amazonaws.com
    region: eu-central-1
    useSDKCreds: true

Can someone check and give me an advice what might we wrong and why it's failing ?