Can't pull docker image from GitHub Packages registry from GitHub Actions workflow - unauthorized error

[gomisha]

Select Topic Area

Question

Body

I have the following GitHub Actions that I'm trying to run but I keep getting an "unauthorized error" when the Action is trying to pull the Docker image from GitHub Packages registry in the repo-sync-gomisha step.

name: Sync Repos

on:
  workflow_dispatch:

permissions:
  packages: read
  actions: write
  checks: write
  contents: write
  deployments: write
  id-token: write
  issues: write
  discussions: write
  pages: write
  pull-requests: write
  repository-projects: write
  security-events: write
  statuses: write

jobs:
  repo-sync-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          persist-credentials: false
          token: ${{ secrets.PAT }}
      - name: repo-sync-gomisha
        uses: gomisha/github-sync@v2.3.1
        with:
          source_repo: "https://github.com/gomisha/repo-sync-test-public"
          source_branch: "main"
          destination_branch: "main"
          github_token: ${{ secrets.PAT }}

I keep getting the following error when trying to pull a Docker image from GitHub Packages registry. The image I'm trying to pull is a forked GitHub action, which I'm able to push to the registry.

Error response from daemon: Head "https://ghcr.io/v2/gomisha/github-sync/manifests/v2.3.1": unauthorized

These are all the failed attempts from the public repo that's trying to use the forked GitHub action:

https://github.com/gomisha/repo-sync-test-private/actions

This is the GitHub Actions workflow file I'm using to pull the docker image of the reusable GitHub Action:

https://github.com/gomisha/repo-sync-test-private/blob/main/.github/workflows/repo-sync.yml

Here is a specific error when I run this workflow:

Pull down action image 'ghcr.io/gomisha/github-sync:v2.3.1'
  /usr/bin/docker pull ghcr.io/gomisha/github-sync:v[2](https://github.com/gomisha/repo-sync-test-private/actions/runs/4060550950/jobs/6989752205#step:2:2).3.1
  Error response from daemon: Head "https://ghcr.io/v2/gomisha/github-sync/manifests/v2.3.1": unauthorized
  Warning: Docker pull failed with exit code 1, back off 4.731 seconds before retry.
  /usr/bin/docker pull ghcr.io/gomisha/github-sync:v2.3.1
  Error response from daemon: Head "https://ghcr.io/v2/gomisha/github-sync/manifests/v2.[3](https://github.com/gomisha/repo-sync-test-private/actions/runs/4060550950/jobs/6989752205#step:2:3).1": unauthorized
  Warning: Docker pull failed with exit code 1, back off 7.07 seconds before retry.
  /usr/bin/docker pull ghcr.io/gomisha/github-sync:v2.3.1
  Error response from daemon: Head "https://ghcr.io/v2/gomisha/github-sync/manifests/v2.3.1": unauthorized
Error: Docker pull failed with exit code 1

I'm not sure why I'm getting this unauthorized error.

I can successfully build the Docker image of the GitHub action and upload it to GitHub Package registry. It's only when I try and pull that image from another GitHub Action that I get the above error.

This is the forked GitHub Action that I can successfully build the Docker image and deploy to GitHub Packages registry:

https://github.com/gomisha/github-sync

Here is the last successful deployment of this GitHub Action to the GitHub Package registry.

Solutions Attempted

Things I have tried (but weren't successful):

• using a Personal Access Token that has the correct permissions to read/write to the GitHub Package registry

• using GITHUB_TOKEN instead of personal access token and giving it very permissive settings so it would have permission to access the GitHub Package registry:

permissions:
  packages: read
  actions: write
  checks: write
  contents: write
  deployments: write
  id-token: write
  issues: write
  discussions: write
  pages: write
  pull-requests: write
  repository-projects: write
  security-events: write
  statuses: write

• going to package settings: https://github.com/orgs/gomisha/packages/container/github-sync/settings
  and adding https://github.com/gomisha/repo-sync-test-private/ repo to have access to the package

• pulling this image locally to my machine
  docker login ghcr.io
docker pull ghcr.io/gomisha/github-sync:v2.3.1
  So the image is there in the registry and I can even pull it locally, but I can't pull it from my GitHub Actions workflow for some reason.

• going to package settings: https://github.com/orgs/gomisha/packages/container/github-sync/settings
  and adding https://github.com/gomisha/repo-sync-test-private/ repo to have access to the package

[OR13]

I hit same issue, I filed: actions/hello-world-docker-action#15

The problem appears to be that the action.yml is processed by the runner before any ability to ensure it will be able to pull from the container registry.

[leandromsft]

I have the same issue here :(
Any workaround?

[joshjohanning]

I think the main issue is that GitHub doesn't natively support pulling in containers in docker actions from a private feed. Some workarounds:

1. make your registry public
2. pre-authenticate / have the container available using a pre-job step on a self-hosted runner
3. use a container or service job where you can specify credentials
4. use a composite action to authenticate, pull, and run the docker image yourself via command line
5. Do something like this where you expect the env vars to be there in order to be able to pull container from the private feed
6. reference the dockerfile directly when using docker actions

runs.image
Required The Docker image to use as the container to run the action. The value can be the Docker base image name, a local Dockerfile in your repository, or a public image in Docker Hub or another registry. To reference a Dockerfile local to your repository, the file must be named Dockerfile and you must use a path relative to your action metadata file. The docker application will execute this file.

---

Edit: see my comment for 2 other options.

[sumanth-kondru]

Having same issue , when I call same action from market place action it pulls image from docker hub and working , when I clone the same repo to org goin account and point to jfrog image it’s failing with authentication needed though I am passing the auth step any one found fix ?

[tremblaysimon]

Workaround are ok, but in the end GitHub Actions should support it natively to be able to use a private regsitry requiring authentication.

[joshjohanning]

I have two more workarounds / solutions to share:

7. Using a composite Action you can sort of bury the Docker action by making it a local action (cloning the docker action repo and running it from a local directory). Then, it won't auth to Docker at the beginning of the job and you can auth to docker yourself before that action runs. There are 2 examples of this in this repo.
8. Use Container Hooks
   • Example code
   • Example action being ran
   • Using container hooks with self-hosted runners would allow you to customize how docker actions are referenced, therefore being able to inject docker auth logic without the end-user having to do it themselves in the action workflow.

For those finding this comment later, I wrote on points 7 and 8 in a blog post here.

[tremblaysimon]

Your point 7 could work with an image already built but it will not work with a Dockerfile (when the param image is like image: "Dockerfile").

[pkieltyka]

Please fix :) it's super unintuitive, and after some Googling, I see many folks are having the same unauthorized issue on public repos with public images. I'm still working through it to try to make it work on https://github.com/goware/jwtutil .. it seems my ghcr.io/goware/jwtutil package is "private", even though the repo is public. It's too bad as this is a very elegant integration I'm sure many GH engineers worked hard to provide, yet it doesn't work if people can't use it easily..

[jhelbig]

this is something that gitlab has had working for about a year now and has a positively huge impact on pipeline development. Not only am I running into this exact problem, but it's been well over a year since it's been brought up AND nobody from github has even responded.

At this point with all the feature bloat elsewhere and lack of support tooling to make developers jobs easier, I can't justify recommending github as a solution to another development team and could easily justify the hours it would take to migrate to gitlab depending on the size of the account.

I'm not mad, just disappointed in the product. It can't even deliver on the basic necessary features that a direct competitor could perform a year ago with what's likely a budget 3/4 the size.

[zachspar]

Any updates on this issue? It's unacceptable that GH doesn't have this built-in, especially when using GHCR.

[sabralod]

Solved the issue by using the 2. workaround with my local development environment, like:

• dockerized github-action in private repo with image on ghcr.io
• action is used by another repo, which got read permissions to the specific ghcr.io package/image
• added ACTIONS_RUNNER_HOOK_JOB_STARTED=exact-path-to-runner-or-script-folder/ghcr-login.sh to local runner's .env file and
• created simple script ghcr-login.sh with the command: echo "xxx" | docker login ghcr.io -u xxx --password-stdin

for now I'm fine with it; cause I can develop my action with private images like the way the action would be consumed.

---

Is it possible to inject / set the ACTIONS_RUNNER_HOOK_JOB_STARTED env var from workflow side?

[joshjohanning]

Awesome! No, sadly you cannot set it from the workflow side :(. That's why the solution only supports self-hosted runners, you have to set it on the .env file on the runner such that the environment variable exists before the job starts.

[zachspar]

@joshjohanning are there any plans to build a feature to pull private package repos in workflow job steps? Particularly from GHCR?

[joshjohanning]

@zachspar sadly nothing on the public roadmap for this :(

[github-actions[bot]]

🕒 Discussion Activity Reminder 🕒

This Discussion has been labeled as dormant by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as out of date at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This dormant notification will only apply to Discussions with the Question label. To learn more, see our recent announcement.

Thank you for helping bring this Discussion to a resolution! 💬

[iamstarkov]

ran in the same issue with crown's docker based action crowdin/crowdin-cli#827

[marcinkubica]

I'm on self-hosted runners and have exactly the same problem. No reasonable solution there apart from Github to start respecting the permissions model where we already allow packages to be pulled.

And we allow it in two places - permissions: in the workflow and the package settings allowing to read for a set repo.

[joshjohanning]

permissions are just what types of resources inside of a repository that a particular workflow can access. The permissions on the package could potentially work, but there are still no "docker login" running ahead of time to authenticate to the container registry in order for the container to be pulled.

See another response here https://github.com/orgs/community/discussions/45981#discussioncomment-10163954

[mhemken-vts]

I've configured all the relevant permissions but still, github actions is unauthorized at ghcr.io?

My use case:

• composite workflows are written as YAML and thus cannot be tested or run locally
• I want to write my CI logic with Go and run as a Container Action
• I follow the documentation to build and push my container to ghcr.io
• I define a workflow to reference that container
• the Pull step happens before any of the user defined steps
• thus I cannot authenticate with ghcr.io from github actions

Trying to get this to work has been a miserable user experience.

[joshjohanning]

It would have to be a public ghcr image. Assuming yours is private, you would have to do one of the workarounds mentioned here to provide authentication.

TLDR, to provide authentication, you can:

1. pre-auth in self-hosted runner environment ahead of time
2. use a pre-step job/task in Actions to run logic before every job
3. use a composite action to hide the docker action and pre-auth before it runs - this would work with github-hosted runners

From an open source / marketplace perspective, having public actions gated around requiring docker creds would be somewhat of a slipper slope, but I do agree for enterprise customers it makes things a touch trickier.

[mhemken-vts]

<vent><rant>

Thanks for the reply, @joshjohanning , I appreciate the documentation you've provided in this thread. It is very helpful. Authentication is a hard problem and I definitely see how it's even harder at GitHub scale.

Unfortunately the reality of the user experience for a quick and dirty proof of concept is that it's bad. I'm trying to get something working to show the team it is possible. My use case is a pre-alpha-v0.0.1 candidate throw-away hello world example. I don't have the time to set up the authentication scaffolding for self-hosted runners. I have to tip-toe around new things because my team is very skeptical of changing things that already work. I have not built the necessary consensus with the team to put it on the roadmap as a solution to issues that the team is happy to ignore for now but have been and will continue to be nagging problems.

One would think a GitHub git repo could have an easy path to authenticate with a GitHub ghcr.io package repo that's created by the same user under the same account with the same credentials.

"Just make it public" is not a reasonable default for users doing work for an employer. I'd even argue it is a bad default for users working on their own but that's besides the point.

Maybe I'll try for option 3. Or use my personal GitHub account. Neither option compares well with what would be the path of least surprise which would be for my git repo to auth with its eponymous ghcr.io package repo.

</vent></rant>

[github-actions[bot]]

🕒 Discussion Activity Reminder 🕒

This Discussion has been labeled as dormant by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as out of date at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This dormant notification will only apply to Discussions with the Question label. To learn more, see our recent announcement.

Thank you for helping bring this Discussion to a resolution! 💬

[zachspar]

Still an issue.