Private repository with Docker for Mac

[armetiz]

Hi there!

I'm trying to install private repository using composer Docker image.
I'm running Docker for mac.

docker run --rm --interactive --tty \
    --volume $PWD:/app \
    --volume $SSH_AUTH_SOCK:/ssh-auth.sock \
    --volume /etc/passwd:/etc/passwd:ro \
    --volume /etc/group:/etc/group:ro \
    --user $(id -u):$(id -g) \
    --env SSH_AUTH_SOCK=/ssh-auth.sock \
    --workdir /app \
    composer:latest update

After the following error, I've added /private/tmp as shared path.

docker: Error response from daemon: Mounts denied:
The path /private/tmp/com.apple.launchd.FYsdWrYgET/Listeners
is not shared from OS X and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> File Sharing.
See https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.

Now; I'm getting the following error :

Cannot create cache directory /tmp/cache/repo/https---packagist.org/, or directory is not writable. Proceeding without cache
Cannot create cache directory /tmp/cache/files/, or directory is not writable. Proceeding without cache
Loading composer repositories with package information

[RuntimeException]
/tmp/cache/vcs does not exist and could not be created.

I'm not mastering Docker at all.
Do you know why /tmpis ready only ?

Regards;

[alcohol]

Can you try starting the container with bash instead, and see what the ownership and permissions on /tmp are?

[armetiz]

Hi @alcohol

➜ booking-engine git:(master) ✗ docker run --rm --interactive --tty
--volume $PWD:/app
--volume $SSH_AUTH_SOCK:/ssh-auth.sock
--volume /etc/passwd:/etc/passwd:ro
--volume /etc/group:/etc/group:ro
--user $(id -u):$(id -g)
--env SSH_AUTH_SOCK=/ssh-auth.sock
--workdir /app
composer:latest bash
bash-4.3$ ls -alh /tmp
total 24
drwxrwxrwt 3 root root 4.0K Aug 10 17:45 .
drwxr-xr-x 61 root root 4.0K Aug 12 21:07 ..
-rw-r--r-- 1 root root 13 Aug 10 17:45 .htaccess
drwxr-xr-x 2 root root 4.0K Aug 10 17:45 cache

As I understand; I've changed the Docker user's id (501), which is not root's id ?
On macOS, sharing SSH Agent seems to be difficult..

[alcohol]

Changing the user id should be fine. That cache directory should not be owned by root though. Unless root previously created it. I'll have to have a look at our base image..

[alcohol]

Ah, my bad. I think that is a remnant of the composer install. It should be cleaned up though, definitely not supposed to be there.

[alcohol]

This should fix it once it gets merged. docker-library/official-images#3321

[armetiz]

Hi @alcohol
Thank you for your reactivity!

Should I wait for Docker hub to build new image or something like that?

[alcohol]

Once that above linked PR gets merged, the hub should have the latest image available.

You can then run docker pull composer to make sure you have the latest versions locally.

[armetiz]

Thanks for tips! I'm subscribing on your PR to get notified when merged. I'll close this issue when OK.

[armetiz]

Hi @alcohol
Your PR was merged.

With the latest composer image;

docker run --rm --interactive --tty \
    --volume $PWD:/app \
    --volume $SSH_AUTH_SOCK:/ssh-auth.sock \
    --volume /etc/passwd:/etc/passwd:ro \
    --volume /etc/group:/etc/group:ro \
    --user $(id -u):$(id -g) \
    --env SSH_AUTH_SOCK=/ssh-auth.sock \
    composer update

Loading composer repositories with package information

[RuntimeException]
Failed to execute git clone --mirror 'git@gitlab.private.com/repo.git' '/tmp/cache/vcs/git-gitlab.private.com/repo.git/'
Cloning into bare repository '/tmp/cache/vcs/git-gitlab.private.com/repo.git'...
No user exists for uid 501
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

Can you tell me about your configuration ? Notice that I'm running Docker for Mac (latest versions).

Regards;

[alcohol]

Can you return the output of the following commands?

docker run --rm --interactive --tty \
    --volume $PWD:/app \
    --volume $SSH_AUTH_SOCK:/ssh-auth.sock \
    --volume /etc/passwd:/etc/passwd:ro \
    --volume /etc/group:/etc/group:ro \
    --user $(id -u):$(id -g) \
    --env SSH_AUTH_SOCK=/ssh-auth.sock \
    composer ls -al /tmp

docker run --rm --interactive --tty \
    --volume $PWD:/app \
    --volume $SSH_AUTH_SOCK:/ssh-auth.sock \
    --volume /etc/passwd:/etc/passwd:ro \
    --volume /etc/group:/etc/group:ro \
    --user $(id -u):$(id -g) \
    --env SSH_AUTH_SOCK=/ssh-auth.sock \
    composer id

docker run --rm --interactive --tty \
    --volume $PWD:/app \
    --volume $SSH_AUTH_SOCK:/ssh-auth.sock \
    --volume /etc/passwd:/etc/passwd:ro \
    --volume /etc/group:/etc/group:ro \
    --user $(id -u):$(id -g) \
    --env SSH_AUTH_SOCK=/ssh-auth.sock \
    composer cat /etc/passwd

[armetiz]

Hi @alcohol

drwxrwxrwt 2 root root 4096 Aug 14 17:55 .
drwxr-xr-x 61 root root 4096 Aug 15 12:21 ..

uid=501 gid=20(dialout)

root:x:0:0:root:/root:/bin/ash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
news:x:9:13:news:/usr/lib/news:/sbin/nologin
uucp:x:10:14:uucp:/var/spool/uucppublic:/sbin/nologin
operator:x:11:0:operator:/root:/bin/sh
man:x:13:15:man:/usr/man:/sbin/nologin
postmaster:x:14:12:postmaster:/var/spool/mail:/sbin/nologin
cron:x:16:16:cron:/var/spool/cron:/sbin/nologin
ftp:x:21:21::/var/lib/ftp:/sbin/nologin
sshd:x:22:22:sshd:/dev/null:/sbin/nologin
at:x:25:25:at:/var/spool/cron/atjobs:/sbin/nologin
squid:x:31:31:Squid:/var/cache/squid:/sbin/nologin
xfs:x:33:33:X Font Server:/etc/X11/fs:/sbin/nologin
games:x:35:35:games:/usr/games:/sbin/nologin
postgres:x:70:70::/var/lib/postgresql:/bin/sh
cyrus:x:85:12::/usr/cyrus:/sbin/nologin
vpopmail:x:89:89::/var/vpopmail:/sbin/nologin
ntp:x:123:123:NTP:/var/empty:/sbin/nologin
smmsp:x:209:209:smmsp:/var/spool/mqueue:/sbin/nologin
guest:x:405:100:guest:/dev/null:/sbin/nologin
nobody:x:65534:65534:nobody:/:/sbin/nologin
chrony:x:100:101:chrony:/var/log/chrony:/sbin/nologin
docker:x:1001:50:Docker:/home/docker:/bin/sh
dockremap:x:1000:1000:Linux User,,,:/home/dockremap:/sbin/nologin

The /etc/passwd from latest CLI is not the same as /etc/passwd from host.

[alcohol]

Right, my bad, you are using OSX.

I have noticed that they have introduced some changes to Docker for Mac. You no longer have to change user and mount the passwd and group files to avoid permission issues. What exactly else these changes introduced (or when) I am not fully aware of yet. I need to do some testing and debugging again to find out how one should run the container now.

[alcohol]

Also, on OSX, this workaround that I documented does not actually work since OSX users aren't actually stored in /etc/passwd and such (see https://superuser.com/a/191333 also).

[armetiz]

Ok; that what I was thinking.
Thank you for your time.

Wait for this issue : sharing SSH Agent feature on Docker for mac

[emnsen]

Hi @armetiz,

Did you solve the problem ?

If you can't solve the problem, remove the
--volume /etc/passwd:/etc/passwd:ro \,
--volume /etc/group:/etc/group:ro \ and
--user $(id -u):$(id -g) and

try again, i solved this way.

[alcohol]

@emnsen is correct. With recent editions of Docker for Mac, permission issues are no longer an issue. Files created by containers through volume binds will have the ownership set to the user running the container from the host. The workaround documented is only necessary on Linux. Unfortunately, I'm not sure if the same can be said for the ownership issues regarding SSH auth socket though. I think that is still problematic. But I could be mistaken.

[armetiz]

Hi both of @emnsen & @alcohol ;)

let's see what's happening

docker run --rm --interactive --tty
--volume $PWD:/app
--volume $SSH_AUTH_SOCK:/ssh-auth.sock
--env SSH_AUTH_SOCK=/ssh-auth.sock
composer update
Loading composer repositories with package information
The authenticity of host 'gitlab.acme.com (xx.xx.xx.xx)' can't be established.
ECDSA key fingerprint is SHA256:blabla/m/Cd5Zx5iD5oKuj2GmM.
Are you sure you want to continue connecting (yes/no)? yes
git@gitlab.acme.com's password:

The SSH Auth is not forwarded :(

[emnsen]

I am using a composer package using gitlab ssh via a different ssh port and it works fine

Try this @armetiz ;

dockercomposer () {
    tty=
    tty -s && tty=--tty
    docker run \
        $tty \
        --interactive \
        --rm \
        --volume $(pwd):/app \
        --volume $SSH_AUTH_SOCK:/ssh-auth.sock \
        --volume ~/.ssh/config:/etc/ssh/ssh_config \
        --volume ~/.ssh:/root/.ssh \
        --env SSH_AUTH_SOCK=/ssh-auth.sock \
        composer "$@"
}

and ~/.ssh/config ;

Host gitlab.custom.url
  IdentityFile ~/.ssh/id_rsa
  IdentitiesOnly yes
  Port 1234

[armetiz]

Hi @emnsen
Exact, it's working. But this method seems to be an ugly workaround; do not ask me why ;) docker/for-mac#410

BTW,
If you're mounting SSH keys with SSH configuration, you could remove the useless SSH Auth section.
Be sure to mount ~/.ssh with read-only permission to avoid the apocalypse.

docker run --rm --interactive --tty
--volume $PWD:/app
--volume ~/.ssh:/root/.ssh:ro
composer update

[emnsen]

Hi @armetiz
Yes it is not a good solution, but it works for now :)

[armetiz]

Hi there! I'm back ;)

One of our teammate have a passphrase.
I'm still mounting ~/.ssh to the container; composer is asking about passphrase, but the entered passphrase is not working at all...

Do you have any clue? I think the problem is coming from the TTY;
BTW, this is a simple passphrase with only alpha / digit.

[emnsen]

Hi :)

I tried to solve the problem, but i didn't.
If your friend remove the password, the problem can be solved. Reference
What does your friend think about removing the password? Because, these are private keys and encrypted keys.

[armetiz]

I don't want to dive into this debate with him ;)
Having a passphrase on private key is a personal opinion. Private keys can allow access to really huge projects; having a protection on it is not a bad opinion.

[emnsen]

:)))

Yes, this personal opinion. I just thought this would be the quickest solution :)

[armetiz]

In fact; the real problem is not the passphrase. I've tried to remove passphrase; and the problem still persist. I'm closing this issue to talk about the real problem.