GCE VM fails to pull modified Datalab image in GCR

[vochicong]

I've followed the instruction in "Using your modified Datalab image on GCE" and found that the VM created by Datalab in GCE failed to pull images from gcr.io/${PROJECT_ID}.

With my modified Datalab image

# On Cloud Shell 1
$ datalab create ${INSTANCE_NAME} --image-name=gcr.io/${PROJECT_ID}/datalab --zone=${ZONE}
# On Cloud Shell 2
$ gcloud compute ssh datalab@${INSTANCE_NAME} --zone=${ZONE}
# Inside the Datalab VM
$ watch -d docker images # my Datalab image never show up
REPOSITORY                             TAG                 IMAGE ID            CREATED             SIZE
gcr.io/google_containers/pause-amd64   3.0                 99e59f495ffa        13 months ago       746.9 kB
gcr.io/google_containers/fluentd-gcp   1.18                e0be8d052951        15 months ago       411.4 MB
$ docker pull gcr.io/${PROJECT_ID}/datalab # actually failed

With gcr.io/cloud-datalab/datalab:commit-latest-master-build it works, as a matter of course.

# On Cloud Shell 1
$ datalab create ${INSTANCE_NAME}  --image-name gcr.io/cloud-datalab/datalab:commit-latest-master-build --zone=${ZONE}
# On Cloud Shell 2
$ gcloud compute ssh datalab@${INSTANCE_NAME} --zone=${ZONE}
# Inside the Datalab VM
$ docker images # we can see the datalab image
REPOSITORY                             TAG                          IMAGE ID            CREATED             SIZE
gcr.io/cloud-datalab/datalab           commit-latest-master-build   35aab0a7073a        7 hours ago         2.665 GB
gcr.io/google_containers/pause-amd64   3.0                          99e59f495ffa        13 months ago       746.9 kB
gcr.io/google_containers/fluentd-gcp   1.18                         e0be8d052951        15 months ago       411.4 MB

My workaround is to use Docker Hub, e.g. the following works.

$ datalab create ${INSTANCE_NAME} --image-name=vochicong/datalab-nlp:latest --zone=${ZONE}

So it seems that we need some fixes so that the VM created by datalab create can pull GCR images in the same GCP project.

[vochicong]

More details about the modified Datalab image is available at https://github.com/vochicong/datalab-nlp/tree/dev#run-on-gcloud-datalab .

[chmeyers]

The most likely difference here is in the permissions of the docker image.

gcr.io/cloud-datalab/datalab:commit-latest-master-build is publicly accessible, and I'm assuming vochicong/datalab-nlp:latest is public too, but gcr.io/${PROJECT_ID}/datalab probably isn't.

The datalab VM uses the COS Image's functionality (https://cloud.google.com/container-optimized-os/docs/) to pull the docker images, so I'm not sure if it uses "docker pull" (no auth) or "gcloud docker -- pull" (auth). The VM generally runs with the "compute" service account for a project which usually has "Editor" permissions, so it should have the ability to pull containers for the same project.