automatically updating tagged images to dockerhub

[ChristophSchranz]

@ChristophSchranz quick Q: are you manually updating the dockerhub?
If so, we can discuss adding directives to
https://github.com/iot-salzburg/gpu-jupyter/blob/master/.github/workflows/main.yml
which perform that function automatically. If this is something you're interested in, open a new issue and I can help close it out. you'll just need to add your dockerhub token to the secrets of this repo.

Originally posted by @mathematicalmichael in #19 (comment)

• automatic build of the "latest" tag (already works)
• automatically build the smaller images

[mathematicalmichael]

I'm currently testing the generation of four different configs, then running the start script. Doesn't that mean I'm already automatically building the smaller images? Or are you referring to something else with respect to building the latest tag? (as in, something that's outside the github workflow)

---

Do you want it to trigger this workflow only when you push a tag (which can be used to create an associated release)? Or just run weekly? daily? On push to master? All/none/some of the above? lol

Should we automatically append these tags to the pushed docker image? Should they contain the v in version # used in the tag?
Do we want a latest docker tag? A minimal one? Other names/aliases that make it easy for people to remember?

Do we upload it to your docker? Create a new account?

---

I will add a step at the end of the workflow that will publish the versions we want. Can you:

• add DOCKER_USERNAME to secrets for this repo
• add DOCKER_PASSWORD to secrets for this repo
• create a checklist of all the names you want them to be pushed under (including username)

---

Here's something I could envision:

Local -> Cloud
git tag v0.1.2 && git push v0.1.2 will trigger job that will push repo/gpu-jupyter-xxx:0.1.2 and ...:latest for xxx = default, nods, noup, slim, (maybe minimal points to slim)

Remote -> Cloud
Someone submits PR you merge in. You then create tag/release via Github, job will trigger and push to dockerhub all the aforementioned images.

[ChristophSchranz]

That are many things to keep in mind.

I'm currently testing the generation of four different configs, then running the start script. Doesn't that mean I'm already automatically building the smaller images? Or are you referring to something else with respect to building the latest tag? (as in, something that's outside the github workflow)

Do you want it to trigger this workflow only when you push a tag (which can be used to create an associated release)? Or just run weekly? daily? On push to master? All/none/some of the above? lol

I'm refering to automatically build the dockerhub tags automatically, like shown here. Currently, only the build of the latest tag is triggered. It would be nice to trigger the build on dockerhub for all four test cases each on a special tag.

Should we automatically append these tags to the pushed docker image? Should they contain the v in version # used in the tag?
Do we want a latest docker tag? A minimal one? Other names/aliases that make it easy for people to remember?
I though of labels like v1.1_cuda-10.1_ubuntu-18.04_python-only, but the version should update only if there are significant changes (from 1.0 to 1.1 there was some restructuring). Is it okay for you to overwrite some tags, or do you want a very granular versioning of dockerhub tags?

Do we upload it to your docker? Create a new account?

Good point. I would definitely keep the repository here in iot-salzburg, but I'm open to create a new organisation on dockerhub. Do you have a suggestion?

I will add a step at the end of the workflow that will publish the versions we want. Can you:

* [ ]  add `DOCKER_USERNAME` to secrets for this repo

* [ ]  add `DOCKER_PASSWORD` to secrets for this repo

* [ ]  create a checklist of all the names you want them to be pushed under (including username)

Depends on the dockerhub decision, what is your id?

Here's something I could envision:

Local -> Cloud
git tag v0.1.2 && git push v0.1.2 will trigger job that will push repo/gpu-jupyter-xxx:0.1.2 and ...:latest for xxx = default, nods, noup, slim, (maybe minimal points to slim)

Remote -> Cloud
Someone submits PR you merge in. You then create tag/release via Github, job will trigger and push to dockerhub all the aforementioned images.

Sounds good 👍

[mathematicalmichael]

Oh interesting, I didn't know docker hub had that feature but I'm not surprised. I would be configuring it to push all tags directly from github, so you don't have to bounce between websites for changing configurations. Thanks for the clarification, I think we'll be able to disable that by the time this is done.

I kind of like the (silly) idea of seeing if gupyter is available... one letter change and your usual docker-stacks references now get GPU capabilities. lol maybe gpupyter (two letter change) is more pronounceable though.

regarding tags... I would treat it like software versioning. no breaking changes in 0.0.X, breaking changes in 0.X.0 and monumental shifts in X.0.0, such as changing the base image (upgrading ubuntu 18 to 20 for example). I don't think including the OS version as part of the tag is useful, as long tags are cumbersome. They should be short version numbers of one/two word aliases.

Basically, I'd like to follow how the jupyter/docker-stacks folks do it as closely as possible with respect to versioning, I could see going so far as to even match their release names for everything except the org name, as an addition to the versioning system we use. However, they're on 20 and we're on 18, so that's already kind of out of the picture unless we bump up our base images as well.

Definitely some decisions to be made, and I'm happy to revise any initial proposal I put together... but yah, looking forward to this undertaking. What I'd like to be able to do is allow anyone with a dockerfile that starts with FROM jupyter/minimal-notebook to simply swap out for a different image and have everything work without further changes (again that would already require switching to focal since the PPAs differ)

[mathematicalmichael]

I do like the idea of having a separate account for this so that someone visiting the hub would see only a selection of relevant images to choose from, and no personal projects.

and yeah, sorry, it is a lot to keep in mind, but good news is that once it's sorted, it'll all just work magically and silently update things in the background in a very organized way. I really enjoy seeing a CI/CD pipeline come together.

[ChristophSchranz]

I haven't thought of gupyter and gpupyter (with a silent p), these names are great! 😃 Yes there are convincing reasons to create a new account that only solves one problem. Which one do you prefer?

I agree to the versioning and the switch to focal. I think an update to a new commit would be good anyway. Next week I would have time for that.

[mathematicalmichael]

let's do gupyter and (eventually) even swap out the logo icon's j for a g. for this visual trick (turning a J into a G is natural, squeezing a P in isn't), my vote is on that one, despite loving how cheeky "(with a silent p)" sounds. maybe the equivalent joke is "(soft g)" lol

I snagged the name on docker hub

[ChristophSchranz]

Great, then gupyter (with a soft g). I hope the Julia Language devs won't be mad about the removed 'J'^^

How to start from here? Shall we copy and adapt the content of cschranz/gpu-jupyter and also adapt the README within this repo? What steps do we need?

[mathematicalmichael]

sorry for the delay, this will be a bit of a work in progress. The machine I test this stuff on was out of commission. And my notifications got filled up with thesis-related github workflows that I only now cleaned up.
I'm not sure I understand your question. Are you referring to another github account? I don't see a problem with keeping it as-is but just publishing under gupyter/minimal-notebook account on dockerhub. usernames don't need to match as far as I can tell.
once it is all working, we can change the README and mention the dockerhub image.

I cloned the repo again on the fresh ubuntu install, ran clone+start-local and noticed it took forever to complete. to that end, I'd argue we use a slimmer image as a default, I think packaging R and Julia is a bit much. From what I can tell, Python is the primary GPU-enabled use case (tflow/torch). Please correct me if I'm wrong in that assessment.

./start-local also requires docker-compose, which I think we shouldn't assume the presence of. I think we can just use a docker run command (again, something I noticed only because I'm on a fresh install).

I think I've figured out the key steps that I'll need, but I still have some testing to do for referring to different base images. once I get a PR set up from my fork, all you'll need to do is add a secret key to this repo. I can do the testing with my own account.

[ChristophSchranz]

First, thank you very much!

sorry for the delay, this will be a bit of a work in progress. The machine I test this stuff on was out of commission. And my notifications got filled up with thesis-related github workflows that I only now cleaned up.
I'm not sure I understand your question. Are you referring to another github account? I don't see a problem with keeping it as-is but just publishing under gupyter/minimal-notebook account on dockerhub. usernames don't need to match as far as I can tell.
once it is all working, we can change the README and mention the dockerhub image.

I referred to the dockerhub account. Yes, that should work. And you are right about the installation of R and Julia. We should start with a slim setup and add Julia later when there is a need to do so.

I cloned the repo again on the fresh ubuntu install, ran clone+start-local and noticed it took forever to complete. to that end, I'd argue we use a slimmer image as a default, I think packaging R and Julia is a bit much. From what I can tell, Python is the primary GPU-enabled use case (tflow/torch). Please correct me if I'm wrong in that assessment.

./start-local also requires docker-compose, which I think we shouldn't assume the presence of. I think we can just use a docker run command (again, something I noticed only because I'm on a fresh install).

Ahh yes, I haven't removed it yet in start-local.sh, only in the README.

I think I've figured out the key steps that I'll need, but I still have some testing to do for referring to different base images. once I get a PR set up from my fork, all you'll need to do is add a secret key to this repo. I can do the testing with my own account.

Super! Let me know when you need the key. It would be great if you could grant me access to the gupyter dockerhub account, to add the secret key.

I'm going now on vaccation and return on 25 September, so I won't be online in the meanwhile.

Hear you later :)

[ChristophSchranz]

Not a solution with commit c833f1f, but I've added at least a script that builds for a given version three images types (full, python-only and slim), pushes them to Dockerhub and zipps them. Maybe this script should be moved to a branch later on.

I'm still interested in how a smooth solution of this issue could look like.

[mathematicalmichael]

nice! I'm not sure I follow why zipping the builds is necessary. What's the thinking there?

[ChristophSchranz]

You are right, this is not necessary!

[ChristophSchranz]

Removed save + zip and also tagged "python-only"-verion in commit 488ebea.