Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add image parameters to compose.yaml #1092

Merged
merged 5 commits into from
Jun 12, 2024
Merged

Conversation

tgeoghegan
Copy link
Contributor

@tgeoghegan tgeoghegan commented Jun 12, 2024

Compose files can be parameterized using Bash-style (sigh) syntax (1). We add variables JANUS_AGGREGATOR_IMAGE, JANUS_MIGRATOR_IMAGE, DIVVIUP_API_IMAGE and DIVVIUP_API_MIGRATOR_IMAGE to allow setting Docker image tags like so:

DIVVIUP_API_IMAGE=myrepository.dev/divviup_api:0.0.1 \
  JANUS_AGGREGATOR_IMAGE=myrepository.dev/janus_aggregator:0.7.8 \
  docker compose up

These variables are interpolated into the image field of the service elements (2). Because we also have a build element on the divviup-api services, the behavior is to pull the specified image and fall back to the build element if it can't be found (3). So if you want to build divviup-api from source and use that, do:

DIVVIUP_API_IMAGE=divviup_api:1 docker compose up

divviup_api:1 will be built from the local context, unless it's already in the local Docker repository, and then launched.

We provide defaults for all these images that pull from Divvi Up owned public artifact repositories. It's somewhat unfortunate that we use a divviup-api version that is behind main, but on the other hand the objective is to enable a demo experience, so it makes sense to pin Janus and divviup-api versions where the demo is known to work end-to-end.

@tgeoghegan tgeoghegan requested a review from a team as a code owner June 12, 2024 00:49
compose.yaml Outdated
@@ -66,6 +66,7 @@ services:
- README.md

divviup_api:
image: ${DIVVIUP_API_IMAGE:-us-west2-docker.pkg.dev/janus-artifacts/divviup-api/divviup_api:0.3.9}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR isn't quite ready to go, because the default image here is not in a public repository. I'm working on separate changes to enable docker-release.yml to push to us-west2-docker.pkg.dev/divviup-artifacts-public and will update this URL to point there before merging this.

@tgeoghegan tgeoghegan force-pushed the timg/docker-compose-parameters branch 3 times, most recently from 6c84f1f to 4c5e6db Compare June 12, 2024 01:05
@divergentdave
Copy link
Contributor

I think an easier entry point would be to create another file, compose.dev.yaml, and use include or extends to import parts of an end-user-focused compose.yaml. The dev file could then use image: !reset null to ensure it builds from source, without requiring the user to make up image names.

@tgeoghegan
Copy link
Contributor Author

Seems reasonable! The simplest thing would be to use merge, but that requires passing multiple compose YAMLs in the correct order at the command line so that merge rules are evaluated correctly (i.e., docker compose -f compose.yaml -f compose.dev.yaml up works, but docker compose -f compose.dev.yaml -f compose.yaml up does not).

So I opted for include. This requires us to add a further overrides file, but it makes the dev setup easier to use.

I also moved the build stanzas to compose.dev.yaml, on the premise that this makes the behavior of compose.yaml less surprising: if someone is using compose.yaml and makes a typo in an image name, I think we want them to get a clear docker pull error rather than have an image with that tag "silently" get built.

@divergentdave
Copy link
Contributor

Unfortunately, Dependabot cannot update image names in compose.yaml for us: dependabot/dependabot-core#390. We could maybe add a GitHub Actions workflow that runs sed and pushes a commit after a release.

Comment on lines 12 to 17
develop:
watch:
- path: migration
action: rebuild
ignore:
- README.md
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The develop stanzas are mixed up, this one should go on divviup_api_migration and vice versa.

@tgeoghegan
Copy link
Contributor Author

Still need #1093 and in turn https://github.com/divviup/janus-ops/pull/1825 before I can merge this.

Compose files can be parameterized using Bash-style (*sigh*) syntax
([1]). We add variables `JANUS_AGGREGATOR_IMAGE`,
`JANUS_MIGRATOR_IMAGE`, `DIVVIUP_API_IMAGE` and
`DIVVIUP_API_MIGRATOR_IMAGE` to allow setting Docker image tags like so:

```sh
DIVVIUP_API_IMAGE=myrepository.dev/divviup_api:0.0.1 \
  JANUS_AGGREGATOR_IMAGE=myrepository.dev/janus_aggregator:0.7.8 \
  docker compose up
```

These variables are interpolated into the `image` field of the `service`
elements ([2]). Because we also have a `build` element on the
`divviup-api` services, the behavior is to pull the specified image and
fall back to the `build` element if it can't be found ([3]). So if you
want to build `divviup-api` from source and use that, do:

```sh
DIVVIUP_API_IMAGE=divviup_api:1 docker compose up
```

`divviup_api:1` will be built from the local context, unless it's
already in the local Docker repository, and then launched.

We provide defaults for all these images that pull from Divvi Up owned
public artifact repositories. It's somewhat unfortunate that we use a
`divviup-api` version that is behind `main`, but on the other hand the
objective is to enable a demo experience, so it makes sense to pin
Janus and divviup-api versions where the demo is known to work
end-to-end.

[1]: https://docs.docker.com/compose/compose-file/12-interpolation/
[2]: https://docs.docker.com/compose/compose-file/05-services/#image
[3]: https://docs.docker.com/compose/compose-file/build/#using-build-and-image
Add a distinct compose.dev.yaml and include the base compose.yaml into
it with some overrides.
@tgeoghegan tgeoghegan force-pushed the timg/docker-compose-parameters branch from 5a0be9a to 4a1426d Compare June 12, 2024 23:05
@tgeoghegan tgeoghegan merged commit b189dbd into main Jun 12, 2024
7 checks passed
@tgeoghegan tgeoghegan deleted the timg/docker-compose-parameters branch June 12, 2024 23:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants