-
Notifications
You must be signed in to change notification settings - Fork 18.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for extends
feature in Compose v3 / docker stack deploy
#31101
Comments
I've added some notes docker/compose#4315 (comment) to me bringing back extends as it exists up to docker compose file 2.1 version is not a good idea but the main feature I miss is to be able to declare abstract services (that should never be runned but could be used to group common properties of services for convenience). |
Agreed on comments from docker/compose#4315 that the way I won't recommend a solution, but FWIW, to show the extent of abuse, here are the conventions that adorn the top of our compose file:
|
I think the extends as in v2.1 is a good choice. Extends is actually simple and understandable, this is too a good pratice for each environnement to have a little and readable transformation between dev, prod and env. Actually, i have
This works, and i don't understand why we should search in this ticket a complete rewrite, but more the keep of an interessing feature. Extends is a good part with special use cases that other techniques can't resolv easily. I'm happy with extends possibilities. |
Blocking us from upgrading to v3.x format as well. We keep our docker container definitions in a folder-per-instance layout, where each folder contains a So when I need to manage a particular container instance, I just need to |
I'm blocked from using version 3 compose files. I use |
I'm not against improving the v2.1 'extends' solution with something more appropriate though. Something like abstract services could be both more safe to use and more powerful. For example (since it's abstract) I could imagine supporting abstract volumes/mountpoints/networks, where the abstract base service defines a local mount point or a required network for a service, without defining the host part - meaning a deriving service could then define/map the host part of the mounts/volumes and networks to whats appropriate in his host/stage environment. That's an example of something we can't do now with 'extends' as far as I know, and would open some interesting possibilities. |
Taking away extends feature was not helpful at all. We have a lot of web services started with the same volumes mapped, environment variables and labels set. They also have same healthcheck policies. And Not to mention ports. Combining compose files using multiple -f option is tedious and error prone if you are dealing with multiple projects in the same host. |
@shin- will this be brought back on 3.2 schema ? |
I really, really would love to get I would not mind loosing the exact |
I have a setup with a common service file and bunch of services extending it, changing mostly volumes, so I would say I rely on
|
+1 I'm hoping a satisfactory solution can be found here to enable better factoring of compose structures! |
At the top of my use case list is to DRYup my configuration managed hoard of services. I thought I'd be able to take advantage of 'extends' if v3. I don't want to go back to v2... and I don't want to have special cases where I must use the -f process work-around. |
Hey, did anyone start working on this? I can't find a PR to keep track. It'd simplify a lot some things for us here as well (use-case very similar to some described above). |
Since version 3 is frozen and I needed a way to share common configuration, I hacked a small workaround, which I think I could share here (I'm not sure if this is the right place but feel free to tell me where else I can share this info :)) I'm not going to elaborate on it here, since there's a readme in the repo. Edit: |
+1 |
2 similar comments
+1 |
+1 |
Thanks for the +1s 💃 Unfortunately, there's no License in that repo. I already wrote a message to the owner on facebk but he didn't answer yet. Maybe he'll add a license if more people query him about it :) |
Something that might help some of the extends use cases (those within a single file) would be support for YAML anchors: https://learnxinyminutes.com/docs/yaml/ It appears that the JSON Schema may be failing validation on them |
Hey, @grayside, yaml anchors do work, at least for me. See my comment above for how I use them. |
Ok but it's too sad to use some noop service no? Especially for env vars, what kind of values those env vars handle? If it's about secrets, use swarm secrets feature (or any other secrets solution). If it's about settings, we're ok that most of the time settings are app/service specific, not intended to be shared between services. If you need to share settings between services, most of the time it's when you launch the same container image but for different runtime purposes/tasks (consumer, producer, http worker, ...). |
I tend to disagree. In the project I'm currently working on, I use it for example for volumes:
Now I can specify these volumes like that:
This makes it way easier to navigate different volumes without having to think about which container you're in. Imho, this is one way to make your docker-compose setup more consistent and easier to use and maintain. |
Ok yes I understand @JanNash but in your example below you don't have any noop service right? |
But anchors are not enough for many cases. My case involves supporting several environments for the same project. You can see a scaffolding for our projects here. When you develop, you use Each one has its own peculiarities:
This simple separation allows to have a very agile and flexible DevOps pipeline, where everybody uses the same code among different stages, with just little tweaks depending on the environment in use.
... or at least we have a way to generate a valid DRY-less format from a DRY-ful solution. I thought ... or we have a different tool that makes everything (I'm keeping an eye on ansible-container too). But surely this is not "the fix". |
The link in #31101 (comment) includes a README with the working example of YAML anchors. Looking it over and trying it again today, it works fine. Not sure what I'm doing differently. @JanNash 👍 |
@yajo I hear you and as said, it's a workaround and it would be better by an order of magnitude if there was a good, built-in, DRY solution supplied by docker/moby/docker-compose (whatever is the correct reference). Let's all hope that will come soon because apart from that, I'm pretty happy with docker compose 👍
Have a nice one, y'all 🚶 |
I forget but is there an actual reason it was taken away?
…On Wed, 6 May 2020, 23:14 Julien Marechal, ***@***.***> wrote:
A little reminder that a lot of people would like this feature :)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#31101 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABOE6GGDIVGATP734YJA4UTRQHOLJANCNFSM4DANZGSQ>
.
|
The way yaml anchors handle this use case is a bit inconvenient compared to An example: # anchors for the service, environment, deploy, and deploy.placement blocks
# you'll need an anchor for every dict that you want to merge into
x-common-app: &common-app
image: app:1.0
environment: &common-app-environment
common_option: a
overwrite_option: b
deploy: &common-app-deploy
max-replicas-per-host: 3
placement: &common-app-deploy-placement
constraints:
- 'node.labels.app_host==true'
services:
myapp: << *common-app
environment: << *common-app-environment
foo: bar
baz: xyzzy
overwrite_option: quz
deploy: << *common-app-deploy
replicas: 15
placement: << *common-app-deploy-placement
preferences:
- spread: node.labels.region
# The above yields the following:
services:
myapp:
image: app:1.0
environment:
common_option: a
overwrite_option: quz
foo: bar
baz: xyzzy
deploy:
replicas: 15
max-replicas-per-host: 3
placement:
constraints:
- 'node.labels.app_host==true'
preferences:
- spread: node.labels.region It might not look so annoying in this example, but it gets annoying and less readable if you're templating multiple (10+) services off one extension block. In my mind an ideal scenario is a combination of the yaml anchor approach and the In my organization we found yaml anchors a bit syntactically sloppy so we basically re-implemented the |
I have done a lot of gitlab CI YAML lately. It has exactly these features, which are sooo nice to achieve readable and manageable templates and final configurations:
This is the exact set of feature that makes these files bearable. |
I arrived at this issue from the docker docs, in my case I wanted to template my For some context, I am using a basic reverse proxy with Let's Encrypt and several application containers (Nextcloud for now, one for me, and some separated ones for friends) - in this case, I wanted to make a template of the Nextcloud containers so that I avoid mistakes and duplication with keystrokes for very similar setups. The following packages were what I tried: ytt seems very comprehensive and is the only option to use YAML natively. It seemed powerful and the right tool for the job, and it uses Starlark, a superset of Python, directly inside the YAML file to performing processing. However after not long, the template became very messy, and the littering of fragments of code and fragments of YAML, plus mixing of Python data types like dictionaries and arrays and YAML fragments (that seem to be somewhat processed like text, a bit like using an HTML template engine that outputs tags like strings) eventually resulted in too many errors and too messy a file. Dhall also seems very comprehensive and uses a unique native language that can be output to a variety of formats; It seems more like a metaprogramming language than a template system, however given that the syntax is functional and it is pretty strictly typed, it quickly became more complex than was worth it for a simple template for unstructured YAML. As what seems a little like a blend of JSON and Haskell, it required too much thought to work what I needed done into the language. Interestingly, sometihng that was difficult with both Dhall and ytt was using parameterised field names, both would have worked well enough otherwise, but I need to have the name of my instance appear in the services names and volume names, and in both of these achieving this was somewhat ugly; using arguments for the values in a hash is easy, but using those arguments in the names of the keys was messy or I couldn't find how to do it neatly, plus Dhall enforces type safety and this simply goes against that concept. With Jsonnet it was a simple as putting the expression in square brackets. CUE and Jsonnet are both JSON-oriented, however running these through a converter is not hard at all, and it seems again, like Dhall, CUE has a lot of powerful features and was born out of shortcomings in Jsonnet, however partway into the documentation, it became apparent it was already overkill; perhaps with much more time to learn it properly it would be the superior option, but it seems CUE is more oriented to validation and schemata than a simple template job and so I quickly moved onto Jsonnet and finished the job quite quickly. Finally, it was only after I had completed all this I realised the whole time I had been comparing these tools to the simplicity of Liquid tags or similar HTML templates, that I could actually probably simply have used Liquid in the first place. I have only used it in the context of a Jekyll site, so it just never ocurred ot me ot obtain a standalone package, however with basic looping and lists, and the ability ot evalate expressions straight into in-place text, this probably would have been a much better too for the job; Jsonify probably superior for JSON, but Liquid could operate in pure YAML and so the file becomes more readable again. |
+1 docker-compose was one inspiration behind the bespoke solution I've implemented at work since this ticket was created to support migrating a large number of test envs to k8s. I was very careful to avoid bells and whistles but pretty quickly justified an analogous feature. The philosophical discussion (composition versus inheritance etc.) looks to me like a distraction from common sense (with the benefit hindsight - still unresolved nearly 3 years later). Evidently it is required by people who might continue to use docker-compose. |
+1 👍 I used this feature heavily before for I'm shocked by deprecating the feature and not replacing it with something that can do something similar. Maybe just allow us to use jinja2 and do what we want. I hope Docker-Compose would be less anti-DRY. :'( |
docker engine does not support extends in compose files... moby/moby#31101
docker engine does not support extends in compose files... moby/moby#31101
Seems that |
One use case where I use the feature heavily, is to version docker images to code. My docker dev and prod compose files both extend from docker-images.yml where only the basic service is listed and a tagged version of the image of the service. Didn't found an easy workaround for this in v3. |
Is this issue also for And using
It would be good to know if it is planned feature for swarm beacuse its presence in new compose. |
It fails for me with
|
Confirmed, works for me with docker-compose 1.29. |
Whoever decided to remove this extremely useful and widely used feature from Docker should be fired. This is an excellent example of how "improvement" of a product actually makes it less usable. This is one for the history books of software development ;) |
I can't believe it's been 4 years I've been tracking this issue and still no solution |
@panique and @SC7639 have you read the recent comments? I'll just summarize to make sure we're all up to date: The Compose Spec is the latest version of compose files and supports Now for Swarm Mode (which this ticket is about), where you can use a compose file for Stacks, the compose file still requires the older Technically I believe this It looks like this CLI issue for moving to Also, please add your thumbs up here for continued Swarm Mode support: docker/roadmap#175 And on these related issues: |
I was following the (docker-compose.yaml) => docker-compose config => (docker-composed.yaml) method to get a yaml that works with docker swarm. Things were working fine a month ago, after digging, I found out my setup was broken because docker-compose version 2.5.0 does not resolve the extends keyword whereas 1.29.2 did. This is even if I specify docker-compose config file version 2.1 in my all yamls. |
Will it be possible to extend I wonder how hard it would be to just add support for the "#include" keyword, that would just merge compose files ... if the end-result would be the same as just doing 'cat *.yaml >> compose.yaml' then we'd solve 99% of the need :) |
As can be seen in docker/compose#4315 , the
extends
feature that exists indocker-compose
seems to be popular among users despite its flaws. However, it has so far not been added in the Engine's implementation of the Compose format. So far, we have advised users to simply flatten their Compose file structure when using v3, but is this the long-term solution we want to go with? How can we provide a clear upgrade path for users who have come to rely on this feature?cc @dnephin @vdemeester
The text was updated successfully, but these errors were encountered: