add --parallel option to build

[graphaelli]

#1652 added parallelism to a number of compose actions. build was left out because of moby/moby#9656.

That doesn't seem to be an issue any more, at least in some cases. For example, adding sleep 5 to the two services introduced in this change results in:

sequential build:

time docker-compose build --no-cache
real	0m17.660s
user	0m0.450s
sys	0m0.121s

parallel build:

time docker-compose build --no-cache --parallel
real	0m9.511s
user	0m0.477s
sys	0m0.119s

Testing on a real set of services reveals similar results.

Resolves #5158

[GordonTheTurtle]

Please sign your commits following these rules:
https://github.com/moby/moby/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:

$ git clone -b "parallel-build" git@github.com:graphaelli/compose.git somewhere
$ cd somewhere
$ git commit --amend -s --no-edit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

[shin-]

Thank you for your submission!

I think we need to proceed with caution here, because even if this is just an option, it can absolutely be detrimental to the general user experience if parallel builds are slower in a majority of cases - which moby/moby#9656 seems to indicate (and I have no reason to believe it was magically fixed without anyone noticing). Notably, the test you added doesn't include a significant build context, which seems to be important in reproducing the bug.

[graphaelli]

@shin- That sounds fair, what are reasonable parameters to work with? moby/moby#9656 (comment) uses a 400M context. On my machine (18.06.0-ce / 0ffa825 on a mac) results in:

build 1: elapsed 3.624
build 2: elapsed 6.677
build 4: elapsed 11.739
build 8: elapsed 23.576
build 16: elapsed 51.491

[shin-]

I mostly want to check in with the engine maintainers to get their thoughts on the issue's status and how this may interact with the buildkit integration. I'll update once I have more info!

[thaJeztah]

Perhaps it would be good to have --parallel accept a number (--parallel=2 for two simultaneous builds)

Having said that, I think the benefits would highly depend on how the builds relate to each other, and what happens during those builds; with buildkit integration, the context will be streamed to the daemon, and will be cached, so a second run of a build will only stream changes to the daemon.

In a situation where multiple builds use the same build-context (I think) running them in parallel will be less performant in the end, as the same build-context will be uploaded to the daemon twice (in parallel) whereas sequential builds would re-use the same uploaded context (only sending changes since the last build).

@tonistiigi and @AkihiroSuda are really the experts in this area though 👍

[graphaelli]

@thaJeztah I'd prefer --parallel took a number too but figured consistency with the boolean (but deprecated) pull --parallel was more important - thoughts? I'm happy to make that change.

[tonistiigi]

In Buildkit this can just be turned on always. No need for a flag.

Perhaps it would be good to have --parallel accept a number (--parallel=2 for two simultaneous builds)

In some BuildKit benchmarks I remember seeing a while back it really didn't make sense on limiting it (unless cpu is 100% but the user can't predict that). Only in some very high numbers the benefits on parallelization started to become meaningless.

In a situation where multiple builds use the same build-context (I think) running them in parallel will be less performant in the end, as the same build-context will be uploaded to the daemon twice (in parallel) whereas sequential builds would re-use the same uploaded context (only sending changes since the last build).

BuildKit does support reusing the context for multiple requests through API. Just needs some work that does a single session request and multiple build requests referencing the session. Until compose sends a tarball of the context and doesn't use the session API it will not benefit of the incremental changes anyway.

[graphaelli]

Only in some very high numbers the benefits on parallelization started to become meaningless.

In that case hard coding seems fine, it's currently 5 - any reason to change that now? @shin- Anything else to clear up before merging this?

[shin-]

I'm fine with the current limit of 5 - we can adjust later based on user feedback if necessary.

Other than that, LGTM.

[hr0915568]

5 is too low since we now have access to 32 cores Threadripper 2990WX in mainstream. Would be nice to make it to accept a number.

[FlorianWendelborn]

What about adding this to docker-compose up --build --build-parallel?

[beaudryj]

I see this is merged? is this in latest compose?

[graphaelli]

It's not in a released version yet, looks like it will be in 1.23 though it's not specifically mentioned in the release notes.

[shin-]

Sorry I let it slip from the release notes! Will make sure it's fixed in the upcoming one.

[marcellodesales]

@shin- one of the best things after a long time!!! Thank you!!! Building multiple apps in parallel is now just a piece of cake!!! Wow, super fast!

[syntaqx]

Is there a way to pass the --parallel flag through to docker-compose up?

[Dragas]

When are the apt repositories gonna be updated with latest docker-compose? It seems that apt still installs 1.17

[cjbush]

Would it be possible to add an environment variable to support building in parallel as well? We currently use a Docker Compose project in Visual Studio and it doesn't support passing command line args to compose. It would be nice if we could just set an environment variable and have it default to always build in parallel.