build(modules): add :modules:reportModuleOrder task

[keturn]

I wanted to know the best order in which to push module updates so that when they build on CI, a module won't try to build before its dependencies have been pushed.

[keturn]

This is one implementation.

It uses JGraphT, which seems like a nice library, but it may be overkill to add another library for handling directed acyclic graphs to the build dependencies. Gradle must have one of these internally, as does gestalt, and guava has some graph traversal functions too.

(I only found out about Guava's when I got in to the JGraphT docs. I'd probably start there and see if that's sufficient.)

I also realized that since CI builds things in parallel, it would be best to have not just any topological sort, but one with more specific properties.

JGraphT defines topological order as:

A topological order is a permutation p of the vertices of a graph such that an edge (i, j) implies that i appears before j in p.

Additionally, we would like

• to maximize the average distance between i and j in p. This increases the chance that i has finished building by the time the queue starts on j.
• to be able to partition p such that there no edges within each partition, that the origin of each edge was in a previous partition. This would allow pushing a batch without concern of which order CI runs the things within the batch, as long as the previous batch had finished.

(if we have the partitioning and it's optimized for the fewest number of partitions that satisfy that constraint, the first quality is redundant. or maybe they're different ways of saying the same thing? I have not done the maths.)

[skaldarnar]

JGraphT looks pretty nice! If I understand correctly this hooks up to gradlew calls. I see that this is the logical place where we have the module dependency graph anyways, but on the other hand I'm not sure whether our central build tooling is the place where this should go. Would this fit into the picoCLI groovyw tooling bucket (once we get there)?

Reasoning for this comment is that I don't think that this info is/should be used by the CI, at least not in the current setup. It is more about developer tooling, isn't it?

Identifying these clusters (or levels) would be really helpful to know when you hit the point where you have to wait before pushing more things. For full automation, we'd need to figure out how to get informed that the build for a commit was finished, and the push-train can move on... not sure whether/how the Github API could help us with that, I might have a look at that 🙃

[keturn]

For full automation, we'd need to figure out how to get informed that the build for a commit was finished, and the push-train can move on... not sure whether/how the Github API could help us with that.

hmm. We know that PRs have checks attached to them, but can be the same be said for other branches? Does GitHub have a way to attach a status report to the current state of develop?

I would look straight to Jenkins without involving GitHub as an intermediary. Or, heck, even the package repository, because we're really waiting until a new SNAPSHOT build has been uploaded, right?

but on the other other hand, I was thinking of this as the quick and dirty kludge that could be done with only local information. If it turns in to a project of integrating with other service APIs and synchronizing to changes in remote server state, I'd be inclined to invest that effort in improving the workflow on the Jenkins side instead.

[skaldarnar]

I think having this makes sense if it helps us in any way. Not sure how customizable or extensible this is in the future (due to being a gradle task).

(hmpf, why's gradle so slow 🙈 running this script takes 5s with a daemon started, and 15s if there's no daemon yet running (in Ωmega workspace).