The Great Project Restructuring of 2018

[vito]

Context: https://discuss.concourse-ci.org/t/psa-the-great-code-restructing-of-2018/543

• Automate creation of "one big repo"
  • Done: https://github.com/concourse/one-big-repo/tree/maintenance
• Get a proof-of-concept going using Go 1.11 modules
  • Done: https://github.com/concourse/one-big-repo/tree/master
• Ship 4.1.0 4.2.1
• Prep workflow around "one big repo"
  • Design pipeline for new structure
    • Figure out how concourse-bosh-release should be built (solution: auto-bump concourse blob)
  • Create a really nice README.md (finally)
  • Preserve CODE_OF_CONDUCT.md, LICENSE.md, NOTICE.md (update attribution header? contact legal)
  • Create a new CONTRIBUTING.md for the newer (hopefully much much simpler) workflow
  • Create a better issue template
    • Instruct users to open "ops" issues on deployment-specific repo (try to limit concourse/concourse to actual bugs and feature requests)
• Create concourse-bosh-release repo containing the current content of concourse/concourse
• Clean up all stale branches under concourse/concourse
  • Migrate any ones we still need to concourse-bosh-release
• Create "one big repo" based on prior automation, this time "for real"
• Push the "one big repo" as a merge (with distinct parents) to the concourse/concourse master branch.
• Archive atc, fly, go-concourse, tsa, web, testflight, ...
• Restructure "one big repo"; look to other "one big repos" for patterns (e.g. pkg/..., cmd/..., web/...) (long-term)

[jama22]

"blocked" on a release....but @vito will keep working on READMEs and what-not

[aegershman]

I just want to tell you both good luck. We're all counting on you.
(and thank you for your OSS work. we the community genuinely appreciate it 👍you're helping make our lives easier and better)

[vito]

@aegershman Hah, thanks!

---

I just realized we're in a bit of a pickle here. The original plan was to just push -f over the concourse repo, and this is so that we can keep all the existing issues here, since they can't be moved.

But, there are also GitHub releases on the concourse repo, and all of our GitHub releases contain all the artifacts for previous versions of Concourse. If I push -f over master of concourse/concourse, all the old tags will go away, and as a result all existing releases will be turned into draft releases and unpublished. We would have to go back and manually re-add them, but all their tags wouldn't even point to anything useful, because the history is completely new and contains the sum of all the different repos, and nothing related to the release itself. I guess the tags would still be present on the concourse-bosh-release repo if we really want to see the code for a given version.

I could rename the repo instead, but that would in turn move all the existing issues to the concourse-bosh-release repo. This maintains the history much better and feels like a smoother transition, but all the issues would be in the wrong place. :/ And all the existing links between issues would break as soon as we add the one big concourse repo and make the name active again.

So we need to pick our poison: lose all release history but preserve tags on concourse-bosh-release, or rename the repo and break GitHub issue links and end up with all the GitHub issues on the wrong repo. Maybe we could also ask GitHub for help here? :/

[vito]

Hmm, actually we might be able to get away with this. When you push -f over master, that'll blow away master with a whole new history, but all the old tags are still there, keeping the old refs and commit history alive. The GitHub releases also stick around, with their artifacts and everything.

Side note: even with all the old tags keeping the history around, the resulting concourse repo is still pretty small. The "one big repo" with everything stripped down was ~26MB, and the push -fed repo that still contains tags and such is ~31MB. Not bad!

[vito]

Other things I'm thinking about while tackling this:

• We should stop checking in compiled assets like bindata.go and elm.js.min. Repository size is a concern now that we've got everything in here together. As part of the "one-big-repo" creation script I've used BFG to clean up the history, to great effect.
  • We should switch from go-bindata to something that's still maintained. statik looks good.
• (Utilize gdn Guardian binary rather than importing their code directly #2574) For the Linux developer workflow, we should make it easier to go from 0 to a fully working Garden-runC Linux worker.
  • Right now we directly import guardiancmd and embed it in the binary go-flags struct. This is kinda cool but is also tightly coupled and means we have to hoist around all of its binary dependencies. There is a self-contained gdn binary now (which hoists around and auto-wires its own dependencies) - should we just use that instead? (opened Utilize gdn Guardian binary rather than importing their code directly #2574 for this)
• (Independently version all "core" resource types #2576) The build process for the concourse binary is fairly obtuse and depends on the BOSH release for embedding resource packages directly into the binary. I think we should probably change this, but I'm not sure what to do yet. It probably makes more sense to just ship the binary alongside the resources rather than embedding it.

[vito]

More questions as I discover all the ways the BOSH release workflow has worked its tendrils into our process. For fear of getting trapped in analysis paralysis, I'm at least going to write them down before coming up with answers. I guess I'll just keep editing this.

• (Use concourse binary in BOSH release #2575) When and how will we build Elm and other static assets in to the binary/BOSH release, if we're not checking it in?
• (Use concourse binary in BOSH release #2575) How do we ensure the BOSH release can build without needing to fetch Go dependencies? (Can we vendor prior to building the release and convince the bosh CLI that the tree isn't dirty somehow?)
• (Use concourse binary in BOSH release #2575) Would all of the above be easier if we just had the BOSH release be a wrapper for the concourse binary? Would the loss in flexibility be worth it? What other benefits might there be?
• (Independently version all "core" resource types #2576) Is it time to just start versioning and shipping the core resources properly?
  • Is this also a good time to 'trim down' the core set of supported resource types?
• How do we cut over to this process without stopping the world and making it impossible to deliver for a while?
  • I don't think you do.

[vito]

I've started switching everything over to packr: https://github.com/gobuffalo/packr

It's a lot better suited than the previously-mentioned statik package because it lets us have an arbitrary number of statically embedded things, rather than just the one single package that statik provides. It also works really well for local development; it'll just read from disk, and then in our pipeline we just run packr to snapshot the state and embed it right in the binary. Neat!

[jchesterpivotal]

Force-push (no, really) the "one big repo" over concourse/concourse master branch.

It's actually possible to merge repos in a way that preserves their independent histories using --allow-unrelated-histories. The outcome is that it looks like a set of merge patches from branches representing each of the repositories.

(Maybe this is what you already intend, I'm just an anxious person)

[vito]

That's the current plan for everything but the concourse repo itself. I'm considering doing it tho. At first I was worried the repo would be huge but the concourse repo is actually relatively small (30mb or so to close), thanks to our use of submodules I guess it stayed lightweight. With everything merged on top it goes up to 60mb (vs 23mb with force push) which really ain't too bad.

…

On Sun, Sep 9, 2018, 9:08 AM Jacques Chester ***@***.***> wrote: Force-push (no, really) the "one big repo" over concourse/concourse master branch. It's actually possible to merge repos in a way that preserves their independent histories using --allow-unrelated-histories. The outcome is that it looks like a set of merge patches from branches representing each of the repositories. (Maybe this is what you already intend, I'm just an anxious person) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2534 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAHWG5Z8Xib2yprS0LrMdlReCJxwCp-ks5uZRLigaJpZM4WOFci> .

[jchesterpivotal]

Can bosh releases live in a subdirectory, or do they always have to sit at the root of a repository? If it can go in a subdirectory that suggests a relatively smooth rearrangement. git mv the current stuff into bosh-release/, then use the merges to bring the other repos in.

[vito]

@jchesterpivotal That'd work but I'd rather have a separate repo for the release anyway as I don't want our core code to appear married to any particular distribution method. I guess we could put them all in there, but then we'd lose GitHub issue fidelity, and I don't think we can do that for e.g. the Helm chart (since it has to be a fork of the charts repo for us to be able to PR changes back).

Current plan is to commit a rm -rf and merge the "one big merge" on top of that.

[vito]

We're now on "one big repo" and have new contributing docs written here:

https://github.com/concourse/concourse/blob/master/.github/CONTRIBUTING.md

Thanks to @tkellen for working with me on the contributing doc!

Here's a running tally of things done so far and things left to do, regarding the dev and contributor workflow:

• replace dev/start with Docker Compose
  • make testflight able to run without needing direct access to workers
    • introduced Mock resource for this
    • this should also make testflight much faster to run, though I haven't done a comparison yet
  • optimize building/rebuilding/restarting
    • pre-bake modules and Go build cache into base image
    • pre-bake resource types into base image
    • auto-bump gdn and dumb-init (and any other deps)
  • auto propagate web assets to container so no restart/rebuild is needed
• make web UI building feel more natural (Makefile -> yarn watch)
• write a new CONTRIBUTING.md
• get unit tests running against master in CI
• get unit tests running for PRs
• get testflight running against master in CI
• get wats running for PRs
• get testflight running for PRs
• switch local baggageclaim driver to overlay (perhaps just switch this everywhere...) to prevent loopback leaks
  • also waiting on make grootfs init optional when embedded cloudfoundry/guardian#119 to prevent leaking two additional loopbacks
• move baggageclaim into its own pipeline
  • add jobs to publish major/minor/patch bumps
• change acceptance tests to wait for ATC to be running prior to running tests
  • testflight (already does this!)
  • wats
  • watsjs
• configure 'test' user for admin team and 'guest' user for non-admin test teams

[jchester]

So I came back and took a look for my own hopes for this epic quest, which is to pull in various parts of Concourse as dependencies. It doesn't quite work, unfortunately.

My best guess, after resorting to pointless superstitious exercises like throwing runes, flipping tarot cards and going over what golang is pleased to call "documentation", is that the various subdirectories also need go.mod files.

Otherwise I get stuff like this:

pkg/check/check.go:8:2: unknown import path "github.com/concourse/concourse/atc": cannot find module providing package github.com/concourse/concourse/atc

Which prods me to try:

$ go get github.com/concourse/concourse/atc@3f6f1170443

go: finding github.com/concourse/concourse/atc 3f6f1170443
go: finding github.com/concourse/concourse 3f6f1170443
go get github.com/concourse/concourse/atc@3f6f1170443: missing github.com/concourse/concourse/atc/go.mod at revision 3f6f11704430

I also suspect, but can't prove without resorting to desperate measures like "effort", that go get doesn't understand the tag format Concourse uses for releases. If I say go get github.com/concourse/concourse@v4.2.1 it chokes.

Returning again to The Sacred Mysteries, I find the advice that "go get github.com/gorilla/mux obtains the latest version with a semver tag". It works for branches, it works for commit SHAs, but apparently just specifying a tag is rejected wholesale as the unclean work of beelzebub, seductive archfiend of convenient but missing features.