Official API for build scripts

[epage]

Problem

Having a Rust API would help with several problems

• Help people migrate through build script best practices (e.g. Cannot extend build directives within cargo #11461)
• Avoid people making type mistakes
• Bring documentation more inline with people's workflow, e.g. IDEs auto-completing and rendering doc comments
• Help improve discoverability of APIs
• Ensures a feedback loop with the cargo team, hoping it avoids some failure modes where compatibility is broken
• Reduce duplicated effort
  • build_script
  • build-rs
  • build-env
  • vergen
  • foreman <=0.4.0 (package name was handed off and used for a different purpose after that version)
  • system-deps
  • rsconf

Proposed Solution

A crate that provides an API for working with build scripts

• Low level API handling writing of directives, reading runtime env variables, etc (see also Official API for reading cargo env variables #12431)
• A higher level API to help with re-run-if-changed

Notes

Things to work out

• Name
• If we co-maintain, how repo and publish permissions work out
• How do we handle the MSRV because the cargo repo currently only handles "latest" (see also fix: Set MSRV for internal packages #12381) see Set and verify all MSRVs in CI #12654

As for timing, it'd help if we had this to recommend before we fully resolve #11461 so we can tell people to migrate once, rather than twice

[epage]

FYI we now have an MSRV policy as of #12654

[codyps]

I'm the person who wrote build-env (and also added the target specific env var naming pattern it uses to the cc crate originally):

The content of build-env was primarily written to allow build scripts to implement target specific env variable reading in a way that matches the cc crate's methodology. In practice though, this has been either reimplemented in other build scripts either in the same way (rarely) or in a manner closer but not the same as the style used by cargo's own target specific env variables.

A build-rs-like API for emitting the directives read by cargo or parsing variables set by cargo seems fine, and I expect this crate could use it easily (if I ever update it). I generally like the idea of having a real API instead of peeking into env variables and printing magic strings.

Some encouragement on increasing the scope though:

I'd worry that not providing some API for external info not known to cargo that is still target specific means that the ecosystem of crates will continue to have various inconsistent ways of providing configuration that is (sometimes) target specific to build scripts. As long as env variables are the only really way to communicate that info (ie: until/unless target specs or some target spec adjacent thing exists to do something similar), we're going to want some pattern in env variable naming that tries to obtain consistency. Given that cargo has its own pattern here (<env>_<uppercase target name with underscores>), it could be sensible to adopt it or something based on it. I don't think not having this should stop shipping some sort of build script helper crate, but given the widespread need for something like this, it's something we'll want to include.

Also: just having something that reads an env var and emits the rerun-if-env-changed directive seems like a fairly basic ask, and is something that I've seen duplicated in tons of build scripts. It's so small, that folks are unlikely to pull in another crate to handle it, but as long as we're adding a crate to enable build script writing, it's something that should be included. (Perhaps this is part of what is meant by "A higher level API to help with re-run-if-changed"?)

[madsmtm]

Could we consider providing such a crate with rustup? Similarly to how we have the built-inproc_macro crate (I know that one needs to be distributed with the compiler, so maybe a bit different).

[epage]

rust-lang/compiler-team#659 could allow optional packages in the sysroot but I don't think thats a route we should go.

Being in the sysroot / distributed with rustup means we can't break compatibility. There are some existing packages in this space but I'm not ready to lock us down for 1.0 for ever at this point, if for no other reason.

[madsmtm]

not ready to lock us down for 1.0

Totally get that, we'd likely need years of experimentation first, I'm mostly arguing considering that as the end goal.

Just speaking personally, I wouldn't use such a crate, since I'm very wary about compile-times, and the extra dependency doesn't seem worth it for a (percieved) small usability improvement. But if I knew that it was on the way to being part of the standard distribution, I'd be willing to take the compile-time perf hit in the meantime.

[Xaeroxe]

Hi, at the risk of overstepping my boundaries, I’d like to initiate a repo for this. If I develop an API crate under my own repo could we arrange for it to be subject to code review and if accepted, transferred to the rust-lang organization?

[Xaeroxe]

Or alternatively we could just add a folder to the cargo repo.

[epage]

We have a crate name / implementation that was planned to be donated but its stuck in the current owner being busy and me being unfamiliar with the process for migrating something like that.

[Xaeroxe]

Well build-rs, the crate that you seem to be talking about, does have extremely permissive licensing.

With that being the case I doubt they'd be upset about me taking the initiative and adding it to the cargo repository. I think there's real utility in developing it alongside cargo, PRs that modify the API and cargo at the same time won't require two separate PRs and thus can be easier to track. I'm not in a position to talk about how we might gain publishing rights for this crate on crates.io. It's possible that's all we really need though.

@CAD97 I'm pinging you to give you an opportunity to give us some input here. It's not my intent to override your will or governance of your project, but I do think my proposal is in line with your goals.

[CAD97]

I'm fully on board with cargo adopting build-rs directly. I've put a small amount of effort into trying to figure out what that'd look like but hadn't really gotten anywhere with that yet.

[Eh2406]

Some people can use crates.io admin privileges to magically move ownership for you if and only if you make it clear that you want it. If you explicitly state that you would like to give published permission to the cargo team we (can probably) talk crates.io admins into making that happen without more intervention on your part.