Parse less JSON on null builds #6880

This was currently getting executed on all builds, even if the directory already exists. There shouldn't be any reason though to exclude the directory from backups on all builds, and after seeing this get a stack sample in a profile I figured it's best to ensure it only executes once in case the backing system implementation isn't the speediest.

This gets called quite a lot and doesn't need to allocate in the first place!

This commit fixes a performance pathology in Cargo today. Whenever Cargo generates a lock file (which happens on all invocations of `cargo build` for example) Cargo will parse the crates.io index to learn about dependencies. Currently, however, when it parses a crate it parses the JSON blob for every single version of the crate. With a lock file, however, or with incremental builds only one of these lines of JSON is relevant. Measured today Cargo building Cargo parses 3700 JSON dependencies in the registry. This commit implements an optimization that brings down the number of parsed JSON lines in the registry to precisely the right number necessary to build a project. For example Cargo has 150 crates in its lock file, so now it only parses 150 JSON lines (a 20x reduction from 3700). This in turn can greatly improve Cargo's null build time. Cargo building Cargo dropped from 120ms to 60ms on a Linux machine and 400ms to 200ms on a Mac. The commit internally has a lot more details about how this is done but the general idea is to have a cache which is optimized for Cargo to read which is maintained automatically by Cargo. Closes rust-lang#6866

This commit updates the locking strategy in Cargo handle the recent addition of creating a cache of the on-disk index also on disk. The goal here is reduce the overhead of locking both cognitively when reading but also performance wise by requiring fewer locks. Previously Cargo had a bunch of fine-grained locks throughout the index and git repositories, but after this commit there's just one global "package cache" lock. This global lock now serves to basically synchronize the entire crate graph resolution step. This shouldn't really take that long unless it's downloading, in which case there's not a ton of benefit to running in parallel anyway. The other intention of this single global lock is to make it much easier on the sources to not worry so much about lock ordering or when to acquire locks, but rather they just assert in their various operations that they're locked. Cargo now has a few coarse-grained locations where locks are held (for example during resolution and during package downloading). These locks are a bit sprinkled about but they have in-code asserts which assert that they're held, so we'll find bugs quickly if any lock isn't held (before a race condition is hit that is)

Removed in the previous commit this now adds dedicated tracking for the last update.

Using mtime information is pretty finnicky across platforms, so instead take a different strategy where we embed the sha that a cache file was generated from into the cache file itself. If the registry's sha has changed then we regenerate the cache file, otherwise we can reuse the cache file. This should make cache file generation more robust (any command can generate a cache file to get used at any time) as well as works better across platforms (doesn't run into issues with coarse mtime systems and the like).

This will largely only get tested during Cargo's own tests, but this commit adds debug assertions where the cache of registry JSON files is always valid and up to date when we consider it being up to date.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parse less JSON on null builds #6880

Parse less JSON on null builds #6880

Commits on May 3, 2019