Enable pipelined compilation by default #7143

alexcrichton · 2019-07-17T21:12:38Z

This commit enables pipelined compilation by default in Cargo now that
the requisite support has been stablized in rust-lang/rust#62766. This
involved minor updates in a number of locations here and there, but
nothing of meat has changed from the original implementation (just
tweaks to how rustc is called).

rust-highfive · 2019-07-17T21:12:48Z

r? @nrc

(rust_highfive has picked a reviewer for you, use r? to override)

alexcrichton · 2019-07-17T21:14:16Z

r? @ehuss

(note that CI will fail and this can't land until rust-lang/rust#62766 propagates to nightly)

This commit stabilizes options in the compiler necessary for Cargo to enable "pipelined compilation" by default. The concept of pipelined compilation, how it's implemented, and what it means for rustc are documented in rust-lang#60988. This PR is coupled with a PR against Cargo (rust-lang/cargo#7143) which updates Cargo's support for pipelined compliation to rustc, and also enables support by default in Cargo. (note that the Cargo PR cannot land until this one against rustc lands). The technical changes performed here were to stabilize the functionality proposed in rust-lang#60419 and rust-lang#60987, the underlying pieces to enable pipelined compilation support in Cargo. The issues have had some discussion during stabilization, but the newly stabilized surface area here is: * A new `--json` flag was added to the compiler. * The `--json` flag can be passed multiple times. * The value of the `--json` flag is a comma-separated list of directives. * The `--json` flag cannot be combined with `--color` * The `--json` flag must be combined with `--error-format=json` * The acceptable list of directives to `--json` are: * `diagnostic-short` - the `rendered` field of diagnostics will have a "short" rendering matching `--error-format=short` * `diagnostic-rendered-ansi` - the `rendered` field of diagnostics will be colorized with ansi color codes embedded in the string field * `artifacts` - JSON blobs will be emitted for artifacts being emitted by the compiler The unstable `-Z emit-artifact-notifications` and `--json-rendered` flags have also been removed during this commit as well. Closes rust-lang#60419 Closes rust-lang#60987 Closes rust-lang#60988

…r=oli-obk rustc: Stabilize options for pipelined compilation This commit stabilizes options in the compiler necessary for Cargo to enable "pipelined compilation" by default. The concept of pipelined compilation, how it's implemented, and what it means for rustc are documented in #60988. This PR is coupled with a PR against Cargo (rust-lang/cargo#7143) which updates Cargo's support for pipelined compliation to rustc, and also enables support by default in Cargo. (note that the Cargo PR cannot land until this one against rustc lands). The technical changes performed here were to stabilize the functionality proposed in #60419 and #60987, the underlying pieces to enable pipelined compilation support in Cargo. The issues have had some discussion during stabilization, but the newly stabilized surface area here is: * A new `--json` flag was added to the compiler. * The `--json` flag can be passed multiple times. * The value of the `--json` flag is a comma-separated list of directives. * The `--json` flag cannot be combined with `--color` * The `--json` flag must be combined with `--error-format=json` * The acceptable list of directives to `--json` are: * `diagnostic-short` - the `rendered` field of diagnostics will have a "short" rendering matching `--error-format=short` * `diagnostic-rendered-ansi` - the `rendered` field of diagnostics will be colorized with ansi color codes embedded in the string field * `artifacts` - JSON blobs will be emitted for artifacts being emitted by the compiler The unstable `-Z emit-artifact-notifications` and `--json-rendered` flags have also been removed during this commit as well. Closes #60419 Closes #60987 Closes #60988

alexcrichton · 2019-07-30T14:49:10Z

Ok rust-lang/rust#62766 has landed, I've rebased this and updated all failing tests. This will fail CI since the compiler isn't on the nightly channel yet, but @ehuss if you're up for it this should be good for review!

ehuss · 2019-07-31T15:39:42Z

I restarted CI, and it looks like stable/beta does not pass.

alexcrichton · 2019-07-31T15:52:37Z

Ah good point, I've updated the PR to just ignore those tests on non-nightly builders for now. Both of their assertions are tweaked when pipelining is enabled (both tests still morally pass on all compilers, it's just a difference of what should be asserted) and looking into the tests it seems not worth it to keep them running on stable/beta. Once pipelining rides to stable and we can remove the feature detection for that rustc flag we can update these tests to run on all channels again.

ehuss · 2019-07-31T16:06:18Z

tests/testsuite/dep_info.rs

@@ -511,6 +511,6 @@ fn canonical_path() {
    assert_deps_contains(
        &p,
        "target/debug/.fingerprint/foo-*/dep-lib-foo-*",
-        &[(1, "src/lib.rs"), (2, "debug/deps/libregdep-*.rlib")],
+        &[(1, "src/lib.rs"), (2, "debug/deps/libregdep-*.r*")],


Can we just change this to .rmeta, or at least leave a comment why the * is there?

Sure yeah, the r* is to handle rlib on stable/beta but rmeta on nightly. I'll just switch this to a nightly-only test though and make it rmeta

ehuss · 2019-07-31T21:49:30Z

r=me with minor nit addressed.

This commit enables pipelined compilation by default in Cargo now that the requisite support has been stablized in rust-lang/rust#62766. This involved minor updates in a number of locations here and there, but nothing of meat has changed from the original implementation (just tweaks to how rustc is called).

alexcrichton · 2019-07-31T21:58:08Z

@bors: r=ehuss

bors · 2019-07-31T21:58:09Z

📌 Commit daa1bce has been approved by ehuss

bors · 2019-07-31T21:58:17Z

⌛ Testing commit daa1bce with merge b21602c...

…=ehuss Enable pipelined compilation by default This commit enables pipelined compilation by default in Cargo now that the requisite support has been stablized in rust-lang/rust#62766. This involved minor updates in a number of locations here and there, but nothing of meat has changed from the original implementation (just tweaks to how rustc is called).

bors · 2019-07-31T22:16:33Z

☀️ Test successful - checks-azure
Approved by: ehuss
Pushing b21602c to master...

Update cargo, rls ## cargo 12 commits in d0f828419d6ce6be21a90866964f58eb2c07cd56..26092da337b948719549cd5ed3d1051fd847afd7 2019-07-23 21:58:59 +0000 to 2019-07-31 23:24:32 +0000 - tests: Enable features to fix unstabilized `#[bench]` (rust-lang/cargo#7198) - Fix excluding target dirs from backups on OSX (rust-lang/cargo#7192) - Handle symlinks to directories (rust-lang/cargo#6817) - Enable pipelined compilation by default (rust-lang/cargo#7143) - Refactor resolve `Method` (rust-lang/cargo#7186) - Update `cargo_compile` module doc. (rust-lang/cargo#7187) - Clean up TargetInfo (rust-lang/cargo#7185) - Fix some issues with absolute paths in dep-info files. (rust-lang/cargo#7137) - Update the `url` crate to 2.0 (rust-lang/cargo#7175) - Tighten requirements for git2 crates (rust-lang/cargo#7176) - Fix a deadlocking test with master libgit2 (rust-lang/cargo#7179) - Fix detection of cyclic dependencies through `[patch]` (rust-lang/cargo#7174) ## rls 1 commits in 70347b5d4dfe78eeb9e6f6db85f773c8d43cd22b..93d9538c6000fcf6c8da763ef4ce7a8d407b7d24 2019-07-30 12:56:38 +0200 to 2019-07-31 21:42:49 +0200 - Update cargo (rust-lang/rls#1529)

Cargo has of rust-lang#7143 enabled pipelined compilation by default which affects how the compiler is invoked, especially with respect to JSON messages. This, in some testing, has proven to cause quite a few issues with rustbuild's current integration with Cargo. This commit is aimed at adding features to Cargo to solve this issue. This commit adds the ability to customize the stream of JSON messages coming from Cargo. The new feature for Cargo is that it now also mirrors rustc in how you can configure the JSON stream. Multiple `--message-format` arguments are now supported and the value specified is a comma-separated list of directives. In addition to the existing `human`, `short`, and `json` directives these new directives have been added: * `json-render-diagnostics` - instructs Cargo to render rustc diagnostics and only print out JSON messages for artifacts and Cargo things. * `json-diagnostic-short` - indicates that the `rendered` field of rustc diagnostics should use the "short" rendering. * `json-diagnostic-rendered-ansi` - indicates that the `rendered` field of rustc diagnostics should embed ansi color codes. The first option here, `json-render-diagnostics`, will be used by rustbuild unconditionally. Additionally `json-diagnostic-short` will be conditionally used based on the input to rustbuild itself. This should be enough for external tools to customize how Cargo is invoked and how all kinds of JSON diagnostics get printed, and it's thought that we can relatively easily tweak this as necessary to extend it and such.

Add support for customizing JSON diagnostics from Cargo Cargo has of #7143 enabled pipelined compilation by default which affects how the compiler is invoked, especially with respect to JSON messages. This, in some testing, has proven to cause quite a few issues with rustbuild's current integration with Cargo. This commit is aimed at adding features to Cargo to solve this issue. This commit adds the ability to customize the stream of JSON messages coming from Cargo. The new feature for Cargo is that it now also mirrors rustc in how you can configure the JSON stream. Multiple `--message-format` arguments are now supported and the value specified is a comma-separated list of directives. In addition to the existing `human`, `short`, and `json` directives these new directives have been added: * `json-render-diagnostics` - instructs Cargo to render rustc diagnostics and only print out JSON messages for artifacts and Cargo things. * `json-diagnostic-short` - indicates that the `rendered` field of rustc diagnostics should use the "short" rendering. * `json-diagnostic-rendered-ansi` - indicates that the `rendered` field of rustc diagnostics should embed ansi color codes. The first option here, `json-render-diagnostics`, will be used by rustbuild unconditionally. Additionally `json-diagnostic-short` will be conditionally used based on the input to rustbuild itself. This should be enough for external tools to customize how Cargo is invoked and how all kinds of JSON diagnostics get printed, and it's thought that we can relatively easily tweak this as necessary to extend it and such.

Pkgsrc changes: * Adapt to the move of the implementation of random numbers. * Remove patch which is no longer relevant (Signals.inc) * Cross-build currently fails due to the still unresolved rust-lang/rust#62558, so bootstrap kits for 1.38.0 have to be built natively, and will follow shortly. * Bump bootstrap requirements to 1.37.0 except for armv7-unknown-netbsd-eabihf which I've neither managed to cross-build nor build natively. Upstream changes: Version 1.38.0 (2019-09-26) ========================== Language -------- - [The `#[global_allocator]` attribute can now be used in submodules.][62735] - [The `#[deprecated]` attribute can now be used on macros.][62042] Compiler -------- - [Added pipelined compilation support to `rustc`.][62766] This will improve compilation times in some cases. For further information please refer to the [_"Evaluating pipelined rustc compilation"_][pipeline-internals] thread. - [Added tier 3\* support for the `aarch64-uwp-windows-msvc`, `i686-uwp-windows-gnu`, `i686-uwp-windows-msvc`, `x86_64-uwp-windows-gnu`, and `x86_64-uwp-windows-msvc` targets.][60260] - [Added tier 3 support for the `armv7-unknown-linux-gnueabi` and `armv7-unknown-linux-musleabi` targets.][63107] - [Added tier 3 support for the `hexagon-unknown-linux-musl` target.][62814] - [Added tier 3 support for the `riscv32i-unknown-none-elf` target.][62784] \* Refer to Rust's [platform support page][forge-platform-support] for more information on Rust's tiered platform support. Libraries --------- - [`ascii::EscapeDefault` now implements `Clone` and `Display`.][63421] - [Derive macros for prelude traits (e.g. `Clone`, `Debug`, `Hash`) are now available at the same path as the trait.][63056] (e.g. The `Clone` derive macro is available at `std::clone::Clone`). This also makes all built-in macros available in `std`/`core` root. e.g. `std::include_bytes!`. - [`str::Chars` now implements `Debug`.][63000] - [`slice::{concat, connect, join}` now accepts `&[T]` in addition to `&T`.][62528] - [`*const T` and `*mut T` now implement `marker::Unpin`.][62583] - [`Arc<[T]>` and `Rc<[T]>` now implement `FromIterator<T>`.][61953] - [Added euclidean remainder and division operations (`div_euclid`, `rem_euclid`) to all numeric primitives.][61884] Additionally `checked`, `overflowing`, and `wrapping` versions are available for all integer primitives. - [`thread::AccessError` now implements `Clone`, `Copy`, `Eq`, `Error`, and `PartialEq`.][61491] - [`iter::{StepBy, Peekable, Take}` now implement `DoubleEndedIterator`.][61457] Stabilized APIs --------------- - [`<*const T>::cast`] - [`<*mut T>::cast`] - [`Duration::as_secs_f32`] - [`Duration::as_secs_f64`] - [`Duration::div_duration_f32`] - [`Duration::div_duration_f64`] - [`Duration::div_f32`] - [`Duration::div_f64`] - [`Duration::from_secs_f32`] - [`Duration::from_secs_f64`] - [`Duration::mul_f32`] - [`Duration::mul_f64`] - [`any::type_name`] Cargo ----- - [Added pipelined compilation support to `cargo`.][cargo/7143] - [You can now pass the `--features` option multiple times to enable multiple features.][cargo/7084] Misc ---- - [`rustc` will now warn about some incorrect uses of `mem::{uninitialized, zeroed}` that are known to cause undefined behaviour.][63346] Compatibility Notes ------------------- - Unfortunately the [`x86_64-unknown-uefi` platform can not be built][62785] with rustc 1.39.0. - The [`armv7-unknown-linux-gnueabihf` platform is also known to have issues][62896] for certain crates such as libc. [60260]: rust-lang/rust#60260 [61457]: rust-lang/rust#61457 [61491]: rust-lang/rust#61491 [61884]: rust-lang/rust#61884 [61953]: rust-lang/rust#61953 [62042]: rust-lang/rust#62042 [62528]: rust-lang/rust#62528 [62583]: rust-lang/rust#62583 [62735]: rust-lang/rust#62735 [62766]: rust-lang/rust#62766 [62784]: rust-lang/rust#62784 [62785]: rust-lang/rust#62785 [62814]: rust-lang/rust#62814 [62896]: rust-lang/rust#62896 [63000]: rust-lang/rust#63000 [63056]: rust-lang/rust#63056 [63107]: rust-lang/rust#63107 [63346]: rust-lang/rust#63346 [63421]: rust-lang/rust#63421 [cargo/7084]: rust-lang/cargo#7084 [cargo/7143]: rust-lang/cargo#7143 [`<*const T>::cast`]: https://doc.rust-lang.org/std/primitive.pointer.html#method.cast [`<*mut T>::cast`]: https://doc.rust-lang.org/std/primitive.pointer.html#method.cast [`Duration::as_secs_f32`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.as_secs_f32 [`Duration::as_secs_f64`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.as_secs_f64 [`Duration::div_duration_f32`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.div_duration_f32 [`Duration::div_duration_f64`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.div_duration_f64 [`Duration::div_f32`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.div_f32 [`Duration::div_f64`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.div_f64 [`Duration::from_secs_f32`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.from_secs_f32 [`Duration::from_secs_f64`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.from_secs_f64 [`Duration::mul_f32`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.mul_f32 [`Duration::mul_f64`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.mul_f64 [`any::type_name`]: https://doc.rust-lang.org/std/any/fn.type_name.html [forge-platform-support]: https://forge.rust-lang.org/platform-support.html [pipeline-internals]: https://internals.rust-lang.org/t/evaluating-pipelined-rustc-compilation/10199

rust-highfive assigned nrc Jul 17, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 17, 2019

alexcrichton mentioned this pull request Jul 17, 2019

rustc: Stabilize options for pipelined compilation rust-lang/rust#62766

Merged

alexcrichton force-pushed the stabilize-pipelined-compilation branch from dbd6ef2 to 2fd1f06 Compare July 17, 2019 21:13

rust-highfive assigned ehuss and unassigned nrc Jul 17, 2019

alexcrichton mentioned this pull request Jul 17, 2019

Tracking issue for stabilizing "pipelined compilation" rust-lang/rust#60988

Closed

7 tasks

alexcrichton added the relnotes Release-note worthy label Jul 17, 2019

alexcrichton force-pushed the stabilize-pipelined-compilation branch from 2fd1f06 to 09de563 Compare July 30, 2019 14:48

ehuss mentioned this pull request Jul 31, 2019

Build with pipelining enabled fails with error: Unrecognized option: 'json-rendered' #7195

Closed

alexcrichton force-pushed the stabilize-pipelined-compilation branch from 09de563 to 0adbf0a Compare July 31, 2019 15:51

ehuss reviewed Jul 31, 2019

View reviewed changes

alexcrichton force-pushed the stabilize-pipelined-compilation branch from 0adbf0a to daa1bce Compare July 31, 2019 21:58

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 31, 2019

bors merged commit daa1bce into rust-lang:master Jul 31, 2019

alexcrichton deleted the stabilize-pipelined-compilation branch July 31, 2019 22:16

This was referenced Jul 31, 2019

tests: Enable features to fix unstabilized #[bench] #7198

Merged

Update cargo, rls rust-lang/rust#63181

Merged

alexcrichton mentioned this pull request Aug 5, 2019

Add support for customizing JSON diagnostics from Cargo #7214

Merged

ehuss added this to the 1.38.0 milestone Feb 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable pipelined compilation by default #7143

Enable pipelined compilation by default #7143

alexcrichton commented Jul 17, 2019 •

edited

Loading

rust-highfive commented Jul 17, 2019

alexcrichton commented Jul 17, 2019

alexcrichton commented Jul 30, 2019

ehuss commented Jul 31, 2019

alexcrichton commented Jul 31, 2019

ehuss Jul 31, 2019

alexcrichton Jul 31, 2019

ehuss commented Jul 31, 2019

alexcrichton commented Jul 31, 2019

bors commented Jul 31, 2019

bors commented Jul 31, 2019

bors commented Jul 31, 2019

Enable pipelined compilation by default #7143

Enable pipelined compilation by default #7143

Conversation

alexcrichton commented Jul 17, 2019 • edited Loading

rust-highfive commented Jul 17, 2019

alexcrichton commented Jul 17, 2019

alexcrichton commented Jul 30, 2019

ehuss commented Jul 31, 2019

alexcrichton commented Jul 31, 2019

ehuss Jul 31, 2019

Choose a reason for hiding this comment

alexcrichton Jul 31, 2019

Choose a reason for hiding this comment

ehuss commented Jul 31, 2019

alexcrichton commented Jul 31, 2019

bors commented Jul 31, 2019

bors commented Jul 31, 2019

bors commented Jul 31, 2019

alexcrichton commented Jul 17, 2019 •

edited

Loading