Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows: Use a pipe relay for chaining pipes #95841

Merged
merged 1 commit into from
Apr 15, 2022

Conversation

ChrisDenton
Copy link
Member

@ChrisDenton ChrisDenton commented Apr 9, 2022

Fixes #95759

This fixes the issue by chaining pipes synchronously and manually pumping messages between them. It's not ideal but it has the advantage of not costing anything if pipes are not chained ("don't pay for what you don't use") and it also avoids breaking existing code that rely on our end of the pipe being asynchronous (which includes rustc's own testing framework).

Libraries can avoid needing this by using their own pipes to chain commands.

@rust-highfive
Copy link
Collaborator

r? @kennytm

(rust-highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Apr 9, 2022
@ChrisDenton
Copy link
Member Author

ChrisDenton commented Apr 10, 2022

I'm not sure if kenneytm is around at the moment and github is suggesting dtolnay so...

r? @dtolnay

Not to rush review or anything but a test is intermittently failing in CI while this issue remains unresolved.

@ChrisDenton
Copy link
Member Author

Ok, so I'm guessing dtolnay is busy too and people are getting frustrated with the CI failures. So erm, let's try...

r? @m-ou-se

I'm totally available to answer any questions or concerns.

@rust-highfive rust-highfive assigned m-ou-se and unassigned dtolnay Apr 13, 2022
@m-ou-se m-ou-se added O-windows Operating system: Windows T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Apr 13, 2022
@m-ou-se
Copy link
Member

m-ou-se commented Apr 13, 2022

rely on our end of the pipe being asynchronous (which includes rustc's own testing framework).

Could you elaborate a bit on how code relies on that?

@m-ou-se
Copy link
Member

m-ou-se commented Apr 13, 2022

Can you point me to the right documentation about asynchronous pipes?

I did a quick search, but only found a page that says:

Asynchronous (overlapped) read and write operations are not supported by anonymous pipes.

@m-ou-se m-ou-se added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 13, 2022
@ChrisDenton
Copy link
Member Author

rely on our end of the pipe being asynchronous (which includes rustc's own testing framework).

Could you elaborate a bit on how code relies on that?

It uses miow to do asynchronous I/O. See

pub fn read2(
out_pipe: ChildStdout,
err_pipe: ChildStderr,
data: &mut dyn FnMut(bool, &mut Vec<u8>, bool),
) -> io::Result<()> {
let mut out = Vec::new();
let mut err = Vec::new();
let port = CompletionPort::new(1)?;
port.add_handle(0, &out_pipe)?;
port.add_handle(1, &err_pipe)?;

I guess it wouldn't be too hard to change but I'm worried that the wider ecosystem is also relying on these being opened for "overlapped" I/O as the handles are made available via the AsRawHandle trait.

Can you point me to the right documentation about asynchronous pipes?

We actually use named pipes

let handle = c::CreateNamedPipeW(

The docs are here Synchronous and Overlapped Pipe I/O

@ChrisDenton
Copy link
Member Author

This comment has a bit of background on the issue of async and sync pipe I/O: #38811 (comment)

After that, Rust switched to using synchronous I/O for the end of the pipe given to the child process. However, we continued to use "Overlapped" (aka asynchronous) I/O for our end. Which is fine until we give our end of the pipe to another process.

@m-ou-se
Copy link
Member

m-ou-se commented Apr 13, 2022

We actually use named pipes

Ah, that explains my confusion.

After that, Rust switched to using synchronous I/O for the end of the pipe given to the child process. However, we continued to use "Overlapped" (aka asynchronous) I/O for our end. Which is fine until we give our end of the pipe to another process.

I see. I dug a bit through history, reading #38835 that switched one end of the pipe to synchonous code, to try to understand this. Here's my summary, please correct me if I'm wrong:


Just like on Unix, it's possible to use pipes for stdin/stdout/stderr of a child process. Windows supports basic anonymous pipes, but those don't support 'overlapped' aka asynchronous mode, which we use to be able to read from both stdout and stderr in parallel. So instead, we open a named pipe (\\.\pipe\__rust_anonymous_pipe1__.{pid}.{random}), because those do support overlapped mode.

Using a pipe end that's in overlapped mode as if it's not in overlapped mode might result in bad things, so we shouldn't pass those to child processes. So we open only one end of the pipe in overlapped mode: the end we keep. The comment on anon_pipe talks about 'our' and 'their' ends of the pipe, making the assumption that 'our' end will always stay on our side:

/// subtly different. Here we'll return two pipes in the `Pipes` return value,
/// but one is intended for "us" where as the other is intended for "someone
/// else".

However, this is a wrong assumption, because 'our' half of the pipe might be given to another subprocess, through .stdin(child.stdout) when spawning another process, possibly resulting in undefined behaviour in that process.

Now, the only reason we enable overlapping mode, is for read2, which reads from two pipes at once:

pub fn read2(p1: AnonPipe, v1: &mut Vec<u8>, p2: AnonPipe, v2: &mut Vec<u8>) -> io::Result<()> {

read2 is only used in one place: in Child::wait_with_output in the case both stdout and stderr are redirected:

(Some(out), Some(err)) => {
let res = read2(out.inner, &mut stdout, err.inner, &mut stderr);

It might be possible to implement read2 without using overlapped mode. If we do that, we can use 'real' anonymous pipes rather than our \\.\pipe\__rust_anonymous_pipe1__ hack, and the issue with passing overlapped mode handles to subprocesses would be fixed.

However, there is at least one case of another crate assuming overlapped mode is enabled for subprocess pipes: our very own compiletest, which implements its own read2 function based on that assumption:

pub fn read2(
out_pipe: ChildStdout,
err_pipe: ChildStderr,
data: &mut dyn FnMut(bool, &mut Vec<u8>, bool),
) -> io::Result<()> {
let mut out = Vec::new();
let mut err = Vec::new();
let port = CompletionPort::new(1)?;
port.add_handle(0, &out_pipe)?;
port.add_handle(1, &err_pipe)?;

Other crates might depend on it too, for example if they use miow::iocp::CompletionPort::add_handle on child.stdout. We don't know if anything other than compiletest does that.


Based on that understanding, I have a few questions:

  • Can we implement read2 for (anonymous) non-overlapped mode pipes?
  • Is is possible to turn off overlapped mode on a handle, or get a clone/duplicate of a handle with overlapped mode disabled? (E.g. through ReOpenFile()?)
  • In which ways would code break that assumes overlapped mode is enabled if we stop enabling it? Will they reliably error, or would that result in undefined/unpredictable behaviour?

@ChrisDenton
Copy link
Member Author

ChrisDenton commented Apr 13, 2022

  • Can we implement read2 for (anonymous) non-overlapped mode pipes?

Yes. It would require spinning up a thread to make sure we don't block either pipe. If a pipe buffer is full then it'll be blocked until drained so we could otherwise end up deadlocked.


  • Is is possible to turn off overlapped mode on a handle, or get a clone/duplicate of a handle with overlapped mode disabled? (E.g. through ReOpenFile()?)

No, it can't be turned off once it's on. Pipes use a server/client model. You can create more than one server for the same pipe (both overlapped and not) but once a client "connects" it must use the same server. It can reconnect to the pipe (thus potentially being given a different server) but the child process won't know that and in any case we can't coordinate a disconnect/reconnect. Essentially once we give a process the "theirs" handle we can't really change either side of the pipe.


  • In which ways would code break that assumes overlapped mode is enabled if we stop enabling it? Will they reliably error, or would that result in undefined/unpredictable behaviour?

For the pipe server it should be a reliable error. I tested this by removing c::FILE_FLAG_OVERLAPPED from:

let mut flags = c::FILE_FLAG_FIRST_PIPE_INSTANCE | c::FILE_FLAG_OVERLAPPED;

This should be equivalent to using anonymous pipes because (from Anonymous Pipe Operations):

Anonymous pipes are implemented using a named pipe with a unique name.

When attempting to use overlapped I/O (i.e via miow), this line:

port.add_handle(0, &out_pipe)?;

Will give the error:

Os { code: 87, kind: InvalidInput, message: "The parameter is incorrect." }

EDIT: In code above, miow is using I/O completion ports which do error if given a handle that wasn't opened in overlapped mode. However, if using ReadFileEx, WriteFileEx, ReadFile or WriteFile without I/O completion ports, the call will simply block. This could be a problem for applications relying on asynchronous I/O.

@m-ou-se
Copy link
Member

m-ou-se commented Apr 14, 2022

Yes. It would require spinning up a thread to make sure we don't block either pipe. If a pipe buffer is full then it'll be blocked until drained so we could otherwise end up deadlocked.

Ah, I see. It looks like that was exactly what we did before we switched to overlapped mode pipes.

So, it looks like we'll always have to spawn threads to handle certain situations:

  • Either we spawn a thread in read2 when waiting on both stdout and stderr in Child::wait_with_output (what we did a long time ago);
  • Or, we spawn a thread when attaching one child process' stdin/out/err to that of another child (what this PR does).

Neither is great, but it seems to me that option 2 (this PR) is the better one, since that's a less common scenario.

@m-ou-se
Copy link
Member

m-ou-se commented Apr 14, 2022

Can we reduce the number of situations in which this is done by only using asynchronous pipes if both stderr and stdout of a child process are pipes? (And never doing it for stdin.)

@m-ou-se
Copy link
Member

m-ou-se commented Apr 14, 2022

I'm approving this for now so the issue gets fixed, but I'd like to see if we can reduce the number of cases in which this workaround is necessary.

@bors r+ rollup=never

@bors
Copy link
Contributor

bors commented Apr 14, 2022

📌 Commit 9013054 has been approved by m-ou-se

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 14, 2022
@ChrisDenton
Copy link
Member Author

Can we reduce the number of situations in which this is done by only using asynchronous pipes if both stderr and stdout of a child process are pipes? (And never doing it for stdin.)

For sure, I can look into this. I'd still worry a bit about people relying on them being asynchronous pipes but I guess there are no guarantees when using as_raw_handle on stdlib types.

@m-ou-se
Copy link
Member

m-ou-se commented Apr 14, 2022

Maybe we can add something like .use_async_pipes() to Windows' CommandExt, and enable that automatically only when both stderr and stdout are pipes. Then users that really need these pipes to be async in other situations can still manually enable it.

@ChrisDenton
Copy link
Member Author

That does sound like a good compromise.

@bors
Copy link
Contributor

bors commented Apr 14, 2022

⌛ Testing commit 9013054 with merge 9f434ff730b39d84ff9c40f47a1396a8fdd758db...

@bors
Copy link
Contributor

bors commented Apr 14, 2022

💥 Test timed out

@bors bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Apr 14, 2022
@ChrisDenton
Copy link
Member Author

@bors retry timeout waiting for a runner to pick up aarch64-gnu, self-hosted, ARM64, linux

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 15, 2022
@RalfJung
Copy link
Member

@bors p=10
higher priority than rollups, because this keeps blocking other PRs

@@ -552,7 +558,7 @@ impl Stdio {

impl From<AnonPipe> for Stdio {
fn from(pipe: AnonPipe) -> Stdio {
Stdio::Handle(pipe.into_handle())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this line the key bug? Stdio::Handle expects a sync pipe but pipe.into_handle() produces an async pipe?
That seems like maybe there should be comments in some places so that this is not reverted in the future when the context of this PR has been lost. (But let's add comments, if any, in a separate PR, so that this one can land ASAP.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I'll add that in a follow up PR.

@bors
Copy link
Contributor

bors commented Apr 15, 2022

⌛ Testing commit 9013054 with merge 69a5ae3...

@rust-log-analyzer
Copy link
Collaborator

A job failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

@RalfJung
Copy link
Member

(I think that is the log from the old failure, the timeout)

@ChrisDenton
Copy link
Member Author

Indeed, CI looks like it is still running. It gave me a scare though. 😌

@bors
Copy link
Contributor

bors commented Apr 15, 2022

☀️ Test successful - checks-actions
Approved by: m-ou-se
Pushing 69a5ae3 to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Apr 15, 2022
@bors bors merged commit 69a5ae3 into rust-lang:master Apr 15, 2022
@rustbot rustbot added this to the 1.62.0 milestone Apr 15, 2022
@ChrisDenton ChrisDenton deleted the pipe-server branch April 15, 2022 15:32
@rust-timer
Copy link
Collaborator

Finished benchmarking commit (69a5ae3): comparison url.

Summary:

  • Primary benchmarks: no relevant changes found
  • Secondary benchmarks: mixed results
Regressions 😿
(primary)
Regressions 😿
(secondary)
Improvements 🎉
(primary)
Improvements 🎉
(secondary)
All 😿 🎉
(primary)
count1 0 1 0 1 0
mean2 N/A 0.7% N/A -0.3% N/A
max N/A 0.7% N/A -0.3% N/A

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

@rustbot label: -perf-regression

Footnotes

  1. number of relevant changes

  2. the arithmetic mean of the percent change

wip-sync pushed a commit to NetBSD/pkgsrc-wip that referenced this pull request Jul 12, 2022
Pkgsrc changes:
 * adapt patches (libc crate version bump)
 * no longer pass -I/usr/pkg/include through via gcc-wrap script.
   Attempt at fixing version skew with curl package vs. internal
   version of curl.
 * new checksums

Upstream changes:

Version 1.62.0 (2022-06-30)
==========================

Language
--------

- [Stabilize `#[derive(Default)]` on enums with a `#[default]` variant][94457]
- [Stop validating some checks in dead code after functions with
  uninhabited return types][93313]
- [Fix constants not getting dropped if part of a diverging expression][94775]
- [Support unit struct/enum variant in destructuring assignment][95380]
- [Remove mutable_borrow_reservation_conflict lint and allow the
  code pattern][96268]

Compiler
--------

- [linker: Stop using whole-archive on dependencies of dylibs][96436]
- [Make `unaligned_references` lint deny-by-default][95372]
  This lint is also a future compatibility lint, and is expected to eventually
  become a hard error.
- [Only add codegen backend to dep info if -Zbinary-dep-depinfo is used][93969]
- [Reject `#[thread_local]` attribute on non-static items][95006]
- [Add tier 3 `aarch64-pc-windows-gnullvm` and `x86_64-pc-windows-gnullvm`
  targets\*][94872]
- [Implement a lint to warn about unused macro rules][96150]
- [Promote `x86_64-unknown-none` target to Tier 2\*][95705]

\* Refer to Rust's [platform support page][platform-support-doc] for more
   information on Rust's tiered platform support.

Libraries
---------

- [Move `CStr` to libcore, and `CString` to liballoc][94079]
- [Windows: Use a pipe relay for chaining pipes][95841]
- [Replace Linux Mutex and Condvar with futex based ones.][95035]
- [Replace RwLock by a futex based one on Linux][95801]
- [std: directly use pthread in UNIX parker implementation][96393]

Stabilized APIs
---------------

- [`bool::then_some`]
- [`f32::total_cmp`]
- [`f64::total_cmp`]
- [`Stdin::lines`]
- [`windows::CommandExt::raw_arg`]
- [`impl<T: Default> Default for AssertUnwindSafe<T>`]
- [`From<Rc<str>> for Rc<[u8]>`][rc-u8-from-str]
- [`From<Arc<str>> for Arc<[u8]>`][arc-u8-from-str]
- [`FusedIterator for EncodeWide`]
- [RDM intrinsics on aarch64][stdarch/1285]

Clippy
------

- [Create clippy lint against unexpectedly late drop for temporaries
  in match scrutinee expressions][94206]

Cargo
-----

- Added the `cargo add` command for adding dependencies to `Cargo.toml` from
  the command-line.
  [docs](https://doc.rust-lang.org/nightly/cargo/commands/cargo-add.html)
- Package ID specs now support `name@version` syntax in addition to the
  previous `name:version` to align with the behavior in `cargo add` and other
  tools. `cargo install` and `cargo yank` also now support this syntax so the
  version does not need to passed as a separate flag.
- The `git` and `registry` directories in Cargo's home directory (usually
  `~/.cargo`) are now marked as cache directories so that they are not
  included in backups or content indexing (on Windows).
- Added automatic `@` argfile support, which will use "response files" if the
  command-line to `rustc` exceeds the operating system's limit.

Compatibility Notes
-------------------

- `cargo test` now passes `--target` to `rustdoc` if the specified target is
  the same as the host target.
  [#10594](rust-lang/cargo#10594)
- [rustdoc: Remove .woff font files][96279]
- [Enforce Copy bounds for repeat elements while considering lifetimes][95819]

Internal Changes
----------------

- [Unify ReentrantMutex implementations across all platforms][96042]

These changes provide no direct user facing benefits, but represent
significant improvements to the internals and overall performance
of rustc and related tools.

[93313]: rust-lang/rust#93313
[93969]: rust-lang/rust#93969
[94079]: rust-lang/rust#94079
[94206]: rust-lang/rust#94206
[94457]: rust-lang/rust#94457
[94775]: rust-lang/rust#94775
[94872]: rust-lang/rust#94872
[95006]: rust-lang/rust#95006
[95035]: rust-lang/rust#95035
[95372]: rust-lang/rust#95372
[95380]: rust-lang/rust#95380
[95431]: rust-lang/rust#95431
[95705]: rust-lang/rust#95705
[95801]: rust-lang/rust#95801
[95819]: rust-lang/rust#95819
[95841]: rust-lang/rust#95841
[96042]: rust-lang/rust#96042
[96150]: rust-lang/rust#96150
[96268]: rust-lang/rust#96268
[96279]: rust-lang/rust#96279
[96393]: rust-lang/rust#96393
[96436]: rust-lang/rust#96436
[96557]: rust-lang/rust#96557

[`bool::then_some`]: https://doc.rust-lang.org/stable/std/primitive.bool.html#method.then_some
[`f32::total_cmp`]: https://doc.rust-lang.org/stable/std/primitive.f32.html#method.total_cmp
[`f64::total_cmp`]: https://doc.rust-lang.org/stable/std/primitive.f64.html#method.total_cmp
[`Stdin::lines`]: https://doc.rust-lang.org/stable/std/io/struct.Stdin.html#method.lines
[`impl<T: Default> Default for AssertUnwindSafe<T>`]: https://doc.rust-lang.org/stable/std/panic/struct.AssertUnwindSafe.html#impl-Default
[rc-u8-from-str]: https://doc.rust-lang.org/stable/std/rc/struct.Rc.html#impl-From%3CRc%3Cstr%3E%3E
[arc-u8-from-str]: https://doc.rust-lang.org/stable/std/sync/struct.Arc.html#impl-From%3CArc%3Cstr%3E%3E
[stdarch/1285]: rust-lang/stdarch#1285
[`windows::CommandExt::raw_arg`]: https://doc.rust-lang.org/stable/std/os/windows/process/trait.CommandExt.html#tymethod.raw_arg
[`FusedIterator for EncodeWide`]: https://doc.rust-lang.org/stable/std/os/windows/ffi/struct.EncodeWide.html#impl-FusedIterator
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Aug 31, 2022
Pkgsrc changes:

 * Bump required GCC to 7 (same as LLVM) to avoid ABI issues
   Fixes native i386 and powerpc 8.x build w/pkgsrc LLVM 14
 * Bump available bootstraps to 1.61.0.
 * Also unlimit stacksize
 * Sync patches over from wip/rust
 * Adjust line number in patches which had non-zero offsets.
 * no longer pass -I/usr/pkg/include through via gcc-wrap script
   when building natively.  Attempt at fixing version skew with curl
   package vs. internal version of curl (may not work...)
 * The NetBSD bootstraps now use .xz compression.
 * Use mk/atomic64.mk.  Still have conditional for libatomic-links.
 * Default to using the internal LLVM when cross-building.


Upstream changes:

Version 1.62.1 (2022-07-19)
==========================

Rust 1.62.1 addresses a few recent regressions in the compiler and standard
library, and also mitigates a CPU vulnerability on Intel SGX.

* [The compiler fixed unsound function coercions involving `impl
  Trait` return types.][98608]
* [The compiler fixed an incremental compilation bug with `async
  fn` lifetimes.][98890]
* [Windows added a fallback for overlapped I/O in synchronous reads
  and writes.][98950]
* [The `x86_64-fortanix-unknown-sgx` target added a mitigation for the
  MMIO stale data vulnerability][98126], advisory [INTEL-SA-00615].

[98608]: rust-lang/rust#98608
[98890]: rust-lang/rust#98890
[98950]: rust-lang/rust#98950
[98126]: rust-lang/rust#98126
[INTEL-SA-00615]: https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00615.html


Version 1.62.0 (2022-06-30)
==========================

Language
--------

- [Stabilize `#[derive(Default)]` on enums with a `#[default]` variant][94457]
- [Stop validating some checks in dead code after functions with
  uninhabited return types][93313]
- [Fix constants not getting dropped if part of a diverging expression][94775]
- [Support unit struct/enum variant in destructuring assignment][95380]
- [Remove mutable_borrow_reservation_conflict lint and allow the
  code pattern][96268]

Compiler
--------

- [linker: Stop using whole-archive on dependencies of dylibs][96436]
- [Make `unaligned_references` lint deny-by-default][95372]
  This lint is also a future compatibility lint, and is expected to eventually
  become a hard error.
- [Only add codegen backend to dep info if -Zbinary-dep-depinfo is used][93969]
- [Reject `#[thread_local]` attribute on non-static items][95006]
- [Add tier 3 `aarch64-pc-windows-gnullvm` and `x86_64-pc-windows-gnullvm`
  targets\*][94872]
- [Implement a lint to warn about unused macro rules][96150]
- [Promote `x86_64-unknown-none` target to Tier 2\*][95705]

\* Refer to Rust's [platform support page][platform-support-doc] for more
   information on Rust's tiered platform support.

Libraries
---------

- [Move `CStr` to libcore, and `CString` to liballoc][94079]
- [Windows: Use a pipe relay for chaining pipes][95841]
- [Replace Linux Mutex and Condvar with futex based ones.][95035]
- [Replace RwLock by a futex based one on Linux][95801]
- [std: directly use pthread in UNIX parker implementation][96393]

Stabilized APIs
---------------

- [`bool::then_some`]
- [`f32::total_cmp`]
- [`f64::total_cmp`]
- [`Stdin::lines`]
- [`windows::CommandExt::raw_arg`]
- [`impl<T: Default> Default for AssertUnwindSafe<T>`]
- [`From<Rc<str>> for Rc<[u8]>`][rc-u8-from-str]
- [`From<Arc<str>> for Arc<[u8]>`][arc-u8-from-str]
- [`FusedIterator for EncodeWide`]
- [RDM intrinsics on aarch64][stdarch/1285]

Clippy
------

- [Create clippy lint against unexpectedly late drop for temporaries
  in match scrutinee expressions][94206]

Cargo
-----

- Added the `cargo add` command for adding dependencies to `Cargo.toml` from
  the command-line.
  [docs](https://doc.rust-lang.org/nightly/cargo/commands/cargo-add.html)
- Package ID specs now support `name@version` syntax in addition to the
  previous `name:version` to align with the behavior in `cargo add` and other
  tools. `cargo install` and `cargo yank` also now support this syntax so the
  version does not need to passed as a separate flag.
- The `git` and `registry` directories in Cargo's home directory (usually
  `~/.cargo`) are now marked as cache directories so that they are not
  included in backups or content indexing (on Windows).
- Added automatic `@` argfile support, which will use "response files" if the
  command-line to `rustc` exceeds the operating system's limit.

Compatibility Notes
-------------------

- `cargo test` now passes `--target` to `rustdoc` if the specified target is
  the same as the host target.
  [#10594](rust-lang/cargo#10594)
- [rustdoc: Remove .woff font files][96279]
- [Enforce Copy bounds for repeat elements while considering lifetimes][95819]

Internal Changes
----------------

- [Unify ReentrantMutex implementations across all platforms][96042]

These changes provide no direct user facing benefits, but represent
significant improvements to the internals and overall performance
of rustc and related tools.

[93313]: rust-lang/rust#93313
[93969]: rust-lang/rust#93969
[94079]: rust-lang/rust#94079
[94206]: rust-lang/rust#94206
[94457]: rust-lang/rust#94457
[94775]: rust-lang/rust#94775
[94872]: rust-lang/rust#94872
[95006]: rust-lang/rust#95006
[95035]: rust-lang/rust#95035
[95372]: rust-lang/rust#95372
[95380]: rust-lang/rust#95380
[95431]: rust-lang/rust#95431
[95705]: rust-lang/rust#95705
[95801]: rust-lang/rust#95801
[95819]: rust-lang/rust#95819
[95841]: rust-lang/rust#95841
[96042]: rust-lang/rust#96042
[96150]: rust-lang/rust#96150
[96268]: rust-lang/rust#96268
[96279]: rust-lang/rust#96279
[96393]: rust-lang/rust#96393
[96436]: rust-lang/rust#96436
[96557]: rust-lang/rust#96557

[`bool::then_some`]: https://doc.rust-lang.org/stable/std/primitive.bool.html#method.then_some
[`f32::total_cmp`]: https://doc.rust-lang.org/stable/std/primitive.f32.html#method.total_cmp
[`f64::total_cmp`]: https://doc.rust-lang.org/stable/std/primitive.f64.html#method.total_cmp
[`Stdin::lines`]: https://doc.rust-lang.org/stable/std/io/struct.Stdin.html#method.lines
[`impl<T: Default> Default for AssertUnwindSafe<T>`]: https://doc.rust-lang.org/stable/std/panic/struct.AssertUnwindSafe.html#impl-Default
[rc-u8-from-str]: https://doc.rust-lang.org/stable/std/rc/struct.Rc.html#impl-From%3CRc%3Cstr%3E%3E
[arc-u8-from-str]: https://doc.rust-lang.org/stable/std/sync/struct.Arc.html#impl-From%3CArc%3Cstr%3E%3E
[stdarch/1285]: rust-lang/stdarch#1285
[`windows::CommandExt::raw_arg`]: https://doc.rust-lang.org/stable/std/os/windows/process/trait.CommandExt.html#tymethod.raw_arg
[`FusedIterator for EncodeWide`]: https://doc.rust-lang.org/stable/std/os/windows/ffi/struct.EncodeWide.html#impl-FusedIterator


Version 1.61.0 (2022-05-19)
==========================

Language
--------

- [`const fn` signatures can now include generic trait bounds][93827]
- [`const fn` signatures can now use `impl Trait` in argument and return
  position][93827]
- [Function pointers can now be created, cast, and passed around in a
  `const fn`][93827]
- [Recursive calls can now set the value of a function's opaque
  `impl Trait` return type][94081]

Compiler
--------

- [Linking modifier syntax in `#[link]` attributes and on the command
  line, as well as the `whole-archive` modifier specifically, are now
  supported][93901]
- [The `char` type is now described as UTF-32 in debuginfo][89887]
- The [`#[target_feature]`][target_feature] attribute
  [can now be used with aarch64 features][90621]
- X86 [`#[target_feature = "adx"]` is now stable][93745]

Libraries
---------

- [`ManuallyDrop<T>` is now documented to have the same layout as `T`][88375]
- [`#[ignore = "#"]` messages are printed when running tests][92714]
- [Consistently show absent stdio handles on Windows as NULL handles][93263]
- [Make `std::io::stdio::lock()` return `'static` handles.][93965]
  Previously, the creation of locked handles to stdin/stdout/stderr would
  borrow the handles being locked, which prevented writing
  `let out = std::io::stdout().lock();` because `out` would outlive
  the return value of `stdout()`.
  Such code now works, eliminating a common pitfall that affected many
  Rust users.
- [`Vec::from_raw_parts` is now less restrictive about its inputs][95016]
- [`std::thread::available_parallelism` now takes cgroup quotas into
  account.][92697] Since `available_parallelism` is often used to create a
  thread pool for parallel computation, which may be CPU-bound for
  performance, `available_parallelism` will return a value consistent with
  the ability to use that many threads continuously, if possible.
  For instance, in a container with 8 virtual CPUs but quotas only allowing
  for 50% usage, `available_parallelism` will return 4.

Stabilized APIs
---------------

- [`Pin::static_mut`]
- [`Pin::static_ref`]
- [`Vec::retain_mut`]
- [`VecDeque::retain_mut`]
- [`Write` for `Cursor<[u8; N]>`][cursor-write-array]
- [`std::os::unix::net::SocketAddr::from_pathname`]
- [`std::process::ExitCode`] and [`std::process::Termination`].
  The stabilization of these two API s now makes it possible for
  programs to return errors from `main` with custom exit codes.
- [`std::thread::JoinHandle::is_finished`]

These APIs are now usable in const contexts:

- [`<*const T>::offset` and `<*mut T>::offset`][ptr-offset]
- [`<*const T>::wrapping_offset` and `<*mut T>::wrapping_offset`]
  [ptr-wrapping_offset]
- [`<*const T>::add` and `<*mut T>::add`][ptr-add]
- [`<*const T>::sub` and `<*mut T>::sub`][ptr-sub]
- [`<*const T>::wrapping_add` and `<*mut T>::wrapping_add`][ptr-wrapping_add]
- [`<*const T>::wrapping_sub` and `<*mut T>::wrapping_sub`][ptr-wrapping_sub]
- [`<[T]>::as_mut_ptr`][slice-as_mut_ptr]
- [`<[T]>::as_ptr_range`][slice-as_ptr_range]
- [`<[T]>::as_mut_ptr_range`][slice-as_mut_ptr_range]

Cargo
-----

No feature changes, but see compatibility notes.

Compatibility Notes
-------------------

- Previously native static libraries were linked as `whole-archive` in
  some cases, but now rustc tries not to use `whole-archive` unless
  explicitly requested. This [change][93901] may result in linking errors
  in some cases. To fix such errors, native libraries linked from the
  command line, build scripts, or [`#[link]` attributes][link-attr] need to
  - (more common) either be reordered to respect dependencies between them
    (if `a` depends on `b` then `a` should go first and `b` second)
  - (less common) or be updated to use the [`+whole-archive`] modifier.
- [Catching a second unwind from FFI code while cleaning up from a Rust
  panic now causes the process to abort][92911]
- [Proc macros no longer see `ident` matchers wrapped in groups][92472]
- [The number of `#` in `r#` raw string literals is now required to be
  less than 256][95251]
- [When checking that a dyn type satisfies a trait bound, supertrait
  bounds are now enforced][92285]
- [`cargo vendor` now only accepts one value for each `--sync` flag]
  [cargo/10448]
- [`cfg` predicates in `all()` and `any()` are always evaluated to detect
  errors, instead of short-circuiting.][94295] The compatibility
  considerations here arise in nightly-only code that used the
  short-circuiting behavior of `all` to write something like
  `cfg(all(feature = "nightly", syntax-requiring-nightly))`, which
  will now fail to compile. Instead, use either `cfg_attr(feature
  = "nightly", ...)` or nested uses of `cfg`.
- [bootstrap: static-libstdcpp is now enabled by default, and can
  now be disabled when llvm-tools is enabled][94832]

Internal Changes
----------------

These changes provide no direct user facing benefits, but represent
significant improvements to the internals and overall performance
of rustc and related tools.

- [debuginfo: Refactor debuginfo generation for types][94261]
- [Remove the everybody loops pass][93913]

[88375]: rust-lang/rust#88375
[89887]: rust-lang/rust#89887
[90621]: rust-lang/rust#90621
[92285]: rust-lang/rust#92285
[92472]: rust-lang/rust#92472
[92697]: rust-lang/rust#92697
[92714]: rust-lang/rust#92714
[92911]: rust-lang/rust#92911
[93263]: rust-lang/rust#93263
[93745]: rust-lang/rust#93745
[93827]: rust-lang/rust#93827
[93901]: rust-lang/rust#93901
[93913]: rust-lang/rust#93913
[93965]: rust-lang/rust#93965
[94081]: rust-lang/rust#94081
[94261]: rust-lang/rust#94261
[94295]: rust-lang/rust#94295
[94832]: rust-lang/rust#94832
[95016]: rust-lang/rust#95016
[95251]: rust-lang/rust#95251
[`+whole-archive`]: https://doc.rust-lang.org/stable/rustc/command-line-arguments.html#linking-modifiers-whole-archive
[`Pin::static_mut`]: https://doc.rust-lang.org/stable/std/pin/struct.Pin.html#method.static_mut
[`Pin::static_ref`]: https://doc.rust-lang.org/stable/std/pin/struct.Pin.html#method.static_ref
[`Vec::retain_mut`]: https://doc.rust-lang.org/stable/std/vec/struct.Vec.html#method.retain_mut
[`VecDeque::retain_mut`]: https://doc.rust-lang.org/stable/std/collections/struct.VecDeque.html#method.retain_mut
[`std::os::unix::net::SocketAddr::from_pathname`]: https://doc.rust-lang.org/stable/std/os/unix/net/struct.SocketAddr.html#method.from_pathname
[`std::process::ExitCode`]: https://doc.rust-lang.org/stable/std/process/struct.ExitCode.html
[`std::process::Termination`]: https://doc.rust-lang.org/stable/std/process/trait.Termination.html
[`std::thread::JoinHandle::is_finished`]: https://doc.rust-lang.org/stable/std/thread/struct.JoinHandle.html#method.is_finished
[cargo/10448]: rust-lang/cargo#10448
[cursor-write-array]: https://doc.rust-lang.org/stable/std/io/struct.Cursor.html#impl-Write-4
[link-attr]: https://doc.rust-lang.org/stable/reference/items/external-blocks.html#the-link-attribute
[ptr-add]: https://doc.rust-lang.org/stable/std/primitive.pointer.html#method.add
[ptr-offset]: https://doc.rust-lang.org/stable/std/primitive.pointer.html#method.offset
[ptr-sub]: https://doc.rust-lang.org/stable/std/primitive.pointer.html#method.sub
[ptr-wrapping_add]: https://doc.rust-lang.org/stable/std/primitive.pointer.html#method.wrapping_add
[ptr-wrapping_offset]: https://doc.rust-lang.org/stable/std/primitive.pointer.html#method.wrapping_offset
[ptr-wrapping_sub]: https://doc.rust-lang.org/stable/std/primitive.pointer.html#method.wrapping_sub
[slice-as_mut_ptr]: https://doc.rust-lang.org/stable/std/primitive.slice.html#method.as_mut_ptr
[slice-as_mut_ptr_range]: https://doc.rust-lang.org/stable/std/primitive.slice.html#method.as_mut_ptr_range
[slice-as_ptr_range]: https://doc.rust-lang.org/stable/std/primitive.slice.html#method.as_ptr_range
[target_feature]: https://doc.rust-lang.org/reference/attributes/codegen.html#the-target_feature-attribute
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
merged-by-bors This PR was explicitly merged by bors. O-windows Operating system: Windows S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-libs Relevant to the library team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Windows: std::process::Command may pass asynchronous pipes to child process
10 participants