Precise capturing

[traviscross]

To fully stabilize, in Rust 2024, the Lifetime Capture Rules 2024 that we accepted in RFC 3498, we need to stabilize some means of precise capturing. This RFC provides that means.

This RFC adds use<..> syntax for specifying which generic parameters should be captured in an opaque RPIT-like impl Trait type, e.g. impl Trait + use<'t, T> (as amended in rust-lang/rust#125836). This solves the problem of overcapturing and will allow the Lifetime Capture Rules 2024 to be fully stabilized for RPIT in Rust 2024.

One way to think about use<..> is that, in Rust use brings things into scope, and here we are bringing certain generic parameters into scope for the hidden type.

For some history about the progress toward this feature predating this RFC, see this comment.

Rendered

Note that the final syntax was left as an open question in this RFC, and was later decided in:

• Syntax for precise capturing: impl Trait + use<..> rust#125836

In the end, we decided that use<..> would appear as a syntactic bound.

Tracking:

• Tracking issue for precise_capturing syntax rust#123432

[juntyr]

It might be nice to allow use<..> from the start as an explicit way to capture all in-scope generic parameters. Using this short-hand together with explicit explicit captures or excluded captures should still remain future work.

Especially in code where precise captures would be frequently used, it might be preferable to always be explicit, even in cases where the implicit capture-all is what we want. Similar to how the syntax is used with struct initialisation, using use<..> could mean that we know that we will always want to capture all generics, even if more are added, whereas manually enumerating each one might mean that we want to explicitly have to choose with every newly added generic.

Furthermore, since the stabilisation strategy suggests that at first only capturing all parameters may be supported, supporting use<..> from the start would make this shorter to type.

[Jules-Bertholet]

An alternative meaning for use<..> is "capture all APITs".

[petrochenkov]

cannot be captured with use<..>

Are they captured implicitly now?
If they are, then what is the opt out of that?

Are unnamed lifetimes implicitly captured by RPIT?
What is the opt out in that case?
(If unnamed early bound lifetimes are possible at all.)

[compiler-errors]

Nope, we don't capture APITs or elided lifetimes with use<'a, T>. The opt-in here is to promote your APITs to real type generics and give your unnamed lifetimes names.

The only caveat here is that you can name the elided lifetime in the output in the same cases you are allowed to normally in RPITs -- i.e. fn hello(x: &u8) -> impl use<'_> Sized { x } works fine.

[petrochenkov]

Nope, we don't capture APITs or elided lifetimes with use<'a, T>.

My first question was whether they are captured if use is not written.
The text says that all generic parameters are captured (on 2024 edition at least), but it's not clear whether APITs and elided lifetimes are included into this "all".

[compiler-errors]

All type and const generic parameters (including APITs) are captured for all RPITs regardless if they're in edition 2021 or 2024.

The only change between 2021 and 2024 involves capturing all lifetimes parameters in scope (incl. elided lifetimes, since those become early- or late-bound lifetime params) -- previously we only captured lifetimes mentioned in the bounds of RPITs. The opt-out here is to use use<'a, T> with the set of lifetimes that previously showed up in your opaque's bounds.

The only corner case is when you have an APIT and a lifetime you want to not capture -- you'll need to turn your APITs into real type parameters to make use of the use<'a, T> syntax.

Regarding opt-out for capturing type or const generic parameters currently, it's not possible to do currently; we will likely support that eventually, but if you see https://github.com/rust-lang/rfcs/blob/TC/precise-capturing/text/3617-precise-capturing.md#stabilization-strategy, we're probably not going to stabilize that initially because it requires exercising new paths of the type system (namely, bivariant unconstrained type parameters), and it's not necessary to mitigate the fallout of the new edition 2024 lifetime capture rules.

[nikomatsakis]

@rfcbot fcp merge

We've tried hard to avoid an explicit syntax like this but I think it's clear by now that it will be useful and it unblocks important Edition work. We had a reasonably thorough deep dive into the syntactic options and I believe the RFC lays out the options pretty well and the tradeoffs around them. Personally while I have some minor qualms about overloading use, I think it's the best option overall.

Note that in this fcp I am explicitly wearing my @rust-lang/lang hat -- I think the @rust-lang/types team should (before stabilization) vet the overall semantics and our implementation thereof but that's not really a question to be answered in the RFC.

[rfcbot]

Team member @nikomatsakis has proposed to merge this. The next step is review by the rest of the tagged team members:

• @joshtriplett
• @nikomatsakis
• @pnkfelix
• @scottmcm
• @tmandry

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

cc @rust-lang/lang-advisors: FCP proposed for lang, please feel free to register concerns.
See this document for info about what commands tagged team members can give me.

[petrochenkov]

I suspect that it's going to get annoying pretty quick (compared to impl<..> Trait), if precise captures become something that is specified often (e.g. by convention to minimize captures).

[Jules-Bertholet]

I don't feel strongly about this syntax either way, but:

This would hardly be the only place where syntax for introducing generic parameters is similar to syntax for using them. fn foo<T>() vs foo::<T>() for example.

Also, there is a symmetry of a sort between the two kinds of impl generics. In a trait implementation, they introduce generic parameters available for use by the type and trait; in RPIT, they would introduce the generics available for use by the hidden type.

[Jules-Bertholet]

Why can't this "just work"? Shouldn't the compiler be able to figure out that the closure is not using T? There should be no semver hazard, because to return the closure you need RPIT or TAIT , at which point you as API designer have the opportunity to specify the captures you commit to in public API.

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[kennytm]

Since this use<..> thing is part of the type signature, could you clarify if the capture list is changed, when it will be considered breaking change or not?

From the caller's point of view, if a function's RPIT (in covariant position only) has a capture removed, the hidden type's potential shortest lifetime is lengthened, which is compatible with existing caller code. So I think semver should allow removing captures from covariant-position RPIT in a minor update.

- fn callee1<'a, 'b>(aaa: &'a u8, bbb: &'b u8) -> &'a u8 { &0 }
+ fn callee1<'a, 'b>(aaa: &'a u8, bbb: &'b u8) -> &'static u8 { &0 }

- fn callee2<'a, 'b>(aaa: &'a u8, bbb: &'b u8) -> impl use<'a> Sized { &0 }
+ fn callee2<'a, 'b>(aaa: &'a u8, bbb: &'b u8) -> impl use<> Sized { &0 }

Meanwhile for TAIT, ATPIT and RPITIT changing the capture list in either direction should be considered a major breaking change.

(For APIT the capture list is irrelevant.)

[jackh726]

So, just procedurally...

While I do think solving the underlying ergonomic and semantic issues here are quite important, I do have to note that this syntax and RFC seem to have moved quite fast. Even for me - who, while doesn't attend lang meetings, does try to keep up to date of active lang things - I could have almost missed this. To put some dates to this on typical milestones in our current processes:

• The tracking issue was opened only 3 weeks ago
• The initial implementation PR a day after that
• The design meeting happened today
• This RFC was opened today
• And both PFCP and FCP have started today on this RFC

I know we're on a time crunch because of the edition, but I worry about this all moving just a bit too fast. Of course, accepting this RFC doesn't necessarily mean that actually stabilizing this feature will also happen quickly, but I worry that the pace so far is a harbinger for that. I hate to sit here and be the one that say "wait, we're moving too fast", especially because not moving fast here puts edition work in jeopardy, but I'd rather this concern be voiced than ignored.

I do want to be clear that my above comments have little reflection on my thoughts on the contents of this RFC or semantics of this feature. My concerns here apply to even the best of features (to put it into perspective: let-else, which imo is a very clear win all around took about a month from RFC open to RFC merge, with a fair amount of prior discussion; and this is what I consider a "fast" RFC process).

[traviscross]

Thanks @aliemjay for those great questions. We've now added examples and discussion to clarify each of those.

Note that the hardest subset of those questions relate to how things would work if we were able to capture less than all of the generic input parameters to the trait (including Self) in the trait definition. This is not likely to be possible in initial rounds of stabilization, and if it ever does become possible, these kind of questions are likely to receive extensive coverage in the stabilization report.

[Evian-Zhang]

Great work! I think this feature is a rather advanced feature and maybe this syntax appearing on simple APIs may discourage beginners when using fundamental functionalities in a crate. However, I have designed the following API pattern multiple times:

fn parse(path: impl Path) -> impl Iterator<Item = Foo> {}

I wonder if I should add a use<> in the return type position to indicate it does not capture anything?

I think this pattern is very common and basic in most crates. Should there be some special treatment in rustdoc for use<> (i.e. capture nothing) to make it more beginner-friendly?

[nikomatsakis]

@Evian-Zhang That's a good question (though not I think one that needs to block progress on the RFC). That said, I think even better would be if we could avoid using impl Trait for those kind of coercions. The right place for them to take place is in the caller side, the callee would prefer to just get a value of known type and hence be (more) monomorphic. We had at some point discussed a syntax option for this (e.g., writing ~String instead of impl Into<String>). That said, I hadn't until now considered the implications for borrow checking when you have an -> impl Trait return value.

(We've also discussed (and even done some exploration of) having the compiler recognize the pattern of a "mostly monomorphic" function that just does transforms in the beginning and avoid code duplication: that's worth doing as a first step, I believe, but it wouldn't have the borrow checker benefits.)

[Evian-Zhang]

@nikomatsakis Thank you for your response!

I agree that the coercions should be in the caller side, since I have found many places where the following code pattern appears in the Rust std source code:

fn foo(path: impl AsRef<Path>) {
    fn inner_foo(path: &Path) { ... }
    inner_foo(path.as_ref())
}

It is more graceful if this pattern can be automatically done by the compiler.

I am not familiar with the RFC discussion guidelines, and I think maybe I should put the rustdoc's use<> problem in the tracking issues after this RFC stabilized? Apologies for putting it here.

[traviscross]

@Evian-Zhang: It'd be better to open a thread on Zulip (or perhaps on IRLO) to discuss that further.

[nikomatsakis]

@Evian-Zhang (...and no apologies required)

[nikomatsakis]

@traviscross I'd like to add an unresolved question about the use<> impl Trait vs impl use<> Trait. My experience was that I thought I didn't care but within a few minutes of using it I found that use<> impl just felt much simpler to me. The best explanation I can give is that it feels less "stacked" -- i.e., I can look at the use and process it as a kind of "prefix" on the impl Trait type, rather than being an inner part that I have to think carefully about. That combined with the different scoping relative to for feels like a very strong argument to me.

[traviscross]

Done. This unresolved question has now been added.

We leave as an open question which of these two syntaxes we should choose:

1. impl use<..> Trait
   • This syntax is used throughout this RFC.
2. use<..> impl Trait
   • This syntax is the worthy challenger.

See the alternatives section above for a detailed comparative analysis of these options.

[traviscross]

The lang team has accepted this RFC and we've now merged it.

Thanks to all those who reviewed this and offered helpful feedback.

For further updates, follow the tracking issue:

• Tracking issue for precise_capturing syntax rust#123432

[oxalica]

This is not clearly documented about what "captures '_ only" means. It reads like it captures all elided lifetimes, but that's not the case.

After some testing on playground, it turns out '_ works as the same way as the '_ in a normal result type, that is:

• Refer to the only elided lifetime, if there is only a single elided lifetime and no named lifetimes.
  • So you cannot use '_ where there are named lifetimes, even if only a single lifetime is elided.
• Refer to the elided lifetime of &self if there is one. Other elided lifetimes are not captured.
• Otherwise, report an ambiguity error.

Ideally we should document this somewhere.