Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] async functions #50850

Closed
wants to merge 7 commits into from
Closed

Conversation

withoutboats
Copy link
Contributor

@withoutboats withoutboats commented May 17, 2018

This PR implements async functions under the async_await feature flag.

Checklist:

  • Parse async fn
    • Don't ICE when someone writes something like async struct Foo;.
  • Parse async closures
  • Parse async blocks
  • Lower async fn and async closures to returning their outer return type.
  • Lower the bodies of async items to be inside a generator/future.
  • Adjust lifetimes of async fn return type after lifetime elision
  • rustdoc integration
  • Test everything well

Currently built off of #50307, will be rebased once that has been merged.

@rust-highfive
Copy link
Collaborator

r? @michaelwoerister

(rust_highfive has picked a reviewer for you, use r? to override)

@rust-highfive
Copy link
Collaborator

warning Warning warning

  • These commits modify submodules.

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 17, 2018
@withoutboats withoutboats added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 17, 2018
@eddyb
Copy link
Member

eddyb commented May 18, 2018

So is the plan to have async fn in HIR, instead of desugaring it in HIR lowering?
How much of the existing generator machinery would be reused?
cc @rust-lang/compiler

@cramertj
Copy link
Member

@eddyb I figured we'd desugar to generators as early as possible at least for the MVP, since that will likely get us results the fastest. @alexcrichton and @aturon even suggested adding a temporary lang item to do TLS so we don't have to worry about threading the context arg through.

@michaelwoerister
Copy link
Member

r? @cramertj who knows more about this than me.

@withoutboats
Copy link
Contributor Author

@eddyb I'd like to desugar them to after lifetime elisions, because the return type is -> impl Future + 'ret, where 'ret is bound by all input lifetimes.

@cramertj
Copy link
Member

cramertj commented May 18, 2018

@withoutboats

You can use a custom visitor like the one we use to collect lifetimes for impl Trait to pick up the input lifetimes and copy them to the output. If you detect elided lifetimes, you can add the elided lifetime name ('_) to the output set.

@withoutboats
Copy link
Contributor Author

Talked with @eddyb and @cramertj and my current understanding of the plan for handling the return type is to transform it into impl Future<Output = $ret>, but keep the async tag, and at a later point, between lifetime resolution and typeck, injecting the 'ret lifetime into any function that is tagged as async.

Then there's the matter of the body. Talking to @eddyb, it seems like we will probably translate everything to async blocks:

  • async fn foo() { } becomes fn foo() { async { } }
  • async || { } becomes || { async { } }

Then there's the question of how to handle async blocks and await, which are essentially sugar for generators yielding (). Still getting there..

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-3.9 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading https://files.pythonhosted.org/packages/c2/1e/f70d1125f5bf6383d2ee7a76aea72bed6ba103c1bb9dd4ca051787552d2b/awscli-1.15.24-py2.py3-none-any.whl (1.3MB)
    0% |▎                               | 10kB 9.4MB/s eta 0:00:01
    1% |▌                               | 20kB 1.2MB/s eta 0:00:01
    2% |▉                               | 30kB 1.5MB/s eta 0:00:01
    3% |█                               | 40kB 1.3MB/s eta 0:00:01
---

[00:05:49] travis_fold:start:tidy
travis_time:start:tidy
tidy check
[00:05:50] tidy error: /checkout/src/librustc_metadata/encoder.rs:1232: line longer than 100 chars
[00:05:50] tidy error: /checkout/src/librustdoc/html/render.rs:2606: line longer than 100 chars
[00:05:50] tidy error: /checkout/src/libsyntax/visit.rs:229: trailing whitespace
[00:05:50] tidy error: /checkout/src/libsyntax/test.rs:130: line longer than 100 chars
[00:05:50] tidy error: Found 1 features without a gate test.
[00:05:50] Expected a gate test for the feature 'async_await'.
[00:05:50] Hint: create a failing test file named 'feature-gate-async_await.rs'
[00:05:50]       in the 'ui' test suite, with its failures due to
[00:05:50]       missing usage of #![feature(async_await)].
[00:05:50] Hint: If you already have such a test and don't want to rename it,
[00:05:50]       you can also add a // gate-test-async_await line to the test file.
[00:05:51] some tidy checks failed
[00:05:51] 
[00:05:51] 
[00:05:51] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor" "--quiet"
[00:05:51] 
[00:05:51] 
[00:05:51] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:05:51] Build completed unsuccessfully in 0:02:30
[00:05:51] Build completed unsuccessfully in 0:02:30
[00:05:51] make: *** [tidy] Error 1
[00:05:51] Makefile:79: recipe for target 'tidy' failed

The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:125a1855
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@eddyb
Copy link
Member

eddyb commented May 19, 2018

injecting the 'ret lifetime into any function that is tagged as async.

Hmm, I thought impl Trait allowed any number of lifetime bounds, wouldn't that be a more straight-forward translation? i.e. extracting the WF lifetimes (cc @nikomatsakis) from the argument types and adding all of them as bounds to the impl Trait definition.

(I haven't thought about the WF bit until now and I think this is one of the best reasons to not try and guess the set of lifetimes at any point before typeck).

@withoutboats
Copy link
Contributor Author

Hmm, I thought impl Trait allowed any number of lifetime bounds, wouldn't that be a more straight-forward translation?

This didn't work in the macro based implementation, but I believe, looking at the code, that the limitation is entirely in parsing, so yea we don't need to manufacture a lifetime, just collect all of the resolved input lifetimes.

@withoutboats
Copy link
Contributor Author

withoutboats commented May 20, 2018

Talked with eddyb more and came up with this (hacky) solution that works, pretty mcuh the elaboration of cramertj's & eddyb's original vision:

  • desugar async fn and async || { } to have async { } blocks in their middle
  • desugar async { } block to be a GenFuture wrapper around a generator, unstable & implemented in libcore
  • expand await! during HIR lowering
  • return type of async fn becomes impl Future<Output = R>, return type of async closure becomes GenFuture<_, R>
  • Inject the lifetimes into the return type of async fn in a separate pass (??)
  • use hardcoded std paths instead of lang items because lang items aren't available in lowering

Then we just piggy back the generator code the rest of the way down. We'll need a pass to make sure that yield is not used inside of an async block before we desugar it to a generator.

@withoutboats
Copy link
Contributor Author

HIR lowering of the return types is now implemented, as well as this "future" API (pending further movement on the lib RFC):

trait Future { }

struct GenFuture<G, R> {
    gen: G,
    marker: PhantomData<R>,
}

impl<G: Generator<Yield = ()>> Future for GenFuture<G, G::Return> { }

fn gen_future<G: Generator>(gen: G) -> GenFuture<G, G::Return> { /* ... */ }

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-3.9 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
    100% |████████████████████████████████| 4.2MB 289kB/s 
Collecting pyasn1>=0.1.3 (from rsa<=3.5.0,>=3.1.2->awscli)
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading https://files.pythonhosted.org/packages/a0/70/2c27740f08e477499ce19eefe05dbcae6f19fdc49e9e82ce4768be0643b9/pyasn1-0.4.3-py2.py3-none-any.whl (72kB)
    14% |████▌                           | 10kB 42.3MB/s eta 0:00:01
    28% |█████████                       | 20kB 40.5MB/s eta 0:00:01
    42% |█████████████▌                  | 30kB 46.6MB/s eta 0:00:01
    56% |██████████████████              | 40kB 50.6MB/s eta 0:00:01
---

[00:04:59] travis_fold:start:tidy
travis_time:start:tidy
tidy check
[00:04:59] tidy error: /checkout/src/libcore/future.rs: incorrect license
[00:04:59] tidy error: /checkout/src/librustc_metadata/encoder.rs:1240: line longer than 100 chars
[00:04:59] tidy error: /checkout/src/librustdoc/html/render.rs:2606: line longer than 100 chars
[00:04:59] tidy error: /checkout/src/libsyntax/visit.rs:230: trailing whitespace
[00:04:59] tidy error: /checkout/src/libsyntax/test.rs:130: line longer than 100 chars
[00:04:59] tidy error: /checkout/src/libsyntax/parse/parser.rs:3305: line longer than 100 chars
[00:04:59] tidy error: /checkout/src/libsyntax/print/pprust.rs:2165: line longer than 100 chars
[00:04:59] tidy error: /checkout/src/librustc/hir/lowering.rs:300: line longer than 100 chars
[00:04:59] tidy error: /checkout/src/librustc/hir/lowering.rs:1744: line longer than 100 chars
[00:04:59] tidy error: /checkout/src/librustc/hir/lowering.rs:1757: line longer than 100 chars
[00:04:59] tidy error: /checkout/src/librustc/hir/lowering.rs:1762: line longer than 100 chars
[00:04:59] tidy error: /checkout/src/librustc/hir/lowering.rs:1798: line longer than 100 chars
[00:04:59] tidy error: /checkout/src/librustc/hir/lowering.rs:1812: line longer than 100 chars
[00:04:59] tidy error: /checkout/src/librustc/hir/lowering.rs:2874: line longer than 100 chars
[00:04:59] tidy error: /checkout/src/librustc/hir/lowering.rs:3199: line longer than 100 chars
[00:04:59] tidy error: /checkout/src/libcore/future.rs:2: mismatches to corresponding lang feature in: ["tracking issue"]
[00:04:59] tidy error: /checkout/src/libcore/future.rs:5: mismatches to corresponding lang feature in: ["tracking issue"]
[00:05:00] tidy error: /checkout/src/libstd/lib.rs:394: mismatches to corresponding lang feature in: ["tracking issue"]
[00:05:00] Expected a gate test for the feature 'async_await'.
[00:05:00] tidy error: Found 1 features without a gate test.
[00:05:00] Hint: create a failing test file named 'feature-gate-async_await.rs'
[00:05:00]       in the 'ui' test suite, with its failures due to
[00:05:00]       missing usage of #![feature(async_await)].
[00:05:00] Hint: If you already have such a test and don't want to rename it,
[00:05:00]       you can also add a // gate-test-async_await line to the test file.
[00:05:00] some tidy checks failed
[00:05:00] 
[00:05:00] 
[00:05:00] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor" "--quiet"
[00:05:00] 
[00:05:00] 
[00:05:00] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:05:00] Build completed unsuccessfully in 0:01:56
[00:05:00] Build completed unsuccessfully in 0:01:56
[00:05:00] Makefile:79: recipe for target 'tidy' failed
[00:05:00] make: *** [tidy] Error 1

The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:3615d82a
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

/// depend on it.
#[allow(missing_debug_implementations, dead_code)]
#[unstable(feature = "gen_future", issue = "50547")]
pub struct GenFuture<G, R> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you add a bound on G that mentions R, I think you can both get rid of the PhantomData, and force the type-checker to infer the return type from the type you put in R, from GenFuture alone, although gen_future below is probably enough either way.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't know you could get rid of PhantomData this way. If that works I'll just put the bound here, make the field public, and get rid of the constructor.


struct NonAsyncRet;
struct AsyncClosureRet;
struct AsyncFunctionRet;
Copy link
Member

@eddyb eddyb May 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any particular reason for not using an enum here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I watched a talk at a conference about how great dependency injection is right before I wrot this code :) (the talk was by Sandi Metz)

I don't mind switching to a method on an enum which then delegates to three other methods, but it introduces the opportunity for irregularity (adding an argument that only applies to 1 case, for example, which I think is usually suspicious). I'd love if Rust made it more natural to write this pattern.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the main thing I don't want to do is pass an enum down and then write the match inside of this function. I think the proliferation of that pattern in lowering has made the code more difficult to understand (but I like OO, so YMMV).

Copy link
Member

@eddyb eddyb May 28, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our mileage does vary - enum represent finite state, whereas a trait introduces an open-world source of ambiguity - every single time there's a trait object involved, all the impls must be found to reason about anything at all.

I can's speak for all of @rust-lang/compiler, but I personally wouldn't use a trait unless the implementers are used in unrelated situations, but they can reuse some common abstraction through the trait (e.g. visitor/folder pattern) - in other words, bridging gaps in an open-world.

And even then I'd almost never employ a trait object (or fn pointers for that matter) except for bypassing polymorphic recursion (typically needed when a recursive function takes a closure) or when polymorphism is somehow prohibited (e.g. the query engine, which does dependency injection with fn pointers, or the codegen backend, which is dynamically loaded nowadays for LLVM reasons). There's also the oddball case of "half of this crate would be monomorphized if you used generics" but AFAIK that's rare.

In this case, all the implementors are in the same location, they're not recursive data types (or any data at all, but most data wouldn't be relevant), they're created from similar match-es on asyncness, and the sole method of the trait has exactly one caller.

I don't mind switching to a method on an enum which then delegates to three other methods, but it introduces the opportunity for irregularity (adding an argument that only applies to 1 case, for example, which I think is usually suspicious).

I don't understand this - are you referring to an argument that would come from a field of the structs (which are currently all units)? Or from the caller of lower? Because in the latter case I don't know how that'd change anything, and in the former you just move all the struct fields to the respective enum variant.

let LoweredNodeId { node_id, hir_id } = ctx.next_id();
let gen_future_path = hir::Path {
segments: segments.map_slice(|mut v| {
v.last_mut().unwrap().parameters = Some(P(hir::PathParameters {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add an Option<P<hir::PathParameters>> argument to std_path instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And then perform this operation there, or is there way to do this eventually without the map_sice if I keep passing the argument down? std_path delegates to something else and I stopped investigating how it worked at that point.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It constructs a hir::Path at some point and map_slice should be avoidable, yes.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's in the implementation of resolve_str_path, turns out:

}).map(hir::PathSegment::from_name).collect(),

I don't know why the path is constructed there instead of its sole caller in hir::lowering, but @petrochenkov might.

@bors
Copy link
Contributor

bors commented May 24, 2018

☔ The latest upstream changes (presumably #50930) made this pull request unmergeable. Please resolve the merge conflicts.

@withoutboats
Copy link
Contributor Author

Now also contains code to lower the bodies of async items to generators

Without Boats added 3 commits May 25, 2018 11:24
This is gated on edition 2018 & the `async_await` feature gate.

The parser will accept `async fn` and `async unsafe fn` as fn
items. Along the same lines as `const fn`, only `async unsafe fn`
is permitted, not `unsafe async fn`.The parser will not accept
`async` functions as trait methods.

To do a little code clean up, four fields of the function type
struct have been merged into the new `FnHeader` struct: constness,
asyncness, unsafety, and ABI.

Also, a small bug in HIR printing is fixed: it previously printed
`const unsafe fn` as `unsafe const fn`, which is grammatically
incorrect.
@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-3.9 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[00:04:45] travis_fold:start:tidy
travis_time:start:tidy
tidy check
[00:04:45] tidy error: /checkout/src/libcore/future.rs: incorrect license
[00:04:45] tidy error: /checkout/src/librustc_metadata/encoder.rs:1240: line longer than 100 chars
[00:04:45] tidy error: /checkout/src/librustdoc/html/render.rs:2606: line longer than 100 chars
[00:04:45] tidy error: /checkout/src/libsyntax/visit.rs:230: trailing whitespace
[00:04:45] tidy error: /checkout/src/libsyntax/test.rs:130: line longer than 100 chars
[00:04:45] tidy error: /checkout/src/libsyntax/parse/parser.rs:3315: line longer than 100 chars
[00:04:45] tidy error: /checkout/src/libsyntax/print/pprust.rs:2161: line longer than 100 chars
[00:04:45] tidy error: /checkout/src/librustc/hir/lowering.rs:300: line longer than 100 chars
[00:04:45] tidy error: /checkout/src/librustc/hir/lowering.rs:832: trailing whitespace
[00:04:45] tidy error: /checkout/src/librustc/hir/lowering.rs:833: trailing whitespace
[00:04:45] tidy error: /checkout/src/librustc/hir/lowering.rs:839: TODO is deprecated; use FIXME
[00:04:45] tidy error: /checkout/src/librustc/hir/lowering.rs:1788: line longer than 100 chars
[00:04:45] tidy error: /checkout/src/librustc/hir/lowering.rs:1801: line longer than 100 chars
[00:04:45] tidy error: /checkout/src/librustc/hir/lowering.rs:1806: line longer than 100 chars
[00:04:45] tidy error: /checkout/src/librustc/hir/lowering.rs:1842: line longer than 100 chars
[00:04:45] tidy error: /checkout/src/librustc/hir/lowering.rs:1856: line longer than 100 chars
[00:04:45] tidy error: /checkout/src/librustc/hir/lowering.rs:2927: line longer than 100 chars
[00:04:45] tidy error: /checkout/src/librustc/hir/lowering.rs:3252: line longer than 100 chars
[00:04:46] tidy error: /checkout/src/libcore/future.rs:2: mismatches to corresponding lang feature in: ["tracking issue"]
[00:04:46] tidy error: /checkout/src/libcore/future.rs:5: mismatches to corresponding lang feature in: ["tracking issue"]
[00:04:46] tidy error: /checkout/src/libstd/lib.rs:394: mismatches to corresponding lang feature in: ["tracking issue"]
[00:04:46] Expected a gate test for the feature 'async_await'.
[00:04:46] tidy error: Found 1 features without a gate test.
[00:04:46] Hint: create a failing test file named 'feature-gate-async_await.rs'
[00:04:46]       in the 'ui' test suite, with its failures due to
[00:04:46]       missing usage of #![feature(async_await)].
[00:04:46] Hint: If you already have such a test and don't want to rename it,
[00:04:46]       you can also add a // gate-test-async_await line to the test file.
[00:04:47] some tidy checks failed
[00:04:47] 
[00:04:47] 
[00:04:47] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor" "--quiet"
[00:04:47] 
[00:04:47] 
[00:04:47] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:04:47] Build completed unsuccessfully in 0:01:49
[00:04:47] Build completed unsuccessfully in 0:01:49
[00:04:47] make: *** [tidy] Error 1
[00:04:47] Makefile:79: recipe for target 'tidy' failed

The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:14703aaa
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@bors
Copy link
Contributor

bors commented May 26, 2018

☔ The latest upstream changes (presumably #51082) made this pull request unmergeable. Please resolve the merge conflicts.

/// depend on it.
#[allow(missing_debug_implementations, dead_code)]
#[unstable(feature = "gen_future", issue = "50547")]
pub struct GenFuture<G: ::ops::Generator<Yield = (), Return = R>, R>(pub G);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use #[doc(hidden)] too.

@eddyb
Copy link
Member

eddyb commented May 28, 2018

@withoutboats Your first commit (accidentally?) modifies submodules. You need to rebase those changes out (I'd use git rebase -i upstream/master to edit the first commit, and remove the changes by amending through git gui). I also suspect the lockfile commit won't be needed then.

@withoutboats
Copy link
Contributor Author

@cramertj is going to take over this work building off this branch

@pietroalbini
Copy link
Member

@withoutboats should we close this in the meantime?

@eddyb eddyb mentioned this pull request Jun 22, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants