Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid unnecessary TokenTree to TokenStream conversions #65455

Conversation

nnethercote
Copy link
Contributor

A TokenStream contains any number of TokenTrees. Therefore, a single TokenTree can be promoted to a TokenStream. But doing so costs two allocations: one for the single-element Vec, and one for the Lrc. (An IsJoint value also must be added; the default is NonJoint.)

The current code converts TokenTrees to TokenStreams unnecessarily in a few places. This PR removes some of these unnecessary conversions, both simplifying the code and speeding it up.

r? @petrochenkov

@nnethercote
Copy link
Contributor Author

@bors try @rust-timer queue

@rust-timer
Copy link
Collaborator

Awaiting bors try build completion

@bors
Copy link
Contributor

bors commented Oct 15, 2019

⌛ Trying commit 08afe1f with merge 881f05b...

bors added a commit that referenced this pull request Oct 15, 2019
…enStream-conversions, r=<try>

Avoid unnecessary `TokenTree` to `TokenStream` conversions

A `TokenStream` contains any number of `TokenTrees`. Therefore, a single `TokenTree` can be promoted to a `TokenStream`. But doing so costs two allocations: one for the single-element `Vec`, and one for the `Lrc`. (An `IsJoint` value also must be added; the default is `NonJoint`.)

The current code converts `TokenTree`s to `TokenStream`s unnecessarily in a few places. This PR removes some of these unnecessary conversions, both simplifying the code and speeding it up.

r? @petrochenkov
@bors
Copy link
Contributor

bors commented Oct 16, 2019

☀️ Try build successful - checks-azure
Build commit: 881f05b (881f05b8576331a868132c5d3201bb7e54a2fff3)

@rust-timer
Copy link
Collaborator

Queued 881f05b with parent 237d54f, future comparison URL.

@petrochenkov petrochenkov added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Oct 16, 2019
@petrochenkov
Copy link
Contributor

r=me after the perf is ready, with or without addressing #65455 (comment)

@petrochenkov petrochenkov added S-waiting-on-perf Status: Waiting on a perf run to be completed. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 16, 2019
@rust-timer
Copy link
Collaborator

Finished benchmarking try commit 881f05b, comparison URL.

@mati865
Copy link
Contributor

mati865 commented Oct 16, 2019

In case perf.rlo dies again the results are good.

Screenshot

Screenshot_2019-10-16 rustc performance data

@nnethercote nnethercote force-pushed the avoid-unnecessary-TokenTree-to-TokenStream-conversions branch from 08afe1f to 0fb82f3 Compare October 16, 2019 19:00
@nnethercote
Copy link
Contributor Author

I added MetaItemKind::token() back in, because it's useful. But MetaItem::tokens() and NestedMetaItem::tokens() aren't necessary, so I didn't add them back in.

@bors r=petrochenkov

@bors
Copy link
Contributor

bors commented Oct 16, 2019

📌 Commit 0fb82f34d3ff58b28d5dcd21f89860b002313a3f has been approved by petrochenkov

@bors bors added the S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. label Oct 16, 2019
@petrochenkov petrochenkov removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Oct 17, 2019
@bors
Copy link
Contributor

bors commented Oct 17, 2019

☔ The latest upstream changes (presumably #65495) made this pull request unmergeable. Please resolve the merge conflicts.

@bors bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Oct 17, 2019
The current code has this impl:
```
impl<T: Into<TokenStream>> iter::FromIterator<T> for TokenStream
```
If given an `IntoIterator<Item = TokenTree>`, it will convert each individual
`TokenTree` to a `TokenStream` (at the cost of two allocations: a `Vec`
and an `Lrc`). It will then merge those `TokenStream`s into a single
`TokenStream`. This is inefficient.

This commit changes the impl to this less general one:
```
impl iter::FromIterator<TokenTree> for TokenStream
```
It collects the `TokenTree`s into a single `Vec` first and then converts that
to a `TokenStream` by wrapping it in a single `Lrc`. The previous generality
was unnecessary; no other code needs changing.

This change speeds up several benchmarks by up to 4%.
Because most of the call sites have an easier time working with a
`TokenTree` instead of a `TokenStream`.
Likewise for `NestedMetaItem::tokens()`. Also, add
`MetaItemKind::token_trees_and_joints()`, which `MetaItemKind::tokens()`
now calls.

This avoids some unnecessary `TokenTree` to `TokenStream` conversions,
and removes the need for the clumsy
`TokenStream::append_to_tree_and_joint_vec()`.
@nnethercote nnethercote force-pushed the avoid-unnecessary-TokenTree-to-TokenStream-conversions branch from 0fb82f3 to e4ec4a6 Compare October 18, 2019 02:28
@nnethercote
Copy link
Contributor Author

I rebased.

@bors r=petrochenkov

@bors
Copy link
Contributor

bors commented Oct 18, 2019

📌 Commit e4ec4a6 has been approved by petrochenkov

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 18, 2019
tmandry added a commit to tmandry/rust that referenced this pull request Oct 18, 2019
…Tree-to-TokenStream-conversions, r=petrochenkov

Avoid unnecessary `TokenTree` to `TokenStream` conversions

A `TokenStream` contains any number of `TokenTrees`. Therefore, a single `TokenTree` can be promoted to a `TokenStream`. But doing so costs two allocations: one for the single-element `Vec`, and one for the `Lrc`. (An `IsJoint` value also must be added; the default is `NonJoint`.)

The current code converts `TokenTree`s to `TokenStream`s unnecessarily in a few places. This PR removes some of these unnecessary conversions, both simplifying the code and speeding it up.

r? @petrochenkov
tmandry added a commit to tmandry/rust that referenced this pull request Oct 18, 2019
…Tree-to-TokenStream-conversions, r=petrochenkov

Avoid unnecessary `TokenTree` to `TokenStream` conversions

A `TokenStream` contains any number of `TokenTrees`. Therefore, a single `TokenTree` can be promoted to a `TokenStream`. But doing so costs two allocations: one for the single-element `Vec`, and one for the `Lrc`. (An `IsJoint` value also must be added; the default is `NonJoint`.)

The current code converts `TokenTree`s to `TokenStream`s unnecessarily in a few places. This PR removes some of these unnecessary conversions, both simplifying the code and speeding it up.

r? @petrochenkov
tmandry added a commit to tmandry/rust that referenced this pull request Oct 18, 2019
…Tree-to-TokenStream-conversions, r=petrochenkov

Avoid unnecessary `TokenTree` to `TokenStream` conversions

A `TokenStream` contains any number of `TokenTrees`. Therefore, a single `TokenTree` can be promoted to a `TokenStream`. But doing so costs two allocations: one for the single-element `Vec`, and one for the `Lrc`. (An `IsJoint` value also must be added; the default is `NonJoint`.)

The current code converts `TokenTree`s to `TokenStream`s unnecessarily in a few places. This PR removes some of these unnecessary conversions, both simplifying the code and speeding it up.

r? @petrochenkov
bors added a commit that referenced this pull request Oct 18, 2019
Rollup of 17 pull requests

Successful merges:

 - #65016 (Always inline `mem::{size_of,align_of}` in debug builds)
 - #65197 (Prepare `MutVisitor`s to handle interned projections)
 - #65201 (Disable Go and OCaml bindings when building LLVM)
 - #65364 (Collect occurrences of empty blocks for mismatched braces diagnostic)
 - #65417 (Add more coherence tests)
 - #65434 (Add long error explanation for E0577)
 - #65455 (Avoid unnecessary `TokenTree` to `TokenStream` conversions)
 - #65472 (Use a sharded dep node to dep node index map)
 - #65480 (Speed up `LexicalResolve::expansion()`)
 - #65496 (properly document panics in div_euclid and rem_euclid)
 - #65508 (add option to ping llvm ice-breakers to triagebot)
 - #65511 (save-analysis: Nest tables when processing impl block definitions)
 - #65513 (reorder fmt docs for more clarity)
 - #65532 (doc: make BitSet intro more short)
 - #65540 (show up some extra info when t!() fails)
 - #65549 (Fix left/right shift typo in wrapping rotate docs)
 - #65552 (Clarify diagnostics when using `~` as a unary op)

Failed merges:

 - #65471 (Add long error explanation for E0578)

r? @ghost
tmandry added a commit to tmandry/rust that referenced this pull request Oct 18, 2019
…Tree-to-TokenStream-conversions, r=petrochenkov

Avoid unnecessary `TokenTree` to `TokenStream` conversions

A `TokenStream` contains any number of `TokenTrees`. Therefore, a single `TokenTree` can be promoted to a `TokenStream`. But doing so costs two allocations: one for the single-element `Vec`, and one for the `Lrc`. (An `IsJoint` value also must be added; the default is `NonJoint`.)

The current code converts `TokenTree`s to `TokenStream`s unnecessarily in a few places. This PR removes some of these unnecessary conversions, both simplifying the code and speeding it up.

r? @petrochenkov
bors added a commit that referenced this pull request Oct 19, 2019
Rollup of 19 pull requests

Successful merges:

 - #65016 (Always inline `mem::{size_of,align_of}` in debug builds)
 - #65197 (Prepare `MutVisitor`s to handle interned projections)
 - #65201 (Disable Go and OCaml bindings when building LLVM)
 - #65334 (Add long error explanation for E0575)
 - #65364 (Collect occurrences of empty blocks for mismatched braces diagnostic)
 - #65455 (Avoid unnecessary `TokenTree` to `TokenStream` conversions)
 - #65472 (Use a sharded dep node to dep node index map)
 - #65480 (Speed up `LexicalResolve::expansion()`)
 - #65493 (Add long error explanation for E0584)
 - #65496 (properly document panics in div_euclid and rem_euclid)
 - #65498 (Plugins deprecation: don’t suggest simply removing the attribute)
 - #65508 (add option to ping llvm ice-breakers to triagebot)
 - #65511 (save-analysis: Nest tables when processing impl block definitions)
 - #65513 (reorder fmt docs for more clarity)
 - #65532 (doc: make BitSet intro more short)
 - #65535 (rustc: arena-allocate the slice in `ty::GenericsPredicate`, not the whole struct.)
 - #65540 (show up some extra info when t!() fails)
 - #65549 (Fix left/right shift typo in wrapping rotate docs)
 - #65552 (Clarify diagnostics when using `~` as a unary op)

Failed merges:

 - #65390 (Add long error explanation for E0576)
 - #65434 (Add long error explanation for E0577)
 - #65471 (Add long error explanation for E0578)

r? @ghost
@bors bors merged commit e4ec4a6 into rust-lang:master Oct 19, 2019
@nnethercote nnethercote deleted the avoid-unnecessary-TokenTree-to-TokenStream-conversions branch October 20, 2019 21:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants