Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rustc_codegen_llvm: add support for writing summary bitcode #125345

Merged
merged 6 commits into from
May 24, 2024

Conversation

durin42
Copy link
Contributor

@durin42 durin42 commented May 20, 2024

Typical uses of ThinLTO don't have any use for this as a standalone file, but distributed ThinLTO uses this to make the linker phase more efficient. With clang you'd do something like clang -flto=thin -fthin-link-bitcode=foo.indexing.o -c foo.c and then get both foo.o (full of bitcode) and foo.indexing.o (just the summary or index part of the bitcode). That's then usable by a two-stage linking process that's more friendly to distributed build systems like bazel, which is why I'm working on this area.

I talked some to @teresajohnson about naming in this area, as things seem to be a little confused between various blog posts and build systems. "bitcode index" and "bitcode summary" tend to be a little too ambiguous, and she tends to use "thin link bitcode" and "minimized bitcode" (which matches the descriptions in LLVM). Since the clang option is thin-link-bitcode, I went with that to try and not add a new spelling in the world.

Per @dtolnay, you can work around the lack of this by using lld --thinlto-index-only to do the indexing on regular .o files of bitcode, but that is a bit wasteful on actions when we already have all the information in rustc and could just write out the matching minimized bitcode. I didn't test that at all in our infrastructure, because by the time I learned that I already had this patch largely written.

@rustbot
Copy link
Collaborator

rustbot commented May 20, 2024

r? @Nadrieril

rustbot has assigned @Nadrieril.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels May 20, 2024
@rust-log-analyzer

This comment has been minimized.

@rustbot
Copy link
Collaborator

rustbot commented May 20, 2024

Some changes occurred in compiler/rustc_codegen_cranelift

cc @bjorn3

@rust-log-analyzer

This comment has been minimized.

@Nadrieril
Copy link
Member

r? compiler

@rustbot rustbot assigned lcnr and unassigned Nadrieril May 21, 2024
@lcnr
Copy link
Contributor

lcnr commented May 21, 2024

r? compiler

@rustbot
Copy link
Collaborator

rustbot commented May 21, 2024

Some changes occurred in compiler/rustc_codegen_gcc

cc @antoyo, @GuillaumeGomez

@rust-log-analyzer

This comment has been minimized.

@durin42
Copy link
Contributor Author

durin42 commented May 21, 2024

Latest CI run failed with [ui] tests/ui/asm/inline-syntax.rs#arm - which I shouldn't have been able to influence at all with this change.

@nnethercote
Copy link
Contributor

This is enabled with --emit=thin-link-bitcode, correct?

The code in the PR looks ok to me, though LTO experts might have opinions -- @bjorn3? @mw?

But I think that adding a new output type is a significant enough change that it should be done via the Major Change Proposal mechanism.

@bjorn3
Copy link
Member

bjorn3 commented May 22, 2024

But I think that adding a new output type is a significant enough change that it should be done via the Major Change Proposal mechanism.

rust-lang/compiler-team#735

@nnethercote
Copy link
Contributor

Ok, thanks for the extra info. @bjorn3 do you want to give the final approval here? You know what's going on here much better than I do.

@rust-log-analyzer

This comment has been minimized.

Typical uses of ThinLTO don't have any use for this as a standalone
file, but distributed ThinLTO uses this to make the linker phase more
efficient. With clang you'd do something like `clang -flto=thin
-fthin-link-bitcode=foo.indexing.o -c foo.c` and then get both foo.o
(full of bitcode) and foo.indexing.o (just the summary or index part of
the bitcode). That's then usable by a two-stage linking process that's
more friendly to distributed build systems like bazel, which is why I'm
working on this area.

I talked some to @teresajohnson about naming in this area, as things
seem to be a little confused between various blog posts and build
systems. "bitcode index" and "bitcode summary" tend to be a little too
ambiguous, and she tends to use "thin link bitcode" and "minimized
bitcode" (which matches the descriptions in LLVM). Since the clang
option is thin-link-bitcode, I went with that to try and not add a new
spelling in the world.

Per @dtolnay, you can work around the lack of this by using `lld
--thinlto-index-only` to do the indexing on regular .o files of
bitcode, but that is a bit wasteful on actions when we already have all
the information in rustc and could just write out the matching minimized
bitcode. I didn't test that at all in our infrastructure, because by the
time I learned that I already had this patch largely written.
This was needed in an older version of this patch, but never got edited
out when it became obsolete.
@durin42
Copy link
Contributor Author

durin42 commented May 22, 2024

I have confirmed that, somehow, the llvm-17 regression here is real and seemingly caused by my change. I'll investigate soon.

If we don't do this, some versions of LLVM (at least 17, experimentally)
will double-emit some error messages, which is how I noticed this. Given
that it seems to be costing some extra work, let's only request the
summary bitcode production if we'll actually bother writing it down,
otherwise skip it.
I did this in the user-facing logic, but I noticed while fixing a minor
defect that I had missed it in a few places in the internal details.
@durin42
Copy link
Contributor Author

durin42 commented May 23, 2024

Fixed the llvm 17 failure. Sorry for the force-push, that ended up being hard to avoid once I got into the weeds debugging.

I also cleaned up a couple of variable names so things are generally consistent in using summary instead of index.

@rust-log-analyzer

This comment has been minimized.

@bjorn3
Copy link
Member

bjorn3 commented May 23, 2024

@bors r+

@bors
Copy link
Contributor

bors commented May 23, 2024

📌 Commit cfe3f77 has been approved by bjorn3

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 23, 2024
bors added a commit to rust-lang-ci/rust that referenced this pull request May 23, 2024
…llaumeGomez

Rollup of 6 pull requests

Successful merges:

 - rust-lang#125263 (rust-lld: fallback to rustc's sysroot if there's no path to the linker in the target sysroot)
 - rust-lang#125345 (rustc_codegen_llvm: add support for writing summary bitcode)
 - rust-lang#125362 (Actually use TAIT instead of emulating it)
 - rust-lang#125412 (Don't suggest adding the unexpected cfgs to the build-script it-self)
 - rust-lang#125445 (Migrate `run-make/rustdoc-with-short-out-dir-option` to `rmake.rs`)
 - rust-lang#125452 (Cleanup check-cfg handling in core and std)

r? `@ghost`
`@rustbot` modify labels: rollup
bors added a commit to rust-lang-ci/rust that referenced this pull request May 24, 2024
…llaumeGomez

Rollup of 6 pull requests

Successful merges:

 - rust-lang#125263 (rust-lld: fallback to rustc's sysroot if there's no path to the linker in the target sysroot)
 - rust-lang#125345 (rustc_codegen_llvm: add support for writing summary bitcode)
 - rust-lang#125362 (Actually use TAIT instead of emulating it)
 - rust-lang#125412 (Don't suggest adding the unexpected cfgs to the build-script it-self)
 - rust-lang#125445 (Migrate `run-make/rustdoc-with-short-out-dir-option` to `rmake.rs`)
 - rust-lang#125452 (Cleanup check-cfg handling in core and std)

r? `@ghost`
`@rustbot` modify labels: rollup
@bors bors merged commit 4ee97fc into rust-lang:master May 24, 2024
6 checks passed
@rustbot rustbot added this to the 1.80.0 milestone May 24, 2024
rust-timer added a commit to rust-lang-ci/rust that referenced this pull request May 24, 2024
Rollup merge of rust-lang#125345 - durin42:thin-link-bitcode, r=bjorn3

rustc_codegen_llvm: add support for writing summary bitcode

Typical uses of ThinLTO don't have any use for this as a standalone file, but distributed ThinLTO uses this to make the linker phase more efficient. With clang you'd do something like `clang -flto=thin -fthin-link-bitcode=foo.indexing.o -c foo.c` and then get both foo.o (full of bitcode) and foo.indexing.o (just the summary or index part of the bitcode). That's then usable by a two-stage linking process that's more friendly to distributed build systems like bazel, which is why I'm working on this area.

I talked some to `@teresajohnson` about naming in this area, as things seem to be a little confused between various blog posts and build systems. "bitcode index" and "bitcode summary" tend to be a little too ambiguous, and she tends to use "thin link bitcode" and "minimized bitcode" (which matches the descriptions in LLVM). Since the clang option is thin-link-bitcode, I went with that to try and not add a new spelling in the world.

Per `@dtolnay,` you can work around the lack of this by using `lld --thinlto-index-only` to do the indexing on regular .o files of bitcode, but that is a bit wasteful on actions when we already have all the information in rustc and could just write out the matching minimized bitcode. I didn't test that at all in our infrastructure, because by the time I learned that I already had this patch largely written.
bjorn3 pushed a commit to bjorn3/rust that referenced this pull request Jun 30, 2024
rustc_codegen_llvm: add support for writing summary bitcode

Typical uses of ThinLTO don't have any use for this as a standalone file, but distributed ThinLTO uses this to make the linker phase more efficient. With clang you'd do something like `clang -flto=thin -fthin-link-bitcode=foo.indexing.o -c foo.c` and then get both foo.o (full of bitcode) and foo.indexing.o (just the summary or index part of the bitcode). That's then usable by a two-stage linking process that's more friendly to distributed build systems like bazel, which is why I'm working on this area.

I talked some to `@teresajohnson` about naming in this area, as things seem to be a little confused between various blog posts and build systems. "bitcode index" and "bitcode summary" tend to be a little too ambiguous, and she tends to use "thin link bitcode" and "minimized bitcode" (which matches the descriptions in LLVM). Since the clang option is thin-link-bitcode, I went with that to try and not add a new spelling in the world.

Per `@dtolnay,` you can work around the lack of this by using `lld --thinlto-index-only` to do the indexing on regular .o files of bitcode, but that is a bit wasteful on actions when we already have all the information in rustc and could just write out the matching minimized bitcode. I didn't test that at all in our infrastructure, because by the time I learned that I already had this patch largely written.
GuillaumeGomez added a commit to GuillaumeGomez/rust that referenced this pull request Jul 10, 2024
rustc_codegen_llvm: add support for writing summary bitcode

Typical uses of ThinLTO don't have any use for this as a standalone file, but distributed ThinLTO uses this to make the linker phase more efficient. With clang you'd do something like `clang -flto=thin -fthin-link-bitcode=foo.indexing.o -c foo.c` and then get both foo.o (full of bitcode) and foo.indexing.o (just the summary or index part of the bitcode). That's then usable by a two-stage linking process that's more friendly to distributed build systems like bazel, which is why I'm working on this area.

I talked some to `@teresajohnson` about naming in this area, as things seem to be a little confused between various blog posts and build systems. "bitcode index" and "bitcode summary" tend to be a little too ambiguous, and she tends to use "thin link bitcode" and "minimized bitcode" (which matches the descriptions in LLVM). Since the clang option is thin-link-bitcode, I went with that to try and not add a new spelling in the world.

Per `@dtolnay,` you can work around the lack of this by using `lld --thinlto-index-only` to do the indexing on regular .o files of bitcode, but that is a bit wasteful on actions when we already have all the information in rustc and could just write out the matching minimized bitcode. I didn't test that at all in our infrastructure, because by the time I learned that I already had this patch largely written.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants