Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't log mass build output #1911

Open
Nemo157 opened this issue Nov 15, 2022 · 3 comments
Open

Don't log mass build output #1911

Nemo157 opened this issue Nov 15, 2022 · 3 comments
Labels
A-builds Area: Building the documentation for a crate

Comments

@Nemo157
Copy link
Member

Nemo157 commented Nov 15, 2022

When building some crates they cause hundreds/thousands of lines of output from rustc/rustdoc which overwhelms journald and it starts dropping messages. One example is when building pallas@0.13.4 for the non-x86-64 targets:

Nov 14 20:57:58 docsrs cratesfyi[10930]: 2022-11-14T20:57:58.209405Z  INFO log: [stderr] 89  |             d = _mm256_permute4x64_epi64(d, _MM_SHUFFLE(1, 0, 3, 2)); log.target="rustwid
Nov 14 20:57:58 docsrs cratesfyi[10930]: 2022-11-14T20:57:58.209424Z  INFO log: [stderr]     |                 ^^^^^^^^^^^^^^^^^^^^^^^^ not found in this scope log.target="rustwide::cm
Nov 14 20:57:58 docsrs cratesfyi[10930]: 2022-11-14T20:57:58.209444Z  INFO log: [stderr] ... log.target="rustwide::cmd" log.module_path="rustwide::cmd" log.file="/home/ubuntu/.cargo/re
Nov 14 20:57:58 docsrs cratesfyi[10930]: 2022-11-14T20:57:58.209464Z  INFO log: [stderr] 312 |     ROUND!(1, load1!()); log.target="rustwide::cmd" log.module_path="rustwide::cmd" log.f
Nov 14 20:57:58 docsrs cratesfyi[10930]: 2022-11-14T20:57:58.209486Z  INFO log: [stderr]     |     ------------------- in this macro invocation log.target="rustwide::cmd" log.module_pa
Nov 14 20:57:58 docsrs cratesfyi[10930]: 2022-11-14T20:57:58.209506Z  INFO log: [stderr]     | log.target="rustwide::cmd" log.module_path="rustwide::cmd" log.file="/home/ubuntu/.cargo/
Nov 14 20:57:58 docsrs cratesfyi[10930]: 2022-11-14T20:57:58.209524Z  INFO log: [stderr]     = note: this error originates in the macro `UNDIAGONALIZE` which comes from the expansion o
Nov 14 20:57:58 docsrs cratesfyi[10930]: 2022-11-14T20:57:58.209545Z  INFO log: [stderr] help: consider importing this function log.target="rustwide::cmd" log.module_path="rustwide::cm
Nov 14 20:57:58 docsrs cratesfyi[10930]: 2022-11-14T20:57:58.209564Z  INFO log: [stderr]     | log.target="rustwide::cmd" log.module_path="rustwide::cmd" log.file="/home/ubuntu/.cargo/
Nov 14 20:57:58 docsrs cratesfyi[10930]: 2022-11-14T20:57:58.209582Z  INFO log: [stderr] 1   | use core::arch::x86::_mm256_permute4x64_epi64; log.target="rustwide::cmd" log.module_path
Nov 14 20:57:58 docsrs cratesfyi[10930]: 2022-11-14T20:57:58.209604Z  INFO log: [stderr]     | log.target="rustwide::cmd" log.module_path="rustwide::cmd" log.file="/home/ubuntu/.cargo/
Nov 14 20:57:58 docsrs cratesfyi[10930]: 2022-11-14T20:57:58.209624Z  INFO log: [stderr]  log.target="rustwide::cmd" log.module_path="rustwide::cmd" log.file="/home/ubuntu/.cargo/regis
Nov 14 20:57:58 docsrs cratesfyi[10930]: 2022-11-14T20:57:58.213346Z  INFO log: [stderr] error[E0425]: cannot find function `_mm256_permute4x64_epi64` in this scope log.target="rustwid
Nov 14 20:57:58 docsrs cratesfyi[10930]: 2022-11-14T20:57:58.213430Z  INFO log: [stderr]    --> /opt/rustwide/cargo-home/registry/src/github.com-1ecc6299db9ec823/cryptoxide-0.4.2/src/h
Nov 14 20:57:58 docsrs cratesfyi[10930]: 2022-11-14T20:57:58.213458Z  INFO log: [stderr]     | log.target="rustwide::cmd" log.module_path="rustwide::cmd" log.file="/home/ubuntu/.cargo/
Nov 14 20:58:20 docsrs cratesfyi[10930]: 2022-11-14T20:58:20.687122Z DEBUG docs_rs::utils::daemon: Checking new crates
Nov 14 20:58:20 docsrs systemd-journald[27648]: Suppressed 7196 messages from docs.rs.service

I would just say we shouldn't log these at all since the build-log stored in S3 will capture them, but I have found it useful to see the output in the logs to diagnose issues on non-default targets while we don't have #787. A halfway point until that is done might be to only log the first 50 lines of build output if possible, that seems like enough to generally diagnose basic issues.

@syphar syphar added the A-builds Area: Building the documentation for a crate label Nov 16, 2022
@syphar
Copy link
Member

syphar commented Nov 16, 2022

debugging directly on the build-server will probably become less and less, especially when we have #787 ( and #1011 where we probably would also log metadata errors).

In the final picture IMO journald (and after we have multiple build-servers, the log aggregation that we probably have) should only log things that are not in the build logs on S3.

A halfway point until that is done might be to only log the first 50 lines of build output if possible, that seems like enough to generally diagnose basic issues.

this sounds like a good compromise

@syphar
Copy link
Member

syphar commented Oct 11, 2023

@Nemo157 just a thought: this is probably mostly a rustwide change, which we then would just use here, right?

@syphar
Copy link
Member

syphar commented Oct 11, 2023

Ah no, the output is coming from our config of the log crate, so that's what we have to tweak.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-builds Area: Building the documentation for a crate
Projects
None yet
Development

No branches or pull requests

2 participants