make bootstrap use wget if available #128459

lolbinarycat · 2024-07-31T19:47:36Z

this improves reliability when downloading over unreliable connections.

without this change, it was basically impossible for me to run the bootstrap tests.

rustbot · 2024-07-31T19:47:44Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @Kobzol (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

@rustbot author: the review is finished, PR author should check the comments and take action accordingly
@rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

rustbot · 2024-07-31T19:59:03Z

Some changes occurred in src/tools/cargo

cc @ehuss

lolbinarycat · 2024-07-31T20:03:24Z

whoops, didn't mean to touch that submodule

Kobzol

Hi, thanks for the PR! Just out of curiosity, if you're having issues with network connection, wouldn't wget -c be useful for you? With that flag wget could actually resume interrupted downloads.

lolbinarycat · 2024-08-06T20:40:39Z

doesn't wget -c only matter if the download previously failed? and wget should automatically retry failures a few times before the whole command fails, right?

just swapping in wget seemed to be enough in my experience.

Kobzol · 2024-08-07T06:08:52Z

Yes, it only matters when you want to resume, I thought that is the situation that you want to improve. So if I understand it correctly, when curl retries a download, it starts from the beginning, but wget restart it where it ended?

Overall, I'm fine with using wget, although I think that we should try to copy some of the options from the curl invocation. For example, there is no connection timeout in wget by default, but for curl we set it to 30s. We should try to get these options for wget and curl to be similar-ish.

lolbinarycat · 2024-08-08T17:46:22Z

Yes, it only matters when you want to resume, I thought that is the situation that you want to improve.

as i understand it, wget will automatically reconnect and resume by default, and you only need to use -c if you want to manually continue after that has failed (or after you did something like a ^C)

Kobzol · 2024-08-08T20:43:40Z

I see. Yeah, that makes sense, ok. Could you please try to configure the wget invocations to have more similar options as the curl invocations, particularly around (connection) timeouts, retry counts and similar?

Kobzol

Thank you! Could you now please squash the commits? There are a lot of tmp/WIP commits in there now.

@onur-ozkan Any concerns against using wget as the default downloader if it is available? I'd personally merge this and if people encounter some issues with wget, we can e.g. hide it behind an opt-in env. var. flag.

src/bootstrap/bootstrap.py

bjorn3 · 2024-08-10T15:13:29Z

Would passing --continue-at - to curl work too? According to the man page that is supposed to restart the download from the last downloaded byte when retrying.

onur-ozkan · 2024-08-11T05:38:16Z

Would passing --continue-at - to curl work too? According to the man page that is supposed to restart the download from the last downloaded byte when retrying.

Some of our CI runners are using very old versions of curl and --continue-at is too recent feature for us to use. There are various useful other flags available, but we cannot use them due to this problem.

@onur-ozkan Any concerns against using wget as the default downloader if it is available? I'd personally merge this and if people encounter some issues with wget, we can e.g. hide it behind an opt-in env. var. flag.

If we use recent features of wget, CI runners will probably fail due to having too old wget version.

Kobzol · 2024-08-11T11:47:27Z

I suppose that the easiest way to check is just to try to merge this.

Could you please squash the commits? Thanks.

this improves reliability when downloading over unreliable connections.

Kobzol · 2024-08-12T18:26:39Z

Okay, let's try.

@bors r+ rollup=never

bors · 2024-08-12T18:26:42Z

📌 Commit 22dad3b has been approved by Kobzol

It is now in the queue for this repository.

make bootstrap use wget if available this improves reliability when downloading over unreliable connections. without this change, it was basically impossible for me to run the bootstrap tests.

bors · 2024-08-13T00:44:13Z

⌛ Testing commit 22dad3b with merge 0c7fa26...

rust-log-analyzer · 2024-08-13T00:50:32Z

The job dist-i686-linux failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

configure: 
configure: run `python /checkout/x.py --help`
##[endgroup]
Attempting with retry: make prepare
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.

spurious failure, trying again
downloading https://static.rust-lang.org/dist/2024-07-26/rust-std-beta-i686-unknown-linux-gnu.tar.xz

---
    expected: 0c9a0717054858fa821e4c418b9e99cb0a0f852ec3c866951c1d1795cc1e8152
Traceback (most recent call last):
  File "/checkout/src/bootstrap/bootstrap.py", line 1231, in <module>
    main()
  File "/checkout/src/bootstrap/bootstrap.py", line 1216, in main
    bootstrap(args)
  File "/checkout/src/bootstrap/bootstrap.py", line 1181, in bootstrap
    build.download_toolchain()
  File "/checkout/src/bootstrap/bootstrap.py", line 618, in download_toolchain
    download_component(download_info)
  File "/checkout/src/bootstrap/bootstrap.py", line 486, in download_component
    verbose=download_info.verbose,
  File "/checkout/src/bootstrap/bootstrap.py", line 71, in get
RuntimeError: failed verification
make: *** [prepare] Error 1
Command failed. Attempt 2/5:
Command failed. Attempt 2/5:
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.

spurious failure, trying again
downloading https://static.rust-lang.org/dist/2024-07-26/rust-std-beta-i686-unknown-linux-gnu.tar.xz

---
    expected: 0c9a0717054858fa821e4c418b9e99cb0a0f852ec3c866951c1d1795cc1e8152
Traceback (most recent call last):
  File "/checkout/src/bootstrap/bootstrap.py", line 1231, in <module>
    main()
  File "/checkout/src/bootstrap/bootstrap.py", line 1216, in main
    bootstrap(args)
  File "/checkout/src/bootstrap/bootstrap.py", line 1181, in bootstrap
    build.download_toolchain()
  File "/checkout/src/bootstrap/bootstrap.py", line 618, in download_toolchain
    download_component(download_info)
  File "/checkout/src/bootstrap/bootstrap.py", line 486, in download_component
    verbose=download_info.verbose,
  File "/checkout/src/bootstrap/bootstrap.py", line 71, in get
RuntimeError: failed verification
make: *** [prepare] Error 1
Command failed. Attempt 3/5:
Command failed. Attempt 3/5:
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.

spurious failure, trying again
downloading https://static.rust-lang.org/dist/2024-07-26/rust-std-beta-i686-unknown-linux-gnu.tar.xz

---
    expected: 0c9a0717054858fa821e4c418b9e99cb0a0f852ec3c866951c1d1795cc1e8152
Traceback (most recent call last):
  File "/checkout/src/bootstrap/bootstrap.py", line 1231, in <module>
    main()
  File "/checkout/src/bootstrap/bootstrap.py", line 1216, in main
    bootstrap(args)
  File "/checkout/src/bootstrap/bootstrap.py", line 1181, in bootstrap
    build.download_toolchain()
  File "/checkout/src/bootstrap/bootstrap.py", line 618, in download_toolchain
    download_component(download_info)
  File "/checkout/src/bootstrap/bootstrap.py", line 486, in download_component
    verbose=download_info.verbose,
  File "/checkout/src/bootstrap/bootstrap.py", line 71, in get
RuntimeError: failed verification
make: *** [prepare] Error 1
Command failed. Attempt 4/5:
Command failed. Attempt 4/5:
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.

spurious failure, trying again
downloading https://static.rust-lang.org/dist/2024-07-26/rust-std-beta-i686-unknown-linux-gnu.tar.xz

---
    expected: 0c9a0717054858fa821e4c418b9e99cb0a0f852ec3c866951c1d1795cc1e8152
Traceback (most recent call last):
  File "/checkout/src/bootstrap/bootstrap.py", line 1231, in <module>
    main()
  File "/checkout/src/bootstrap/bootstrap.py", line 1216, in main
    bootstrap(args)
  File "/checkout/src/bootstrap/bootstrap.py", line 1181, in bootstrap
    build.download_toolchain()
  File "/checkout/src/bootstrap/bootstrap.py", line 618, in download_toolchain
    download_component(download_info)
  File "/checkout/src/bootstrap/bootstrap.py", line 486, in download_component
    verbose=download_info.verbose,
  File "/checkout/src/bootstrap/bootstrap.py", line 71, in get
RuntimeError: failed verification
make: *** [prepare] Error 1
Command failed. Attempt 5/5:
Command failed. Attempt 5/5:
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.

spurious failure, trying again
downloading https://static.rust-lang.org/dist/2024-07-26/rust-std-beta-i686-unknown-linux-gnu.tar.xz

---
    expected: 0c9a0717054858fa821e4c418b9e99cb0a0f852ec3c866951c1d1795cc1e8152
Traceback (most recent call last):
  File "/checkout/src/bootstrap/bootstrap.py", line 1231, in <module>
    main()
  File "/checkout/src/bootstrap/bootstrap.py", line 1216, in main
    bootstrap(args)
  File "/checkout/src/bootstrap/bootstrap.py", line 1181, in bootstrap
    build.download_toolchain()
  File "/checkout/src/bootstrap/bootstrap.py", line 618, in download_toolchain
    download_component(download_info)
  File "/checkout/src/bootstrap/bootstrap.py", line 486, in download_component
    verbose=download_info.verbose,
  File "/checkout/src/bootstrap/bootstrap.py", line 71, in get
RuntimeError: failed verification
make: *** [prepare] Error 1
The command has failed after 5 attempts.
  local time: Tue Aug 13 00:50:23 UTC 2024

bors · 2024-08-13T00:51:07Z

💔 Test failed - checks-actions

Kobzol · 2024-08-13T05:52:16Z

Hmm, unfortunate. I wonder if we could just test if curl supports the resume flag, and use it in that case, instead of introducing a full additional way of downloading. Or, if it's annoying or impossible to check properly, just add an environment variable to opt into this, which could then be used by users whose network connection is not stable.

onur-ozkan · 2024-08-13T06:23:37Z

I wonder if we could just test if curl supports the resume flag

That flag is quite recent feature for curl (or at least, for the one we use in our CI runners). I am quite sure that trying that flag will fail too.

just add an environment variable to opt into this, which could then be used by users whose network connection is not stable.

Makes sense.

Kobzol · 2024-08-13T06:27:14Z

I didn't mean trying the flag for CI, but for local usage (on CI the flag would fail), assuming that recent distros already have new enough curl.

onur-ozkan · 2024-08-13T07:48:15Z

Actually, it seems that the flag was introduced in early versions of curl. Even our CI runners use newer versions than that (as far as I remember, the major version was either 6 or 7 but definitely not lower or higher). We could try using it unconditionally (as there is no distro using that old version).

Kobzol · 2024-08-13T08:23:39Z

@lolbinarycat If you're able to replicate your network issues, could you please try if the --continue-at curl flag resolves your issue?

lolbinarycat · 2024-08-15T17:05:22Z

@Kobzol --continue-at works, but only in conjunction with --retry-all-errors, otherwise a broken pipe or connection terminated by peer error will be incorrectly treated as a non-transient error.

Kobzol · 2024-08-15T17:13:54Z

Oh, shoot. That is sadly a quite new flag.

onur-ozkan · 2024-08-15T19:16:23Z

We can check the curl version and enable it only if it's newer than 7.70.0 (as it was introduced with 7.71.0). Then, we can close #110178 as well.

alternative to rust-lang#128459 fixes rust-lang#110178

onur-ozkan · 2024-08-25T05:52:05Z

Closing this one as we are landing #129134.

lolbinarycat · 2024-08-25T06:38:02Z

i believe this actually improves the documentation of what the existing curl flags do, and those comments never got added to the accepted PR

should i add those comments in a seperate PR? or is a whole pr for a couple of comments too frivolous?

onur-ozkan · 2024-08-25T06:42:36Z

Making a separate PR sounds good to me.

bootstrap: improve error recovery flags to curl alternative to rust-lang#128459 fixes rust-lang#110178 r? `@Kobzol`

bootstrap: improve error recovery flags to curl alternative to rust-lang#128459 fixes rust-lang#110178 r? ``@Kobzol``

bootstrap: improve error recovery flags to curl alternative to rust-lang#128459 fixes rust-lang#110178 r? ```@Kobzol```

bootstrap: improve error recovery flags to curl alternative to rust-lang#128459 fixes rust-lang#110178 r? ````@Kobzol````

Rollup merge of rust-lang#129134 - lolbinarycat:continue-at, r=Kobzol bootstrap: improve error recovery flags to curl alternative to rust-lang#128459 fixes rust-lang#110178 r? ````@Kobzol````

rustbot assigned Kobzol Jul 31, 2024

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Jul 31, 2024

This comment has been minimized.

Sign in to view

lolbinarycat force-pushed the wget branch from 7fb88ca to cf7688e Compare July 31, 2024 20:03

This comment has been minimized.

Sign in to view

Kobzol reviewed Aug 6, 2024

View reviewed changes

This comment has been minimized.

Sign in to view

lolbinarycat force-pushed the wget branch from 026da0b to 165ea58 Compare August 9, 2024 22:40

Kobzol reviewed Aug 10, 2024

View reviewed changes

src/bootstrap/bootstrap.py Show resolved Hide resolved

make bootstrap use wget if available

22dad3b

this improves reliability when downloading over unreliable connections.

lolbinarycat force-pushed the wget branch from 165ea58 to 22dad3b Compare August 11, 2024 23:45

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 12, 2024

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Aug 13, 2024

onur-ozkan added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 13, 2024

lolbinarycat pushed a commit to lolbinarycat/rust that referenced this pull request Aug 15, 2024

bootstrap: improve error recovery flags to curl

e2aef71

alternative to rust-lang#128459 fixes rust-lang#110178

lolbinarycat pushed a commit to lolbinarycat/rust that referenced this pull request Aug 15, 2024

bootstrap: improve error recovery flags to curl

ac8a698

alternative to rust-lang#128459 fixes rust-lang#110178

lolbinarycat mentioned this pull request Aug 15, 2024

bootstrap: improve error recovery flags to curl #129134

Merged

lolbinarycat pushed a commit to lolbinarycat/rust that referenced this pull request Aug 16, 2024

bootstrap: improve error recovery flags to curl

4a4db1b

alternative to rust-lang#128459 fixes rust-lang#110178

lolbinarycat pushed a commit to lolbinarycat/rust that referenced this pull request Aug 16, 2024

bootstrap: improve error recovery flags to curl

12e6389

alternative to rust-lang#128459 fixes rust-lang#110178

onur-ozkan closed this Aug 25, 2024

tgross35 added a commit to tgross35/rust that referenced this pull request Aug 25, 2024

Rollup merge of rust-lang#129134 - lolbinarycat:continue-at, r=Kobzol

cff48b7

bootstrap: improve error recovery flags to curl alternative to rust-lang#128459 fixes rust-lang#110178 r? `@Kobzol`

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Aug 25, 2024

Rollup merge of rust-lang#129134 - lolbinarycat:continue-at, r=Kobzol

fa9d460

bootstrap: improve error recovery flags to curl alternative to rust-lang#128459 fixes rust-lang#110178 r? ``@Kobzol``

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Aug 25, 2024

Rollup merge of rust-lang#129134 - lolbinarycat:continue-at, r=Kobzol

3d47d24

bootstrap: improve error recovery flags to curl alternative to rust-lang#128459 fixes rust-lang#110178 r? ```@Kobzol```

make bootstrap use wget if available #128459

make bootstrap use wget if available #128459

Conversation

lolbinarycat commented Jul 31, 2024

rustbot commented Jul 31, 2024

This comment has been minimized.

rustbot commented Jul 31, 2024

lolbinarycat commented Jul 31, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Kobzol left a comment

Choose a reason for hiding this comment

lolbinarycat commented Aug 6, 2024

Kobzol commented Aug 7, 2024

lolbinarycat commented Aug 8, 2024

Kobzol commented Aug 8, 2024

This comment has been minimized.

Kobzol left a comment

Choose a reason for hiding this comment

bjorn3 commented Aug 10, 2024

onur-ozkan commented Aug 11, 2024 • edited Loading

Kobzol commented Aug 11, 2024

Kobzol commented Aug 12, 2024

bors commented Aug 12, 2024

bors commented Aug 13, 2024

rust-log-analyzer commented Aug 13, 2024

bors commented Aug 13, 2024

Kobzol commented Aug 13, 2024

onur-ozkan commented Aug 13, 2024

Kobzol commented Aug 13, 2024

onur-ozkan commented Aug 13, 2024 • edited Loading

Kobzol commented Aug 13, 2024

lolbinarycat commented Aug 15, 2024

Kobzol commented Aug 15, 2024

onur-ozkan commented Aug 15, 2024

onur-ozkan commented Aug 25, 2024

lolbinarycat commented Aug 25, 2024

onur-ozkan commented Aug 25, 2024

onur-ozkan commented Aug 11, 2024 •

edited

Loading

onur-ozkan commented Aug 13, 2024 •

edited

Loading