Skip to content

Commit

Permalink
Download wheel to disk when streaming unzip failed with HTTP streamin…
Browse files Browse the repository at this point in the history
…g error (#5094)

## Summary

Workaround the `stream_wheel` not retry issue
[found](#3514 (comment))
in #3514, it's not a perfect solution but I think it's acceptable
because the error should not occur frequently.

## Test Plan

Manually using `iptables -A OUTPUT -p tcp -dport 3128 -j REJECT
--reject-with tcp-reset` to inject connection reset error to the HTTP
proxy that proxies PyPI requests.

```
error: Failed to prepare distributions
  Caused by: Failed to fetch wheel: piqp==0.4.1
  Caused by: Request failed after 3 retries
  Caused by: error sending request for url (https://files.pythonhosted.org/packages/94/4d/09ade94dfda5b57c1ca43564541871bd1a0d89dfd3c368ac505b6ca09831/piqp-0.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl)
  Caused by: client error (Connect)
  Caused by: tcp connect error: Connection refused (os error 111)
  Caused by: Connection refused (os error 111)
```
  • Loading branch information
messense authored Jul 16, 2024
1 parent 545bbf9 commit 38504dc
Show file tree
Hide file tree
Showing 4 changed files with 27 additions and 4 deletions.
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 10 additions & 4 deletions crates/uv-distribution/src/distribution_database.rs
Original file line number Diff line number Diff line change
Expand Up @@ -208,10 +208,16 @@ impl<'a, Context: BuildContext> DistributionDatabase<'a, Context> {
hashes: archive.hashes,
filename: wheel.filename.clone(),
}),
Err(Error::Extract(err)) if err.is_http_streaming_unsupported() => {
warn!(
"Streaming unsupported for {dist}; downloading wheel to disk ({err})"
);
Err(Error::Extract(err)) => {
if err.is_http_streaming_unsupported() {
warn!(
"Streaming unsupported for {dist}; downloading wheel to disk ({err})"
);
} else if err.is_http_streaming_failed() {
warn!("Streaming failed for {dist}; downloading wheel to disk ({err})");
} else {
return Err(Error::Extract(err));
}

// If the request failed because streaming is unsupported, download the
// wheel directly.
Expand Down
1 change: 1 addition & 0 deletions crates/uv-extract/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ fs-err = { workspace = true, features = ["tokio"] }
futures = { workspace = true }
md-5.workspace = true
rayon = { workspace = true }
reqwest = { workspace = true }
rustc-hash = { workspace = true }
sha2 = { workspace = true }
thiserror = { workspace = true }
Expand Down
15 changes: 15 additions & 0 deletions crates/uv-extract/src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,19 @@ impl Error {
Self::AsyncZip(async_zip::error::ZipError::FeatureNotSupported(_))
)
}

/// Returns `true` if the error is due to HTTP streaming request failed.
pub fn is_http_streaming_failed(&self) -> bool {
match self {
Self::AsyncZip(async_zip::error::ZipError::UpstreamReadError(_)) => true,
Self::Io(err) => {
if let Some(inner) = err.get_ref() {
inner.downcast_ref::<reqwest::Error>().is_some()
} else {
false
}
}
_ => false,
}
}
}

0 comments on commit 38504dc

Please sign in to comment.