Workarounds for copy_file_range issues #75428

the8472 · 2020-08-11T23:31:52Z

…ttemted on a NFS mount under RHEL/CentOS 7. The syscall is supposed to return ENOSYS in most cases but when calling it on NFS it may leak through EOPNOTSUPP even though that's supposed to be handled by the kernel and not returned to userspace. Since it returns ENOSYS in some cases anyway this will trip the HAS_COPY_FILE_RANGE detection anyway, so treat EOPNOTSUPP as if it were a ENOSYS. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.8_release_notes/deprecated_functionality#the_literal_copy_file_range_literal_call_has_been_disabled_on_local_file_systems_and_in_nfs https://bugzilla.redhat.com/show_bug.cgi?id=1783554

rust-highfive · 2020-08-11T23:31:55Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @joshtriplett (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

joshtriplett · 2020-08-12T01:06:37Z

@bors r+

bors · 2020-08-12T01:06:38Z

📌 Commit 1316c78 has been approved by joshtriplett

library/std/src/sys/unix/fs.rs

bors · 2020-08-12T11:37:16Z

⌛ Testing commit 1316c78 with merge c0e06fb4d88cf4063d23e407c155a2096b4ac611...

rust-log-analyzer · 2020-08-12T11:41:27Z

Your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

bors · 2020-08-12T11:57:18Z

💔 Test failed - checks-actions

the8472 · 2020-08-12T18:11:40Z

The bors error is not helpful, spurious failure?

library/std/src/sys/unix/fs.rs

… on file size This solves several problems - race conditions where a file is truncated while copying from it. if we blindly trusted the file size this would lead to an infinite loop - proc files appearing empty to copy_file_range but not to read/write coreutils/coreutils@4b04a0c - copy_file_range returning 0 for some filesystems (overlay? bind mounts?) inside docker, again leading to an infinite loop

the8472 · 2020-08-14T21:19:10Z

Updated to address an additional copy_file_range issue.

the8472 · 2020-08-14T21:59:55Z

ping @joshtriplett

tavianator · 2020-08-15T19:42:37Z

library/std/src/sys/unix/fs.rs

        let copy_result = if has_copy_file_range {
-            let bytes_to_copy = cmp::min(len - written, usize::MAX as u64) as usize;
+            let bytes_to_copy = cmp::min(max_len - written, usize::MAX as u64) as usize;


Is it really necessary to keep track of max_len - written accurately at this point? It's probably okay just to pass a large constant size on every call.

It avoids overflowing an u64. Plus I want to reuse the code in #75272 which will require exact max sizes

bors · 2020-09-05T04:15:45Z

⌛ Testing commit 4ddedd5 with merge 596b8ed70633bf181a3b8966fb235e91a67ae146...

bors · 2020-09-05T08:30:45Z

💥 Test timed out

Dylan-DPC-zz · 2020-09-05T12:43:14Z

@bors retry

…triplett Workarounds for copy_file_range issues fixes rust-lang#75387 fixes rust-lang#75446

bors · 2020-09-05T13:55:35Z

⌛ Testing commit 4ddedd5 with merge f9b36230f023ae6f396f3cc4284cd41a0583b819...

bors · 2020-09-05T14:59:24Z

💔 Test failed - checks-actions

the8472 · 2020-09-05T16:43:43Z

I don't see how linux-only code changes can make a mingw build fail. Doubly so when it's doc tests.

nagisa · 2020-09-05T18:52:01Z

@bors retry

bors · 2020-09-05T19:09:27Z

⌛ Testing commit 4ddedd5 with merge de921ab...

bors · 2020-09-05T21:02:11Z

☀️ Test successful - checks-actions, checks-azure
Approved by: joshtriplett
Pushing de921ab to master...

implement better availability probing for copy_file_range Followup to rust-lang#75428 (comment) Previously syscall detection was overly pessimistic. Any attempt to copy to an immutable file (EPERM) would disable copy_file_range support for the whole process. The change tries to copy_file_range on invalid file descriptors which will never run into the immutable file case and thus we can clearly distinguish syscall availability.

rust-highfive assigned joshtriplett Aug 11, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Aug 11, 2020

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 12, 2020

tesuji reviewed Aug 12, 2020

View reviewed changes

library/std/src/sys/unix/fs.rs Outdated Show resolved Hide resolved

tesuji reviewed Aug 12, 2020

View reviewed changes

library/std/src/sys/unix/fs.rs Outdated Show resolved Hide resolved

Tavi-Kohn reviewed Aug 12, 2020

View reviewed changes

library/std/src/sys/unix/fs.rs Show resolved Hide resolved

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Aug 12, 2020

more concise error matching

f078363

nagisa reviewed Aug 13, 2020

View reviewed changes

library/std/src/sys/unix/fs.rs Show resolved Hide resolved

the8472 changed the title ~~Workaround for copy_file_range spuriously returning EOPNOTSUPP on CentOS 7~~ Workarounds for copy_file_range issues Aug 14, 2020

tavianator reviewed Aug 15, 2020

View reviewed changes

the8472 mentioned this pull request Sep 4, 2020

specialize io::copy to use copy_file_range, splice or sendfile #75272

Merged

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 5, 2020

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Sep 5, 2020

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 5, 2020

Dylan-DPC-zz pushed a commit to Dylan-DPC-zz/rust that referenced this pull request Sep 5, 2020

Rollup merge of rust-lang#75428 - the8472:fix-copy-eopnotsupp, r=josh…

5e60cf8

…triplett Workarounds for copy_file_range issues fixes rust-lang#75387 fixes rust-lang#75446

Dylan-DPC-zz mentioned this pull request Sep 5, 2020

Rollup of 14 pull requests #76373

Closed

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Sep 5, 2020

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 5, 2020

bors added the merged-by-bors This PR was explicitly merged by bors. label Sep 5, 2020

bors merged commit de921ab into rust-lang:master Sep 5, 2020

rustbot added this to the 1.48.0 milestone Sep 5, 2020

the8472 mentioned this pull request Nov 21, 2020

implement better availability probing for copy_file_range #79274

Merged

leomao10 mentioned this pull request May 19, 2021

Very specific set of circumstances leads to zero-byte (empty) file being created docker/for-linux#1015

Closed

3 tasks

the8472 mentioned this pull request Jun 29, 2021

Use copy_file_range(2) only on Linux >= 5.3 #86718

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Workarounds for copy_file_range issues #75428

Workarounds for copy_file_range issues #75428

the8472 commented Aug 11, 2020 •

edited

Loading

rust-highfive commented Aug 11, 2020

joshtriplett commented Aug 12, 2020

bors commented Aug 12, 2020

bors commented Aug 12, 2020

rust-log-analyzer commented Aug 12, 2020

bors commented Aug 12, 2020

the8472 commented Aug 12, 2020

the8472 commented Aug 14, 2020

the8472 commented Aug 14, 2020

tavianator Aug 15, 2020

the8472 Aug 15, 2020

bors commented Sep 5, 2020

bors commented Sep 5, 2020

Dylan-DPC-zz commented Sep 5, 2020

bors commented Sep 5, 2020

bors commented Sep 5, 2020

the8472 commented Sep 5, 2020

nagisa commented Sep 5, 2020

bors commented Sep 5, 2020

bors commented Sep 5, 2020

Workarounds for copy_file_range issues #75428

Workarounds for copy_file_range issues #75428

Conversation

the8472 commented Aug 11, 2020 • edited Loading

rust-highfive commented Aug 11, 2020

joshtriplett commented Aug 12, 2020

bors commented Aug 12, 2020

bors commented Aug 12, 2020

rust-log-analyzer commented Aug 12, 2020

bors commented Aug 12, 2020

the8472 commented Aug 12, 2020

the8472 commented Aug 14, 2020

the8472 commented Aug 14, 2020

tavianator Aug 15, 2020

Choose a reason for hiding this comment

the8472 Aug 15, 2020

Choose a reason for hiding this comment

bors commented Sep 5, 2020

bors commented Sep 5, 2020

Dylan-DPC-zz commented Sep 5, 2020

bors commented Sep 5, 2020

bors commented Sep 5, 2020

the8472 commented Sep 5, 2020

nagisa commented Sep 5, 2020

bors commented Sep 5, 2020

bors commented Sep 5, 2020

the8472 commented Aug 11, 2020 •

edited

Loading