[stable9] Allow chunk GC mtime tolerance for unfinished part chunks #24676
Conversation
|
By analyzing the blame information on this pull request, we identified @DeepDiver1975, @LukasReschke and @MorrisJobke to be potential reviewers |
PVince81
changed the title
|
Please don't merge yet, target is stable9 for now for testability. |
|
CC @karlitschek @cmonteroluque for backport to 9.0, 8.2, 8.1, 8.0 This should solve most of the good old "Expected filesize X got 0" (the one with zero) errors... |
|
Awesome. Please backport. |
|
Awww, forgot to remove it. It was useful for testing. |
Whenever part chunks are written, every fwrite in the write loop will reset the mtime to the current mtime. Only at the end will the touch() operation set the mtime to now + ttl, in the future. However the GC code is expecting that every chunk with mtime < now are old and must be deleted. This causes the GC to sometimes delete part chunks in which the write loop is slow. To fix this, a tolerance value is added in the GC code to allow for more time before a part chunk gets deleted.
|
That's what I figured. Or we leave it in and claim a massive performance boost in the next release when we remove it ;-) |
PVince81
force-pushed
the
stable9-fixchunkttl
branch
from
90c91cd
to
dfc0a7a
Compare
|
Fixes the "expected filesize 5242880 got 0" errors for me on 9.0.2.2. Tested uploading 6 x 100MB files. |
|
yeah if you can't upload 8k in an hour then you have bigger things to worry about! 👍 |
|
Should also go into master I assume? |
|
Yes, I'll take care of submitting all the PRs soon |
|
master PR here #24691 I'll list the other backports there |
|
👍 |
|
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Whenever part chunks are written, every fwrite in the write loop will
reset the mtime to the current mtime. Only at the end will the touch()
operation set the mtime to now + ttl, in the future.
However the GC code is expecting that every chunk with mtime < now are
old and must be deleted. This causes the GC to sometimes delete part
chunks in which the write loop is slow.
To fix this, a tolerance value is added in the GC code to allow for
more time before a part chunk gets deleted.
Please review/test @icewind1991 @Marginal @MorrisJobke @nickvergessen @schiesbn
With these steps and the fix I could not reproduce the issue: #24653 (comment)