-
Notifications
You must be signed in to change notification settings - Fork 434
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(rtc): maintain dirty and uploaded IndexPart #7833
Conversation
3198 tests run: 3059 passed, 0 failed, 139 skipped (full report)Code coverage* (full report)
* collected from Rust tests only The comment gets automatically updated with the latest test results
a2463ca at 2024-06-04T11:50:59.590Z :recycle: |
Perhaps cloning this to a |
8edd2d7
to
d8e56e7
Compare
Once upon a time, we used to have duplicated types for runtime IndexPart and whatever we stored. Because of the serde fixes in #5335 we have no need for duplicated IndexPart type anymore, but the `IndexLayerMetadata` stayed. - remove the type - remove LayerFileMetadata::file_size() in favor of direct field access Split off from #7833. Cc: #3072.
da3618b
to
02f225a
Compare
on serialization failure we are pretty much dead in the water, but as we don't have a good way to make tenants broken on errors, we will stay dead until error is detected. of course the serialization should never fail, but...
dcl = disk_consistent_lsn. it was only being used by mistake in http endpoints and pagectl. it should only be used in scrubber.
02f225a
to
f9ae1e9
Compare
This reverts commit b202c4a35e71a32e1e67890cd3c931790a9b9f4b.
f9ae1e9
to
63f238c
Compare
it can now be read from UploadQueueInitialized::clean
63f238c
to
a130cc2
Compare
in a situation of continously having index uploads, those should complete one by one and the completing should be the one spawning the next task. this means we cannot race to the lock by multiple completing index part uploads, which means we can turn the monotonicity check into an assertion.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't have a lot of context here so someone else should have a look, but looks reasonable.
RemoteTimelineClient maintains a copy of "next IndexPart" as a number of fields which are like an IndexPart but this is not immediately obvious. Instead of multiple fields, maintain a
dirty
("next IndexPart") andclean
("uploaded IndexPart") fields.Additional cleanup:
IndexPart::disk_consistent_lsn
accessorduplicated_disk_consistent_lsn
UploadQueueInitialized::dirty
of which the uploaded snapshot is serializedclean
IndexPart with an assertion which is not expected to fireContinued work from #7860 towards next step of #6994.