Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(rtc): maintain dirty and uploaded IndexPart #7833

Merged
merged 16 commits into from
Jun 4, 2024

Conversation

koivunej
Copy link
Member

@koivunej koivunej commented May 21, 2024

RemoteTimelineClient maintains a copy of "next IndexPart" as a number of fields which are like an IndexPart but this is not immediately obvious. Instead of multiple fields, maintain a dirty ("next IndexPart") and clean ("uploaded IndexPart") fields.

Additional cleanup:

  • rename IndexPart::disk_consistent_lsn accessor duplicated_disk_consistent_lsn
    • no one except scrubber should be looking at it, even scrubber is a stretch
    • remove usage elsewhere (pagectl used by tests, metadata scan endpoint)
  • serialize index part before the index upload operation
    • avoid upload operation being retried because of serialization error
    • serialization error is fatal anyway for timeline -- it can only make transient local progress after that, at least the error is bubbled up now
  • gather exploded IndexPart fields into single actual UploadQueueInitialized::dirty of which the uploaded snapshot is serialized
  • implement the long wished monotonicity check with the clean IndexPart with an assertion which is not expected to fire

Continued work from #7860 towards next step of #6994.

Copy link

github-actions bot commented May 21, 2024

3198 tests run: 3059 passed, 0 failed, 139 skipped (full report)


Flaky tests (1)

Postgres 14

  • test_hot_standby_gc[True]: debug

Code coverage* (full report)

  • functions: 31.4% (6563 of 20873 functions)
  • lines: 48.4% (50784 of 104942 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
a2463ca at 2024-06-04T11:50:59.590Z :recycle:

@koivunej koivunej changed the title wip: clean up rtc index uploads, read-commited reads clean up rtc index uploads, read-commited reads May 22, 2024
@koivunej
Copy link
Member Author

Perhaps cloning this to a clean on completed upload isn't a problem, I was remembering we would use String in IndexPart::layer_metadata: HashMap<_, LayerFileMetadata> but that is the fixed length LayerName.

Base automatically changed from joonas/openapi_tad to main May 22, 2024 13:55
koivunej added a commit that referenced this pull request May 23, 2024
Once upon a time, we used to have duplicated types for runtime IndexPart
and whatever we stored. Because of the serde fixes in #5335 we have no
need for duplicated IndexPart type anymore, but the `IndexLayerMetadata`
stayed.

- remove the type
- remove LayerFileMetadata::file_size() in favor of direct field access

Split off from #7833. Cc: #3072.
on serialization failure we are pretty much dead in the water, but as we
don't have a good way to make tenants broken on errors, we will stay
dead until error is detected.

of course the serialization should never fail, but...
dcl = disk_consistent_lsn. it was only being used by mistake in http
endpoints and pagectl. it should only be used in scrubber.
it can now be read from UploadQueueInitialized::clean
@koivunej koivunej changed the title clean up rtc index uploads, read-commited reads feat(rtc): maintain dirty and uploaded IndexPart Jun 3, 2024
@koivunej koivunej marked this pull request as ready for review June 3, 2024 13:15
@koivunej koivunej requested a review from a team as a code owner June 3, 2024 13:15
@koivunej koivunej requested a review from arssher June 3, 2024 13:15
@koivunej koivunej requested a review from jcsp June 3, 2024 13:42
in a situation of continously having index uploads, those should
complete one by one and the completing should be the one spawning the
next task. this means we cannot race to the lock by multiple completing
index part uploads, which means we can turn the monotonicity check into
an assertion.
Copy link
Contributor

@arssher arssher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't have a lot of context here so someone else should have a look, but looks reasonable.

@koivunej koivunej merged commit 0112097 into main Jun 4, 2024
65 checks passed
@koivunej koivunej deleted the joonas/rtc_index_uploads branch June 4, 2024 14:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants