-
Notifications
You must be signed in to change notification settings - Fork 409
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(pageserver): persist aux file policy in index part #7668
Conversation
876a814
to
41fd310
Compare
41fd310
to
6401330
Compare
3096 tests run: 2969 passed, 0 failed, 127 skipped (full report)Flaky tests (3)Postgres 16Code coverage* (full report)
* collected from Rust tests only The comment gets automatically updated with the latest test results
24a864a at 2024-05-17T19:30:46.315Z :recycle: |
03038da
to
396b9cf
Compare
2f73656
to
2ee9021
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pre-emptively for https://github.com/neondatabase/neon/pull/7668/files#r1596792568.
93b234b
to
abbdfda
Compare
Updated the pull request with storing the field in index part, and waiting for CI runs. The pull request body also gets updated correspondingly. |
abbdfda
to
41453ac
Compare
Signed-off-by: Alex Chi Z <chi@neon.tech>
Signed-off-by: Alex Chi Z <chi@neon.tech>
Signed-off-by: Alex Chi Z <chi@neon.tech>
Signed-off-by: Alex Chi Z <chi@neon.tech>
Signed-off-by: Alex Chi Z <chi@neon.tech>
Co-Authored-By: Joonas Koivunen <joonas@neon.tech> Signed-off-by: Alex Chi Z <chi@neon.tech>
I also took the test cases from #7781 and adapted to my proposed semantics of these flags. |
And to add, I think the main confusing in this PR is that we don't plan to support v1->v2 migration for customers. New customers go into cross validation mode first and go into v2 when we remove the read/write path of v1; existing customers will jump start from v2 and not being able to access their v1 replication state. v1->v2 migration is possible but it needs careful design, and with the persistent flag, it is still possible for us to do so, but I decided not to do that for now. |
Signed-off-by: Alex Chi Z <chi@neon.tech>
added |
Signed-off-by: Alex Chi Z <chi@neon.tech>
Part of #7462 ## Summary of changes Tenant config is not persisted unless it's attached on the storage controller. In this pull request, we persist the aux file policy flag in the `index_part.json`. Admins can set `switch_aux_file_policy` in the storage controller or using the page server API. Upon the first aux file gets written, the write path will compare the aux file policy target with the current policy. If it is switch-able, we will do the switch. Otherwise, the original policy will be used. The test cases show what the admins can do / cannot do. The `last_aux_file_policy` is stored in `IndexPart`. Updates to the persisted policy are done via `schedule_index_upload_for_aux_file_policy_update`. On the write path, the writer will update the field. --------- Signed-off-by: Alex Chi Z <chi@neon.tech> Co-authored-by: Joonas Koivunen <joonas@neon.tech>
Problem
Part of #7462
Summary of changes
Tenant config is not persisted unless it's attached on the storage controller. In this pull request, we persist the aux file policy flag in the
index_part.json
.Admins can set
switch_aux_file_policy
in the storage controller or using the page server API. Upon the first aux file gets written, the write path will compare the aux file policy target with the current policy. If it is switch-able, we will do the switch. Otherwise, the original policy will be used. The test cases show what the admins can do / cannot do.The
last_aux_file_policy
is stored inIndexPart
. Updates to the persisted policy are done viaschedule_index_upload_for_aux_file_policy_update
. On the write path, the writer will update the field.Checklist before requesting a review
Checklist before merging