Upload initdb results to S3 #5390
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
arpad-m
requested review from
conradludgate and
koivunej
and removed request for
a team
koivunej
reviewed
Sep 27, 2023
koivunej
reviewed
Sep 27, 2023
koivunej
reviewed
Sep 27, 2023
koivunej
reviewed
Oct 3, 2023
koivunej
reviewed
Oct 3, 2023
koivunej
reviewed
Oct 3, 2023
This made the archive size go down by a lot in experiments: 4002122 to 1604253 according to wc --bytes in a test run with --preserve-database-files enabled.
2412 tests run: 2284 passed, 0 failed, 128 skipped (full report)Flaky tests (5)Postgres 16
Postgres 15
Postgres 14
Code coverage (full report)
The comment gets automatically updated with the latest test results
1f571b1 at 2023-11-23T18:11:49.929Z :recycle: |
it has only very little benefit but non-zero risk for bugs
This fixes read errors when extracting the .tar.zst files
it is green now!
arpad-m
force-pushed
the
arpad/upload_initdb_result
branch
from
9baea2d
to
e08c7eb
Compare
koivunej
reviewed
Nov 23, 2023
koivunej
reviewed
Nov 23, 2023
koivunej
reviewed
Nov 23, 2023
Co-authored-by: Joonas Koivunen <joonas@neon.tech>
shanyp
approved these changes
Nov 23, 2023
koivunej
approved these changes
Nov 23, 2023
This was referenced Nov 25, 2023
arpad-m
added a commit
that referenced
this pull request
Nov 30, 2023
This PR adds an `existing_initdb_timeline_id` option to timeline creation APIs, taking an optional timeline ID. Follow-up of #5390. If the `existing_initdb_timeline_id` option is specified via the HTTP API, the pageserver downloads the existing initdb archive from the given timeline ID and extracts it, instead of running initdb itself. --------- Co-authored-by: Christian Schwarz <christian@neon.tech>
arpad-m
added a commit
that referenced
this pull request
Dec 1, 2023
If `index_part.json` is (verifiably) not present on remote storage, we should regard the timeline as inexistent. This lets `clean_up_timelines` purge the partial local disk state, which is important in the case of incomplete creations leaving behind state that hinders retries. For incomplete deletions, we also want the timeline's local disk content be gone completely. The PR removes the allowed warnings added by #5390 and #5912, as we now are only supposed to issue info level messages. It also adds a reproducer for #6007, by parametrizing the `test_timeline_init_break_before_checkpoint_recreate` test added by #5390. If one reverts the .rs changes, the "cannot create its uninit mark file" log line occurs once one comments out the failing checks for the local disk state being actually empty. Closes #6007 --------- Co-authored-by: Joonas Koivunen <joonas@neon.tech>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
See #2592
Summary of changes
Compresses the results of initdb into a .tar.zst file and uploads them to S3, to enable usage in recovery from lsn.
Generations should not be involved I think because we do this only once at the very beginning of a timeline.