Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pageserver: remove import/export script previously used for breaking format changes #7458

Merged
merged 2 commits into from
Apr 23, 2024

Conversation

VladLazar
Copy link
Contributor

Problem

The fullbackup command enabled us to do major storage format changes in the past.
If we have to do such breaking changes in the future this approach wouldn't be suitable because:

  1. It doesn't scale to the current size of the fleet
  2. It loses history

Summary of changes

Remove the fullbackup command handling and it's only user, the import/export script.

Note that I've kept the import basebackup and import wal commands since:

  1. They enable a valid use case IMO: importing from a vanilla postgres instance (used by at least one test)
  2. They enable the neon_local import command

Closes https://github.com/neondatabase/cloud/issues/11648

Checklist before requesting a review

  • I have performed a self-review of my code.
  • If it is a core feature, I have added thorough tests.
  • Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
  • If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

  • Do not forget to reformat commit message to not include the above checklist

@VladLazar VladLazar changed the title Vlad/remove fullbackup and friends pageserver: remove fullbackup command handling Apr 22, 2024
@hlinnaka
Copy link
Contributor

-1, it's very handy while debugging. @kelvich and I used it just last week when debugging a production issue. And we will eventually need the functionality for a user-exposed export feature. It was discussed as part of pg_upgrade, too.

I agree it's not how we'd want to do major storage format changes in the future. Some comment cleanup would be in order, to explain why the command still exists.

@hlinnaka
Copy link
Contributor

It was discussed as part of pg_upgrade, too.

scratch that part, that was about the import function, not fullbackup

Copy link

github-actions bot commented Apr 22, 2024

2760 tests run: 2642 passed, 0 failed, 118 skipped (full report)


Code coverage* (full report)

  • functions: 28.1% (6464 of 23041 functions)
  • lines: 46.8% (45676 of 97593 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
7656930 at 2024-04-22T17:30:15.468Z :recycle:

@VladLazar
Copy link
Contributor Author

-1, it's very handy while debugging. @kelvich and I used it just last week when debugging a production issue.

Ok, let's leave it in that case. Did you use the export_import_between_pageservers.py script or did you do it manually? Trying to figure out what should stay and what's dead code.

@hlinnaka
Copy link
Contributor

-1, it's very handy while debugging. @kelvich and I used it just last week when debugging a production issue.

Ok, let's leave it in that case. Did you use the export_import_between_pageservers.py script or did you do it manually? Trying to figure out what should stay and what's dead code.

I don't remember; @kelvich do you? The import part of the script certainly seems less interesting. And the add_missing_rels hack in the script seems obsolete for sure.

@kelvich
Copy link
Contributor

kelvich commented Apr 22, 2024

I don't remember; @kelvich do you? The import part of the script certainly seems less interesting. And the add_missing_rels hack in the script seems obsolete for sure.

I usually use it manually: psql -c 'fullbackup <> <> <> <>' > backup.tar

@VladLazar
Copy link
Contributor Author

I don't remember; @kelvich do you? The import part of the script certainly seems less interesting. And the add_missing_rels hack in the script seems obsolete for sure.

I usually use it manually: psql -c 'fullbackup <> <> <> <>' > backup.tar

Would be nice to have this procedure written down somewhere. The immediate next question is, what do you do with the fullbackup after you got it. I guess you'd use import basebackup if you wanted to load it into another pageserver.

@VladLazar
Copy link
Contributor Author

Oke, so based on the feedback above, I will:

  • remove the script
  • update the comments around basebackup to make its purpose a bit clearer

@VladLazar VladLazar force-pushed the vlad/remove-fullbackup-and-friends branch from 162141d to 7656930 Compare April 22, 2024 16:52
@VladLazar VladLazar requested a review from hlinnaka April 22, 2024 16:54
@VladLazar VladLazar marked this pull request as ready for review April 22, 2024 16:54
@VladLazar VladLazar requested a review from a team as a code owner April 22, 2024 16:54
@VladLazar VladLazar requested a review from jcsp April 22, 2024 16:54
@VladLazar VladLazar changed the title pageserver: remove fullbackup command handling pageserver: remove import/export script previously used for breaking format changes Apr 22, 2024
Copy link
Contributor

@hlinnaka hlinnaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked if we still have test coverage for the fullbackup method; yes we do, tests test_import_from_pageserver_*.

@VladLazar VladLazar merged commit d551bfe into main Apr 23, 2024
52 of 53 checks passed
@VladLazar VladLazar deleted the vlad/remove-fullbackup-and-friends branch April 23, 2024 10:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants