Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scrubber: migrate tenant_snapshot to remote_storage #8830

Open
arpad-m opened this issue Aug 26, 2024 · 0 comments
Open

scrubber: migrate tenant_snapshot to remote_storage #8830

arpad-m opened this issue Aug 26, 2024 · 0 comments
Labels
c/storage/scrubber Component: s3_scrubber c/storage Component: storage

Comments

@arpad-m
Copy link
Member

arpad-m commented Aug 26, 2024

With #8673 and #8761 merged, there is only one remaining part of the scrubber that uses remote_storage: the tenant_snapshot feature.

This issue tracks migration of that last piece to remote_storage, which still uses the aws sdk.

I've talked about it with John and the plan was to first extend GenericRemoteStorage:

  • addition of a DownloadOptions struct to the download function containing an optional and default off VersionId parameter. The struct can also be used for etag support, i.e. save downloading the file unless it actually changed (pageserver: optimize heatmap downloads with S3 conditional GET #6199)
  • add a list_object_versions call that returns VersionId data

Then, those two functionalities can be used in the scrubber.

I'm breaking this out of #7547 as it is not the top priority.

@arpad-m arpad-m added c/storage Component: storage c/storage/scrubber Component: s3_scrubber labels Aug 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c/storage/scrubber Component: s3_scrubber c/storage Component: storage
Projects
None yet
Development

No branches or pull requests

1 participant