-
Notifications
You must be signed in to change notification settings - Fork 394
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Epic: Support Azure Blob storage in the scrubber #7547
Labels
Comments
5 tasks
5 tasks
arpad-m
added a commit
that referenced
this issue
Jul 22, 2024
Starts using the `remote_storage` crate in the S3 scrubber for the `PurgeGarbage` subcommand. The `remote_storage` crate is generic over various backends and thus using it gives us the ability to run the scrubber against all supported backends. Start with the `PurgeGarbage` subcommand as it doesn't use `stream_tenants`. Part of #7547.
This week:
|
arpad-m
added a commit
that referenced
this issue
Jul 24, 2024
This adds the ability to list many prefixes in a streaming fashion to both the `RemoteStorage` trait as well as `GenericRemoteStorage`. * The `list` function of the `RemoteStorage` trait is implemented by default in terms of `list_streaming`. * For the production users (S3, Azure), `list_streaming` is implemented and the default `list` implementation is used. * For `LocalFs`, we keep the `list` implementation and make `list_streaming` call it. The `list_streaming` function is implemented for both S3 and Azure. A TODO for later is retries, which the scrubber currently has while the `list_streaming` implementations lack them. part of #8457 and #7547
arpad-m
added a commit
that referenced
this issue
Jul 24, 2024
Implements the TODO from #8466 about retries: now the user of the stream returned by `list_streaming` is able to obtain the next item in the stream as often as they want, and retry it if it is an error. Also adds extends the test for paginated listing to include a dedicated test for `list_streaming`. follow-up of #8466 fixes #8457 part of #7547 --------- Co-authored-by: Joonas Koivunen <joonas@neon.tech>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Motivation
We want to run the same tools on Azure Blob storage that we run on S3.
DoD
The scrubber supports Azure Blob storage as well as S3.
Implementation ideas
I'd implement this via extending the
remote_storage
crate to cover the use cases of the scrubber and porting the scrubber to it.This doesn't have to happen in one go, but can go command-by-command. For the transition period, the scrubber could create both S3 and
remote_storage
contexts and then have each command use whatever it supports.Tasks
PurgeGarbage
: Start using remote_storage in S3 scrubber for PurgeGarbage #7932FindGarbage
: storage_scrubber: migrate FindGarbage to remote_storage #8548ScanMetadata
: storage_scrubber: migrate scan_safekeeper_metadata to remote_storage #8595TenantSnapshot
PageserverPhysicalGc
FindLargeObjects
: scrubber: add remote_storage based listing APIs and use them in find-large-objects #8541Other related tasks and Epics
The text was updated successfully, but these errors were encountered: