Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scrubber: more robust metadata consistency check #8128

Open
2 tasks
jcsp opened this issue Jun 21, 2024 · 0 comments
Open
2 tasks

scrubber: more robust metadata consistency check #8128

jcsp opened this issue Jun 21, 2024 · 0 comments
Assignees
Labels
c/storage/scrubber Component: s3_scrubber t/feature Issue type: feature, for new features or requests

Comments

@jcsp
Copy link
Contributor

jcsp commented Jun 21, 2024

The scrubber's scan_metadata command will flag some inconsistencies, but it's not quite robust enough to trust at scale:

  • It needs to handle objects being deleted while it scans, by re-reading the index after scan and checking that any objects that vanished have also vanished from the index.
  • It needs a way of reporting results that we can alert on at scale. Perhaps this is a log, perhaps it's a write to a database, or a write to a scrub report in S3.

Basically the code in branch_cleanup_and_check_errors is somewhat old and crufty and needs a fresh set of eyes to make sure we are robustly checking a well understood set of things, before we can use it at scale to increase confidence that our metadata's integrity is okay.

Tasks

@jcsp jcsp added t/feature Issue type: feature, for new features or requests c/storage/scrubber Component: s3_scrubber labels Jun 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c/storage/scrubber Component: s3_scrubber t/feature Issue type: feature, for new features or requests
Projects
None yet
Development

No branches or pull requests

2 participants