Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(scrubber): more robust metadata consistency checks #8344

Merged
merged 17 commits into from
Jul 22, 2024
Merged

fix(scrubber): more robust metadata consistency checks #8344

merged 17 commits into from
Jul 22, 2024

Commits on Jul 15, 2024

  1. fix(scrubber): more robust metadata consistency checks 

    Signed-off-by: Yuchen Liang <yuchen@neon.tech>
    @yliang412
    yliang412 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    455c5d6 View commit details
    Browse the repository at this point in the history

  2. add tests for scrubber check 

    Signed-off-by: Yuchen Liang <yuchen@neon.tech>
    @yliang412
    yliang412 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    eaf3a3b View commit details
    Browse the repository at this point in the history

  3. add comments on test 

    Signed-off-by: Yuchen Liang <yuchen@neon.tech>
    @yliang412
    yliang412 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    b228fef View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2024

  1. fix lints 

    Signed-off-by: Yuchen Liang <yuchen@neon.tech>
    @yliang412
    yliang412 committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    7cdb5ac View commit details
    Browse the repository at this point in the history

  2. fix tenant path 

    Signed-off-by: Yuchen Liang <yuchen@neon.tech>
    @yliang412
    yliang412 committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    3773a94 View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2024

  1. make is_l0 a common function taking in key range 

    Signed-off-by: Yuchen Liang <yuchen@neon.tech>
    @yliang412
    yliang412 committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    ae46fa8 View commit details
    Browse the repository at this point in the history

  2. use remote_layer_path to construct path 

    Signed-off-by: Yuchen Liang <yuchen@neon.tech>
    @yliang412
    yliang412 committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    d98b5ad View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into yuchen/scrubber-robust-metadata-check

    @yliang412
    yliang412 committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    b6dad33 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2024

  1. ignore initdb-preserved.tar.zst 

    Signed-off-by: Yuchen Liang <yuchen@neon.tech>
    @yliang412
    yliang412 committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    6fdd72e View commit details
    Browse the repository at this point in the history

  2. move remote_storage utility function to S3Storage; wordsmith 

    Signed-off-by: Yuchen Liang <yuchen@neon.tech>
    @yliang412
    yliang412 committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    fd83fb2 View commit details
    Browse the repository at this point in the history

  3. add a control metadata scan before injecting corruption 

    Signed-off-by: Yuchen Liang <yuchen@neon.tech>
    @yliang412
    yliang412 committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    e696884 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into yuchen/scrubber-robust-metadata-check

    @yliang412
    yliang412 committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    e77d064 View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2024

  1. scrubber: remove metadata histograms (#8424) 

    The histograms in the metadata summary were added experimentally. We
will remove them since we can get most of the insights from prometheus
stats. Also reduce noise when maximum value go out of the range.

---------

Signed-off-by: Yuchen Liang <yuchen@neon.tech>
    @yliang412
    yliang412 committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    0237221 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into yuchen/scrubber-robust-metadata-check

    @yliang412
    yliang412 committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    5516dea View commit details
    Browse the repository at this point in the history

  3. rebase scrubber scan_metadata test 

    Signed-off-by: Yuchen Liang <yuchen@neon.tech>
    @yliang412
    yliang412 committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    c67da97 View commit details
    Browse the repository at this point in the history

  4. only analyze s with highest shard count; skip deleted timeline 

    Signed-off-by: Yuchen Liang <yuchen@neon.tech>
    @yliang412
    yliang412 committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    ed6cfdb View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2024

  1. Merge branch 'main' into yuchen/scrubber-robust-metadata-check

    @yliang412
    yliang412 committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    a6cfed9 View commit details
    Browse the repository at this point in the history