Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pageserver: apply shard filtering to blocks ingested during initdb #7319

Merged
merged 2 commits into from
Apr 5, 2024

Conversation

jcsp
Copy link
Collaborator

@jcsp jcsp commented Apr 4, 2024

Problem

Ingest filtering wasn't being applied to timeline creations, so a timeline created on a sharded tenant would use 20MB+ on each shard (each shard got a full copy). This didn't break anything, but is inefficient and leaves the system in a harder-to-validate state where shards initially have some data that they will eventually drop during compaction.

Closes: #6649

Summary of changes

  • in import_rel, filter block-by-block with is_key_local
  • During test_sharding_smoke, check that per-shard physical sizes are as expected
  • Also extend the test to check deletion works as expected (this was an outstanding tech debt task)

Checklist before requesting a review

  • I have performed a self-review of my code.
  • If it is a core feature, I have added thorough tests.
  • Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
  • If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

  • Do not forget to reformat commit message to not include the above checklist

@jcsp jcsp added c/storage/pageserver Component: storage: pageserver a/tech_debt Area: related to tech debt labels Apr 4, 2024
Copy link

github-actions bot commented Apr 4, 2024

2796 tests run: 2672 passed, 0 failed, 124 skipped (full report)


Flaky tests (1)

Postgres 14

  • test_storcon_cli: debug

Code coverage* (full report)

  • functions: 27.9% (6382 of 22848 functions)
  • lines: 46.8% (44999 of 96058 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
c83b04a at 2024-04-05T08:58:22.912Z :recycle:

@jcsp jcsp marked this pull request as ready for review April 4, 2024 16:29
@jcsp jcsp requested a review from a team as a code owner April 4, 2024 16:29
@jcsp jcsp requested review from koivunej and problame April 4, 2024 16:29
@jcsp jcsp merged commit 4fc95d2 into main Apr 5, 2024
59 checks passed
@jcsp jcsp deleted the jcsp/sharding-initdb-filter branch April 5, 2024 17:07
problame added a commit that referenced this pull request Apr 5, 2024
problame added a commit that referenced this pull request Apr 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a/tech_debt Area: related to tech debt c/storage/pageserver Component: storage: pageserver
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Revise initdb handling on shard >0
2 participants