metrics for new aux file storage #7443
Assignees
Labels
c/storage/pageserver
Component: storage: pageserver
Comments
arpad-m
added
t/bug
5 tasks
skyzh
added a commit
that referenced
this issue
May 13, 2024
ref #7443 ## Summary of changes This pull request adds a size estimator for aux files. Each timeline stores a cached `isize` for the estimated total size of aux files. It gets reset on basebackup, and gets updated for each aux file modification. TODO: print a warning when it exceeds the size. The size metrics is not accurate. Race between `on_basebackup` and other functions could create a negative basebackup size, but the chance is rare. Anyways, this does not impose any extra I/Os to the storage as everything is computed in-memory. The aux files are only stored on shard 0. As basebackups are only generated on shard 0, only shard 0 will report this metrics. --------- Signed-off-by: Alex Chi Z <chi@neon.tech>
a-masterov
pushed a commit
that referenced
this issue
May 20, 2024
ref #7443 ## Summary of changes This pull request adds a size estimator for aux files. Each timeline stores a cached `isize` for the estimated total size of aux files. It gets reset on basebackup, and gets updated for each aux file modification. TODO: print a warning when it exceeds the size. The size metrics is not accurate. Race between `on_basebackup` and other functions could create a negative basebackup size, but the chance is rare. Anyways, this does not impose any extra I/Os to the storage as everything is computed in-memory. The aux files are only stored on shard 0. As basebackups are only generated on shard 0, only shard 0 will report this metrics. --------- Signed-off-by: Alex Chi Z <chi@neon.tech>
arpad-m
changed the title
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Even with the new aux file storage we are interested in how many items we store in aux storage and how large their impact is. This is important because it determines the size of basebackup tarballs and beyond just knowing how large they can be in theory, it's also useful to know how large they are usually in order to help us verify that the approach we use now works great.
Tasks:
The text was updated successfully, but these errors were encountered: