ancient shrink bug fix - remove skip slots #2927
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
HaoranYi
changed the title
|
I caused this here: |
|
seems like it would be great to have a test. in the meantime, this is clearly wrong and needs to be merged asap. |
|
It looks like we already have a test case to cover the delete for skipping slots. I have updated the test to cover the correct deleting behavior. |
HaoranYi
force-pushed
the
fix_ancient_pack
branch
from
66c911d
to
b7169d2
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
There is a bug in ancient shrinking - not removing skipped slots from actual shrinking.
Summary of Changes
remove skipped slots from actual shrinking.
before the fix (num_slots_shrunk=0):
[2024-09-13T17:17:36.603654713Z INFO solana_metrics::metrics] datapoint: shrink_ancient_stats num_slots_shrunk=0i index_scan_returned_none=0i index_scan_returned_some=3854447i storage_read_elapsed=2570935i num_duplicated_accounts=0i index_read_elapsed=8062383i create_and_insert_store_elapsed=0i store_accounts_elapsed=0i update_index_elapsed=0i handle_reclaims_elapsed=0i remove_old_stores_shrink_us=0i rewrite_elapsed=0i unpackable_slots_count=70i newest_alive_packed_count=228941i drop_storage_entries_elapsed=0i accounts_removed=574179i bytes_removed=224574008i bytes_written=1303311744i alive_accounts=3280268i dead_accounts=288404i accounts_loaded=3854447i ancient_append_vecs_shrunk=0i random=25i slots_eligible_to_shrink=116326i total_dead_bytes=3194867160i total_alive_bytes=24936296088i slots_considered=349305i ancient_scanned=0i total_us=12413822i bytes_ancient_created=0i bytes_from_must_shrink=794468832i bytes_from_smallest_storages=547390160i bytes_from_newest_storages=318152i many_ref_slots_skipped=70i slots_cannot_move_count=20i many_refs_old_alive=24i purged_zero_lamports_count=0i accounts_not_found_in_index=0iafter the fix (num_slots_shrunk=143688i):
[2024-09-14T01:41:36.336810701Z INFO solana_metrics::metrics] datapoint: shrink_ancient_stats num_slots_shrunk=143688i index_scan_returned_none=0i index_scan_returned_some=6624140i storage_read_elapsed=37268722i num_duplicated_accounts=0i index_read_elapsed=172293636i create_and_insert_store_elapsed=9091913i store_accounts_elapsed=354280809i update_index_elapsed=64029479i handle_reclaims_elapsed=0i remove_old_stores_shrink_us=41921345i rewrite_elapsed=418431086i unpackable_slots_count=83i newest_alive_packed_count=281390i drop_storage_entries_elapsed=1000i accounts_removed=3040391i bytes_removed=427606104i bytes_written=956024832i alive_accounts=3583749i dead_accounts=201141i accounts_loaded=6624140i ancient_append_vecs_shrunk=182761i random=38i slots_eligible_to_shrink=16790i total_dead_bytes=17117208i total_alive_bytes=119581136i slots_considered=355591i ancient_scanned=0i total_us=405999472i bytes_ancient_created=955343832i bytes_from_must_shrink=65777712i bytes_from_smallest_storages=1275587248i bytes_from_newest_storages=797872i many_ref_slots_skipped=78i slots_cannot_move_count=143681i many_refs_old_alive=381509i purged_zero_lamports_count=2838313i accounts_not_found_in_index=0iFixes #