P-chain state prune + height index

[dhrubabasu]

Why this should be merged

Closes #1623.

This PR is a remake (with significant changes) of the original P-chain height index PR: #1447. We reverted the original PR as it was stalling nodes by populating the height index synchronously.

At the end of the process kicked off in this PR:

• All existing non-accepted blocks in blockDB will be purged from disk.
  • As of stop persisting rejected blocks on P-chain #1696, we only write accepted blocks to disk. However, if a node ran prior to this change, it will have non-accepted blocks on disk taking up unnecessary space.
• blockDB will be a map of blockIDs to Blocks.
  • Since we are only storing accepted blocks on disk, we can simplify blockDB by removing stateBlk which takes up unnecessary space.

How this works

Kicks off a background process that goes through each block in blockDB. Per block, it does this logic:

• Is the block accepted?
  • No
    • Remove it from disk.
  • Yes
    • Update the height to blockID index (blockIDDb)
    • If it's stored as a stateBlk in blockDB, replace it with block bytes

To prevent OOMing nodes and to minimize disruption, we commit periodically (currently set to every 1024 blocks) and sleep after each commit to prevent burning the DB too much.

How this was tested

• CI
• Manually on Fuji
• Manually on Mainnet
• Probe nodes

[patrick-ogrady]

Where is the early exit if the node shuts down?

[dhrubabasu]

The VM takes care of shutting down here: https://github.com/ava-labs/avalanchego/blob/master/vms/platformvm/vm.go#L315-L352

[StephenButtolph]

I don't think we've really added any of these for any of our indexing - it'll exit when the DB errors due to being closed.. (or when the process exits 🤷)

[patrick-ogrady]

And to ask a question I think I know the answer to, partial pruning will be handled gracefully (and the pruned value will only be written once we've iterated over all elements)?

[StephenButtolph]

Hmmmm this should be the case. But I think we should probably check for blockIterator.Error() before trying to write the pruned value. Specifically, because it doesn't seem like the corruptabledb has any handling for iterators.

[StephenButtolph]

This should have been:

if err := s.blockDB.Put(blkID[:], blk.Bytes()); err != nil {