Optimistic elections

[pwojcikdev]

When bootstrapping, a node needs to cement a large number of blocks (~175 million so far), so it is critical to ensure this process is as efficient and quick as possible. Currently, it is done in a very naive way of confirming blocks in account chains one by one, ie. election scheduler always activates election for account's confirmation frontier + 1. This means that the time to confirm all blocks is more or less proportional to number of all blocks in the ledger.

This PR introduces a helper mechanism to speed up this cementing process. An optimistic scheduler randomly samples accounts with unconfirmed blocks and activates an election for the account head block (the latest unconfirmed block in an account chain). This means that the time to confirm all blocks will be proportional to number of all accounts in the ledger. The number of optimistic elections is limited to 500, so it doesn't replace or impact the default election scheduling behavior.

This is a very simple algorithm that can definitely be improved further. However, even is such a basic form it provides a significant speedup to the cementing process.

The following test results are thanks to @gr0vity-dev:

Live network:

The testcase uses a fully cemented live ledger and then resets the confirmation height.

• The first 90m blcoks were synced in 3 hours
• The remaining 85m blocks took 4 days (https://kutt.bnano.info/L2azCQ):

Beta network:

• Optimistic elections = 100% cemented in about 5 hours (2 runs https://kutt.bnano.info/cprjwa)
• V24.0 = 40% cemented after 20 hours (https://kutt.bnano.info/prrqU2)

I suspect the main reason for such a big difference between live and beta performance is the greater variety of PRs, thus it's harder to gather enough votes for block confirmation.

[qwahzi]

@pwojcikdev Does the optimistic confirmation process apply per account-chain, or does it apply cross-chain too (and/or could that be an avenue for future optimization)?

For example, if account_c has receives from account_a and account_b, if account_c's latest frontier is confirmed, are transactions from account_a and account_b (up to the send associated with account_c) confirmed too?

---

Semi-related, does this change replace/modify previous automatic dependent block confirmations via confirmation_height? Or is it completely separate?

When confirmation height is set, any blocks below that height in the account are also considered confirmed by extension, as well as any dependent blocks from other account chains. This means the node can stop any elections for these lower blocks and remove them from any processing queues. This will save processing and bandwidth resources.

[qwahzi]

@pwojcikdev - I've been testing this for the last few days with good results, but it looks like my node just switched out of bootstrapping mode, even though it wasn't finished. Unchecked is suddenly capped to 256,000 (unchecked memory table?) and count/cemented drastically slowed down to ~20/hour (from 200k-300k/hr). It's probably not directly related to this branch/change, but I figured I'd report it just in case:

Sun Feb 12 08:00:01 AM CST 2023
{
    "count": "168740012",
    "unchecked": "11596953",
    "cemented": "168513871"
}

Sun Feb 12 12:56:01 PM CST 2023
{
    "count": "169643611",
    "unchecked": "256000",
    "cemented": "169559033"
}

Sun Feb 12 01:09:43 PM CST 2023
{
    "count": "169643635",
    "unchecked": "256000",
    "cemented": "169568762"
}

EDIT:

Per some Discord discussion, this is probably just the legacy bootstrap implementation - I'll wait for the updated ascending bootstrap client

[dsiganos]

The latest bootstrap cutoff point is: 169643594.

[pwojcikdev]

@qwahzi This PR actually builds upon the improvements introduced in dependent block confirmation. It applies cross chain, so if we happen to confirm account A that has receives from B & C, then exactly as you said, both B & C chains will be confirmed (up to the associated sends). The problem before introduction of this PR was that there was no reliable mechanism to fully utilise those dependent elections.

It's worth noting that the current mechanism is still not working to the fullest potential. The biggest bottleneck I see is the vote requesting, which will be improved by vote generator refactor & vote storage.

[pwojcikdev]

@clemahieu Merge introduced a small bug here, it should pass in 'behavior' parameter instead of fixed enum value.