Abort query if not enough stake is connected

[yacovm]

This pull request makes the snowman engine to abort early and avoid sending a query if the node considers the connected stake as insufficient to achieve a successful poll.

The rational for this, is to prevent the node to try to agree on blocks when it is in a partition.

The intent here is twofold: (1) To prevent needless resource expenditure, and (2) to prevent snowball from increasing its preference strength while the protocol has no way to make progress.

The higher preference strength in a partition, the longer it takes to recover and achieve consensus once the partition heals.

Why this should be merged

Without this change, nodes in a partition consistently try to poll each other and depending on the configuration, it may be that all nodes in a partition manage to increase the preference strength over time in snowball (but fail to finalize blocks, because they cannot consistently achieve the required confidence thresholds).
Once the nodes have increased the preference strength of the next block they prefer, even if the partition heals, it will take a long time for the nodes to change their preference, and the consensus will stall for a long time or until a restart of the nodes.

With this change, the nodes will avoid sending polls once they detect they are in a partition, and thus the preference strength of snowball won't increase needlessly.

How this works

Before sending a query, a node checks if it senses enough stake is reachable, and aborts early if it is not the case.

How this was tested

I ran a Fuji node with verbose logging level, and blocked its communication to 33% of the stake via iptables rules.
Then, without this change, the node still tries to query other nodes.
However, with this change, the node aborts and does not query anymore:

Instead, it prints that it failed fast and aborted the query:

I also did a more comprehensive test:

I deployed a 4 node subnet on Fuji (testnet) and then setup a network partition where two nodes are on one side and the other two nodes are on the other.
Then I submitted transactions to some nodes on each side of the partition, and monitored a custom metric I added which measures the preference strength of snowball. Afterwards, I let the partition heal and attempted to submit more transactions to the network.

I repeated the experiment above for both a build from master and a build based on this fix.

In the build based on master, i observed the snowball preference strength go wild, and the network was not functional after the partition healed:

In the build based on this fix, the metric didn't change as the queries are not being sent in the first place.
After the partition healed, conflicting blocks were rejected as consensus was re-established: