Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache and retransmit bls changes if submitted early #5031

Closed
wants to merge 13 commits into from

Conversation

g11tech
Copy link
Contributor

@g11tech g11tech commented Jan 21, 2023

A user (most likely ) is expected to start submitting bls to execution changes earlier than capella hf . This would result in gossip error as the topic is not yet been subscribed by peers.

This PR would cache the bls changes and e re-gossip if there is any gossip publishing error.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 21, 2023

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 6166395 Previous: b84c234 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 896.56 us/op 1.3667 ms/op 0.66
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 58.174 us/op 151.39 us/op 0.38
BLS verify - blst-native 2.1693 ms/op 2.4930 ms/op 0.87
BLS verifyMultipleSignatures 3 - blst-native 4.4869 ms/op 5.0148 ms/op 0.89
BLS verifyMultipleSignatures 8 - blst-native 9.7016 ms/op 10.402 ms/op 0.93
BLS verifyMultipleSignatures 32 - blst-native 35.257 ms/op 38.946 ms/op 0.91
BLS aggregatePubkeys 32 - blst-native 46.338 us/op 52.320 us/op 0.89
BLS aggregatePubkeys 128 - blst-native 182.75 us/op 201.02 us/op 0.91
getAttestationsForBlock 76.036 ms/op 121.79 ms/op 0.62
isKnown best case - 1 super set check 434.00 ns/op 542.00 ns/op 0.80
isKnown normal case - 2 super set checks 425.00 ns/op 531.00 ns/op 0.80
isKnown worse case - 16 super set checks 422.00 ns/op 539.00 ns/op 0.78
CheckpointStateCache - add get delete 7.9900 us/op 10.869 us/op 0.74
validate gossip signedAggregateAndProof - struct 5.0192 ms/op 5.5591 ms/op 0.90
validate gossip attestation - struct 2.3558 ms/op 2.5304 ms/op 0.93
pickEth1Vote - no votes 2.1828 ms/op 2.8838 ms/op 0.76
pickEth1Vote - max votes 15.594 ms/op 22.851 ms/op 0.68
pickEth1Vote - Eth1Data hashTreeRoot value x2048 13.027 ms/op 16.473 ms/op 0.79
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 19.669 ms/op 28.980 ms/op 0.68
pickEth1Vote - Eth1Data fastSerialize value x2048 1.0988 ms/op 1.6399 ms/op 0.67
pickEth1Vote - Eth1Data fastSerialize tree x2048 9.1618 ms/op 13.499 ms/op 0.68
bytes32 toHexString 864.00 ns/op 1.3450 us/op 0.64
bytes32 Buffer.toString(hex) 630.00 ns/op 799.00 ns/op 0.79
bytes32 Buffer.toString(hex) from Uint8Array 965.00 ns/op 1.2060 us/op 0.80
bytes32 Buffer.toString(hex) + 0x 640.00 ns/op 797.00 ns/op 0.80
Object access 1 prop 0.28600 ns/op 0.39700 ns/op 0.72
Map access 1 prop 0.28000 ns/op 0.33900 ns/op 0.83
Object get x1000 10.757 ns/op 17.507 ns/op 0.61
Map get x1000 0.94300 ns/op 1.0790 ns/op 0.87
Object set x1000 65.730 ns/op 124.88 ns/op 0.53
Map set x1000 42.801 ns/op 87.145 ns/op 0.49
Return object 10000 times 0.43870 ns/op 0.45310 ns/op 0.97
Throw Error 10000 times 6.5179 us/op 8.4388 us/op 0.77
fastMsgIdFn sha256 / 200 bytes 4.9070 us/op 5.6200 us/op 0.87
fastMsgIdFn h32 xxhash / 200 bytes 529.00 ns/op 623.00 ns/op 0.85
fastMsgIdFn h64 xxhash / 200 bytes 671.00 ns/op 837.00 ns/op 0.80
fastMsgIdFn sha256 / 1000 bytes 15.633 us/op 17.725 us/op 0.88
fastMsgIdFn h32 xxhash / 1000 bytes 699.00 ns/op 829.00 ns/op 0.84
fastMsgIdFn h64 xxhash / 1000 bytes 791.00 ns/op 943.00 ns/op 0.84
fastMsgIdFn sha256 / 10000 bytes 136.95 us/op 157.09 us/op 0.87
fastMsgIdFn h32 xxhash / 10000 bytes 2.6170 us/op 2.8970 us/op 0.90
fastMsgIdFn h64 xxhash / 10000 bytes 1.8050 us/op 2.1550 us/op 0.84
enrSubnets - fastDeserialize 64 bits 2.2740 us/op 3.2380 us/op 0.70
enrSubnets - ssz BitVector 64 bits 900.00 ns/op 1.1430 us/op 0.79
enrSubnets - fastDeserialize 4 bits 323.00 ns/op 399.00 ns/op 0.81
enrSubnets - ssz BitVector 4 bits 904.00 ns/op 1.1750 us/op 0.77
prioritizePeers score -10:0 att 32-0.1 sync 2-0 149.02 us/op 207.72 us/op 0.72
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 175.38 us/op 259.39 us/op 0.68
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 250.10 us/op 392.91 us/op 0.64
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 415.94 us/op 656.15 us/op 0.63
prioritizePeers score 0:0 att 64-1 sync 4-1 471.98 us/op 750.78 us/op 0.63
array of 16000 items push then shift 51.607 us/op 5.8750 us/op 8.78
LinkedList of 16000 items push then shift 12.120 ns/op 22.689 ns/op 0.53
array of 16000 items push then pop 177.46 ns/op 267.86 ns/op 0.66
LinkedList of 16000 items push then pop 11.959 ns/op 19.800 ns/op 0.60
array of 24000 items push then shift 77.362 us/op 8.3441 us/op 9.27
LinkedList of 24000 items push then shift 12.260 ns/op 21.688 ns/op 0.57
array of 24000 items push then pop 187.87 ns/op 271.59 ns/op 0.69
LinkedList of 24000 items push then pop 12.017 ns/op 20.547 ns/op 0.58
intersect bitArray bitLen 8 21.727 ns/op 30.517 ns/op 0.71
intersect array and set length 8 118.30 ns/op 214.51 ns/op 0.55
intersect bitArray bitLen 128 70.540 ns/op 98.199 ns/op 0.72
intersect array and set length 128 1.6047 us/op 2.5106 us/op 0.64
Buffer.concat 32 items 5.1440 us/op 6.2760 us/op 0.82
Uint8Array.set 32 items 3.6710 us/op 5.3340 us/op 0.69
pass gossip attestations to forkchoice per slot 5.5501 ms/op 8.2954 ms/op 0.67
computeDeltas 4.2307 ms/op 6.4436 ms/op 0.66
computeProposerBoostScoreFromBalances 2.5064 ms/op 2.9605 ms/op 0.85
altair processAttestation - 250000 vs - 7PWei normalcase 3.2911 ms/op 5.8098 ms/op 0.57
altair processAttestation - 250000 vs - 7PWei worstcase 5.1187 ms/op 8.8258 ms/op 0.58
altair processAttestation - setStatus - 1/6 committees join 173.20 us/op 270.80 us/op 0.64
altair processAttestation - setStatus - 1/3 committees join 348.45 us/op 515.24 us/op 0.68
altair processAttestation - setStatus - 1/2 committees join 514.98 us/op 739.43 us/op 0.70
altair processAttestation - setStatus - 2/3 committees join 661.01 us/op 956.55 us/op 0.69
altair processAttestation - setStatus - 4/5 committees join 930.14 us/op 1.3458 ms/op 0.69
altair processAttestation - setStatus - 100% committees join 1.1279 ms/op 1.6552 ms/op 0.68
altair processBlock - 250000 vs - 7PWei normalcase 23.077 ms/op 30.969 ms/op 0.75
altair processBlock - 250000 vs - 7PWei normalcase hashState 35.080 ms/op 45.633 ms/op 0.77
altair processBlock - 250000 vs - 7PWei worstcase 71.959 ms/op 114.41 ms/op 0.63
altair processBlock - 250000 vs - 7PWei worstcase hashState 98.933 ms/op 130.00 ms/op 0.76
phase0 processBlock - 250000 vs - 7PWei normalcase 3.0739 ms/op 5.1863 ms/op 0.59
phase0 processBlock - 250000 vs - 7PWei worstcase 50.107 ms/op 62.138 ms/op 0.81
altair processEth1Data - 250000 vs - 7PWei normalcase 629.20 us/op 1.2163 ms/op 0.52
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 21.630 us/op 25.103 us/op 0.86
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 45.687 us/op 49.507 us/op 0.92
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 11.345 us/op 30.459 us/op 0.37
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 19.202 us/op 24.463 us/op 0.78
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 168.86 us/op 150.19 us/op 1.12
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 1.0061 ms/op 1.3579 ms/op 0.74
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 1.2354 ms/op 1.6999 ms/op 0.73
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 1.3134 ms/op 1.7271 ms/op 0.76
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 4.5254 ms/op 5.9391 ms/op 0.76
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 2.4746 ms/op 3.5727 ms/op 0.69
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 7.3161 ms/op 9.6367 ms/op 0.76
Tree 40 250000 create 585.55 ms/op 1.1504 s/op 0.51
Tree 40 250000 get(125000) 273.14 ns/op 341.92 ns/op 0.80
Tree 40 250000 set(125000) 1.7522 us/op 3.5213 us/op 0.50
Tree 40 250000 toArray() 24.824 ms/op 40.359 ms/op 0.62
Tree 40 250000 iterate all - toArray() + loop 25.451 ms/op 39.242 ms/op 0.65
Tree 40 250000 iterate all - get(i) 102.48 ms/op 137.86 ms/op 0.74
MutableVector 250000 create 11.768 ms/op 18.528 ms/op 0.64
MutableVector 250000 get(125000) 10.571 ns/op 15.430 ns/op 0.69
MutableVector 250000 set(125000) 480.83 ns/op 905.08 ns/op 0.53
MutableVector 250000 toArray() 5.5826 ms/op 8.5046 ms/op 0.66
MutableVector 250000 iterate all - toArray() + loop 5.6675 ms/op 8.1986 ms/op 0.69
MutableVector 250000 iterate all - get(i) 3.4631 ms/op 4.2861 ms/op 0.81
Array 250000 create 5.2263 ms/op 8.0145 ms/op 0.65
Array 250000 clone - spread 2.3575 ms/op 3.5581 ms/op 0.66
Array 250000 get(125000) 1.1360 ns/op 1.5660 ns/op 0.73
Array 250000 set(125000) 1.1300 ns/op 1.5020 ns/op 0.75
Array 250000 iterate all - loop 152.84 us/op 153.32 us/op 1.00
effectiveBalanceIncrements clone Uint8Array 300000 38.242 us/op 116.44 us/op 0.33
effectiveBalanceIncrements clone MutableVector 300000 727.00 ns/op 1.0550 us/op 0.69
effectiveBalanceIncrements rw all Uint8Array 300000 245.54 us/op 307.17 us/op 0.80
effectiveBalanceIncrements rw all MutableVector 300000 146.76 ms/op 278.28 ms/op 0.53
phase0 afterProcessEpoch - 250000 vs - 7PWei 205.09 ms/op 220.77 ms/op 0.93
phase0 beforeProcessEpoch - 250000 vs - 7PWei 55.170 ms/op 144.43 ms/op 0.38
altair processEpoch - mainnet_e81889 549.65 ms/op 702.68 ms/op 0.78
mainnet_e81889 - altair beforeProcessEpoch 74.538 ms/op 175.46 ms/op 0.42
mainnet_e81889 - altair processJustificationAndFinalization 33.243 us/op 73.841 us/op 0.45
mainnet_e81889 - altair processInactivityUpdates 8.6145 ms/op 13.157 ms/op 0.65
mainnet_e81889 - altair processRewardsAndPenalties 128.78 ms/op 190.89 ms/op 0.67
mainnet_e81889 - altair processRegistryUpdates 3.3620 us/op 14.898 us/op 0.23
mainnet_e81889 - altair processSlashings 824.00 ns/op 3.3600 us/op 0.25
mainnet_e81889 - altair processEth1DataReset 791.00 ns/op 3.8830 us/op 0.20
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.0339 ms/op 3.1905 ms/op 0.64
mainnet_e81889 - altair processSlashingsReset 5.5810 us/op 22.535 us/op 0.25
mainnet_e81889 - altair processRandaoMixesReset 7.4700 us/op 28.069 us/op 0.27
mainnet_e81889 - altair processHistoricalRootsUpdate 748.00 ns/op 4.2810 us/op 0.17
mainnet_e81889 - altair processParticipationFlagUpdates 2.7260 us/op 18.270 us/op 0.15
mainnet_e81889 - altair processSyncCommitteeUpdates 899.00 ns/op 3.7630 us/op 0.24
mainnet_e81889 - altair afterProcessEpoch 195.27 ms/op 229.01 ms/op 0.85
phase0 processEpoch - mainnet_e58758 473.23 ms/op 753.78 ms/op 0.63
mainnet_e58758 - phase0 beforeProcessEpoch 172.05 ms/op 345.84 ms/op 0.50
mainnet_e58758 - phase0 processJustificationAndFinalization 23.688 us/op 70.298 us/op 0.34
mainnet_e58758 - phase0 processRewardsAndPenalties 68.144 ms/op 146.98 ms/op 0.46
mainnet_e58758 - phase0 processRegistryUpdates 16.282 us/op 37.982 us/op 0.43
mainnet_e58758 - phase0 processSlashings 1.1550 us/op 3.4020 us/op 0.34
mainnet_e58758 - phase0 processEth1DataReset 898.00 ns/op 3.9380 us/op 0.23
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.7577 ms/op 2.5562 ms/op 0.69
mainnet_e58758 - phase0 processSlashingsReset 5.1730 us/op 19.258 us/op 0.27
mainnet_e58758 - phase0 processRandaoMixesReset 7.9660 us/op 27.889 us/op 0.29
mainnet_e58758 - phase0 processHistoricalRootsUpdate 1.0770 us/op 3.8970 us/op 0.28
mainnet_e58758 - phase0 processParticipationRecordUpdates 5.7610 us/op 26.900 us/op 0.21
mainnet_e58758 - phase0 afterProcessEpoch 162.44 ms/op 188.10 ms/op 0.86
phase0 processEffectiveBalanceUpdates - 250000 normalcase 2.1045 ms/op 2.5426 ms/op 0.83
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.3995 ms/op 2.8978 ms/op 0.83
altair processInactivityUpdates - 250000 normalcase 31.712 ms/op 52.646 ms/op 0.60
altair processInactivityUpdates - 250000 worstcase 39.579 ms/op 67.631 ms/op 0.59
phase0 processRegistryUpdates - 250000 normalcase 10.220 us/op 30.054 us/op 0.34
phase0 processRegistryUpdates - 250000 badcase_full_deposits 543.07 us/op 678.30 us/op 0.80
phase0 processRegistryUpdates - 250000 worstcase 0.5 172.26 ms/op 273.89 ms/op 0.63
altair processRewardsAndPenalties - 250000 normalcase 96.188 ms/op 166.40 ms/op 0.58
altair processRewardsAndPenalties - 250000 worstcase 93.142 ms/op 112.14 ms/op 0.83
phase0 getAttestationDeltas - 250000 normalcase 11.182 ms/op 15.766 ms/op 0.71
phase0 getAttestationDeltas - 250000 worstcase 11.287 ms/op 15.863 ms/op 0.71
phase0 processSlashings - 250000 worstcase 5.5813 ms/op 7.4301 ms/op 0.75
altair processSyncCommitteeUpdates - 250000 287.30 ms/op 362.25 ms/op 0.79
BeaconState.hashTreeRoot - No change 482.00 ns/op 594.00 ns/op 0.81
BeaconState.hashTreeRoot - 1 full validator 66.192 us/op 77.712 us/op 0.85
BeaconState.hashTreeRoot - 32 full validator 706.21 us/op 766.14 us/op 0.92
BeaconState.hashTreeRoot - 512 full validator 7.3852 ms/op 8.4109 ms/op 0.88
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 89.643 us/op 103.35 us/op 0.87
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.3009 ms/op 1.5390 ms/op 0.85
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 16.916 ms/op 18.953 ms/op 0.89
BeaconState.hashTreeRoot - 1 balances 68.123 us/op 76.239 us/op 0.89
BeaconState.hashTreeRoot - 32 balances 644.82 us/op 704.79 us/op 0.91
BeaconState.hashTreeRoot - 512 balances 6.8603 ms/op 6.1056 ms/op 1.12
BeaconState.hashTreeRoot - 250000 balances 98.766 ms/op 118.91 ms/op 0.83
aggregationBits - 2048 els - zipIndexesInBitList 28.996 us/op 39.560 us/op 0.73
regular array get 100000 times 60.657 us/op 62.286 us/op 0.97
wrappedArray get 100000 times 60.562 us/op 63.545 us/op 0.95
arrayWithProxy get 100000 times 29.032 ms/op 36.468 ms/op 0.80
ssz.Root.equals 981.00 ns/op 1.2430 us/op 0.79
byteArrayEquals 969.00 ns/op 1.2400 us/op 0.78
shuffle list - 16384 els 11.512 ms/op 13.203 ms/op 0.87
shuffle list - 250000 els 167.73 ms/op 186.23 ms/op 0.90
processSlot - 1 slots 14.081 us/op 19.931 us/op 0.71
processSlot - 32 slots 2.0006 ms/op 2.5032 ms/op 0.80
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 331.52 us/op 434.02 us/op 0.76
getCommitteeAssignments - req 1 vs - 250000 vc 5.4192 ms/op 6.1448 ms/op 0.88
getCommitteeAssignments - req 100 vs - 250000 vc 7.9568 ms/op 8.1651 ms/op 0.97
getCommitteeAssignments - req 1000 vs - 250000 vc 8.4879 ms/op 9.2701 ms/op 0.92
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 7.7300 ns/op 10.780 ns/op 0.72
state getBlockRootAtSlot - 250000 vs - 7PWei 1.2201 us/op 1.6839 us/op 0.72
computeProposers - vc 250000 17.251 ms/op 21.588 ms/op 0.80
computeEpochShuffling - vc 250000 170.07 ms/op 191.52 ms/op 0.89
getNextSyncCommittee - vc 250000 287.70 ms/op 349.57 ms/op 0.82

by benchmarkbot/action

@@ -33,6 +33,7 @@ export enum Bucket {
phase0_proposerSlashing = 14, // ValidatorIndex -> ProposerSlashing
phase0_attesterSlashing = 15, // Root -> AttesterSlashing
capella_blsToExecutionChange = 16, // ValidatorIndex -> SignedBLSToExecutionChange
capella_blsToExecutionChangeCache = 17, // ValidatorIndex -> SignedBLSToExecutionChange
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should be able to recycle the existing in-memory + DB pool. Why is this extra bucket required?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

umm i guess the current db pool before capella can be used as to recycle, i was going for caching any generic publishing errors but i guess its over kill

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the problem could be, once capella is hit, it would become unclear which ones are received from gossip and which ones have been submitted, as there is always a possibility of restarts while the gossip job is not complete, so would like to retain this as a separate repo

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can clean it post capella

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does not matter where an object is received. You persist to db changes that are still valid to be included in a block

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the problem could be, once capella is hit, it would become unclear which ones are received from gossip and which ones have been submitted

That's a fair point. However, it's not a consensus issue to re-broadcast other validator messages. After thinking about it I'm not opposed to mark those objects in some way as "submitted pre-capella"

if (chain.clock.currentEpoch >= chain.config.CAPELLA_FORK_EPOCH) {
await network.gossip.publishBlsToExecutionChange(blsToExecutionChange);
} else {
await chain.cacheBlsToExecutionChanges(blsToExecutionChange);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it necessary to explicitly call this? chain.opPool.insertBlsToExecutionChange should eventually write to DB on shutdown

this.cachedBlsChangesPromise = this.gossipCachedBlsChanges().then(() => {
this.cachedBlsChangesPromise = null;
});
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just move this code above under the // On fork transition comment, with some retry in case there are no peers

@g11tech
Copy link
Contributor Author

g11tech commented Jan 26, 2023

closing in favor of #5049

@g11tech g11tech closed this Jan 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants