Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Re-gossip bls changes if submitted earlier than capella #5049

Merged
merged 18 commits into from
Jan 27, 2023
Merged

Re-gossip bls changes if submitted earlier than capella #5049

merged 18 commits into from
Jan 27, 2023

Conversation

g11tech
Copy link
Contributor

@g11tech g11tech commented Jan 24, 2023 •
edited
Loading

replacement for a working version

once tested/confimed will replace the above PR

from devnet 6
image

@github-actions
Copy link
Contributor

github-actions bot commented Jan 24, 2023 •
edited
Loading

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 9d0bded Previous: c7abf8e Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 1.1364 ms/op 1.2627 ms/op 0.90
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 61.647 us/op 70.867 us/op 0.87
BLS verify - blst-native 2.1842 ms/op 1.8592 ms/op 1.17
BLS verifyMultipleSignatures 3 - blst-native 4.4912 ms/op 3.8098 ms/op 1.18
BLS verifyMultipleSignatures 8 - blst-native 9.6965 ms/op 8.1992 ms/op 1.18
BLS verifyMultipleSignatures 32 - blst-native 35.262 ms/op 29.744 ms/op 1.19
BLS aggregatePubkeys 32 - blst-native 46.507 us/op 39.309 us/op 1.18
BLS aggregatePubkeys 128 - blst-native 182.45 us/op 153.74 us/op 1.19
getAttestationsForBlock 82.122 ms/op 91.108 ms/op 0.90
isKnown best case - 1 super set check 448.00 ns/op 450.00 ns/op 1.00
isKnown normal case - 2 super set checks 434.00 ns/op 411.00 ns/op 1.06
isKnown worse case - 16 super set checks 434.00 ns/op 419.00 ns/op 1.04
CheckpointStateCache - add get delete 8.4710 us/op 8.5170 us/op 0.99
validate gossip signedAggregateAndProof - struct 5.0149 ms/op 4.2604 ms/op 1.18
validate gossip attestation - struct 2.3612 ms/op 2.0270 ms/op 1.16
pickEth1Vote - no votes 2.5398 ms/op 2.2513 ms/op 1.13
pickEth1Vote - max votes 17.803 ms/op 17.672 ms/op 1.01
pickEth1Vote - Eth1Data hashTreeRoot value x2048 13.522 ms/op 12.039 ms/op 1.12
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 21.989 ms/op 21.323 ms/op 1.03
pickEth1Vote - Eth1Data fastSerialize value x2048 1.1448 ms/op 1.2998 ms/op 0.88
pickEth1Vote - Eth1Data fastSerialize tree x2048 11.179 ms/op 10.465 ms/op 1.07
bytes32 toHexString 981.00 ns/op 1.1150 us/op 0.88
bytes32 Buffer.toString(hex) 642.00 ns/op 603.00 ns/op 1.06
bytes32 Buffer.toString(hex) from Uint8Array 971.00 ns/op 925.00 ns/op 1.05
bytes32 Buffer.toString(hex) + 0x 661.00 ns/op 605.00 ns/op 1.09
Object access 1 prop 0.30300 ns/op 0.29900 ns/op 1.01
Map access 1 prop 0.30900 ns/op 0.25100 ns/op 1.23
Object get x1000 10.755 ns/op 16.189 ns/op 0.66
Map get x1000 0.94900 ns/op 1.0050 ns/op 0.94
Object set x1000 81.969 ns/op 113.75 ns/op 0.72
Map set x1000 49.448 ns/op 73.088 ns/op 0.68
Return object 10000 times 0.43060 ns/op 0.37540 ns/op 1.15
Throw Error 10000 times 6.7011 us/op 7.1264 us/op 0.94
fastMsgIdFn sha256 / 200 bytes 4.8730 us/op 4.3520 us/op 1.12
fastMsgIdFn h32 xxhash / 200 bytes 551.00 ns/op 522.00 ns/op 1.06
fastMsgIdFn h64 xxhash / 200 bytes 748.00 ns/op 663.00 ns/op 1.13
fastMsgIdFn sha256 / 1000 bytes 15.817 us/op 13.670 us/op 1.16
fastMsgIdFn h32 xxhash / 1000 bytes 718.00 ns/op 648.00 ns/op 1.11
fastMsgIdFn h64 xxhash / 1000 bytes 853.00 ns/op 738.00 ns/op 1.16
fastMsgIdFn sha256 / 10000 bytes 137.24 us/op 117.49 us/op 1.17
fastMsgIdFn h32 xxhash / 10000 bytes 2.6000 us/op 2.3420 us/op 1.11
fastMsgIdFn h64 xxhash / 10000 bytes 1.8220 us/op 1.6940 us/op 1.08
enrSubnets - fastDeserialize 64 bits 2.6850 us/op 2.5290 us/op 1.06
enrSubnets - ssz BitVector 64 bits 994.00 ns/op 854.00 ns/op 1.16
enrSubnets - fastDeserialize 4 bits 323.00 ns/op 311.00 ns/op 1.04
enrSubnets - ssz BitVector 4 bits 1.0020 us/op 850.00 ns/op 1.18
prioritizePeers score -10:0 att 32-0.1 sync 2-0 147.66 us/op 173.90 us/op 0.85
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 183.54 us/op 189.38 us/op 0.97
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 249.84 us/op 316.73 us/op 0.79
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 448.21 us/op 530.60 us/op 0.84
prioritizePeers score 0:0 att 64-1 sync 4-1 497.17 us/op 619.41 us/op 0.80
array of 16000 items push then shift 51.638 us/op 2.8202 us/op 18.31
LinkedList of 16000 items push then shift 12.798 ns/op 17.424 ns/op 0.73
array of 16000 items push then pop 194.59 ns/op 217.02 ns/op 0.90
LinkedList of 16000 items push then pop 12.413 ns/op 16.537 ns/op 0.75
array of 24000 items push then shift 77.404 us/op 3.9786 us/op 19.46
LinkedList of 24000 items push then shift 13.351 ns/op 20.300 ns/op 0.66
array of 24000 items push then pop 197.71 ns/op 220.01 ns/op 0.90
LinkedList of 24000 items push then pop 12.344 ns/op 17.852 ns/op 0.69
intersect bitArray bitLen 8 21.754 ns/op 29.355 ns/op 0.74
intersect array and set length 8 139.37 ns/op 169.00 ns/op 0.82
intersect bitArray bitLen 128 70.540 ns/op 118.67 ns/op 0.59
intersect array and set length 128 1.7526 us/op 2.2998 us/op 0.76
Buffer.concat 32 items 5.1830 us/op 6.3400 us/op 0.82
Uint8Array.set 32 items 4.0460 us/op 4.1180 us/op 0.98
pass gossip attestations to forkchoice per slot 6.1313 ms/op 4.2627 ms/op 1.44
computeDeltas 5.0798 ms/op 6.4168 ms/op 0.79
computeProposerBoostScoreFromBalances 2.5055 ms/op 2.8372 ms/op 0.88
altair processAttestation - 250000 vs - 7PWei normalcase 4.1197 ms/op 4.3175 ms/op 0.95
altair processAttestation - 250000 vs - 7PWei worstcase 5.6750 ms/op 6.5531 ms/op 0.87
altair processAttestation - setStatus - 1/6 committees join 181.07 us/op 211.27 us/op 0.86
altair processAttestation - setStatus - 1/3 committees join 359.86 us/op 409.45 us/op 0.88
altair processAttestation - setStatus - 1/2 committees join 520.38 us/op 568.88 us/op 0.91
altair processAttestation - setStatus - 2/3 committees join 681.50 us/op 728.87 us/op 0.94
altair processAttestation - setStatus - 4/5 committees join 966.54 us/op 1.0092 ms/op 0.96
altair processAttestation - setStatus - 100% committees join 1.1481 ms/op 1.1914 ms/op 0.96
altair processBlock - 250000 vs - 7PWei normalcase 24.168 ms/op 26.554 ms/op 0.91
altair processBlock - 250000 vs - 7PWei normalcase hashState 39.491 ms/op 42.515 ms/op 0.93
altair processBlock - 250000 vs - 7PWei worstcase 78.535 ms/op 81.101 ms/op 0.97
altair processBlock - 250000 vs - 7PWei worstcase hashState 119.45 ms/op 107.92 ms/op 1.11
phase0 processBlock - 250000 vs - 7PWei normalcase 3.3697 ms/op 3.5570 ms/op 0.95
phase0 processBlock - 250000 vs - 7PWei worstcase 51.686 ms/op 46.462 ms/op 1.11
altair processEth1Data - 250000 vs - 7PWei normalcase 731.31 us/op 811.72 us/op 0.90
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 9.6820 us/op 9.4300 us/op 1.03
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 22.985 us/op 28.138 us/op 0.82
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 11.526 us/op 12.377 us/op 0.93
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 8.5460 us/op 9.5760 us/op 0.89
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 89.669 us/op 100.76 us/op 0.89
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 855.88 us/op 998.84 us/op 0.86
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 1.1415 ms/op 1.3408 ms/op 0.85
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 1.1207 ms/op 1.3877 ms/op 0.81
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 4.0549 ms/op 4.3950 ms/op 0.92
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 2.3097 ms/op 3.1524 ms/op 0.73
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 7.2970 ms/op 6.9340 ms/op 1.05
Tree 40 250000 create 756.83 ms/op 658.28 ms/op 1.15
Tree 40 250000 get(125000) 251.52 ns/op 291.14 ns/op 0.86
Tree 40 250000 set(125000) 2.6905 us/op 2.1169 us/op 1.27
Tree 40 250000 toArray() 27.747 ms/op 30.892 ms/op 0.90
Tree 40 250000 iterate all - toArray() + loop 27.915 ms/op 31.018 ms/op 0.90
Tree 40 250000 iterate all - get(i) 104.23 ms/op 105.10 ms/op 0.99
MutableVector 250000 create 15.240 ms/op 17.369 ms/op 0.88
MutableVector 250000 get(125000) 10.793 ns/op 13.227 ns/op 0.82
MutableVector 250000 set(125000) 572.70 ns/op 583.19 ns/op 0.98
MutableVector 250000 toArray() 6.0964 ms/op 6.6166 ms/op 0.92
MutableVector 250000 iterate all - toArray() + loop 6.2990 ms/op 6.7978 ms/op 0.93
MutableVector 250000 iterate all - get(i) 2.7824 ms/op 3.5143 ms/op 0.79
Array 250000 create 6.3209 ms/op 6.4207 ms/op 0.98
Array 250000 clone - spread 2.8116 ms/op 2.5646 ms/op 1.10
Array 250000 get(125000) 1.3550 ns/op 1.1300 ns/op 1.20
Array 250000 set(125000) 1.3190 ns/op 1.1610 ns/op 1.14
Array 250000 iterate all - loop 152.00 us/op 169.98 us/op 0.89
effectiveBalanceIncrements clone Uint8Array 300000 57.024 us/op 78.118 us/op 0.73
effectiveBalanceIncrements clone MutableVector 300000 880.00 ns/op 838.00 ns/op 1.05
effectiveBalanceIncrements rw all Uint8Array 300000 243.95 us/op 252.75 us/op 0.97
effectiveBalanceIncrements rw all MutableVector 300000 180.26 ms/op 197.15 ms/op 0.91
phase0 afterProcessEpoch - 250000 vs - 7PWei 189.62 ms/op 206.55 ms/op 0.92
phase0 beforeProcessEpoch - 250000 vs - 7PWei 56.922 ms/op 67.978 ms/op 0.84
altair processEpoch - mainnet_e81889 585.74 ms/op 568.10 ms/op 1.03
mainnet_e81889 - altair beforeProcessEpoch 124.28 ms/op 143.73 ms/op 0.86
mainnet_e81889 - altair processJustificationAndFinalization 20.622 us/op 29.155 us/op 0.71
mainnet_e81889 - altair processInactivityUpdates 9.5487 ms/op 10.536 ms/op 0.91
mainnet_e81889 - altair processRewardsAndPenalties 78.700 ms/op 90.578 ms/op 0.87
mainnet_e81889 - altair processRegistryUpdates 3.8080 us/op 3.4590 us/op 1.10
mainnet_e81889 - altair processSlashings 815.00 ns/op 671.00 ns/op 1.21
mainnet_e81889 - altair processEth1DataReset 864.00 ns/op 930.00 ns/op 0.93
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.0581 ms/op 2.4998 ms/op 0.82
mainnet_e81889 - altair processSlashingsReset 6.9120 us/op 4.9210 us/op 1.40
mainnet_e81889 - altair processRandaoMixesReset 5.8490 us/op 5.4570 us/op 1.07
mainnet_e81889 - altair processHistoricalRootsUpdate 1.0260 us/op 899.00 ns/op 1.14
mainnet_e81889 - altair processParticipationFlagUpdates 3.1400 us/op 2.8600 us/op 1.10
mainnet_e81889 - altair processSyncCommitteeUpdates 954.00 ns/op 652.00 ns/op 1.46
mainnet_e81889 - altair afterProcessEpoch 217.95 ms/op 194.86 ms/op 1.12
phase0 processEpoch - mainnet_e58758 539.57 ms/op 520.72 ms/op 1.04
mainnet_e58758 - phase0 beforeProcessEpoch 187.86 ms/op 226.47 ms/op 0.83
mainnet_e58758 - phase0 processJustificationAndFinalization 20.613 us/op 23.629 us/op 0.87
mainnet_e58758 - phase0 processRewardsAndPenalties 99.408 ms/op 136.55 ms/op 0.73
mainnet_e58758 - phase0 processRegistryUpdates 10.181 us/op 10.001 us/op 1.02
mainnet_e58758 - phase0 processSlashings 833.00 ns/op 671.00 ns/op 1.24
mainnet_e58758 - phase0 processEth1DataReset 660.00 ns/op 695.00 ns/op 0.95
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.6573 ms/op 1.9393 ms/op 0.85
mainnet_e58758 - phase0 processSlashingsReset 5.6280 us/op 4.7760 us/op 1.18
mainnet_e58758 - phase0 processRandaoMixesReset 6.8920 us/op 7.5460 us/op 0.91
mainnet_e58758 - phase0 processHistoricalRootsUpdate 842.00 ns/op 1.1270 us/op 0.75
mainnet_e58758 - phase0 processParticipationRecordUpdates 5.9350 us/op 5.9790 us/op 0.99
mainnet_e58758 - phase0 afterProcessEpoch 162.00 ms/op 160.30 ms/op 1.01
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.9708 ms/op 2.9482 ms/op 0.67
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.4216 ms/op 3.3714 ms/op 0.72
altair processInactivityUpdates - 250000 normalcase 40.651 ms/op 44.292 ms/op 0.92
altair processInactivityUpdates - 250000 worstcase 40.931 ms/op 43.464 ms/op 0.94
phase0 processRegistryUpdates - 250000 normalcase 9.3610 us/op 12.547 us/op 0.75
phase0 processRegistryUpdates - 250000 badcase_full_deposits 376.12 us/op 532.61 us/op 0.71
phase0 processRegistryUpdates - 250000 worstcase 0.5 186.43 ms/op 206.37 ms/op 0.90
altair processRewardsAndPenalties - 250000 normalcase 76.582 ms/op 119.25 ms/op 0.64
altair processRewardsAndPenalties - 250000 worstcase 85.672 ms/op 97.319 ms/op 0.88
phase0 getAttestationDeltas - 250000 normalcase 12.981 ms/op 12.644 ms/op 1.03
phase0 getAttestationDeltas - 250000 worstcase 13.126 ms/op 12.945 ms/op 1.01
phase0 processSlashings - 250000 worstcase 5.2989 ms/op 5.6883 ms/op 0.93
altair processSyncCommitteeUpdates - 250000 291.44 ms/op 269.26 ms/op 1.08
BeaconState.hashTreeRoot - No change 478.00 ns/op 416.00 ns/op 1.15
BeaconState.hashTreeRoot - 1 full validator 69.889 us/op 63.371 us/op 1.10
BeaconState.hashTreeRoot - 32 full validator 650.15 us/op 722.63 us/op 0.90
BeaconState.hashTreeRoot - 512 full validator 8.9214 ms/op 6.0390 ms/op 1.48
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 87.268 us/op 76.496 us/op 1.14
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.2917 ms/op 1.1150 ms/op 1.16
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 17.128 ms/op 15.356 ms/op 1.12
BeaconState.hashTreeRoot - 1 balances 65.656 us/op 62.174 us/op 1.06
BeaconState.hashTreeRoot - 32 balances 632.85 us/op 571.62 us/op 1.11
BeaconState.hashTreeRoot - 512 balances 6.4700 ms/op 5.5818 ms/op 1.16
BeaconState.hashTreeRoot - 250000 balances 93.416 ms/op 88.960 ms/op 1.05
aggregationBits - 2048 els - zipIndexesInBitList 34.173 us/op 29.385 us/op 1.16
regular array get 100000 times 60.588 us/op 67.293 us/op 0.90
wrappedArray get 100000 times 60.590 us/op 67.345 us/op 0.90
arrayWithProxy get 100000 times 26.739 ms/op 27.897 ms/op 0.96
ssz.Root.equals 1.0120 us/op 1.0910 us/op 0.93
byteArrayEquals 1.0340 us/op 1.0830 us/op 0.95
shuffle list - 16384 els 11.426 ms/op 11.205 ms/op 1.02
shuffle list - 250000 els 168.35 ms/op 166.32 ms/op 1.01
processSlot - 1 slots 13.172 us/op 11.725 us/op 1.12
processSlot - 32 slots 1.9454 ms/op 1.7003 ms/op 1.14
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 321.58 us/op 335.68 us/op 0.96
getCommitteeAssignments - req 1 vs - 250000 vc 5.4671 ms/op 5.6014 ms/op 0.98
getCommitteeAssignments - req 100 vs - 250000 vc 8.0006 ms/op 7.5184 ms/op 1.06
getCommitteeAssignments - req 1000 vs - 250000 vc 8.5522 ms/op 8.0820 ms/op 1.06
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 8.9300 ns/op 9.7100 ns/op 0.92
state getBlockRootAtSlot - 250000 vs - 7PWei 1.3559 us/op 1.2156 us/op 1.12
computeProposers - vc 250000 17.499 ms/op 16.015 ms/op 1.09
computeEpochShuffling - vc 250000 171.90 ms/op 170.19 ms/op 1.01
getNextSyncCommittee - vc 250000 290.83 ms/op 268.78 ms/op 1.08

by benchmarkbot/action

@g11tech g11tech marked this pull request as ready for review January 26, 2023 07:39
@g11tech g11tech requested a review from a team as a code owner January 26, 2023 07:39
@g11tech g11tech mentioned this pull request Jan 26, 2023
Closed
dapplion
dapplion approved these changes Jan 27, 2023
View reviewed changes
Copy link
Contributor

@dapplion dapplion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@g11tech g11tech merged commit aaf134a into unstable Jan 27, 2023
@g11tech g11tech deleted the g11tech/re-transmit-blschanges-new branch January 27, 2023 15:43
wemeetagain pushed a commit that referenced this pull request Feb 27, 2023
@g11tech @wemeetagain
Re-gossip bls changes if submitted earlier than capella (#5049)
4bfa8b6 
* Cache and retransmit bls changes if submitted early

* regossip when synced on/post capella

* fix tests

* modify publish vs cache condition

* add logging

* add error logging

* batchify the processing

* fix some of the cases

* fix log issues

* also delete included keys

* fix lint

* simplify gossip loop

* fix build

* refac to a versioned bls change store

* update

* logging improvements

* improve logging and finish behavior

* cleanup
dapplion added a commit that referenced this pull request Feb 28, 2023
@dapplion @wemeetagain
@g11tech @nflaig
Merge stable to unstable no conflicts (#5216)
d4b9201 
* v1.5.0

* Add validatior option to specify builder block selection strategy (#5064)

* Add validatior option to specify builder block selection strategy

* cleanup and improvements

* server api fx

* fix test

* fix the testcase

* Re-gossip bls changes if submitted earlier than capella (#5049)

* Cache and retransmit bls changes if submitted early

* regossip when synced on/post capella

* fix tests

* modify publish vs cache condition

* add logging

* add error logging

* batchify the processing

* fix some of the cases

* fix log issues

* also delete included keys

* fix lint

* simplify gossip loop

* fix build

* refac to a versioned bls change store

* update

* logging improvements

* improve logging and finish behavior

* cleanup

* Rename eip4844 fork as Deneb (#5082)

* Fix and extract payload to header conversion (#5101)

* Remove unused eslint disable directives (#5046)

* Add execution payload header to light client header capella onwards (#5027)

* Add execution payload header to light client header capella onwards

* add more forks

* add allfork types

* make other types multifork

* make events multifork compatible

* make lightclient routes multifork

* update light-client transport

* upgrade the blocktype on lightclient server import block

* fixes

* fix test

* fix event serialization/deserilization type

* update proof constants

* fix payload index

* update the light client update generation to multi fork

* fix types

* add spec override to pass specs

* make event types allforks

* make req/resp multifork

* make types multifork throughout

* fix the update serialize/deserialize

* fix error log

* better fork determination for update

* fix the fork of finalized

* fix tests

* fix header upgradation

* Fix zero finalized header in lightclient update gen (#5121)

* Schedule capella on sepolia (#5127)

* Get the capella builder functional (#5179)

* Allow blsChange reprocess/republish via API submission (#5185)

* Fix gossiping blsChange on submit if post capella

* comment improvement

* Use new db bucket for light client updates (#5205)

---------

Co-authored-by: Cayman <caymannava@gmail.com>
Co-authored-by: g11tech <develop@g11tech.io>
Co-authored-by: Nico Flaig <nflaig@protonmail.com>
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.6.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dapplion dapplion dapplion approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@g11tech @wemeetagain @dapplion