feat: add historical state regen #6033

wemeetagain · 2023-10-11T16:57:55Z

Motivation

see Not able to download finalized state #5846

Description

Add a historical state regen module which can regenerate old states
Optional to avoid breaking unrelated tests / mocks
Uses a dedicated worker thread with a JobFnQueue inside
nodejs, state transition, job queue metrics wired up
Add historical state dashboard, which includes VM, job queue, and stfn panels
Update VM dashboard to include historical state worker panels and update cross-worker panels
PubkeyIndexMap persisted between runs for faster repeated use
Deployed to feat3 on Jan 22

TODO

pull out any necessary configurability to CLI level - no CLI configurability is needed (?)
tests
dashboard

github-actions · 2023-10-11T17:41:21Z

Performance Report

✔️ no performance regression detected

Full benchmark results

Benchmark suite	Current: `bb8fa49`	Previous: `59f72d0`	Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc	2.0047 ms/op	2.8144 ms/op	0.71
getPubkeys - validatorsArr - req 1000 vs - 250000 vc	44.379 us/op	58.733 us/op	0.76
BLS verify - blst	1.4394 ms/op	1.0293 ms/op	1.40
BLS verifyMultipleSignatures 3 - blst	2.2384 ms/op	1.5085 ms/op	1.48
BLS verifyMultipleSignatures 8 - blst	2.4675 ms/op	2.2218 ms/op	1.11
BLS verifyMultipleSignatures 32 - blst	7.4301 ms/op	7.0741 ms/op	1.05
BLS verifyMultipleSignatures 64 - blst	10.596 ms/op	12.137 ms/op	0.87
BLS verifyMultipleSignatures 128 - blst	16.714 ms/op	20.810 ms/op	0.80
BLS deserializing 10000 signatures	682.46 ms/op	739.80 ms/op	0.92
BLS deserializing 100000 signatures	6.8180 s/op	7.6259 s/op	0.89
BLS verifyMultipleSignatures - same message - 3 - blst	1.2572 ms/op	1.0784 ms/op	1.17
BLS verifyMultipleSignatures - same message - 8 - blst	1.1432 ms/op	1.2278 ms/op	0.93
BLS verifyMultipleSignatures - same message - 32 - blst	1.6899 ms/op	1.7701 ms/op	0.95
BLS verifyMultipleSignatures - same message - 64 - blst	2.6680 ms/op	2.6861 ms/op	0.99
BLS verifyMultipleSignatures - same message - 128 - blst	4.4026 ms/op	4.3882 ms/op	1.00
BLS aggregatePubkeys 32 - blst	19.475 us/op	21.434 us/op	0.91
BLS aggregatePubkeys 128 - blst	68.147 us/op	76.979 us/op	0.89
notSeenSlots=1 numMissedVotes=1 numBadVotes=10	75.714 ms/op	101.51 ms/op	0.75
notSeenSlots=1 numMissedVotes=0 numBadVotes=4	68.768 ms/op	67.655 ms/op	1.02
notSeenSlots=2 numMissedVotes=1 numBadVotes=10	37.164 ms/op	40.916 ms/op	0.91
getSlashingsAndExits - default max	95.665 us/op	166.98 us/op	0.57
getSlashingsAndExits - 2k	397.85 us/op	496.12 us/op	0.80
proposeBlockBody type=full, size=empty	5.8558 ms/op	6.7414 ms/op	0.87
isKnown best case - 1 super set check	304.00 ns/op	500.00 ns/op	0.61
isKnown normal case - 2 super set checks	288.00 ns/op	535.00 ns/op	0.54
isKnown worse case - 16 super set checks	304.00 ns/op	442.00 ns/op	0.69
InMemoryCheckpointStateCache - add get delete	5.4830 us/op	6.7200 us/op	0.82
validate api signedAggregateAndProof - struct	1.8052 ms/op	1.7594 ms/op	1.03
validate gossip signedAggregateAndProof - struct	1.9093 ms/op	1.6286 ms/op	1.17
validate gossip attestation - vc 640000	1.0628 ms/op	1.1567 ms/op	0.92
batch validate gossip attestation - vc 640000 - chunk 32	129.02 us/op	141.22 us/op	0.91
batch validate gossip attestation - vc 640000 - chunk 64	111.53 us/op	125.17 us/op	0.89
batch validate gossip attestation - vc 640000 - chunk 128	109.03 us/op	114.95 us/op	0.95
batch validate gossip attestation - vc 640000 - chunk 256	108.69 us/op	117.75 us/op	0.92
pickEth1Vote - no votes	1.0222 ms/op	1.1676 ms/op	0.88
pickEth1Vote - max votes	7.6123 ms/op	12.274 ms/op	0.62
pickEth1Vote - Eth1Data hashTreeRoot value x2048	11.643 ms/op	17.013 ms/op	0.68
pickEth1Vote - Eth1Data hashTreeRoot tree x2048	19.004 ms/op	29.311 ms/op	0.65
pickEth1Vote - Eth1Data fastSerialize value x2048	467.84 us/op	558.93 us/op	0.84
pickEth1Vote - Eth1Data fastSerialize tree x2048	3.6400 ms/op	4.5172 ms/op	0.81
bytes32 toHexString	430.00 ns/op	483.00 ns/op	0.89
bytes32 Buffer.toString(hex)	240.00 ns/op	256.00 ns/op	0.94
bytes32 Buffer.toString(hex) from Uint8Array	360.00 ns/op	384.00 ns/op	0.94
bytes32 Buffer.toString(hex) + 0x	251.00 ns/op	278.00 ns/op	0.90
Object access 1 prop	0.13700 ns/op	0.14400 ns/op	0.95
Map access 1 prop	0.13100 ns/op	0.13400 ns/op	0.98
Object get x1000	5.9030 ns/op	6.0310 ns/op	0.98
Map get x1000	6.3930 ns/op	6.7980 ns/op	0.94
Object set x1000	33.167 ns/op	77.701 ns/op	0.43
Map set x1000	24.307 ns/op	51.335 ns/op	0.47
Return object 10000 times	0.28960 ns/op	0.32530 ns/op	0.89
Throw Error 10000 times	3.3706 us/op	4.0236 us/op	0.84
fastMsgIdFn sha256 / 200 bytes	2.1630 us/op	2.5380 us/op	0.85
fastMsgIdFn h32 xxhash / 200 bytes	236.00 ns/op	331.00 ns/op	0.71
fastMsgIdFn h64 xxhash / 200 bytes	270.00 ns/op	312.00 ns/op	0.87
fastMsgIdFn sha256 / 1000 bytes	7.1470 us/op	8.2050 us/op	0.87
fastMsgIdFn h32 xxhash / 1000 bytes	365.00 ns/op	423.00 ns/op	0.86
fastMsgIdFn h64 xxhash / 1000 bytes	341.00 ns/op	369.00 ns/op	0.92
fastMsgIdFn sha256 / 10000 bytes	63.363 us/op	68.185 us/op	0.93
fastMsgIdFn h32 xxhash / 10000 bytes	1.8070 us/op	1.9830 us/op	0.91
fastMsgIdFn h64 xxhash / 10000 bytes	1.1940 us/op	1.3210 us/op	0.90
send data - 1000 256B messages	12.196 ms/op	14.074 ms/op	0.87
send data - 1000 512B messages	16.340 ms/op	18.766 ms/op	0.87
send data - 1000 1024B messages	26.685 ms/op	27.920 ms/op	0.96
send data - 1000 1200B messages	26.413 ms/op	33.507 ms/op	0.79
send data - 1000 2048B messages	30.980 ms/op	36.791 ms/op	0.84
send data - 1000 4096B messages	31.351 ms/op	37.187 ms/op	0.84
send data - 1000 16384B messages	70.633 ms/op	73.300 ms/op	0.96
send data - 1000 65536B messages	216.99 ms/op	244.60 ms/op	0.89
enrSubnets - fastDeserialize 64 bits	1.0370 us/op	1.3780 us/op	0.75
enrSubnets - ssz BitVector 64 bits	349.00 ns/op	466.00 ns/op	0.75
enrSubnets - fastDeserialize 4 bits	152.00 ns/op	186.00 ns/op	0.82
enrSubnets - ssz BitVector 4 bits	343.00 ns/op	411.00 ns/op	0.83
prioritizePeers score -10:0 att 32-0.1 sync 2-0	155.68 us/op	225.14 us/op	0.69
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25	139.51 us/op	189.71 us/op	0.74
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5	290.91 us/op	398.79 us/op	0.73
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75	531.92 us/op	595.66 us/op	0.89
prioritizePeers score 0:0 att 64-1 sync 4-1	659.40 us/op	1.1148 ms/op	0.59
array of 16000 items push then shift	1.6620 us/op	1.8577 us/op	0.89
LinkedList of 16000 items push then shift	7.2680 ns/op	8.7960 ns/op	0.83
array of 16000 items push then pop	114.98 ns/op	155.81 ns/op	0.74
LinkedList of 16000 items push then pop	6.9230 ns/op	8.2190 ns/op	0.84
array of 24000 items push then shift	2.3600 us/op	3.2274 us/op	0.73
LinkedList of 24000 items push then shift	6.9110 ns/op	17.365 ns/op	0.40
array of 24000 items push then pop	129.91 ns/op	268.90 ns/op	0.48
LinkedList of 24000 items push then pop	6.7250 ns/op	10.238 ns/op	0.66
intersect bitArray bitLen 8	6.2520 ns/op	7.8480 ns/op	0.80
intersect array and set length 8	43.939 ns/op	112.91 ns/op	0.39
intersect bitArray bitLen 128	29.073 ns/op	34.438 ns/op	0.84
intersect array and set length 128	650.22 ns/op	1.3290 us/op	0.49
bitArray.getTrueBitIndexes() bitLen 128	2.4560 us/op	2.7600 us/op	0.89
bitArray.getTrueBitIndexes() bitLen 248	3.4650 us/op	3.9640 us/op	0.87
bitArray.getTrueBitIndexes() bitLen 512	10.167 us/op	8.1840 us/op	1.24
Buffer.concat 32 items	907.00 ns/op	988.00 ns/op	0.92
Uint8Array.set 32 items	2.0360 us/op	1.6320 us/op	1.25
Buffer.copy	2.1340 us/op	1.7750 us/op	1.20
Uint8Array.set - with subarray	2.8080 us/op	2.7890 us/op	1.01
Uint8Array.set - without subarray	1.6290 us/op	2.0330 us/op	0.80
getUint32 - dataview	230.00 ns/op	315.00 ns/op	0.73
getUint32 - manual	145.00 ns/op	260.00 ns/op	0.56
Set add up to 64 items then delete first	2.1090 us/op	2.7753 us/op	0.76
OrderedSet add up to 64 items then delete first	3.1543 us/op	3.9113 us/op	0.81
Set add up to 64 items then delete last	2.3932 us/op	3.6145 us/op	0.66
OrderedSet add up to 64 items then delete last	3.7489 us/op	5.1222 us/op	0.73
Set add up to 64 items then delete middle	2.4483 us/op	2.7298 us/op	0.90
OrderedSet add up to 64 items then delete middle	5.0520 us/op	6.0286 us/op	0.84
Set add up to 128 items then delete first	4.9201 us/op	6.8297 us/op	0.72
OrderedSet add up to 128 items then delete first	7.0971 us/op	10.442 us/op	0.68
Set add up to 128 items then delete last	4.8271 us/op	5.5755 us/op	0.87
OrderedSet add up to 128 items then delete last	7.3519 us/op	9.9522 us/op	0.74
Set add up to 128 items then delete middle	4.7540 us/op	5.2590 us/op	0.90
OrderedSet add up to 128 items then delete middle	13.080 us/op	14.926 us/op	0.88
Set add up to 256 items then delete first	10.102 us/op	13.789 us/op	0.73
OrderedSet add up to 256 items then delete first	14.544 us/op	20.545 us/op	0.71
Set add up to 256 items then delete last	9.5392 us/op	12.554 us/op	0.76
OrderedSet add up to 256 items then delete last	14.909 us/op	15.758 us/op	0.95
Set add up to 256 items then delete middle	9.5571 us/op	10.667 us/op	0.90
OrderedSet add up to 256 items then delete middle	38.959 us/op	46.727 us/op	0.83
transfer serialized Status (84 B)	1.3950 us/op	1.5170 us/op	0.92
copy serialized Status (84 B)	1.1360 us/op	1.3080 us/op	0.87
transfer serialized SignedVoluntaryExit (112 B)	1.5340 us/op	1.6990 us/op	0.90
copy serialized SignedVoluntaryExit (112 B)	1.2230 us/op	1.3360 us/op	0.92
transfer serialized ProposerSlashing (416 B)	1.9780 us/op	2.1890 us/op	0.90
copy serialized ProposerSlashing (416 B)	2.0020 us/op	2.2500 us/op	0.89
transfer serialized Attestation (485 B)	2.0080 us/op	2.4170 us/op	0.83
copy serialized Attestation (485 B)	2.0440 us/op	2.2720 us/op	0.90
transfer serialized AttesterSlashing (33232 B)	2.8310 us/op	2.7010 us/op	1.05
copy serialized AttesterSlashing (33232 B)	6.0700 us/op	8.8250 us/op	0.69
transfer serialized Small SignedBeaconBlock (128000 B)	3.8330 us/op	2.8150 us/op	1.36
copy serialized Small SignedBeaconBlock (128000 B)	14.040 us/op	24.634 us/op	0.57
transfer serialized Avg SignedBeaconBlock (200000 B)	4.2260 us/op	3.6840 us/op	1.15
copy serialized Avg SignedBeaconBlock (200000 B)	20.134 us/op	25.755 us/op	0.78
transfer serialized BlobsSidecar (524380 B)	2.9910 us/op	2.6350 us/op	1.14
copy serialized BlobsSidecar (524380 B)	77.520 us/op	96.379 us/op	0.80
transfer serialized Big SignedBeaconBlock (1000000 B)	3.1340 us/op	3.8180 us/op	0.82
copy serialized Big SignedBeaconBlock (1000000 B)	159.48 us/op	183.77 us/op	0.87
pass gossip attestations to forkchoice per slot	3.2519 ms/op	3.5957 ms/op	0.90
forkChoice updateHead vc 100000 bc 64 eq 0	594.34 us/op	552.73 us/op	1.08
forkChoice updateHead vc 600000 bc 64 eq 0	3.0001 ms/op	3.2941 ms/op	0.91
forkChoice updateHead vc 1000000 bc 64 eq 0	5.1774 ms/op	6.1188 ms/op	0.85
forkChoice updateHead vc 600000 bc 320 eq 0	3.0030 ms/op	3.3470 ms/op	0.90
forkChoice updateHead vc 600000 bc 1200 eq 0	3.0512 ms/op	4.8470 ms/op	0.63
forkChoice updateHead vc 600000 bc 7200 eq 0	3.9113 ms/op	4.5896 ms/op	0.85
forkChoice updateHead vc 600000 bc 64 eq 1000	10.500 ms/op	11.543 ms/op	0.91
forkChoice updateHead vc 600000 bc 64 eq 10000	10.647 ms/op	11.883 ms/op	0.90
forkChoice updateHead vc 600000 bc 64 eq 300000	13.843 ms/op	23.222 ms/op	0.60
computeDeltas 500000 validators 300 proto nodes	3.4649 ms/op	4.0673 ms/op	0.85
computeDeltas 500000 validators 1200 proto nodes	3.4483 ms/op	4.2413 ms/op	0.81
computeDeltas 500000 validators 7200 proto nodes	3.4200 ms/op	4.6100 ms/op	0.74
computeDeltas 750000 validators 300 proto nodes	5.1447 ms/op	6.7896 ms/op	0.76
computeDeltas 750000 validators 1200 proto nodes	5.2273 ms/op	6.7521 ms/op	0.77
computeDeltas 750000 validators 7200 proto nodes	5.1799 ms/op	6.3722 ms/op	0.81
computeDeltas 1400000 validators 300 proto nodes	9.6368 ms/op	12.043 ms/op	0.80
computeDeltas 1400000 validators 1200 proto nodes	9.5534 ms/op	11.148 ms/op	0.86
computeDeltas 1400000 validators 7200 proto nodes	9.7561 ms/op	11.517 ms/op	0.85
computeDeltas 2100000 validators 300 proto nodes	14.647 ms/op	15.815 ms/op	0.93
computeDeltas 2100000 validators 1200 proto nodes	14.788 ms/op	15.677 ms/op	0.94
computeDeltas 2100000 validators 7200 proto nodes	14.860 ms/op	15.222 ms/op	0.98
altair processAttestation - 250000 vs - 7PWei normalcase	1.5750 ms/op	1.7476 ms/op	0.90
altair processAttestation - 250000 vs - 7PWei worstcase	2.3775 ms/op	2.6841 ms/op	0.89
altair processAttestation - setStatus - 1/6 committees join	87.642 us/op	92.773 us/op	0.94
altair processAttestation - setStatus - 1/3 committees join	178.88 us/op	180.33 us/op	0.99
altair processAttestation - setStatus - 1/2 committees join	247.18 us/op	265.68 us/op	0.93
altair processAttestation - setStatus - 2/3 committees join	332.73 us/op	333.22 us/op	1.00
altair processAttestation - setStatus - 4/5 committees join	496.10 us/op	507.92 us/op	0.98
altair processAttestation - setStatus - 100% committees join	586.84 us/op	604.03 us/op	0.97
altair processBlock - 250000 vs - 7PWei normalcase	5.1146 ms/op	4.5631 ms/op	1.12
altair processBlock - 250000 vs - 7PWei normalcase hashState	24.950 ms/op	32.407 ms/op	0.77
altair processBlock - 250000 vs - 7PWei worstcase	39.152 ms/op	39.460 ms/op	0.99
altair processBlock - 250000 vs - 7PWei worstcase hashState	83.361 ms/op	85.532 ms/op	0.97
phase0 processBlock - 250000 vs - 7PWei normalcase	2.8043 ms/op	2.7890 ms/op	1.01
phase0 processBlock - 250000 vs - 7PWei worstcase	31.735 ms/op	28.085 ms/op	1.13
altair processEth1Data - 250000 vs - 7PWei normalcase	308.31 us/op	467.26 us/op	0.66
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15	8.0680 us/op	7.1460 us/op	1.13
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219	30.391 us/op	30.398 us/op	1.00
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42	11.093 us/op	13.232 us/op	0.84
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18	8.7520 us/op	8.4650 us/op	1.03
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020	70.060 us/op	119.36 us/op	0.59
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777	1.0778 ms/op	916.20 us/op	1.18
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384	1.5569 ms/op	1.2645 ms/op	1.23
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384	904.86 us/op	1.1508 ms/op	0.79
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384	2.4844 ms/op	3.0927 ms/op	0.80
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384	1.5291 ms/op	2.1243 ms/op	0.72
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384	3.8221 ms/op	4.8473 ms/op	0.79
Tree 40 250000 create	203.04 ms/op	483.00 ms/op	0.42
Tree 40 250000 get(125000)	140.51 ns/op	179.48 ns/op	0.78
Tree 40 250000 set(125000)	627.13 ns/op	1.6828 us/op	0.37
Tree 40 250000 toArray()	15.328 ms/op	25.753 ms/op	0.60
Tree 40 250000 iterate all - toArray() + loop	15.475 ms/op	26.396 ms/op	0.59
Tree 40 250000 iterate all - get(i)	50.189 ms/op	74.884 ms/op	0.67
MutableVector 250000 create	10.827 ms/op	16.958 ms/op	0.64
MutableVector 250000 get(125000)	6.5120 ns/op	7.9280 ns/op	0.82
MutableVector 250000 set(125000)	218.74 ns/op	300.55 ns/op	0.73
MutableVector 250000 toArray()	3.8412 ms/op	5.0849 ms/op	0.76
MutableVector 250000 iterate all - toArray() + loop	3.7757 ms/op	5.4807 ms/op	0.69
MutableVector 250000 iterate all - get(i)	1.5639 ms/op	1.7584 ms/op	0.89
Array 250000 create	2.9853 ms/op	4.8963 ms/op	0.61
Array 250000 clone - spread	1.5255 ms/op	3.6722 ms/op	0.42
Array 250000 get(125000)	0.39200 ns/op	0.51300 ns/op	0.76
Array 250000 set(125000)	0.41800 ns/op	0.53800 ns/op	0.78
Array 250000 iterate all - loop	106.08 us/op	123.48 us/op	0.86
effectiveBalanceIncrements clone Uint8Array 300000	39.044 us/op	78.460 us/op	0.50
effectiveBalanceIncrements clone MutableVector 300000	124.00 ns/op	167.00 ns/op	0.74
effectiveBalanceIncrements rw all Uint8Array 300000	195.72 us/op	221.00 us/op	0.89
effectiveBalanceIncrements rw all MutableVector 300000	70.850 ms/op	179.53 ms/op	0.39
phase0 afterProcessEpoch - 250000 vs - 7PWei	87.138 ms/op	99.330 ms/op	0.88
Array.fill - length 1000000	3.4439 ms/op	7.9360 ms/op	0.43
Array push - length 1000000	21.190 ms/op	26.478 ms/op	0.80
Array.get	0.28615 ns/op	0.31746 ns/op	0.90
Uint8Array.get	0.46409 ns/op	0.49242 ns/op	0.94
phase0 beforeProcessEpoch - 250000 vs - 7PWei	27.160 ms/op	30.087 ms/op	0.90
altair processEpoch - mainnet_e81889	425.38 ms/op	385.46 ms/op	1.10
mainnet_e81889 - altair beforeProcessEpoch	38.605 ms/op	36.538 ms/op	1.06
mainnet_e81889 - altair processJustificationAndFinalization	13.293 us/op	18.921 us/op	0.70
mainnet_e81889 - altair processInactivityUpdates	7.1656 ms/op	6.3351 ms/op	1.13
mainnet_e81889 - altair processRewardsAndPenalties	48.096 ms/op	56.362 ms/op	0.85
mainnet_e81889 - altair processRegistryUpdates	2.3470 us/op	3.3880 us/op	0.69
mainnet_e81889 - altair processSlashings	1.0980 us/op	847.00 ns/op	1.30
mainnet_e81889 - altair processEth1DataReset	930.00 ns/op	732.00 ns/op	1.27
mainnet_e81889 - altair processEffectiveBalanceUpdates	2.1365 ms/op	2.3132 ms/op	0.92
mainnet_e81889 - altair processSlashingsReset	2.9310 us/op	8.3150 us/op	0.35
mainnet_e81889 - altair processRandaoMixesReset	3.6070 us/op	8.5160 us/op	0.42
mainnet_e81889 - altair processHistoricalRootsUpdate	411.00 ns/op	1.2700 us/op	0.32
mainnet_e81889 - altair processParticipationFlagUpdates	2.5380 us/op	3.6310 us/op	0.70
mainnet_e81889 - altair processSyncCommitteeUpdates	520.00 ns/op	1.0030 us/op	0.52
mainnet_e81889 - altair afterProcessEpoch	92.329 ms/op	106.20 ms/op	0.87
capella processEpoch - mainnet_e217614	1.1733 s/op	1.9567 s/op	0.60
mainnet_e217614 - capella beforeProcessEpoch	135.51 ms/op	174.03 ms/op	0.78
mainnet_e217614 - capella processJustificationAndFinalization	23.774 us/op	32.672 us/op	0.73
mainnet_e217614 - capella processInactivityUpdates	18.297 ms/op	23.799 ms/op	0.77
mainnet_e217614 - capella processRewardsAndPenalties	259.80 ms/op	293.13 ms/op	0.89
mainnet_e217614 - capella processRegistryUpdates	18.942 us/op	22.909 us/op	0.83
mainnet_e217614 - capella processSlashings	387.00 ns/op	821.00 ns/op	0.47
mainnet_e217614 - capella processEth1DataReset	320.00 ns/op	1.2620 us/op	0.25
mainnet_e217614 - capella processEffectiveBalanceUpdates	12.707 ms/op	25.334 ms/op	0.50
mainnet_e217614 - capella processSlashingsReset	5.3460 us/op	8.8370 us/op	0.60
mainnet_e217614 - capella processRandaoMixesReset	6.8520 us/op	14.103 us/op	0.49
mainnet_e217614 - capella processHistoricalRootsUpdate	586.00 ns/op	1.2750 us/op	0.46
mainnet_e217614 - capella processParticipationFlagUpdates	2.3660 us/op	8.0520 us/op	0.29
mainnet_e217614 - capella afterProcessEpoch	274.58 ms/op	412.89 ms/op	0.67
phase0 processEpoch - mainnet_e58758	435.24 ms/op	800.70 ms/op	0.54
mainnet_e58758 - phase0 beforeProcessEpoch	110.77 ms/op	218.61 ms/op	0.51
mainnet_e58758 - phase0 processJustificationAndFinalization	21.505 us/op	41.852 us/op	0.51
mainnet_e58758 - phase0 processRewardsAndPenalties	39.973 ms/op	58.574 ms/op	0.68
mainnet_e58758 - phase0 processRegistryUpdates	10.091 us/op	20.839 us/op	0.48
mainnet_e58758 - phase0 processSlashings	562.00 ns/op	1.3770 us/op	0.41
mainnet_e58758 - phase0 processEth1DataReset	319.00 ns/op	1.3080 us/op	0.24
mainnet_e58758 - phase0 processEffectiveBalanceUpdates	1.0982 ms/op	2.0452 ms/op	0.54
mainnet_e58758 - phase0 processSlashingsReset	4.0760 us/op	7.3930 us/op	0.55
mainnet_e58758 - phase0 processRandaoMixesReset	6.3760 us/op	13.948 us/op	0.46
mainnet_e58758 - phase0 processHistoricalRootsUpdate	413.00 ns/op	1.1550 us/op	0.36
mainnet_e58758 - phase0 processParticipationRecordUpdates	3.5730 us/op	20.240 us/op	0.18
mainnet_e58758 - phase0 afterProcessEpoch	78.206 ms/op	137.32 ms/op	0.57
phase0 processEffectiveBalanceUpdates - 250000 normalcase	1.2082 ms/op	2.0470 ms/op	0.59
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5	2.6702 ms/op	7.7077 ms/op	0.35
altair processInactivityUpdates - 250000 normalcase	21.794 ms/op	29.936 ms/op	0.73
altair processInactivityUpdates - 250000 worstcase	21.671 ms/op	34.112 ms/op	0.64
phase0 processRegistryUpdates - 250000 normalcase	8.0150 us/op	20.105 us/op	0.40
phase0 processRegistryUpdates - 250000 badcase_full_deposits	368.82 us/op	507.66 us/op	0.73
phase0 processRegistryUpdates - 250000 worstcase 0.5	128.68 ms/op	168.90 ms/op	0.76
altair processRewardsAndPenalties - 250000 normalcase	48.016 ms/op	46.280 ms/op	1.04
altair processRewardsAndPenalties - 250000 worstcase	42.888 ms/op	46.715 ms/op	0.92
phase0 getAttestationDeltas - 250000 normalcase	8.1329 ms/op	12.549 ms/op	0.65
phase0 getAttestationDeltas - 250000 worstcase	8.4655 ms/op	9.9824 ms/op	0.85
phase0 processSlashings - 250000 worstcase	121.55 us/op	110.95 us/op	1.10
altair processSyncCommitteeUpdates - 250000	116.36 ms/op	146.44 ms/op	0.79
BeaconState.hashTreeRoot - No change	266.00 ns/op	358.00 ns/op	0.74
BeaconState.hashTreeRoot - 1 full validator	114.02 us/op	129.24 us/op	0.88
BeaconState.hashTreeRoot - 32 full validator	1.1610 ms/op	1.7672 ms/op	0.66
BeaconState.hashTreeRoot - 512 full validator	13.625 ms/op	15.259 ms/op	0.89
BeaconState.hashTreeRoot - 1 validator.effectiveBalance	155.85 us/op	189.02 us/op	0.82
BeaconState.hashTreeRoot - 32 validator.effectiveBalance	2.2020 ms/op	2.7816 ms/op	0.79
BeaconState.hashTreeRoot - 512 validator.effectiveBalance	30.107 ms/op	34.706 ms/op	0.87
BeaconState.hashTreeRoot - 1 balances	127.06 us/op	140.65 us/op	0.90
BeaconState.hashTreeRoot - 32 balances	1.1569 ms/op	1.5437 ms/op	0.75
BeaconState.hashTreeRoot - 512 balances	10.029 ms/op	10.871 ms/op	0.92
BeaconState.hashTreeRoot - 250000 balances	173.14 ms/op	164.90 ms/op	1.05
aggregationBits - 2048 els - zipIndexesInBitList	23.682 us/op	28.413 us/op	0.83
byteArrayEquals 32	52.092 ns/op	56.290 ns/op	0.93
Buffer.compare 32	19.618 ns/op	17.995 ns/op	1.09
byteArrayEquals 1024	1.5541 us/op	1.6530 us/op	0.94
Buffer.compare 1024	27.173 ns/op	26.526 ns/op	1.02
byteArrayEquals 16384	24.675 us/op	26.347 us/op	0.94
Buffer.compare 16384	191.67 ns/op	189.44 ns/op	1.01
byteArrayEquals 123687377	184.12 ms/op	212.68 ms/op	0.87
Buffer.compare 123687377	6.5002 ms/op	12.212 ms/op	0.53
byteArrayEquals 32 - diff last byte	51.920 ns/op	59.934 ns/op	0.87
Buffer.compare 32 - diff last byte	17.071 ns/op	19.879 ns/op	0.86
byteArrayEquals 1024 - diff last byte	1.5742 us/op	1.8055 us/op	0.87
Buffer.compare 1024 - diff last byte	25.138 ns/op	29.804 ns/op	0.84
byteArrayEquals 16384 - diff last byte	25.026 us/op	28.535 us/op	0.88
Buffer.compare 16384 - diff last byte	197.98 ns/op	227.59 ns/op	0.87
byteArrayEquals 123687377 - diff last byte	188.18 ms/op	221.27 ms/op	0.85
Buffer.compare 123687377 - diff last byte	6.0954 ms/op	11.603 ms/op	0.53
byteArrayEquals 32 - random bytes	4.9150 ns/op	5.9520 ns/op	0.83
Buffer.compare 32 - random bytes	16.499 ns/op	20.094 ns/op	0.82
byteArrayEquals 1024 - random bytes	4.9420 ns/op	5.8630 ns/op	0.84
Buffer.compare 1024 - random bytes	16.450 ns/op	22.471 ns/op	0.73
byteArrayEquals 16384 - random bytes	4.9140 ns/op	6.1990 ns/op	0.79
Buffer.compare 16384 - random bytes	16.513 ns/op	19.813 ns/op	0.83
byteArrayEquals 123687377 - random bytes	6.0600 ns/op	7.8500 ns/op	0.77
Buffer.compare 123687377 - random bytes	17.640 ns/op	22.730 ns/op	0.78
regular array get 100000 times	31.545 us/op	41.934 us/op	0.75
wrappedArray get 100000 times	31.370 us/op	41.472 us/op	0.76
arrayWithProxy get 100000 times	13.210 ms/op	19.437 ms/op	0.68
ssz.Root.equals	45.581 ns/op	55.656 ns/op	0.82
byteArrayEquals	44.956 ns/op	54.337 ns/op	0.83
Buffer.compare	10.250 ns/op	12.061 ns/op	0.85
shuffle list - 16384 els	6.2957 ms/op	7.3299 ms/op	0.86
shuffle list - 250000 els	92.077 ms/op	112.99 ms/op	0.81
processSlot - 1 slots	14.254 us/op	22.444 us/op	0.64
processSlot - 32 slots	3.2231 ms/op	4.3905 ms/op	0.73
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei	42.216 ms/op	54.205 ms/op	0.78
getCommitteeAssignments - req 1 vs - 250000 vc	2.1382 ms/op	2.5613 ms/op	0.83
getCommitteeAssignments - req 100 vs - 250000 vc	4.1095 ms/op	5.3312 ms/op	0.77
getCommitteeAssignments - req 1000 vs - 250000 vc	4.3760 ms/op	5.2454 ms/op	0.83
findModifiedValidators - 10000 modified validators	245.48 ms/op	625.53 ms/op	0.39
findModifiedValidators - 1000 modified validators	160.38 ms/op	450.90 ms/op	0.36
findModifiedValidators - 100 modified validators	159.12 ms/op	345.77 ms/op	0.46
findModifiedValidators - 10 modified validators	134.45 ms/op	286.71 ms/op	0.47
findModifiedValidators - 1 modified validators	141.64 ms/op	303.21 ms/op	0.47
findModifiedValidators - no difference	146.10 ms/op	324.18 ms/op	0.45
compare ViewDUs	3.1927 s/op	4.3672 s/op	0.73
compare each validator Uint8Array	1.5798 s/op	2.2950 s/op	0.69
compare ViewDU to Uint8Array	948.44 ms/op	1.6281 s/op	0.58
migrate state 1000000 validators, 24 modified, 0 new	553.89 ms/op	683.43 ms/op	0.81
migrate state 1000000 validators, 1700 modified, 1000 new	894.03 ms/op	1.0534 s/op	0.85
migrate state 1000000 validators, 3400 modified, 2000 new	1.1205 s/op	1.2155 s/op	0.92
migrate state 1500000 validators, 24 modified, 0 new	659.84 ms/op	762.66 ms/op	0.87
migrate state 1500000 validators, 1700 modified, 1000 new	864.49 ms/op	832.10 ms/op	1.04
migrate state 1500000 validators, 3400 modified, 2000 new	1.0142 s/op	1.0789 s/op	0.94
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei	4.8400 ns/op	4.7300 ns/op	1.02
state getBlockRootAtSlot - 250000 vs - 7PWei	932.36 ns/op	949.94 ns/op	0.98
computeProposers - vc 250000	9.2434 ms/op	7.6173 ms/op	1.21
computeEpochShuffling - vc 250000	98.265 ms/op	96.464 ms/op	1.02
getNextSyncCommittee - vc 250000	143.67 ms/op	123.15 ms/op	1.17
computeSigningRoot for AttestationData	21.426 us/op	24.217 us/op	0.88
hash AttestationData serialized data then Buffer.toString(base64)	1.6382 us/op	1.5446 us/op	1.06
toHexString serialized data	1.0419 us/op	927.19 ns/op	1.12
Buffer.toString(base64)	220.96 ns/op	194.50 ns/op	1.14

by benchmarkbot/action

packages/beacon-node/src/chain/historicalState/getHistoricalState.ts

packages/beacon-node/src/chain/historicalState/index.ts

packages/beacon-node/src/chain/chain.ts

wemeetagain · 2023-10-16T23:12:34Z

Running into issues opening the db in the worker thread.

It seems the leveldb way of allowing concurrency is have the consumer pass the pointer to the db handle to another thread. That's not currently possible with the leveldb package as it currently exists.

wemeetagain · 2023-10-17T14:51:19Z

Converting to draft until the multithreaded database access issue is resolved

packages/api/src/beacon/routes/debug.ts

packages/beacon-node/src/api/impl/beacon/state/utils.ts

packages/beacon-node/src/chain/historicalState/worker.ts

packages/beacon-node/src/node/nodejs.ts

packages/beacon-node/src/chain/historicalState/worker.ts

packages/beacon-node/src/chain/historicalState/getHistoricalState.ts

packages/beacon-node/src/chain/historicalState/index.ts

philknows · 2024-03-01T21:11:49Z

Any update on the status of this PR? Are we blocked on it somewhere externally?

wemeetagain · 2024-03-04T16:58:33Z

Any update on the status of this PR?

waiting on me to fix this in response to @matthewkeil 's review

wemeetagain · 2024-07-18T18:18:23Z

Ready for another round of review

packages/beacon-node/src/chain/historicalState/getHistoricalState.ts

packages/beacon-node/src/chain/historicalState/index.ts

packages/beacon-node/src/chain/historicalState/getHistoricalState.ts

packages/beacon-node/src/chain/historicalState/index.ts

twoeths · 2024-07-24T02:49:22Z

this PR implements the fundamental work for historical state regen on worker thread, however I'm not sure how likely it could be fully functional in production, a request could take so long given archiveStateEpochFrequency = 1024, it could be up to 1024 * 10s ~= 2.8h given ~10s to process 32 blocks of an epoch like what I saw in n-historical state testing

we could explore storing incremental state diff with 2 purposes:

save disc space
faster time to restart
this historical state regen: apply some diffs until the checkpoint state, and reprocess blocks like in this PR

cc @nazarhussain @philknows @wemeetagain

nazarhussain · 2024-07-27T23:10:15Z

we could explore storing incremental state diff with 2 purposes:

That is already in progress and will be complete very soon.

matthewkeil

LGTM! I think all the important stuff was addressed 🎸

nflaig

LGTM - just a few minor comments / questions

nflaig · 2024-07-30T07:48:03Z

packages/beacon-node/src/chain/chain.ts

    }
+
+    return {state: stateSerialized, executionOptimistic: false, finalized: true};


we have access to the block here, shouldn't this rather be executionOptimistic: isOptimisticBlock(finalizedBlock), afaik a finalized block can still be optimistic

nflaig · 2024-07-30T07:50:48Z

packages/beacon-node/src/chain/historicalState/worker.ts

+// TODO: Pass options from main thread for logging
+// TODO: Logging won't be visible in file loggers


aren't both of these todos stale? file logging works and opts are passed from main thread via workerData

nflaig · 2024-07-30T07:54:54Z

packages/beacon-node/src/chain/historicalState/worker.ts

+    regenTime: metricsRegister.histogram({
+      name: "lodestar_historical_state_regen_time_seconds",
+      help: "Time to regenerate a historical state in seconds",
+      // Historical state regen can take up to 3h as of Aug 2024


We really need to tell people in the release notes that they should reduce archive state frequency if they want access to historical data, do we have a recommended value which keep state regen times below 30s while not blowing up storage requirements?

Also it might be a good time to unhide the flag as per #6908

in my opinion we should not inform, we need to test this work carefully first
I really want to make sure we have a great performance before we inform
see #6033 (comment)

wemeetagain · 2024-08-08T21:22:51Z

🎉 This PR is included in v1.21.0 🎉

feat: add historical state regen

583bc18

wemeetagain requested a review from a team as a code owner October 11, 2023 16:57

chore: wire up metrics

8c1790b

twoeths reviewed Oct 12, 2023

View reviewed changes

packages/beacon-node/src/chain/historicalState/getHistoricalState.ts Outdated Show resolved Hide resolved

nflaig reviewed Oct 13, 2023

View reviewed changes

packages/beacon-node/src/chain/historicalState/index.ts Outdated Show resolved Hide resolved

wemeetagain added 4 commits October 13, 2023 16:19

chore: make historical state regen module optional

93c1981

chore: persist pubkey cache across historical state regen runs

06e58af

chore: cleanup worker termination

623b58c

Merge branch 'unstable' into cayman/historical-state-regen

deba946

dapplion reviewed Oct 16, 2023

View reviewed changes

packages/beacon-node/src/chain/chain.ts Outdated Show resolved Hide resolved

wemeetagain marked this pull request as draft October 17, 2023 14:51

chore: fix worker usage

b8d6113

nflaig mentioned this pull request Oct 31, 2023

No state found for id 'finalized' #4254

Open

matthewkeil mentioned this pull request Nov 5, 2023

Bug in getState and getStateV2 #5440

Closed

matthewkeil added 2 commits November 6, 2023 22:04

fix: swap Level for ClassicLevel for multithreading

0315e1a

fix: getStateV2 state handling hack

cdf3732

philknows mentioned this pull request Nov 7, 2023

Add finalized property to HTTP API responses #5693

Closed

matthewkeil mentioned this pull request Dec 6, 2023

test: fix race condition Level/classic-level#90

Merged

wemeetagain added 9 commits January 22, 2024 10:32

Merge branch 'unstable' into cayman/historical-state-regen

a8f719d

chore: update classic-level

6d9d256

chore: fix build errors

5a812e1

chore: add comments

dabd767

chore: fix test worker path

51fc29a

chore: simplify function naming

21bb96b

chore: optimize getSlotFromOffset

dc45097

chore: refactor to avoid needless deserialization

d32f103

fix: update metrics names

25b1c9f

feat: transfer state across worker boundary

6001521

matthewkeil requested changes Feb 8, 2024

View reviewed changes

wemeetagain added this to the v1.20.0 milestone Jun 4, 2024

wemeetagain added 3 commits July 9, 2024 15:53

Merge branch 'unstable' into cayman/historical-state-regen

41c8b5d

chore: address some pr comments

7ea9c1a

chore: clean module close

4a297dd

philknows modified the milestones: v1.20.0, v1.21.0 Jul 10, 2024

Merge branch 'unstable' into cayman/historical-state-regen

73c1b7c

wemeetagain mentioned this pull request Jul 17, 2024

chore: refactor block and state api utils #6963

Merged

Merge branch 'unstable' into cayman/historical-state-regen

cc4197f

nflaig self-requested a review July 23, 2024 15:03

Merge branch 'unstable' into cayman/historical-state-regen

80debf1

twoeths reviewed Jul 24, 2024

View reviewed changes

packages/beacon-node/src/chain/historicalState/getHistoricalState.ts Outdated Show resolved Hide resolved

twoeths reviewed Jul 24, 2024

View reviewed changes

packages/beacon-node/src/chain/historicalState/index.ts Show resolved Hide resolved

twoeths reviewed Jul 24, 2024

View reviewed changes

packages/beacon-node/src/chain/historicalState/getHistoricalState.ts Outdated Show resolved Hide resolved

twoeths reviewed Jul 24, 2024

View reviewed changes

packages/beacon-node/src/chain/historicalState/index.ts Show resolved Hide resolved

feat: add metrics

e8f4d97

twoeths approved these changes Jul 30, 2024

View reviewed changes

matthewkeil approved these changes Jul 30, 2024

View reviewed changes

twoeths merged commit c23d70c into unstable Jul 30, 2024
20 checks passed

twoeths deleted the cayman/historical-state-regen branch July 30, 2024 06:12

nflaig reviewed Jul 30, 2024

View reviewed changes

twoeths mentioned this pull request Aug 4, 2024

Historical block / state pruning #6869

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add historical state regen #6033

feat: add historical state regen #6033

wemeetagain commented Oct 11, 2023 •

edited

Loading

github-actions bot commented Oct 11, 2023 •

edited

Loading

wemeetagain commented Oct 16, 2023

wemeetagain commented Oct 17, 2023

philknows commented Mar 1, 2024

wemeetagain commented Mar 4, 2024

wemeetagain commented Jul 18, 2024

twoeths commented Jul 24, 2024 •

edited

Loading

nazarhussain commented Jul 27, 2024

matthewkeil left a comment

nflaig left a comment

nflaig Jul 30, 2024

nflaig Jul 30, 2024

nflaig Jul 30, 2024

twoeths Jul 31, 2024

wemeetagain commented Aug 8, 2024

		}

		return {state: stateSerialized, executionOptimistic: false, finalized: true};

		// TODO: Pass options from main thread for logging
		// TODO: Logging won't be visible in file loggers

feat: add historical state regen #6033

feat: add historical state regen #6033

Conversation

wemeetagain commented Oct 11, 2023 • edited Loading

github-actions bot commented Oct 11, 2023 • edited Loading

Performance Report

wemeetagain commented Oct 16, 2023

wemeetagain commented Oct 17, 2023

philknows commented Mar 1, 2024

wemeetagain commented Mar 4, 2024

wemeetagain commented Jul 18, 2024

twoeths commented Jul 24, 2024 • edited Loading

nazarhussain commented Jul 27, 2024

matthewkeil left a comment

Choose a reason for hiding this comment

nflaig left a comment

Choose a reason for hiding this comment

nflaig Jul 30, 2024

Choose a reason for hiding this comment

nflaig Jul 30, 2024

Choose a reason for hiding this comment

nflaig Jul 30, 2024

Choose a reason for hiding this comment

twoeths Jul 31, 2024

Choose a reason for hiding this comment

wemeetagain commented Aug 8, 2024

wemeetagain commented Oct 11, 2023 •

edited

Loading

github-actions bot commented Oct 11, 2023 •

edited

Loading

twoeths commented Jul 24, 2024 •

edited

Loading