feat(lib/babe): implement secondary vrf slot block production #2307

kishansagathiya · 2022-02-17T15:01:45Z

Changes

implement secondary vrf slot block production
Also merge slot claim logic

Tests

Run a cross-client devnet with 2 nodes (with polkadot 0.9.10), https://hackmd.io/LfRK6aG6T7mMCE4Uybr-Mg. You will have to edit allowed slots in genesis-spec.json file to "allowed_slots": "PrimaryAndSecondaryVRFSlots". Use the modifies genesis-spec.json to create genesis.json
You will see blocks getting imported and built. On polkadot side, you should see messages suggesting that secondary vrf slots are being verified correctly. Something like below

2022-02-18 16:04:26.807 DEBUG tokio-runtime-worker babe: Verifying secondary VRF block #18 at slot: 274196743    
2022-02-18 16:04:26.807 TRACE             async-io async_io::driver: main_loop: waiting on I/O    
2022-02-18 16:04:26.807 TRACE             async-io async_io::reactor: process_timers: 0 ready wakers    
2022-02-18 16:04:26.807 TRACE             async-io polling: Poller::wait(_, Some(230.265526807s))    
2022-02-18 16:04:26.807 TRACE             async-io polling::epoll: wait: epoll_fd=22, timeout=Some(230.265526807s)    
2022-02-18 16:04:26.807 TRACE             async-io polling::epoll: modify: epoll_fd=22, fd=24, ev=Event { key: 18446744073709551615, readable: true, writable: false }    
2022-02-18 16:04:26.807 TRACE tokio-runtime-worker trie: Committing trie changes to db.

lib/babe: implement secondary slot block production #2216

Primary Reviewer

@noot

kishansagathiya · 2022-02-18T10:39:02Z

Finally saw that magical log message suggesting that secondary vrf slots are being verified correctly

2022-02-18 16:04:34.092 TRACE tokio-runtime-worker babe: Verifying origin: NetworkBroadcast header: Header { parent_hash: 0x234028f29838f9e2251ed4ad046577b0da5ba4f23f765cc62bf68958c14a55c0, number: 19, state_root: 0xf365086251b5add92c26bc6ddbd7cb689ef436610c436374d67e655213856dbb, extrinsics_root: 0xf384a65025bb41419c7a3161c18e9d1998c1985528c0203cd3c04239bfaa01a5, digest: Digest { logs: [DigestItem::PreRuntime([66, 65, 66, 69], [3, 1, 0, 0, 0, 9, 233, 87, 16, 0, 0, 0, 0, 168, 200, 165, 140, 114, 229, 72, 133, 197, 72, 124, 118, 89, 74, 60, 215, 140, 76, 254, 164, 188, 104, 17, 182, 28, 237, 223, 63, 102, 47, 121, 37, 107, 33, 207, 158, 68, 68, 115, 23, 150, 182, 147, 83, 232, 127, 196, 158, 175, 36, 105, 206, 112, 197, 208, 90, 165, 212, 203, 138, 14, 65, 1, 12, 101, 40, 57, 139, 67, 35, 222, 11, 144, 26, 141, 23, 99, 233, 4, 171, 173, 106, 44, 115, 21, 70, 59, 173, 41, 142, 58, 229, 201, 81, 243, 8]), DigestItem::Seal([66, 65, 66, 69], [100, 222, 149, 189, 250, 93, 86, 50, 253, 153, 162, 172, 145, 230, 120, 198, 156, 188, 48, 79, 227, 97, 228, 150, 234, 152, 225, 99, 95, 197, 64, 19, 222, 163, 193, 219, 172, 28, 62, 237, 112, 220, 182, 7, 139, 143, 70, 12, 125, 254, 81, 90, 51, 187, 100, 165, 64, 54, 178, 133, 121, 3, 127, 131])] } } justification(s): None body: Some([0403000b09c3650c7f01])    
2022-02-18 16:04:34.092 DEBUG tokio-runtime-worker babe: We have 2 logs in this header    
2022-02-18 16:04:34.092 TRACE tokio-runtime-worker babe: Checking log DigestItem::PreRuntime([66, 65, 66, 69], [3, 1, 0, 0, 0, 9, 233, 87, 16, 0, 0, 0, 0, 168, 200, 165, 140, 114, 229, 72, 133, 197, 72, 124, 118, 89, 74, 60, 215, 140, 76, 254, 164, 188, 104, 17, 182, 28, 237, 223, 63, 102, 47, 121, 37, 107, 33, 207, 158, 68, 68, 115, 23, 150, 182, 147, 83, 232, 127, 196, 158, 175, 36, 105, 206, 112, 197, 208, 90, 165, 212, 203, 138, 14, 65, 1, 12, 101, 40, 57, 139, 67, 35, 222, 11, 144, 26, 141, 23, 99, 233, 4, 171, 173, 106, 44, 115, 21, 70, 59, 173, 41, 142, 58, 229, 201, 81, 243, 8]), looking for pre runtime digest    
2022-02-18 16:04:34.092 TRACE tokio-runtime-worker babe: Checking log DigestItem::Seal([66, 65, 66, 69], [100, 222, 149, 189, 250, 93, 86, 50, 253, 153, 162, 172, 145, 230, 120, 198, 156, 188, 48, 79, 227, 97, 228, 150, 234, 152, 225, 99, 95, 197, 64, 19, 222, 163, 193, 219, 172, 28, 62, 237, 112, 220, 182, 7, 139, 143, 70, 12, 125, 254, 81, 90, 51, 187, 100, 165, 64, 54, 178, 133, 121, 3, 127, 131]), looking for pre runtime digest    
2022-02-18 16:04:34.092 TRACE tokio-runtime-worker babe: Ignoring digest not meant for us    
2022-02-18 16:04:34.092 TRACE tokio-runtime-worker babe: Checking header    
2022-02-18 16:04:34.092 DEBUG tokio-runtime-worker header: Retrieving mutable reference to digest    
2022-02-18 16:04:34.092 DEBUG tokio-runtime-worker babe: Verifying secondary VRF block #19 at slot: 274196745    
2022-02-18 16:04:34.092 TRACE             async-io async_io::driver: main_loop: waiting on I/O    
2022-02-18 16:04:34.092 TRACE             async-io async_io::reactor: process_timers: 0 ready wakers    
2022-02-18 16:04:34.092 TRACE             async-io polling: Poller::wait(_, Some(222.980433576s))    
2022-02-18 16:04:34.092 TRACE             async-io polling::epoll: wait: epoll_fd=22, timeout=Some(222.980433576s)    
2022-02-18 16:04:34.092 TRACE             async-io polling::epoll: modify: epoll_fd=22, fd=24, ev=Event { key: 18446744073709551615, readable: true, writable: false }    
2022-02-18 16:04:34.093 TRACE tokio-runtime-worker trie: Committing trie changes to db.

- Also merge slot claim logic

codecov · 2022-02-18T11:02:46Z

Codecov Report

Merging #2307 (0701543) into kishan/feat/secondary-slot-block-production (c7f1730) will decrease coverage by 0.01%.
The diff coverage is 16.48%.

@@                               Coverage Diff                               @@
##           kishan/feat/secondary-slot-block-production    #2307      +/-   ##
===============================================================================
- Coverage                                        57.55%   57.53%   -0.02%     
===============================================================================
  Files                                              211      211              
  Lines                                            27937    27947      +10     
===============================================================================
+ Hits                                             16078    16079       +1     
- Misses                                           10226    10234       +8     
- Partials                                          1633     1634       +1

Flag	Coverage Δ
unit-tests	`57.53% <16.48%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
dot/types/babe_digest.go	`24.44% <0.00%> (+4.80%)`	⬆️
lib/babe/crypto.go	`59.43% <0.00%> (-14.69%)`	⬇️
lib/babe/types.go	`0.00% <ø> (ø)`
lib/babe/epoch.go	`27.61% <23.40%> (+2.89%)`	⬆️
dot/state/test_helpers.go	`50.00% <33.33%> (ø)`
lib/babe/epoch_handler.go	`77.50% <50.00%> (+12.32%)`	⬆️
dot/sync/test_helpers.go	`80.00% <100.00%> (ø)`
dot/telemetry/mailer.go	`58.73% <0.00%> (-4.77%)`	⬇️
dot/network/host.go	`64.31% <0.00%> (-1.07%)`	⬇️
dot/network/service.go	`55.88% <0.00%> (+0.45%)`	⬆️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c7f1730...0701543. Read the comment docs.

…n/feat/secondary-slot-vrf

noot · 2022-02-22T19:29:14Z

@kishansagathiya what version of polkadot are you using to test this? on my machine it doesn't look like polkadot is able to import gossamer's blocks

noot · 2022-02-22T19:44:41Z

lib/babe/crypto.go

@@ -84,6 +84,37 @@ func checkPrimaryThreshold(randomness Randomness,
 	return inoutUint.Compare(threshold) < 0, nil
 }

+func claimSecondarySlotVRF(randomness Randomness,


this is interesting, the VRF output isn't compared to any sort of threshold? I'm wondering what the point of the VRF is.. could you link to the substrate code? just wondering if they have any explanation of this

@kishansagathiya not something that needs to be changed code-wise, but just curious!

It seems like this vrf can be consumed by parachains, I am not sure why. https://github.com/paritytech/substrate/blob/master/client/consensus/babe/README.md

This is the relevant code
~~https://github.com/paritytech/substrate/blob/c22fce5a311beede13479c9a00cca85d823b6b00/client/consensus/babe/src/authorship.rs#L158-L166~~
https://github.com/paritytech/substrate/blob/c22fce5a311beede13479c9a00cca85d823b6b00/client/consensus/babe/src/authorship.rs#L140-L155

noot

nice, looks very clean! let me know the versions you used to test so i can try it out

kishansagathiya · 2022-02-23T10:44:47Z

@kishansagathiya what version of polkadot are you using to test this?

0.9.10 works. So, you can follow the same steps described in https://hackmd.io/LfRK6aG6T7mMCE4Uybr-Mg.

on my machine it doesn't look like polkadot is able to import gossamer's blocks

Check if gossamer got disconnected. Looked for dropped in polkadot logs. If so, you you will have to restart both nodes.

dot/types/babe_digest.go

qdm12 · 2022-02-24T14:37:51Z

dot/types/babe_digest.go

+	if err != nil {
+		return nil, err
+	}
+	enc, err := scale.Marshal(digest)


nit babeDigestEncoding would be nice. I'm an explicit/full words maniac so let me know if i'm bothering as well 😅

qdm12 · 2022-02-24T14:38:53Z

lib/babe/crypto.go

+	keypair *sr25519.Keypair,
+	authorityIndex uint32,
+) (*VrfOutputAndProof, error) {
+


Suggested change

lib/babe/epoch.go

lib/babe/crypto.go

noot

tried on my home machine with v0.9.10 and it works! nice :D

feat(lib/babe): implement secondary vrf slot

b6f481d

- Also merge slot claim logic

kishansagathiya force-pushed the kishan/feat/secondary-slot-vrf branch from f274ac3 to b6f481d Compare February 18, 2022 10:42

kishansagathiya added 4 commits February 18, 2022 17:08

remove comments

127cf93

Merge branch 'development' into kishan/feat/secondary-slot-vrf

1003959

fixing tests+lint

0737413

Merge branch 'kishan/feat/secondary-slot-block-production' into kisha…

e696792

…n/feat/secondary-slot-vrf

kishansagathiya marked this pull request as ready for review February 18, 2022 12:31

kishansagathiya requested review from noot, edwardmack, timwu20, EclesioMeloJunior, jimjbrettj and qdm12 as code owners February 18, 2022 12:31

noot reviewed Feb 22, 2022

View reviewed changes

timwu20 mentioned this pull request Feb 23, 2022

feat(devnet): add substrate docker images to dockerfile #2263

Merged

qdm12 reviewed Feb 24, 2022

View reviewed changes

noot approved these changes Feb 24, 2022

View reviewed changes

kishansagathiya added 4 commits February 25, 2022 15:12

Changing ToPreRuntimeDigest method to function

7927a97

applied quentin's suggestion

ee31e80

ornamentary change

48a2a58

fixing linting errors

0701543

kishansagathiya force-pushed the kishan/feat/secondary-slot-vrf branch from 7342519 to 0701543 Compare February 25, 2022 11:49

kishansagathiya merged commit 34f2605 into kishan/feat/secondary-slot-block-production Feb 25, 2022

kishansagathiya deleted the kishan/feat/secondary-slot-vrf branch February 25, 2022 12:27

kishansagathiya mentioned this pull request Sep 30, 2022

Member Request polkadot-fellows/seeding#31

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(lib/babe): implement secondary vrf slot block production #2307

feat(lib/babe): implement secondary vrf slot block production #2307

kishansagathiya commented Feb 17, 2022 •

edited

Loading

kishansagathiya commented Feb 18, 2022

codecov bot commented Feb 18, 2022 •

edited

Loading

noot commented Feb 22, 2022

noot Feb 22, 2022

noot Feb 24, 2022

kishansagathiya Feb 25, 2022

kishansagathiya Feb 25, 2022 •

edited

Loading

noot left a comment

kishansagathiya commented Feb 23, 2022

qdm12 Feb 24, 2022

qdm12 Feb 24, 2022

noot left a comment

feat(lib/babe): implement secondary vrf slot block production #2307

feat(lib/babe): implement secondary vrf slot block production #2307

Conversation

kishansagathiya commented Feb 17, 2022 • edited Loading

Changes

Tests

Primary Reviewer

kishansagathiya commented Feb 18, 2022

codecov bot commented Feb 18, 2022 • edited Loading

Codecov Report

noot commented Feb 22, 2022

noot Feb 22, 2022

Choose a reason for hiding this comment

noot Feb 24, 2022

Choose a reason for hiding this comment

kishansagathiya Feb 25, 2022

Choose a reason for hiding this comment

kishansagathiya Feb 25, 2022 • edited Loading

Choose a reason for hiding this comment

noot left a comment

Choose a reason for hiding this comment

kishansagathiya commented Feb 23, 2022

qdm12 Feb 24, 2022

Choose a reason for hiding this comment

qdm12 Feb 24, 2022

Choose a reason for hiding this comment

noot left a comment

Choose a reason for hiding this comment

kishansagathiya commented Feb 17, 2022 •

edited

Loading

codecov bot commented Feb 18, 2022 •

edited

Loading

kishansagathiya Feb 25, 2022 •

edited

Loading