`chain-spec`: support for `json` config/patch (and `GenesisBuilder` API) #14562

michalkucharczyk · 2023-07-12T12:56:00Z

This PR:

adds support for:
- JSON based GenesisConfig to ChainSpec allowing interaction with runtime GenesisBuilder API.
- interacting with arbitrary runtime wasm blob to chain-spec-builder command line util,
- GenesisBuilder API in kitchensink runtime
removes code from system_pallet
adds code to the ChainSpec
deprecates ChainSpec::from_genesis, but also changes the signature of this method extending it with code argument. ChainSpec::builder() should be used instead.
implements GenesisBuilder API for node-template-runtime and kitchensink-runtime.

`code` moved into `ChainSpec`

Having the explicit code field in ChainSpec would allow for duplication of the wasm code within the raw version of the ChainSpec (as the raw storage may contain wasm code under genesis::top::raw::0x3a636f6465 key).

To avoid this redundancy/ambiguity following measure was implemented. When re-exporting the raw version of ChainSpec the code field will be converted into genesis::top::raw::0x3a636f6465. It will be removed from the ChainSpec. If the raw ChainSpec already contains genesis::top::raw::0x3a636f6465 entry, it will be overwritten with the value of the code field (if present).

Similarly, when building genesis state (actually calling ChainSpec::assimilate_storage) the existing 0x3a636f6465 will also be overwritten with code field (if present).

Example:

{
  "name": "TestName",
  "id": "test_id",
  "chainType": "Local",
   ...
  "genesis": {
    "raw": {
      "top": {
        "0x3a636f6465": "0x010101"
      },
      "childrenDefault": {}
    }
  },
  "code": "0x060708"
}

is equivalent of:

{
  "name": "TestName",
  "id": "test_id",
  "chainType": "Local",
   ...
  "genesis": {
    "raw": {
      "top": {
        "0x3a636f6465": "0x060708"
      },
      "childrenDefault": {}
    }
  }
}

JSON based GenesisConfig in `ChainSpec`:

Patch

The ChainSpec can now be built using genesis config JSON patch (which contains some key-value pairs meant to override runtime's genesis config default values). This can be achieved with with_genesis_patch method of the builder:

let chain_spec = ChainSpec::builder()
    .with_code(substrate_test_runtime::wasm_binary_unwrap())
    .with_genesis_config_patch(json!({
        "babe": {
            "epochConfig": {
                "c": [
                    7,
                    10
                ],
                "allowed_slots": "PrimaryAndSecondaryPlainSlots"
            }
        }
    }))
    .build()

Resulting ChainSpec instance can be converted to raw version of chain spec JSON file. This was not changed and can be done with chain_spec.as_json(true) method. Sample output is here. The runtime's GenesisBuilder::build_config API is called during this conversion.

The ChainSpec instance can also be written to chain spec JSON file in human readable form. The resulting chain spec file will contain the genesis config patch (partial genesis config). Sample output is here

Full Config

It is also possible to build ChainSpec using full genesis config JSON (containing all the genesis config key-value pairs). No defaults will be used in this approach. The sample code is as follows:

let chain_spec = ChainSpec::builder()
    .with_code(substrate_test_runtime::wasm_binary_unwrap())
    .with_genesis_config(json!({
        "babe": {
            "authorities": [
                [ "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", 1 ],
                ...
            ],
            "epochConfig": {
                "allowed_slots": "PrimaryAndSecondaryPlainSlots",
                "c": [ 3, 10 ]
            }
        },
        "balances": {
            "balances": [
                [ "5D34dL5prEUaGNQtPPZ3yN5Y6BnkfXunKXXz6fo7ZJbLwRRH", 100000000000000000 ],
            ],
            ...
        },
        "substrateTest": {
            "authorities": [
                "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
                ...
            ]
        },
        "system": {}
    }))
    .build()

Again, resulting ChainSpec instance can be converted to the raw version of chain spec JSON file (which involves calling GenesisBuilder::build_config runtime method) .
It can be also stored in human readable version, sample output here.

`chain-spec-builder` util

New commands allowing to interact with arbitrary WASM runtime blob were added. Use cases are as follows:

Get default config from runtime

Queries the default genesis config from the provided runtime.wasm and uses it in the chain spec. Can also store runtime's default genesis config in given file (-d):

chain-spec-builder runtime -r runtime.wasm default -d /dev/stdout

Note: GenesisBuilder::create_default_config runtime function is called.

Generate raw storage chain spec using genesis config patch

Patches the runtime's default genesis config with provided patch.json and generates raw storage (-s) version of chain spec:

chain-spec-builder runtime -s -r runtime.wasm patch -p patch.json

Note: GenesisBuilder::build_config runtime function is called.

Generate raw storage chain spec using full genesis config

Build the chain spec using provided full genesis config json file. No defaults will be used:

chain-spec-builder runtime -s -r runtime.wasm full -c full-genesis-config.json

Note: GenesisBuilder::build_config runtime function is called.

Generate human readable chain spec using genesis config patch

chain-spec-builder runtime -r runtime.wasm patch -p patch.json

Note: No runtime API is called.

Generate human readable chain spec using full genesis config

chain-spec-builder runtime -r runtime.wasm full -c full-genesis-config.json

Note: No runtime API is called.

Some extra utils:

verify: allows to verify if human readable chain spec is valid (precisely: all required fields in genesis config are initialized),
edit, allows to:
- update the code in given chain spec,
- convert given chain spec to the raw chain spec,

Some open questions/issues:

~~naming .with_no_genesis_defaults + in chain spec json keys: JsonPatch / JsonFull,~~
~~GenesisSource source for patch and full config~~.
support for New/Generate commands in `chain-spec-builder? (IMO we can remove them).

Step towards: paritytech/polkadot-sdk#25

polkadot companion: paritytech/polkadot#7508
cumulus companion: paritytech/cumulus#2936

- json test added (wip)

- ChainSpecBuilder used - tests using RuntimeGenesisConfig added to verify patching approach

michalkucharczyk · 2023-08-01T12:25:15Z

bot clean

michalkucharczyk · 2023-08-01T13:44:44Z

bin/utils/chain-spec-builder/src/lib.rs

+/// accounts. Only works for kitchen-sink runtime
+#[derive(Parser, Debug)]
+#[command(rename_all = "kebab-case")]
+pub struct NewCmd {


IMO we could remove it as it works only with kitchensink runtime.

I think that can be removed.
If I understood correctly the same result can be accomplished by:

using the generate command to build the patch

then using the runtime to patch the kitchensink wasm (passed from the command line)

But I also noted that the GenerateCmd also uses generate_chain_spec. And thus also uses the kitchensink runtime.

What abut NOT using the kitchensink runtime at all here? I.e. use this Generate command just to build a patch without the code?

But this also requires the ChainSpecBuilder::with_code to be optional (optional code field for patches)

test-utils/runtime/src/lib.rs

bin/node/cli/src/chain_spec.rs

client/chain-spec/src/chain_spec.rs

client/chain-spec/src/genesis_config_builder.rs

client/chain-spec/src/chain_spec.rs

bin/node/runtime/src/lib.rs

client/chain-spec/src/chain_spec.rs

bin/node/cli/src/chain_spec.rs

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

michalkucharczyk · 2023-08-02T16:53:50Z

bot fmt

michalkucharczyk · 2023-08-02T18:37:26Z

bot rebase

…ort-for-genesis-builder-api

michalkucharczyk · 2023-08-02T18:37:45Z

bot clean

…-api

davxy · 2023-08-03T09:41:11Z

bin/node-template/node/src/chain_spec.rs

-		},
-		balances: BalancesConfig {
+) -> serde_json::Value {
+	serde_json::json!({


By using serde_json::Value we are loosing the static type checking advantages we can have when using the types defined by the native runtime (i.e. RuntimeGenesisConfig).

(Maybe I'm going to say something stupid, not 100% confident with metadata) but isn't possible to have some sanity check using the runtime metadata?
Maybe for this we have to investigate (scale_info::meta_type::<Runtime>())

davxy · 2023-08-03T09:49:05Z

bin/node-template/runtime/Cargo.toml

@@ -114,3 +117,6 @@ try-runtime = [
 	"pallet-transaction-payment/try-runtime",
 	"sp-runtime/try-runtime"
 ]
+
+#Enabling this flag will disable GenesisBuilder API implementation in runtime.
+disable-genesis-builder = []


Just an idea to open a discussion (not saying this is better).

What about instead using a feature called genesis-builder that is part of the default = ["std", "genesis-builder"]?

Then define genesis-builder feature to also enable serde_json and sp-genesis-builder which may be set as optional dependencies.

client/chain-spec/src/chain_spec.rs

bin/utils/chain-spec-builder/bin/main.rs

davxy · 2023-08-03T17:20:08Z

bin/utils/chain-spec-builder/src/lib.rs

+/// accounts. Only works for kitchen-sink runtime
+#[derive(Parser, Debug)]
+#[command(rename_all = "kebab-case")]
+pub struct NewCmd {


I think that can be removed.
If I understood correctly the same result can be accomplished by:

using the generate command to build the patch

then using the runtime to patch the kitchensink wasm (passed from the command line)

davxy · 2023-08-04T08:26:58Z

client/chain-spec/src/chain_spec.rs

+	/// into `RawGenesis` as `genesis::top::raw::0x3a636f6465` (which is
+	/// [`sp_core::storage::well_known_keys::CODE`]). If the spec is already in `raw` format, and
+	/// contains `genesis::top::raw::0x3a636f6465` field it will be updated with content of `code`
+	/// field (if present).


Dumb question. Do we care somewhere about preserving field order in the json? Looks not... but I just wanted to open the discussion

davxy · 2023-08-04T08:37:02Z

client/chain-spec/Cargo.toml

@@ -13,15 +13,30 @@ readme = "README.md"
 targets = ["x86_64-unknown-linux-gnu"]

 [dependencies]
+codec = { package = "parity-scale-codec", version = "3.6.1", default-features = false, features = ["derive"] }
+json-patch = { version = "1.0.0", default-features = false }


I'd just add this few lines function in chain-spec/lib.rs and remove the dependency.
IMO adding a dependency to use a trivial function is not good (again, this is my opinion).

client/chain-spec/src/lib.rs

davxy · 2023-08-04T10:09:02Z

frame/system/src/lib.rs

@@ -671,7 +669,6 @@ pub mod pallet {
 			<UpgradedToU32RefCount<T>>::put(true);
 			<UpgradedToTripleRefCount<T>>::put(true);

-			sp_io::storage::set(well_known_keys::CODE, &self.code);
 			sp_io::storage::set(well_known_keys::EXTRINSIC_INDEX, &0u32.encode());


Looking at this now I wonder what is the point of setting to 0 the well known EXTRINSIC_INDEX during build.

maybe we can modify this with an unwrap_or_default() (thus not returning an Option).

On block initialization this is always set to 0

client/chain-spec/res/substrate_test_runtime_from_patch__raw.json

michalkucharczyk added 20 commits July 12, 2023 15:03

frame::system: code removed

52d6a4f

chain-spec: mod genesis -> genesis_block

e333e16

chain-spec: genesis_config_builder (wasm based)

ea2e252

chain-spec: support for code and config patching

b4f5267

chain-spec: runtime caller added

baacd30

chain-spec: doc + minor improvements

b40193f

chain-spec: set_code added

dc6c927

chain-spec-builder: from_genesis removed

ae9b2a8

chain-spec-builder: boot_nodes call optional

e166fec

chain-spec-builder: support for runtime wasm blobs

c200e26

bin/node: GenesisBuilder implementation added

e64b7d1

- json test added (wip)

bin/node-template: serde enabled in runtime deps

5d450a4

bin/node/runtime: arbitrary_precision feature added to serde_json

b945d4d

bin/node: fixes

ae9415f

bin/node: from_genesis moved to ChainSpecBuilder

98a9a58

- ChainSpecBuilder used - tests using RuntimeGenesisConfig added to verify patching approach

bin/node: boot_nodes call optional

f49362e

bin/node: GenesisBuilder helper struct removed

f62284d

bin/node: frame_system::code removed

12dac2c

node-template: frame_system::code removed

ab4ae6b

client/cli: from_genesis - code arg added

fc55a8d

michalkucharczyk force-pushed the mku-chain-spec-support-for-genesis-builder-api branch from 18330f8 to fc55a8d Compare July 12, 2023 13:03

michalkucharczyk requested a review from bkchr July 12, 2023 16:27

michalkucharczyk requested a review from pepoviola July 12, 2023 16:31

michalkucharczyk added 2 commits July 17, 2023 15:19

chain-spec: naming fixed

dbe0c8b

node-template: ChainSpecBuilder used

01f10b9

michalkucharczyk marked this pull request as ready for review August 1, 2023 12:15

michalkucharczyk requested review from a team August 1, 2023 12:15

michalkucharczyk added A0-please_review Pull request needs code review. and removed A3-in_progress Pull request is in progress. No review needed at this stage. labels Aug 1, 2023

command-bot and others added 2 commits August 1, 2023 12:17

".git/.scripts/commands/fmt/fmt.sh"

2bd3fab

Cargo.lock updated

d788003

command-bot bot deleted a comment from paritytech-processbot bot Aug 1, 2023

michalkucharczyk commented Aug 1, 2023

View reviewed changes

minor fixes

298305b

skunert reviewed Aug 2, 2023

View reviewed changes

michalkucharczyk commented Aug 2, 2023

View reviewed changes

bin/node/cli/src/chain_spec.rs Outdated Show resolved Hide resolved

michalkucharczyk commented Aug 2, 2023

View reviewed changes

bin/node/cli/src/chain_spec.rs Outdated Show resolved Hide resolved

michalkucharczyk commented Aug 2, 2023

View reviewed changes

bin/node/cli/src/chain_spec.rs Outdated Show resolved Hide resolved

michalkucharczyk commented Aug 2, 2023

View reviewed changes

bin/node/cli/src/chain_spec.rs Outdated Show resolved Hide resolved

michalkucharczyk and others added 2 commits August 2, 2023 18:07

Apply suggestions from code review

97e5b1b

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

cleanup

5f60a4b

".git/.scripts/commands/fmt/fmt.sh"

0f58273

Merge remote-tracking branch 'origin/master' into mku-chain-spec-supp…

1096d36

…ort-for-genesis-builder-api

command-bot bot deleted a comment from paritytech-processbot bot Aug 2, 2023

removing unneeded refs to runtime PalletGenesisConfig types

e1c7e11

michalkucharczyk requested a review from a team August 3, 2023 12:06

Merge branch 'master' into mku-chain-spec-support-for-genesis-builder…

fc33dfb

…-api

davxy reviewed Aug 4, 2023

View reviewed changes

michalkucharczyk mentioned this pull request Aug 29, 2023

chain-spec: getting ready for native-runtime-free world paritytech/polkadot-sdk#1256

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`chain-spec`: support for `json` config/patch (and `GenesisBuilder` API) #14562

`chain-spec`: support for `json` config/patch (and `GenesisBuilder` API) #14562

michalkucharczyk commented Jul 12, 2023 •

edited

Loading

michalkucharczyk commented Aug 1, 2023

michalkucharczyk Aug 1, 2023

davxy Aug 3, 2023

davxy Aug 3, 2023

michalkucharczyk commented Aug 2, 2023

michalkucharczyk commented Aug 2, 2023

michalkucharczyk commented Aug 2, 2023

davxy Aug 3, 2023

davxy Aug 3, 2023

davxy Aug 3, 2023

davxy Aug 4, 2023

davxy Aug 4, 2023

davxy Aug 4, 2023 •

edited

Loading

chain-spec: support for json config/patch (and GenesisBuilder API) #14562

Are you sure you want to change the base?

chain-spec: support for json config/patch (and GenesisBuilder API) #14562

Conversation

michalkucharczyk commented Jul 12, 2023 • edited Loading

code moved into ChainSpec

JSON based GenesisConfig in ChainSpec:

Patch

Full Config

chain-spec-builder util

Get default config from runtime

Generate raw storage chain spec using genesis config patch

Generate raw storage chain spec using full genesis config

Generate human readable chain spec using genesis config patch

Generate human readable chain spec using full genesis config

Some extra utils:

Some open questions/issues:

michalkucharczyk commented Aug 1, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michalkucharczyk commented Aug 2, 2023

michalkucharczyk commented Aug 2, 2023

michalkucharczyk commented Aug 2, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davxy Aug 4, 2023 • edited Loading

Choose a reason for hiding this comment

`chain-spec`: support for `json` config/patch (and `GenesisBuilder` API) #14562

`chain-spec`: support for `json` config/patch (and `GenesisBuilder` API) #14562

michalkucharczyk commented Jul 12, 2023 •

edited

Loading

`code` moved into `ChainSpec`

JSON based GenesisConfig in `ChainSpec`:

`chain-spec-builder` util

davxy Aug 4, 2023 •

edited

Loading