contracts: Multi block migrations

[pgherveou]

Multi-block migration framework for pallet-contracts.

Fix #13638

Test migration with try-runtime

Steps to test against contracts-rococo:

#1 build and run cumulus and move the binary to /usr/local/bin/polkadot-parachain
cargo build --release --bin polkadot-parachain
mv target/release/polkadot-parachain /usr/local/bin/polkadot-parachain

#2 sync it, and keep it running in one terminal tab
polkadot-parachain \
 --chain contracts-rococo \
 --base-path ~/Documents/db/contracts-rococo

#3 compile cumulus against local checkout, and build it with try-runtime feature
diener patch --crates-to-patch /path/to/substrate --substrate
cargo build -p contracts-rococo-runtime --features try-runtime --release

#4 run the migration with 
RUST_LOG="runtime::contracts=debug,try-runtime::cli=debug" ./target/release/polkadot-parachain try-runtime \
 --runtime target/release/wbuild/contracts-rococo-runtime/contracts_rococo_runtime.wasm \
 --chain contracts-rococo \
 on-runtime-upgrade live \
 --uri ws://localhost:9944

Instead of testing against a live chain, a snapshot of contracts-rococo running on version 8, can be used for testing.
One could test the migration, with the following snapshot.
contracts-rococo-9420@ed7ddd5b2bc635ff096f3457e18412cf8b7a7d0ca9375ad42aad65dae42c3077.snap.zip

It was generated with this command:

RUST_LOG="try-runtime::cli=debug" ./target/release/polkadot-parachain try-runtime \
 --runtime existing \
 --chain contracts-rococo \
create-snapshot \
--at  0xed7ddd5b2bc635ff096f3457e18412cf8b7a7d0ca9375ad42aad65dae42c3077  \
--uri ws://localhost:9944

It can be tested with this command

RUST_LOG="runtime::contracts=debug,try-runtime::cli=debug" ./target/release/polkadot-parachain try-runtime \
 --runtime target/release/wbuild/contracts-rococo-runtime/contracts_rococo_runtime.wasm \
 --chain contracts-rococo \
 on-runtime-upgrade snap --snapshot-path ./contracts-rococo-9420@60070dc60358594d0132b82aca52724f0913120002d46d07a72a51850ef282e8.snap

Test migration with zombienet

For testing this migration, I checked out polkadot-v0.9.32 that runs v8 of pallet-contracts so we can test all migrations steps included in this PR.

# go to polkadot dir
cargo build --bin polkadot --release --features fast-runtime --bin polkadot
mv target/release/polkadot /usr/local/bin/polkadot

# go to cumulus dir
git checkout polkadot-v0.9.32
cargo build --bin polkadot-parachain --release
mv target/release/polkadot-parachain /usr/local/bin/polkadot-parachain

You can now configure zombienet with

[relaychain]
default_command = "polkadot"
default_args = [ "-lparachain=debug" ]
chain = "rococo-local"

  [[relaychain.nodes]]
  name = "alice"
  validator = true

  [[relaychain.nodes]]
  name = "bob"
  validator = true

[[parachains]]
id = 1000
chain = "contracts-rococo-local"
cumulus_based = true

  [[parachains.collators]]
  name = "collator-1"
  command = "polkadot-parachain"
  args = ["-lparachain=debug", "--force-authoring" ]

start zombienet with

# assuming the config is saved under ~/Desktop/zombienet.toml
zombienet spawn --provider native ~/Desktop/zombienet.toml

Before running the migration, I use Polkadot-JS app and upload a dummy smart contract, so we can have some state to migrate. You could try to compile something with Ink, but that might be a bit tricky to get something that can work on such an old version. An easier way is to use wat2wasm to compile the dummy contract from pallet-contracts fixtures folder:

wat2wasm substrate/frame/contracts/fixtures/dummy.wat -o dummy.wasm

Then using Polkadot-JS: Developer -> extrinsics -> instantiateWithCode.

In Developer -> Chain state. You should be able to see this data for your storage:

contracts.contractInfoOf: Option<PalletContractsStorageContractInfo>
[
  [
    [
      5HjrdiRuv59XjnX8FxqKKvdkdxhJPa4DeR6zvyYZG2DNeVdo
    ]
    {
      trieId: 0xcc42a3f978ac2433dcc004700c6c7b3abac7e0b5f58bb21994b7e79b8ade5c28
      codeHash: 0x13585c8379d4578083cdb00ce4c6938ca8469e51e9a7ddc5129d3abb2c3d8786
      storageBytes: 0
      storageItems: 0
      storageByteDeposit: 0
      storageItemDeposit: 0
      storageBaseDeposit: 670,699,953
    }
  ]
]

system.account: FrameSystemAccountInfo
[
    [
      5HjrdiRuv59XjnX8FxqKKvdkdxhJPa4DeR6zvyYZG2DNeVdo
    ]
    {
      nonce: 0
      consumers: 0
      providers: 1
      sufficients: 0
      data: {
        free: 0
        reserved: 670,699,953
        miscFrozen: 0
        feeFrozen: 0
      }
    }
  ]

Now that we have setup the stage, we can go ahead and compile the new version of contracts-rococo
Before compiling make sure you bump the spec version:

parachains/runtimes/contracts/contracts-rococo/src/lib.rs
@@ -134,7 +134,7 @@ pub const VERSION: RuntimeVersion = RuntimeVersion {
  spec_name: create_runtime_str!("contracts-rococo"),
  impl_name: create_runtime_str!("contracts-rococo"),
  authoring_version: 1,
- spec_version: 9420,
+ spec_version: 9421,

Then compile with:

cd parachains/runtimes/contracts/contracts-rococo
cargo build --release

target/release/wbuild/contracts-rococo-runtime/contracts_rococo_runtime.compact.compressed.wasm is the wasm file you want to upload from polkadot.js to test the migration

You do that, we can use Polkadot-JS app again:

• first do: Developer -> Extrinsics -> sudo > parachainSystem > authorizeUpgrade(codeHash)
• Then: Developer -> Extrinsics -> parachainSystem > enactAuthorizedUpgrade(code)

Once the migration is passed you should be able to read this new state:

contracts.palletVersion: u16
11

contracts.contractInfoOf: Option<PalletContractsStorageContractInfo>
[
  [
    [
      5HjrdiRuv59XjnX8FxqKKvdkdxhJPa4DeR6zvyYZG2DNeVdo
    ]
    {
      trieId: 0xcc42a3f978ac2433dcc004700c6c7b3abac7e0b5f58bb21994b7e79b8ade5c28
      depositAccount: 5FvfZVJNAH938c21AQxqPv6fRpDvfTdUwAAc27XSyffwzqRR
      codeHash: 0x13585c8379d4578083cdb00ce4c6938ca8469e51e9a7ddc5129d3abb2c3d8786
      storageBytes: 0
      storageItems: 0
      storageByteDeposit: 0
      storageItemDeposit: 0
      storageBaseDeposit: 637,366,620
    }
  ]
]

system.account: FrameSystemAccountInfo
{
  nonce: 0
  consumers: 0
  providers: 2
  sufficients: 0
  data: {
    free: 33,333,333
    reserved: 0
    frozen: 0
    flags: 170,141,183,460,469,231,731,687,303,715,884,105,728
  }
}

system.account: FrameSystemAccountInfo
{
  nonce: 0
  consumers: 0
  providers: 1
  sufficients: 0
  data: {
    free: 637,366,620
    reserved: 0
    frozen: 0
    flags: 170,141,183,460,469,231,731,687,303,715,884,105,728
  }
}

Find blocks involved in a migration.

I built a custom CLI for that see repo here

> contracts-query --url "wss://rococo-contracts-rpc.polkadot.io:443" print-migrating-blocks --target-version 8
Fetching migration blocks:
2023-06-05 18:29:00.138 +02:00 -> BlockInfo { block_hash: 0xc5a879739b995b8b69655607f3ae59f8707c6e92026e953f9d474131808cf9e1, block_number: 2738932, version: 10, migration_in_progress: true }
2023-06-05 18:28:48.079 +02:00 -> BlockInfo { block_hash: 0x27224b9b37a031bedf507fa37c0aad108480a711a1b5ab572aa3a7680aa79bc8, block_number: 2738931, version: 9, migration_in_progress: true }
2023-06-05 18:28:00.135 +02:00 -> BlockInfo { block_hash: 0x60070dc60358594d0132b82aca52724f0913120002d46d07a72a51850ef282e8, block_number: 2738928, version: 8, migration_in_progress: true }
2023-06-05 18:26:24.076 +02:00 -> BlockInfo { block_hash: 0xed7ddd5b2bc635ff096f3457e18412cf8b7a7d0ca9375ad42aad65dae42c3077, block_number: 2738922, version: 8, migration_in_progress: false }
Summary:
Version 10 -> 11 took 01 block(s), from blocks 2738932 to 2738932
Version 09 -> 10 took 03 block(s), from blocks 2738929 to 2738931
Version 08 -> 09 took 06 block(s), from blocks 2738923 to 2738928

Steps to make it a generic migration framework

The migration framework as it is build in this PR lives inside pallet-contracts.
To pull it out of the pallet and create a reusable framework, the following updates should be made:

• We are currently relying on MigrationInProgress storage.

#[pallet::storage]
pub(crate) type MigrationInProgress<T: Config> =
    StorageValue<_, migration::Cursor, OptionQuery>;

This storage defines a cursor for the current migration step. If it exists, a migration is in progress.
We check the existence of this value before each dispatchable:

#[pallet::call_index(...)]
#[pallet::weight(...)]
pub fn my_dispatchable(
   origin: OriginFor<T>,
   ...
) -> DispatchResult{
   Migration::<T>::ensure_migrated()?;
   ensure!(!MigrationInProgress::<T>::exists(), Error::<T>::MigrationInProgress);
   // ...

If we move this code to a library, we need to find a way to create this storage from something that is not a pallet
Or have Pallet implements a new trait, let's say MultiblockMigration

trait MultiBlockMigration {
    fn is_migration_in_progress() -> bool;
    fn set_migration_in_progress();
}

Ideally we implement some macros as well so that every Pallet that implements this trait get the ensure code automatically generated.

Once we have that, people can configure their Migration type that they pass to Executive with something like this

Migration<my_pallet::Pallet, (my_pallet::migration::v8, my_pallet::migration::v9, ...)>

[command-bot]

@pgherveou Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" pallet dev pallet_contracts has finished. Result: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2843031 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2843031/artifacts/download.

[pgherveou]

weights => https://weights.tasty.limo/compare?unit=weight&ignore_errors=true&threshold=10&method=asymptotic&repo=substrate&old=master&new=pg%2Fmigration&path_pattern=frame%2F**%2Fsrc%2Fweights.rs

[pgherveou]

note from talk with Sasha: should we give a different DispatchClass to this call?

[kianenigma]

just asking based on title: could this have been made into something general and reusable?

[pgherveou]

just asking based on title: could this have been made into something general and reusable?

@kianenigma, it was easier to build it inside the pallet first, but its pretty generic already and I am looking forward to making it a broader solution that can be used by other pallets, once it has been battle tested.

[agryaznov]

pls merge master into your branch

[pgherveou]

bot merge

[agryaznov]

why subtract min_balance here? if old_deposit < reducible_balance, then tranferring it away from the account should leave it with more than min_balance, isn't it?

[pgherveou]

mmm we could test it again, but I think you need to remove the min_balance that was added in the original deposit to create the account.