check missing migrations in the past

[brenzi]

runnting try-runtime-cli after using zepter discovered outdated pallet storage versions

balances pallet should be at v1:

• check this: API for registering inactive funds paritytech/substrate#12813

[brenzi]

manual try-runtime:

cargo build --release -p integritee-runtime --features try-runtime
try-runtime --runtime ./target/release/wbuild/integritee-runtime/integritee_runtime.compact.compressed.wasm on-runtime-upgrade --checks=pre-and-post --disable-spec-version-check --disable-idempotency-checks live --uri wss://kusama.api.integritee.network:443

yields discrepancies:

[2024-01-17T18:39:42Z INFO  runtime::frame-support] ⚠️ XcmpQueue declares internal migrations (which *might* execute). On-chain `StorageVersion(3)` vs current storage version `StorageVersion(2)`
[2024-01-17T18:39:42Z INFO  runtime::frame-support] ⚠️ PolkadotXcm declares internal migrations (which *might* execute). On-chain `StorageVersion(0)` vs current storage version `StorageVersion(1)`
[2024-01-17T18:39:42Z INFO  runtime::frame-support] ⚠️ DmpQueue declares internal migrations (which *might* execute). On-chain `StorageVersion(2)` vs current storage version `StorageVersion(1)`

but that's only the ip of the iceberg:

pallet	onchain v38	polkadot-v0.9.42	Plan
balances	0	1	👍 migrate
scheduler	0	4	fake-migrate. our storage is already at V1
xcmpQueue	3	2	wait for polkadot-v1.0.0 release runtime
dmpQueue	2	1	wait for polkadot-v1.0.0 release runtime
collective	0	4	don't touch. multi-instance trouble
democracy	0	1	👍 migrate
multisig	0	1	takes 500ms: wait for async backing to migrate, if at all
PolkadotXcm	0	1	fake-migrate. our storage is already at V1 de-facto

[brenzi]

several issues when attempting to catch up migrations

[2024-01-17T20:54:28Z INFO  try_runtime_core::commands::on_runtime_upgrade] 🔬 TryRuntime_on_runtime_upgrade succeeded! Running it again without checks for weight measurements.
[2024-01-17T20:54:29Z INFO  runtime::fix::scheduler::migration] migrating from StorageVersion(0) to 4
[2024-01-17T20:54:29Z ERROR runtime::storage] Corrupted state at `[227, 143, 24, 82, 7, 73, 138, 187, 92, 33, 61, 15, 176, 89, 179, 216, 216, 55, 28, 116, 117, 23, 21, 29, 134, 63, 146, 111, 129, 151, 24, 223, 191, 178, 127, 30, 174, 240, 107, 185, 3, 0, 0, 0, 136, 254, 198, 134, 61, 79, 41, 111, 121, 197, 118, 145, 220, 86, 29, 133, 3, 1, 1, 0, 65, 31]: Error`
[2024-01-17T20:54:29Z ERROR frame_support::storage] old key failed to decode at [227, 143, 24, 82, 7, 73, 138, 187, 92, 33, 61, 15, 176, 89, 179, 216, 216, 55, 28, 116, 117, 23, 21, 29, 134, 63, 146, 111, 129, 151, 24, 223, 191, 178, 127, 30, 174, 240, 107, 185, 3, 0, 0, 0, 136, 254, 198, 134, 61, 79, 41, 111, 121, 197, 118, 145, 220, 86, 29, 133, 3, 1, 1, 0, 65, 31]
[2024-01-17T20:54:29Z ERROR runtime::storage] Corrupted state at `[227, 143, 24, 82, 7, 73, 138, 187, 92, 33, 61, 15, 176, 89, 179, 216, 216, 55, 28, 116, 117, 23, 21, 29, 134, 63, 146, 111, 129, 151, 24, 223, 191, 178, 127, 30, 174, 240, 107, 185, 3, 0, 0, 0, 213, 106, 79, 126, 209, 233, 88, 171, 125, 160, 40, 120, 180, 152, 74, 255, 3, 1, 1, 0, 157, 31]: Error`
[2024-01-17T20:54:29Z ERROR frame_support::storage] old key failed to decode at [227, 143, 24, 82, 7, 73, 138, 187, 92, 33, 61, 15, 176, 89, 179, 216, 216, 55, 28, 116, 117, 23, 21, 29, 134, 63, 146, 111, 129, 151, 24, 223, 191, 178, 127, 30, 174, 240, 107, 185, 3, 0, 0, 0, 213, 106, 79, 126, 209, 233, 88, 171, 125, 160, 40, 120, 180, 152, 74, 255, 3, 1, 1, 0, 157, 31]
[2024-01-17T20:54:29Z INFO  pallet_xcm::migration::v1] v1 applied successfully
[2024-01-17T20:54:29Z INFO  runtime::balances] Storage to version 1
[2024-01-17T20:54:29Z INFO  runtime::frame-support] ⚠️ ParachainSystem declares internal migrations (which *might* execute). On-chain `StorageVersion(2)` vs current storage version `StorageVersion(2)`
[2024-01-17T20:54:29Z INFO  runtime::frame-support] ⚠️ XcmpQueue declares internal migrations (which *might* execute). On-chain `StorageVersion(3)` vs current storage version `StorageVersion(2)`
[2024-01-17T20:54:29Z INFO  runtime::frame-support] ⚠️ PolkadotXcm declares internal migrations (which *might* execute). On-chain `StorageVersion(1)` vs current storage version `StorageVersion(1)`
[2024-01-17T20:54:29Z INFO  runtime::frame-support] ⚠️ DmpQueue declares internal migrations (which *might* execute). On-chain `StorageVersion(2)` vs current storage version `StorageVersion(1)`
[2024-01-17T20:54:29Z INFO  try_runtime_core::commands::on_runtime_upgrade] ℹ Skipping idempotency check
[2024-01-17T20:54:29Z INFO  try-runtime::cli] PoV size (zstd-compressed compact proof): 3.5 KB. For parachains, it's your responsibility to verify that a PoV of this size fits within any relaychain constraints.
[2024-01-17T20:54:29Z INFO  try-runtime::cli] Consumed ref_time: 0.500825s (100.17% of max 0.5s)
[2024-01-17T20:54:29Z WARN  try-runtime::cli] Consumed ref_time is >= 80% of the max allowed ref_time. Please ensure the migration is not be too computationally expensive to be fit in a single block.
[2024-01-17T20:54:29Z ERROR try-runtime::cli] ❌ Weight safety issues detected.
[2024-01-17T20:54:29Z ERROR try_runtime_core::commands::on_runtime_upgrade] ❌ Issues detected, exiting non-zero. See logs.

most likely, corrupt storage is this (which may explain XCM trouble:

[brenzi]

applied v41 on Rococo and migration went through:
https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Frococo.api.integritee.network#/explorer/query/0x74fc38de72d7de24ed81b12652c4aec6b9411c61133d32737264a7d69f38a043

now, the storage versions look like this:

Rococo

pallet	onchain v38	polkadot-v0.9.42	Plan
balances	1	1	👍
scheduler	4	4	👍 1 agenda still there
xcmpQueue	3	2	untouched
dmpQueue	2	1	untouched
collective	0	4	untouched
democracy	1	1	👍
multisig	0	1	untouched
PolkadotXcm	1	1	👍

[brenzi]

upgrading to polkadot v1.0.0, migrations (coming from v41) look like this:

pallet	onchain v41	polkadot-v1.0.0	Plan
bounties	0	4	fake-migrate (See issue)
xcmpQueue	3	3	noop
dmpQueue	2	2	noop
collective	0	4	brute-force migrate both instances
multisig	0	1	👍 migrate

[brenzi]

applied https://github.com/integritee-network/parachain/releases/tag/1.7.5 on rococo v41 -> v42

pallet	onchain v41	polkadot-v1.0.0	Plan
bounties	4	4	👍
xcmpQueue	3	3	👍
dmpQueue	2	2	👍
council	4	4	👍
technicalCommittee	4	4	👍
multisig	1	1	👍

[brenzi]

all this executed well on Kusama mainnet as well (v41 -> v42):