Investigate Genesis Mismatch Issue

[distributedstatemachine]

Description

Users are unable to sync mainnet , they are the nodes returns the following error

Thread 'tokio-runtime-worker' panicked at 'Storage root must match that calculated.', /root/.cargo/git/checkouts/polkadot-sdk-cff69157b985ed76/7049c3c/substrate/frame/executive/src/lib.rs:834

Tasks

• Investigate issue and resolve
• Document findings

Related Links

• Discord Support Thread

[distributedstatemachine]

@open-junius can you please update this ticket with your findings. Let also track the issue here and not on discord

[open-junius]

I checked the both test and finney environments, actually the genesis hash is the same for old node and new polkadot-sdk v1.10 based node. The error is not from simple comparing the hash value.
Then I did some experiment in local env, build three different nodes based on v0.9.37, v1.0.0 and v1.10.0. I can see the v1.0.0 node can talk with v0.9.37 without any mismatched genesis error. It happened when I use the v1.10.0 node to connect v0.9.37 node, lots of mismatched genesis error from different peers.
Finally, I checked the network from the same v1.10.0 nodes, it is ok without any error.