Add CheckMetadataHash extension

[bkchr]

This signed extension enables the new metadata hash verification feature that was approved as
RFC78. This will bring support for the new generic ledger hardware wallet app and further hardware wallets in the Polkadot ecosystem.

[davxy]

A couple of trivial things to be fixed. But basically looks good

[bkchr]

A couple of trivial things to be fixed. But basically looks good

@davxy what do you want to have fixed? :D

[muharem]

How do we communicate such breaking upgrade for clients? And how we make sure they all ready when this get released? I need to learn this.

[davxy]

A couple of trivial things to be fixed. But basically looks good

@davxy what do you want to have fixed? :D

When instancing SignedExtra it requires the extra metadata. But for sure you already noticed that :-)

[bkchr]

When instancing SignedExtra it requires the extra metadata. But for sure you already noticed that :-)

I still don't get it? Do you mean when constructing the transaction? You can also just pass 0 to disable this.

[bkchr]

How do we communicate such breaking upgrade for clients? And how we make sure they all ready when this get released? I need to learn this.

I hope that @SBalaguer informed everybody, as I told him :D @anaelleltd generally this is something we should ensure that it gets communicated.

[github-actions]

Review required! Latest push from author must always be reviewed

[SBalaguer]

I hope that @SBalaguer informed everybody,

Yes, indeed we are taking care of it! The main thing we struggle with is providing a clear ETA for teams that need to integrate this, so that they spend the time to develop what needs to be developed and then they release it when it needs to be released. Currently we are working with end of June as a target.

[michalisFr]

I guess this issue on Rococo is related?
paritytech/polkadot-sdk#4659

[bkchr]

/merge

[fellowship-merge-bot]

Enabled auto-merge in Pull Request

Available commands

• /merge: Enables auto-merge for Pull Request
• /merge cancel: Cancels auto-merge for Pull Request
• /merge help: Shows this menu

For more information see the documentation

[AlastairHolmes]

I hope that @SBalaguer informed everybody,

Yes, indeed we are taking care of it! The main thing we struggle with is providing a clear ETA for teams that need to integrate this, so that they spend the time to develop what needs to be developed and then they release it when it needs to be released. Currently we are working with end of June as a target.

Where are these type of breaking changes communicated? I ask as we have not been able to find any consistent location to monitor.

[SBalaguer]

Hey @AlastairHolmes, this has been communicated in the release notes on the runtime upgrade, on the Polkadot Forum, and also we reached out to individuals that could be directly affected by this change, to the best of our knowledge.

[niso1985]

Hey @AlastairHolmes, this has been communicated in the release notes on the runtime upgrade, on the Polkadot Forum, and also we reached out to individuals that could be directly affected by this change, to the best of our knowledge.

I feel the same way as @AlastairHolmes
I have been looking at Forum and Runtime upgrade commits for a few days now, but I could not notice this update at all.
I guess I still don't have enough love for Polkadot. Or maybe my love is being tested.
I will now stay up all night working on this response. Thanks for the great gift.

[symtor]

Note to wallet maintainers
To update the tx encoding: In its simplest form (i.e. to skip checking the metadata hash), the new tx binary format requires two changes:

• in the transaction, add one byte 0x00 after the tip field
• in the signature payload, add one byte 0x00 after the tip field and another byte 0x00 after the era hash

Note to the maintainers here
I share the sentiment expressed by @AlastairHolmes and @niso1985 .

RFC78 was intended to benefit the users of hardware wallets.

• Polkadot transactions contain a "version" field (currently with value "4", it seems), which can help disambiguating between different encodings. This field is capable of representing a new tx format without breaking the existing standard.
• Even though the changes of this PR are minor and default values exist, the PR chose to ignore versioning and break backward compatibility.
• As a consequence, users of hardware wallets are locked out of their funds (at least until they update their device firmware, if at all possible).

Is this an accurate assessment of the situation? A big, red, flashy warning in the release notes would be on the lower end of expectations in such case.

[bkchr]

• Polkadot transactions contain a "version" field (currently with value "4", it seems), which can help disambiguating between different encodings. This field is capable of representing a new tx format without breaking the existing standard.

This is the version of the underlying transaction format, not related to this change here. But I get what you mean and I'm already thinking on how we can improve this!

Even though the changes of this PR are minor and default values exist, the PR chose to ignore versioning and break backward compatibility.

See my response above.

Is this an accurate assessment of the situation? A big, red, flashy warning in the release notes would be on the lower end of expectations in such case.

Yes we will do this in the future. We actually started to inform everybody more than two months ago, but apparently we missed some people. We will improve the messaging when such breaking changes are going to be merged to ensure that more people are aware of this.

[bkchr]

@symtor can you provide us some channel where we can reach you the best to keep you informed next time better?

[symtor]

I'm already thinking on how we can improve this!

Thanks for taking this seriously! Backward compatibility of the transaction format is important, and activating the CheckMetadataHash extension on Polkadot mainnet changed the format by adding a field. One positive example here is ETH, where a new transaction format (EIP-1559) was introduced through a version prefix (EIP-2718). Even though the ETH maintainers hadn't initially designed for different transaction versions, they've been maintaining both versions since 4 years and still get traffic on v0.

can you provide us some channel where we can reach you the best to keep you informed next time better?

We read the release notes and RFCs attentively, so they are our primary channel of information. But in the Polkadot v1.13.0 release notes this feature is described as an optional add-on rather than a breaking change. If breaking changes to the tx format, node API or node operation on the Polkadot network could be highlighted in the release notes, this would already help us a lot.

For a direct communication channel, I'm going to reach out to you in private communication.

[bkchr]

If breaking changes to the tx format, node API or node operation on the Polkadot network could be highlighted in the release notes, this would already help us a lot.

Yeah! I'm sorry for this! We should have made this more clear.

[bkchr]

polkadot-fellows/RFCs#99 I opened this RFC to ensure that we can stay backwards compatible all the time and this doesn't happen anymore.