Add staging runtime api #5048

tdimitrov · 2022-03-07T17:57:08Z

Quoting @rphmeier (with small edits):

The broader idea I have around runtime API upgrades is that we'd
maintain a vstaging which is based on the latest API version and has a
special STAGING_VERSION (u32::MAX). We'd be able to conditionally enable
this on runtimes for testing. And then we'd eventually set the next
runtime API version to be equal to vstaging and start a fresh vstaging

cumulus companion: paritytech/cumulus#1108

bkchr · 2022-03-11T15:19:05Z

Where would this staging runtime api being used? What is the idea behind using u32::MAX as version?

If you will try to implement the staging + the normal runtime api in the same runtime, the macro will start to complain that this doesn't work.

ordian · 2022-03-11T15:37:31Z

I think we should only enable one of them (either v2 or vstaging) conditionally.
The general idea is that we want to continue adding methods to the runtime API on master without affecting releases of polkadot/kusama/westend/rococo.

Quoting some previous comments from @rphmeier

Depends what we want from the staging API, tbh. If we require that it's always 'complete' in master, then it makes sense to deploy it to Rococo always. But I suspect that the following are more likely to be true:

Runtime API upgrades are fairly infrequent (in the mid-term of stronger protocol development, maybe 2-3 a year and after that about 1 a year)

A staging API builds in master over time and can be added to.

Staging is frozen, tested, and then released.

In this case, I would say that we shouldn't deploy staging to any network except ZombieNet tests most of the time, and only Versi once it's frozen.

bkchr · 2022-03-11T16:40:58Z

Okay thank you.

You will need some sort of node code anyway to handle staging vs v2/vx/etc. You could also just add the methods to the normal runtime api. As your node code will need to be prepared for the staging version anyway, you can use that "to version". Aka you just add new methods but they are not being used by the normal node, just by staging stuff. If you then want to stabilize a method, you bump the trait version and change your node code to make this runtime api function usable when version X is present.

You could also do something like name the method unstable_xy and then already have them being present in all the runtimes and when the time comes that you want to stabilize something you just need to put a renamed_in() above the method and do a node release.

But these are just some random ideas by me.

rphmeier · 2022-03-11T17:03:58Z

@bkchr

Thanks, that's a good suggestion.

@tdimitrov - what I suggest, then, is that we move the declaration of ParachainHost to not be inside of v2/vx and instead be one level above. explicitly refer to the types in its methods as v2::SomeType or vstaging::OtherType. staging types should only be used for unstable_ methods.

tdimitrov · 2022-03-15T09:47:32Z

@rphmeier @bkchr could you please review?

EDIT: I need to fix the usage here in parallel to this PR.

eskimor

Looks good overall. One thing struck me though:

ParachainsHost is versioned .... so should not staging be implemented in the following way:

Have staging correspond to the next version number that is not yet used - we merely just name it staging to make it clear that we don't want that version on production networks yet.
Only support the latest stable version of ParachainHost on production network, but support the staging version number (3 currently) on test networks.
Staging gets stabilized, by letting staging point to the next version number (4) and by implementing/supporting the old staging ParachainHost (3) on production networks.

primitives/src/runtime_api.rs

tdimitrov · 2022-03-21T14:06:11Z

Only support the latest stable version of ParachainHost on production network, but support the staging version number (3 currently) on test networks.

I think this is a problem, because two runtime API versions can't easily coexist in the code. For this reason the ParachainHost trait is moved outside v2.

`ParachainHost` is no longer versioned and is in `runtime_api` module. This is a companion for paritytech/polkadot#5048

ordian · 2022-03-23T18:58:16Z

How do we prevent an unstable_ API accidentally leaking into a release version of polkadot/kusama/westend runtime, which is one of the main points of vstaging idea?

bkchr · 2022-03-23T19:03:07Z

How do we prevent an unstable_ API accidentally leaking into a release version of polkadot/kusama/westend runtime, which is one of the main points of vstaging idea?

There would be no problem of having a unstable_ API as part of a release. You are doing the versioning on the client anyway. This means, if you don't call the api, nothing bad happens ;)

ordian · 2022-03-23T19:08:23Z

How do we prevent an unstable_ API accidentally leaking into a release version of polkadot/kusama/westend runtime, which is one of the main points of vstaging idea?

There would be no problem of having a unstable_ API as part of a release. You are doing the versioning on the client anyway. This means, if you don't call the api, nothing bad happens ;)

But how the client is supposed to quiery this API? We use versioning here

polkadot/node/core/runtime-api/src/lib.rs

Lines 427 to 430 in 6ec8ede

    
           Request::Authorities(sender) => query!(Authorities, authorities(), ver = 1, sender), 
        
           Request::Validators(sender) => query!(Validators, validators(), ver = 1, sender), 
        
           Request::ValidatorGroups(sender) => 
        
           	query!(ValidatorGroups, validator_groups(), ver = 1, sender),

When introducing an unstable API, what version should be put there? And how's the client will be using it? Like any other runtime API? Or are we going to feature gate the unstable functionality on the client side? Then the worry is accidental feature leaking.

primitives/src/runtime_api.rs

`ParachainHost` is no longer versioned and is in `runtime_api` module. This is a companion for paritytech/polkadot#5048

`trait ParachainHost` is no longer part of a specific primitives version. Instead there is a single trait for stable and staging api versions. The trait contains stable AND staging methods. The latter are explicitly marked as unstable.

`polkadot_primitives::v2` becomes `polkadot_primitives::runtime_api`

Introduces the concept for 'staging functions' in runtime API. These functions are still in testing and they are meant to be used only within test networks (Westend). They coexist with the stable calls for technical reasons - maintaining different runtime APIs for different networks is hard to implement. Check the doc comments in source files for more details how the staging API should be used.

Add `staging_get_session_disputes` to `ParachainHost` as the first method of the staging API.

…es()

ordian · 2022-04-12T15:27:38Z

bot merge

paritytech-processbot · 2022-04-12T15:27:46Z

Error: pr-custom-review is not passing for paritytech/cumulus#1108

ordian · 2022-04-12T15:47:38Z

bot merge

* Handle relocation of `ParachainHost` in Polkadot `ParachainHost` is no longer versioned and is in `runtime_api` module. This is a companion for paritytech/polkadot#5048 * Update dependencies

* master: (62 commits) Bump tracing from 0.1.32 to 0.1.33 (#5299) Add staging runtime api (#5048) CI: rename ambiguous jobs (#5313) Prepare for rust 1.60 (#5282) Bump proc-macro2 from 1.0.36 to 1.0.37 (#5265) Fixes the dead lock when any of the channels get at capacity. (#5297) Bump syn from 1.0.90 to 1.0.91 (#5273) create .github/workflows/pr-custom-review.yml (#5280) [ci] fix pallet benchmark script (#5247) (#5292) bump spec_version to 9190 (#5291) bump version to 0.9.19 (#5290) Session is actually ancient not unknown. (#5286) [ci] point benchmark script to sub-commands (#5247) (#5289) Remove Travis CI (#5287) Improve error handling in approval voting block import (#5283) fix .github/pr-custom-review.yml (#5279) Co #11164: Sub-commands for `benchmark` (#5247) Bump clap from 3.1.6 to 3.1.8 (#5240) Custom Slots Migration for Fund Index Change (#5227) create pr-custom-review.yml (#5253) ...

* Handle relocation of `ParachainHost` in Polkadot `ParachainHost` is no longer versioned and is in `runtime_api` module. This is a companion for paritytech/polkadot#5048 * Update dependencies

tdimitrov marked this pull request as draft March 7, 2022 18:08

tdimitrov added C1-low PR touches the given topic and has a low impact on builders. A0-please_review Pull request needs code review. B0-silent Changes should not be mentioned in any release notes labels Mar 7, 2022

tdimitrov force-pushed the runtime-staging-api branch from 9dd0cb4 to 05d8d59 Compare March 11, 2022 13:37

tdimitrov force-pushed the runtime-staging-api branch from 05d8d59 to 3771cf7 Compare March 15, 2022 09:42

tdimitrov marked this pull request as ready for review March 15, 2022 09:45

tdimitrov requested review from tomusdrw, svyatonik, acatangiu and antonio-dropulic as code owners March 15, 2022 09:45

tdimitrov force-pushed the runtime-staging-api branch from 3771cf7 to 3eda74f Compare March 15, 2022 09:46

eskimor reviewed Mar 21, 2022

View reviewed changes

primitives/src/runtime_api.rs Outdated Show resolved Hide resolved

tdimitrov force-pushed the runtime-staging-api branch from 3eda74f to 2be2d06 Compare March 22, 2022 07:28

tdimitrov added the D2-breaksapi label Mar 22, 2022

tdimitrov added a commit to tdimitrov/cumulus that referenced this pull request Mar 23, 2022

Handle relocation of ParachainHost in Polkadot

3ddcc2e

`ParachainHost` is no longer versioned and is in `runtime_api` module. This is a companion for paritytech/polkadot#5048

tdimitrov mentioned this pull request Mar 23, 2022

Handle relocation of ParachainHost in Polkadot paritytech/cumulus#1108

Merged

tdimitrov added a commit to tdimitrov/cumulus that referenced this pull request Mar 23, 2022

Handle relocation of ParachainHost in Polkadot

417a9d9

`ParachainHost` is no longer versioned and is in `runtime_api` module. This is a companion for paritytech/polkadot#5048

tdimitrov force-pushed the runtime-staging-api branch 3 times, most recently from 44e00a6 to 660680b Compare March 23, 2022 13:46

eskimor approved these changes Apr 10, 2022

View reviewed changes

paritytech-ci requested review from a team April 11, 2022 10:30

tdimitrov force-pushed the runtime-staging-api branch 2 times, most recently from 4467629 to 425104e Compare April 11, 2022 14:05

joao-paulo-parity mentioned this pull request Apr 11, 2022

Approvals are not counted correctly paritytech/pr-custom-review#79

Closed

bkchr approved these changes Apr 11, 2022

View reviewed changes

paritytech-ci requested review from a team April 11, 2022 19:27

ordian reviewed Apr 12, 2022

View reviewed changes

primitives/src/runtime_api.rs Outdated Show resolved Hide resolved

primitives/src/runtime_api.rs Outdated Show resolved Hide resolved

tdimitrov added a commit to tdimitrov/cumulus that referenced this pull request Apr 12, 2022

Handle relocation of ParachainHost in Polkadot

43c29ef

`ParachainHost` is no longer versioned and is in `runtime_api` module. This is a companion for paritytech/polkadot#5048

tdimitrov added 7 commits April 12, 2022 17:50

Fix use primitives

690cc4e

`polkadot_primitives::v2` becomes `polkadot_primitives::runtime_api`

Add new staging method - get_session_disputes()

235629b

Add `staging_get_session_disputes` to `ParachainHost` as the first method of the staging API.

Hide vstaging runtime api implementations behind feature flag

b873a8c

Fix test runtime

43ed02f

fn staging_get_session_disputes() is renamed to fn staging_get_disput…

2f84976

…es()

tdimitrov force-pushed the runtime-staging-api branch from 219a57f to 2f84976 Compare April 12, 2022 14:50

paritytech-processbot bot merged commit 722936d into paritytech:master Apr 12, 2022

tdimitrov deleted the runtime-staging-api branch April 12, 2022 15:48

rphmeier mentioned this pull request Apr 21, 2022

Improve ChainSelectionSubsystem to revert based on disputes concluding off-chain #4804

Closed

2 tasks

tdimitrov mentioned this pull request Jun 2, 2022

Staging Runtime API implementation paritytech/substrate#11577

Closed

rphmeier mentioned this pull request Sep 12, 2022

Runtime API Subsystem should not do version negotiation internally paritytech/polkadot-sdk#858

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add staging runtime api #5048

Add staging runtime api #5048

tdimitrov commented Mar 7, 2022 •

edited

Loading

bkchr commented Mar 11, 2022

ordian commented Mar 11, 2022

bkchr commented Mar 11, 2022

rphmeier commented Mar 11, 2022 •

edited

Loading

tdimitrov commented Mar 15, 2022 •

edited

Loading

eskimor left a comment

tdimitrov commented Mar 21, 2022

ordian commented Mar 23, 2022

bkchr commented Mar 23, 2022

ordian commented Mar 23, 2022

ordian commented Apr 12, 2022

paritytech-processbot bot commented Apr 12, 2022

ordian commented Apr 12, 2022

Add staging runtime api #5048

Add staging runtime api #5048

Conversation

tdimitrov commented Mar 7, 2022 • edited Loading

bkchr commented Mar 11, 2022

ordian commented Mar 11, 2022

bkchr commented Mar 11, 2022

rphmeier commented Mar 11, 2022 • edited Loading

tdimitrov commented Mar 15, 2022 • edited Loading

eskimor left a comment

Choose a reason for hiding this comment

tdimitrov commented Mar 21, 2022

ordian commented Mar 23, 2022

bkchr commented Mar 23, 2022

ordian commented Mar 23, 2022

ordian commented Apr 12, 2022

paritytech-processbot bot commented Apr 12, 2022

ordian commented Apr 12, 2022

tdimitrov commented Mar 7, 2022 •

edited

Loading

rphmeier commented Mar 11, 2022 •

edited

Loading

tdimitrov commented Mar 15, 2022 •

edited

Loading