Metadata V15: Add Runtime API metadata

[lexnv]

This PR collects the runtime API information for the metadata V15 and
exposes the metadata under metadata_at_version(u32::MAX).

Considering both sp-api and frame-support crates need access to the
metadata_ir (an intermediate representation of the metadata that can be converted
into multiple metadata versions), and frame-support depends on sp-api, the
metadata_ir has been moved to sp-api.

Implementation Details

decl_runtime_apis

Collects the runtime metadata of each declared trait using the metadata IR
and exposes the runtime_metadata. The collection of the metadata happens
at this stage because it preserves trait and method documentation, as well as
method parameter names.

mod runtime_decl_for_core {
	// Metadata collected for the `Core` trait. 
	pub fn runtime_metadata<Block, ...>() -> RuntimeApiMetadataIR {
		...
	}
}

impl_runtime_apis

This is the stage at which we definitely know which runtime traits are implemented for
the given Runtime. The macro collects all the RuntimeApiMetadataIR by calling
runtime_metadata of each trait.
The runtime_metadata function is exposed using the InternalImplRuntimeApis trait.

trait InternalImplRuntimeApis {
	fn runtime_metadata(&self) -> vec<RuntimeApiMetadataIR> {
		...
	}
}
impl InternalImplRuntimeApis for Runtime {}

InternalImplRuntimeApis and InternalConstructRuntime

There are many tests that use independently just the impl_runtime_apis or just construct_runtime.
However, only the impl_runtime_apis has access to the runtime metadata API.

To avoid a breaking change, the following behavior is desired:

• when only construct_runtime is used a runtime_metadata method should return an empty runtime metadata API
• when construct_runtime and impl_runtime_apis are used together, the runtime_metadata method should be deduced from the impl_runtime_apis macro

This is indeed the behavior exposed by rust unstable specialization: rust-lang/rust#31844 feature.

One way of circumventing this is by introducing 2 traits

trait InternalImplRuntimeApis {
	fn runtime_metadata(&self) -> vec<RuntimeApiMetadataIR> {
                actual_metadata 
	}
}

trait InternalConstructRuntime {
	fn runtime_metadata(&self) -> vec<RuntimeApiMetadataIR> {
                vec![ ]
	}
}

impl_runtime_apis implements InternalImplRuntimeApis on the Runtime itself (strongest pick by the compiler for runtime_metadata() fn). While construct_runtime implements InternalConstructRuntime on the & Runtime.

Using the Deref guarantees, if we are only using construct_runtime: the runtime will return an empty runtime metadata (avoid breaking change). Otherwise, when implementing both traits the runtime_metadata is deduced to be part of the InternalImplRuntimeApis that contains the complete runtime metadata API.

construct_runtime

This macro puts together the metadata IR and implements the InternalConstructRuntime for avoiding breaking changes.

trait InternalConstructRuntime {
	#[inline(always)]
	fn runtime_metadata(&self) ->vec<TraitMetadata> {
		...
	}
}

// Implemented for &Runtime - lower priority than InternalImplRuntimeApis
impl InternalConstructRuntime for & Runtime {}

let rt = Runtime;
let runtime_md = (&rt).runtime_metadata();

Part of: #12939.

Testing Done

Frame-metadata PR: paritytech/frame-metadata#48
Subxt PoC branch for generating the runtime interface subxt/lexnv/metadata_v15, example from subxt/lexnv/runtime_calls

    // Subxt PoC
    
    // Build payload
    let api_tx = polkadot::runtime_api::Metadata::metadata_versions();
    // Submit payload
    let bytes = api.runtime_api().at(None).await?.call(api_tx).await?;
    println!("Metadata::metadata_versions: {:?}", bytes);

    // Build payload
    let alice = AccountKeyring::Alice.to_account_id().into();
    let api_tx = polkadot::runtime_api::AccountNonceApi::account_nonce(alice);
    // Submit payload
    let bytes = api.runtime_api().at(None).await?.call(api_tx).await?;
    println!("AccountNonceApi::account_nonce: {:?}", bytes);

Runtime API metadata extracted from substrate for the Metadata trait:

RuntimeApiMetadata {
    name: "Metadata",
    methods: [
        RuntimeApiMethodMetadata {
            name: "metadata",
            inputs: [],
            output: UntrackedSymbol {
                id: 782,
                marker: PhantomData<fn() -> core::any::TypeId>,
            },
            docs: [
                " Returns the metadata of a runtime.",
            ],
        },
        RuntimeApiMethodMetadata {
            name: "metadata_at_version",
            inputs: [
                RuntimeApiMethodParamMetadata {
                    name: "version",
                    ty: UntrackedSymbol {
                        id: 4,
                        marker: PhantomData<fn() -> core::any::TypeId>,
                    },
                },
            ],
            output: UntrackedSymbol {
                id: 783,
                marker: PhantomData<fn() -> core::any::TypeId>,
            },
            docs: [
                " Returns the metadata at a given version.",
                "",
                " If the given `version` isn't supported, this will return `None`.",
                " Use [`Self::metadata_versions`] to find out about supported metadata version of the runtime.",
            ],
        },
        RuntimeApiMethodMetadata {
            name: "metadata_versions",
            inputs: [],
            output: UntrackedSymbol {
                id: 105,
                marker: PhantomData<fn() -> core::any::TypeId>,
            },
            docs: [
                " Returns the supported metadata versions.",
                "",
                " This can be used to call `metadata_at_version`.",
            ],
        },
    ],
    docs: [
        " The `Metadata` api trait that returns metadata for the runtime.",
    ],
}

Next Steps

• Expose documentation under docs flag similar to frame
• Adjust and add testing for the new metadata
• Construct runtime metadata entirely in decl_runtime_api (poc commit)
• Avoid breaking changes by abusing Deref (poc commit)

cumulus companion: paritytech/cumulus#2357

cc @paritytech/subxt-team

[bkchr]

Thank you! Nice work! :)

[lexnv]

bot rebase

[paritytech-processbot]

Rebased

[lexnv]

bot merge

[Polkadot-Forum]

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/april-updates-for-substrate-and-polkadot-devs/2764/1

[Polkadot-Forum]

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/stablising-v15-metadata/2819/1