Proposal: introduce logical HRMP channels

[svyatonik]

Rationale: HRMP has builtin backpressure mechanism. When the target chain fails to process messages in a timely fashion, it sends the suspend signal and source chain stops sending new messages - they start piling up at the source chain instead. This works perfectly fine when the communication happens between two chains - source and target. But if target chain is the bridge hub, message processing is actually inserting messages to some other outbound bridge queues (heading e.g. from Kusama Bridge Hub to some Polkadot parachain or Ethereum).

We wanted to use the HRMP backpressure to implement dynamic fees - when the bridge queue is "full", we stop processing messages from the source chain. And messages from this chain to sibling brideg hub start piling upe there. But it means that if bridge hub has outbound queues with multiple remote consensus systems (Kusama -> Polkadot, Kusama -> Ethereum), then one lagging bridge may halt the other - simply because they share the single HRMP channel.

What we need: we need a way to suspend (and resume) a substream of messages, sent over the HRMP channel between source chain and its sibling bridge hub. E.g.: let's assume Kusama.AssetHub will have two bridges - with Polkadot.AssetHub and Ethereum. Both bridges are "maintained" by the single sibling bridge hub. So if e.g. relayer for Kusama.AssetHub -> Polkadot.AssetHub bridge is not running, we will send a "suspend-bridge-with-Polkadot-AssetHub" signal to the Kusama.AssetHub. Messages will start piling up at the Kusama.AssetHub and we'll start taking an increased fee for every further message. Meanwhile, messages to Ethereum will be delivered.

Current workaround: to suspend and resume message susbtreams, we use the XCM Transact instruction, which calls the Suspend and Resume methods on our pallet-xcm-bridge-hub-router. But every source chain will have its own runtime call encodings, so we will need to maintain a map of source chain => call format. Also it means that the source chain must enable unpaid execution for bridge hubs, which may be not good for non-system parachains.

Implementation suggestion:

1. Add the custom signal to the ChannelSignal enum of the HRMP pallet:

pub enum ChannelSignal<T> {
	Suspend,
	Resume,
	Custom(T),
}

2. Add the custom signal handler to the HRMP pallet:

pub trait OnCustomSignal<T> {
	fn process_custom_signal(signal: T);
}

3. Implement OnCustomSignal for our pallet-xcm-bridge-hub-router. It'll handle custom signals like that:

enum BridgeSignal {
	Suspend(BridgeId),
	Resume(BridgeId),
}

When the router receives the Suspend signal, then instead of sending messages over HRMP, it just enqueues the
message in the local queue. When Resume signal is received, all enqueued messages are sent over HRMP channel.

[acatangiu]

The proposal looks sound to me, I say we go ahead with it.

[svyatonik]

I'm gonna do all required changes in our repo now and will prepare a Cumulus (monorepo) PR after MQ PR is merged and monorepo is live.

[svyatonik]

We could also think of adding the TransportFeeUpdate(T) signal from after implementing the paritytech/parity-bridges-common#2486 and paritytech/parity-bridges-common#2424. It'll report the updated transport fees to the sending chain, possibly making the fee that user pays for the message lower (c) @acatangiu

It needs to be carefully designed if it'll be implemented

[svyatonik]

Another thing that needs to be added later - it is the process of "punishing" bridges that are misbehaving and keep sending messages after they receive the suspension signal. Two options here:

1. (easy) drop new messages (and close the bridge?) when we see more than M messages in the outbound queue. So we'll have three thresholds (like XCMP pallet do) - suspend_threshold, resume_threshold and drop_threshold;
2. have an offchain fishermen that will submit "proof of sending after suspension" and thus will close the bridge (and get some reward for that out of reserved funds).

[ggwpez]

Not sure if there is anything specific to bridges here, but about HRMP backpressure in general there is the idea to have some "protocol credits" that need to be actively replenished by the receiving side: #2059 (point 3)

The issue is otherwise that the suspend signal is not guaranteed to arrive in congested situations - or arrives too late.

[acatangiu]

Closing in favor of #2059