Revisit the executor cache

[pepyakin]

Currently the executor cache has several layers.

The first layer caches different instantiations of the runtime. That is, a specific code hash of the runtime and specific values of heap_pages and the method. Requesting a runtime with the same code hash but different params will be treated as a different runtime. If the cache is at its capacity then the least recently used instantiation is evicted. The cache is hardcoded to have 2 instantiations at the same time. At the same time there is no time based expiry. Once created, the instantiation lives on until it's evicted.

Then the instantiation hosts a vector of wasm instances ready to be executed. By default it keeps 8 instances. This way the executor can serve the runtime calls with very low latency.

This very same cache is shared among different parts of the system. The cache is used by syncing, offchain workers, RPC, block authoring. Some of them are on the critical path and others are more tolerable.

Examples.

Major syncing will need one instantiation with multiple wasm runtime instances in it. Bumping into a runtime upgrade we can pay the price of flushing the runtime caches.

During the idle syncing the behavior differs. Typically there is one block to import. However, e.g. for BABE, sometimes there can be up to 3-4 forks. Furthermore, for BABE, there are two runtime calls are required to be performed. Hence the 8 instances. This only matters for the nodes that care about latency such as block producers.

Sometimes, block producers run offchain workers. Running those should not get in the way of the syncing and block producing. The same applies for the transaction validation.

An archive node may need to be able to answer lots of requests across multiple runtimes. Those RPC requests should not lead to eviction of the runtimes dedicated for syncing.

Problems. There are problems besides the executor cache itself. Runtime is entered through the Runtime APIs. This mechanism abstracts the low level details of calling into wasm and whatnot. Unfortunately, it also doesn't provide the way to specify either wasm executor or caching specifics.

Possibly related #8251