Add eth_multicall , support array of eth_call for simulation across multiple blocks #383

epheph · 2023-02-20T22:53:54Z

This spec is based heavily on #312 from @s1na (eth_batchCall), but encompasses some of the ideas presented by @MicahZoltu and others. The original spec supported a single block override, and did not allow transactions to appear on different blocks with different states. The original spec allowed you to submit:

a block to target (where to build your transactions from)
a block override (different block number, feeRecipient, randDao, etc)
array of storage overrides (delta or full replacement)
array of unsigned transaction descriptions

This spec changes the spec to encapslate most of the parameters into an array:

a block to target
an Array of zero or more of the following:
- a block override
- array of storage overrides
- array of unsigned transaction descriptions

In addition, I added log return and gasUsed in the array response of call result/error (although I have not confirmed if this is difficult to implement).

The idea here is that simulation needs are getting more advanced:

Searchers need to trigger oracle update and wait a number of blocks
Researchers need to trigger a price change and wait a number of blocks for a TWAP adjustment
Wallets are beginning to provide advanced simulation to the user
Block builders are beginning to explore multi-block MEV

Lacking an API like this will just lead to more non-standard proprietary solutions (Alchemy forking mainnet, Flashbots eth_callBundle, etc) that will not be accessible on self-hosted nodes.

For the "storage overrides" (which includes the ability to replace code in accounts), we should allow replacing code at precompiles, allowing for ecrecover to serve false results. More dapps are utilizing off-chain EIP-712 signatures (notably, Uniswap), and overriding tx.from is no longer enough to perform simulations without a private key. Alternatively, we could add an ecrecoverOverrides (at the same level as storage overrides above).

…lls, does not change execution)

epheph · 2023-02-20T22:59:53Z

openrpc.json diff:

		{
			"name": "eth_multicall",
			"summary": "Executes a sequence of message calls building on each other's state without creating transactions on the block chain, optionally overriding block and state data",
			"params": [
				{
					"name": "Version",
					"schema": {
						"title": "Version",
						"type": "string",
						"pattern": "^0x([1-9a-f]+[0-9a-f]*|0)$"
					}
				},
				{
					"name": "BlockCalls",
					"schema": {
						"title": "Arguments for multi call",
						"type": "array",
						"items": {
							"title": "Array of calls to be executed at specific, optional block/state",
							"properties": {
								"blockOverride": {
									"title": "Block override",
									"type": "object",
									"properties": {
										"number": {
											"title": "Number",
											"type": "string",
											"pattern": "^0x([1-9a-f]+[0-9a-f]{0,31})|0$"
										},
										"prevRandao": {
											"title": "Randomness beacon",
											"type": "string",
											"pattern": "^0x([1-9a-f]+[0-9a-f]{0,31})|0$"
										},
										"time": {
											"title": "Time",
											"type": "string",
											"pattern": "^0x([1-9a-f]+[0-9a-f]{0,31})|0$"
										},
										"gasLimit": {
											"title": "Gas limit",
											"type": "string",
											"pattern": "^0x([1-9a-f]+[0-9a-f]{0,15})|0$"
										},
										"feeRecipient": {
											"title": "feeRecipient",
											"type": "string",
											"pattern": "^0x[0-9,a-f,A-F]{40}$"
										},
										"baseFee": {
											"title": "Base fee",
											"type": "string",
											"pattern": "^0x([1-9a-f]+[0-9a-f]{0,31})|0$"
										}
									}
								},
								"stateOverrides": {
									"title": "State overrides",
									"schema": {
										"title": "Arguments for multi call",
										"type": "array",
										"items": {
											"type": "object",
											"additionalProperties": {
												"$ref": "#/components/schemas/AccountOverride"
											}
										}
									}
								},
								"calls": {
									"type": "array",
									"title": "List of transactions to execute at this block/state",
									"items": {
										"type": "object",
										"title": "Transaction object type for call",
										"properties": {
											"type": {
												"title": "type",
												"type": "string",
												"pattern": "^0x([0-9,a-f,A-F]?){1,2}$"
											},
											"nonce": {
												"title": "nonce",
												"type": "string",
												"pattern": "^0x([1-9a-f]+[0-9a-f]*|0)$"
											},
											"to": {
												"title": "to address",
												"type": "string",
												"pattern": "^0x[0-9,a-f,A-F]{40}$"
											},
											"from": {
												"title": "from address",
												"type": "string",
												"pattern": "^0x[0-9,a-f,A-F]{40}$"
											},
											"gas": {
												"title": "gas limit",
												"type": "string",
												"pattern": "^0x([1-9a-f]+[0-9a-f]*|0)$"
											},
											"value": {
												"title": "value",
												"type": "string",
												"pattern": "^0x([1-9a-f]+[0-9a-f]*|0)$"
											},
											"input": {
												"title": "input data",
												"type": "string",
												"pattern": "^0x[0-9a-f]*$"
											},
											"gasPrice": {
												"title": "gas price",
												"type": "string",
												"pattern": "^0x([1-9a-f]+[0-9a-f]*|0)$",
												"description": "The gas price willing to be paid by the sender in wei"
											},
											"maxPriorityFeePerGas": {
												"title": "max priority fee per gas",
												"type": "string",
												"pattern": "^0x([1-9a-f]+[0-9a-f]*|0)$",
												"description": "Maximum fee per gas the sender is willing to pay to miners in wei"
											},
											"maxFeePerGas": {
												"title": "max fee per gas",
												"type": "string",
												"pattern": "^0x([1-9a-f]+[0-9a-f]*|0)$",
												"description": "The maximum total fee per gas the sender is willing to pay (includes the network / base fee and miner / priority fee) in wei"
											},
											"accessList": {
												"title": "accessList",
												"type": "array",
												"description": "EIP-2930 access list",
												"items": {
													"title": "Access list entry",
													"type": "object",
													"properties": {
														"address": {
															"title": "hex encoded address",
															"type": "string",
															"pattern": "^0x[0-9,a-f,A-F]{40}$"
														},
														"storageKeys": {
															"type": "array",
															"items": {
																"title": "32 byte hex value",
																"type": "string",
																"pattern": "^0x[0-9a-f]{64}$"
															}
														}
													}
												}
											}
										}
									}
								}
							}
						}
					}
				},
				{
					"name": "Block",
					"required": false,
					"schema": {
						"title": "Block number, tag, or block hash",
						"anyOf": [
							{
								"title": "Block number",
								"type": "string",
								"pattern": "^0x([1-9a-f]+[0-9a-f]*|0)$"
							},
							{
								"title": "Block tag",
								"type": "string",
								"enum": [
									"earliest",
									"finalized",
									"safe",
									"latest",
									"pending"
								],
								"description": "`earliest`: The lowest numbered block the client has available; `finalized`: The most recent crypto-economically secure block, cannot be re-orged outside of manual intervention driven by community coordination; `safe`: The most recent block that is safe from re-orgs under honest majority and certain synchronicity assumptions; `latest`: The most recent block in the canonical chain observed by the client, this block may be re-orged out of the canonical chain even under healthy/normal conditions; `pending`: A sample next block built by the client on top of `latest` and containing the set of transactions usually taken from local mempool. Before the merge transition is finalized, any call querying for `finalized` or `safe` block MUST be responded to with `-39001: Unknown block` error"
							},
							{
								"title": "Block hash",
								"type": "string",
								"pattern": "^0x[0-9a-f]{64}$"
							}
						]
					}
				}
			],
			"result": {
				"name": "Result of calls",
				"schema": {
					"title": "Results of multi call",
					"type": "array",
					"items": {
						"anyOf": [
							{
								"title": "Result of call failure",
								"type": "object",
								"required": [
									"status",
									"return",
									"error",
									"gasUsed"
								],
								"properties": {
									"status": {
										"title": "Call Status Failure",
										"type": "string",
										"pattern": "^0x0$"
									},
									"return": {
										"title": "Return data",
										"type": "string",
										"pattern": "^0x[0-9a-f]*$"
									},
									"gasUsed": {
										"title": "Return gasUsed",
										"type": "string",
										"pattern": "^0x([1-9a-f]+[0-9a-f]*|0)$"
									},
									"error": {
										"title": "Error Object",
										"type": "object",
										"properties": {
											"code": {
												"title": "Error code",
												"type": "number",
												"pattern": "^-?[0-9]+$"
											},
											"message": {
												"title": "Error Message (execution reverted, out of gas, etc)",
												"type": "string"
											},
											"data": {
												"title": "Reverted, with optional message",
												"type": "string"
											}
										}
									}
								}
							},
							{
								"title": "Result of call success",
								"type": "object",
								"required": [
									"status",
									"return",
									"gasUsed",
									"logs"
								],
								"properties": {
									"status": {
										"title": "Call Status Success",
										"type": "string",
										"pattern": "^0x1$"
									},
									"return": {
										"title": "Return data",
										"type": "string",
										"pattern": "^0x[0-9a-f]*$"
									},
									"gasUsed": {
										"title": "Return gasUsed",
										"type": "string",
										"pattern": "^0x([1-9a-f]+[0-9a-f]*|0)$"
									},
									"logs": {
										"title": "Return logs",
										"type": "array",
										"items": {
											"title": "log",
											"type": "object",
											"properties": {
												"logIndex": {
													"title": "log index",
													"type": "string",
													"pattern": "^0x([1-9a-f]+[0-9a-f]*|0)$"
												},
												"blockHash": {
													"title": "block hash",
													"type": "string",
													"pattern": "^0x[0-9a-f]{64}$"
												},
												"blockNumber": {
													"title": "block number",
													"type": "string",
													"pattern": "^0x([1-9a-f]+[0-9a-f]*|0)$"
												},
												"address": {
													"title": "address",
													"type": "string",
													"pattern": "^0x[0-9,a-f,A-F]{40}$"
												},
												"data": {
													"title": "data",
													"type": "string",
													"pattern": "^0x[0-9a-f]*$"
												},
												"topics": {
													"title": "topics",
													"type": "array",
													"items": {
														"title": "32 hex encoded bytes",
														"type": "string",
														"pattern": "^0x[0-9a-f]{64}$"
													}
												}
											}
										}
									}
								}
							},
							{
								"title": "Result of call not being valid (nonce, baseFee, etc)",
								"type": "object",
								"required": [
									"status",
									"error"
								],
								"properties": {
									"status": {
										"title": "Call Status Invalid",
										"type": "string",
										"pattern": "^0x2$"
									},
									"error": {
										"title": "Error Object",
										"type": "object",
										"properties": {
											"message": {
												"title": "Reason the transaction could not be included on-chain (baseFee, nonce too high, etc)",
												"type": "string"
											},
											"code": {
												"title": "decimal signed integer",
												"type": "number",
												"pattern": "^-?[0-9]+$"
											}
										}
									}
								}
							}
						]
					}
				}
			}
		}

…H balance to a random account

s1na · 2023-04-07T13:14:04Z

Hey @epheph cool that you took on this project. First I want to add a clarification, please correct me if I'm wrong: State overrides of each block are applied to the post state of the previous block. I.e. if you want to have a test contract you don't need to override it in every block separately.

On another point: I think it's nice to have call results share the same block-structure. It would be easier to map a response to the request.

epheph · 2023-04-19T22:19:56Z

Yes exactly, state overrides are conceptually similar to a transaction type that raw updates storage slots.

I went back and forth on the flattening of the calls. One argument in support of your suggestion about retaining block array structure is that it would give you a logical place to put block-level information (randDao, fee recipient) which might be useful. @MicahZoltu any opposition to adding structure to the response?

s1na · 2023-04-20T09:23:33Z

As I see it you do retain the block-structure for the calls. The results are flat however. This is the inconsistency I was pointing out.

epheph · 2023-04-20T16:30:00Z

Right, definitely need it for calls, I can easily see the argument for maintaining that structure in the response. I'll propose a change to see how it looks.

MicahZoltu · 2023-04-25T11:38:21Z

Thinking more on this, I think I would like to see the response be shaped as close to an eth_getBlockByNumber call. Perhaps we can add a receipt property to each transaction and put the return data in there? This would make it so existing tools would be able to easily parse the response with only minor tweaks to extract the new information?

Generally speaking though, I agree with @s1na that the result should not be flat. Each block in the result should have the call results nested somewhere inside of it.

KillariDev · 2023-05-10T14:29:39Z

Would this allow us to know which contracts were created in the multicall? And what about what storage addresses are changed? Or would getting of this information be too heavy for nodes?

epheph · 2023-05-11T05:33:39Z

Good question, i imagine that is helpful, especially contract deployments. i wonder where it would go? Inside the call object, next to logs? @KillariDev

…itself

src/eth/execute.yaml

src/schemas/execute.yaml

src/eth/execute.yaml

KillariDev · 2023-05-25T14:37:22Z

src/schemas/execute.yaml

+    stateOverrides:
+      title: State overrides
+      $ref: '#/components/schemas/StateOverrides'
+    calls:


I guess blockOverride and calls should be mandatory variables here and stateOverrides optional?

No I'd leave blockOverride also optional. For a multi-call single-block simulation users shouldn't need to override block fields.

I personally prefer empty arrays over missing/null arrays. I find it leads to improved developer ergonomics (no need for conditional branching on presence, you can just map/loop over the array). Functionally, I think both are fine though.

if blockoverride is empty, which block is used? latest I guess?

The third parameter is a "block tag" which serves to determine the fallback block fields (as well as base state). Something we have to consider is when there are multiple batches of calls (without block override fields):

They all inherit the "base block" fields. In this case if user provides 2 batches without fields, it will be equivalent to appending the second batch of calls to the first.

We magically update the fields of future batches

I'm in favor of 1. I think we should keep this method dumb rather than do something unexpected for users.

src/schemas/execute.yaml

s1na · 2023-05-31T13:19:28Z

So I've implemented this spec in geth: https://github.com/s1na/go-ethereum/tree/multicall. Now planning on writing more tests.

Thinking more on this, I think I would like to see the response be shaped as close to an eth_getBlockByNumber call[...]This would make it so existing tools would be able to easily parse the response with only minor tweaks to extract the new information?

Sorry for voicing my opinion on this with delay, but I think this method is different enough from existing ones that tooling will have to write new logic for it. Repeating request fields in the response seems wasteful.

MicahZoltu · 2023-06-01T05:24:00Z

I agree that this will require some tooling to be written. However, it would be very nice to just have something like MulticallBlock extends Block { ... } rather than having to define a totally unique structure. Same for transactions, receipts, etc.

s1na · 2023-06-01T09:32:34Z

However, it would be very nice to just have something like MulticallBlock extends Block { ... } rather than having to define a totally unique structure.

If we want to use existing structures, IMO the result should be [][]Receipt. After all receipts are the outcome of a tx. If all of the information there can easily returned by clients is another question.

MicahZoltu · 2023-06-01T10:23:23Z

In many cases, I think the client will be filling in much of the block data and it feels like it is a good idea to let the user know what was filled in. For example, the base fee will be calculated and set by the client and we should be returning that to the user in the response somewhere.

docs/multicall-notes.md

Co-authored-by: Micah Zoltu <micah@zoltu.net>

docs/multicall-notes.md

MicahZoltu · 2023-10-30T13:26:06Z

docs/multicall-notes.md

+| nonce | Defaults to correct nonce |
+| to | null |
+| from | 0x0 |
+| gas limit | Remaining gas in the curent block |


We should make it clear that this is calculated just before the transaction is executed, after gas has been consumed by previous transactions in the block.

Co-authored-by: Micah Zoltu <micah@zoltu.net>

docs/multicall-notes.md

MicahZoltu · 2023-11-04T12:18:51Z

Superceded by #484.

epheph · 2023-11-27T14:38:35Z

Closing in favor of #484

epheph added 2 commits February 20, 2023 15:26

Add eth_multicall

5556499

Remove chain-id from call-type (no replay protection necessary for ca…

95822af

…lls, does not change execution)

MarekM25 mentioned this pull request Feb 23, 2023

Analyze & prototype eth_multicall RPC endpoint NethermindEth/nethermind#5333

Closed

epheph added 6 commits March 27, 2023 14:52

Merge remote-tracking branch 'origin/main' into eth_multicall

602d2bf

Add tests

6ce85e2

Add test error case

2fcbf1d

Fix account override adding address to target

046502a

Add example resetting WETH to 1ETH totalSupply and assigning that 1ET…

e4ec926

…H balance to a random account

Add another account delta

0fbdf40

OlegJakushkin mentioned this pull request Apr 3, 2023

Add eth_simulate , support array of eth_call for simulation across multiple blocks NethermindEth/nethermind#5530

Merged

28 tasks

epheph added 2 commits April 3, 2023 15:26

Add example resetting WETH to 1ETH totalSupply and assigning that 1ET…

ccfc70b

…H balance to a random account

block override with simple contract that just returns block number

7590b80

Change response to keep the block-based structure similar to request …

bbf34e8

…itself

s1na suggested changes May 17, 2023

View reviewed changes

src/eth/execute.yaml Outdated Show resolved Hide resolved

src/schemas/execute.yaml Outdated Show resolved Hide resolved

MicahZoltu mentioned this pull request May 18, 2023

Come up with uniform versioning scheme for methods #408

Open

KillariDev reviewed May 25, 2023

View reviewed changes

src/schemas/execute.yaml Outdated Show resolved Hide resolved

src/schemas/execute.yaml Outdated Show resolved Hide resolved

KillariDev reviewed May 25, 2023

View reviewed changes

src/schemas/execute.yaml Outdated Show resolved Hide resolved

src/schemas/execute.yaml Outdated Show resolved Hide resolved

MicahZoltu reviewed Oct 24, 2023

View reviewed changes

docs/multicall-notes.md Outdated Show resolved Hide resolved

MicahZoltu reviewed Oct 24, 2023

View reviewed changes

docs/multicall-notes.md Outdated Show resolved Hide resolved

MicahZoltu reviewed Oct 24, 2023

View reviewed changes

docs/multicall-notes.md Outdated Show resolved Hide resolved

MicahZoltu reviewed Oct 24, 2023

View reviewed changes

docs/multicall-notes.md Outdated Show resolved Hide resolved

KillariDev and others added 9 commits October 25, 2023 02:41

Update docs/multicall-notes.md

be26261

Co-authored-by: Micah Zoltu <micah@zoltu.net>

Update docs/multicall-notes.md

0630df2

Co-authored-by: Micah Zoltu <micah@zoltu.net>

Update docs/multicall-notes.md

e861a4a

Co-authored-by: Micah Zoltu <micah@zoltu.net>

Update docs/multicall-notes.md

cba6d4e

Co-authored-by: Micah Zoltu <micah@zoltu.net>

Update docs/multicall-notes.md

2196a29

Co-authored-by: Micah Zoltu <micah@zoltu.net>

Update docs/multicall-notes.md

40daadb

Co-authored-by: Micah Zoltu <micah@zoltu.net>

fix according to comments

d831c5f

add new error for gas limit and block number limits

995acb2

add failures comment

88f1f7b

MicahZoltu reviewed Oct 30, 2023

View reviewed changes

docs/multicall-notes.md Outdated Show resolved Hide resolved

MicahZoltu reviewed Oct 30, 2023

View reviewed changes

docs/multicall-notes.md Outdated Show resolved Hide resolved

KillariDev reviewed Oct 30, 2023

View reviewed changes

docs/multicall-notes.md Outdated Show resolved Hide resolved

MicahZoltu reviewed Oct 30, 2023

View reviewed changes

KillariDev and others added 3 commits October 30, 2023 15:33

Update docs/multicall-notes.md

235c40c

Update docs/multicall-notes.md

e817b74

Co-authored-by: Micah Zoltu <micah@zoltu.net>

Update docs/multicall-notes.md

78b3a92

Co-authored-by: Micah Zoltu <micah@zoltu.net>

KillariDev reviewed Oct 30, 2023

View reviewed changes

docs/multicall-notes.md Outdated Show resolved Hide resolved

Update docs/multicall-notes.md

cb5d419

MicahZoltu mentioned this pull request Oct 31, 2023

eth_call underspecified return error message #463

Open

KillariDev added 2 commits November 3, 2023 05:24

add info about overriding default values and restrictions on them

8d3b4d4

add note

5f09367

KillariDev mentioned this pull request Nov 4, 2023

add eth_simulateV1 #484

Open

epheph closed this Nov 27, 2023

banteg mentioned this pull request Apr 8, 2024

implement eth_simulate api erigontech/erigon#9881

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add eth_multicall , support array of eth_call for simulation across multiple blocks #383

Add eth_multicall , support array of eth_call for simulation across multiple blocks #383

epheph commented Feb 20, 2023

epheph commented Feb 20, 2023

s1na commented Apr 7, 2023

epheph commented Apr 19, 2023

s1na commented Apr 20, 2023

epheph commented Apr 20, 2023

MicahZoltu commented Apr 25, 2023

KillariDev commented May 10, 2023

epheph commented May 11, 2023

KillariDev May 25, 2023

s1na May 31, 2023

MicahZoltu May 31, 2023

KillariDev Jun 1, 2023

s1na Jun 1, 2023

s1na commented May 31, 2023

MicahZoltu commented Jun 1, 2023

s1na commented Jun 1, 2023

MicahZoltu commented Jun 1, 2023

MicahZoltu Oct 30, 2023

MicahZoltu commented Nov 4, 2023

epheph commented Nov 27, 2023

Add eth_multicall , support array of eth_call for simulation across multiple blocks #383

Add eth_multicall , support array of eth_call for simulation across multiple blocks #383

Conversation

epheph commented Feb 20, 2023

epheph commented Feb 20, 2023

s1na commented Apr 7, 2023

epheph commented Apr 19, 2023

s1na commented Apr 20, 2023

epheph commented Apr 20, 2023

MicahZoltu commented Apr 25, 2023

KillariDev commented May 10, 2023

epheph commented May 11, 2023

KillariDev May 25, 2023

Choose a reason for hiding this comment

s1na May 31, 2023

Choose a reason for hiding this comment

MicahZoltu May 31, 2023

Choose a reason for hiding this comment

KillariDev Jun 1, 2023

Choose a reason for hiding this comment

s1na Jun 1, 2023

Choose a reason for hiding this comment

s1na commented May 31, 2023

MicahZoltu commented Jun 1, 2023

s1na commented Jun 1, 2023

MicahZoltu commented Jun 1, 2023

MicahZoltu Oct 30, 2023

Choose a reason for hiding this comment

MicahZoltu commented Nov 4, 2023

epheph commented Nov 27, 2023