Use hivechain to generate test chain

[fjl]

In this PR, I'm trying to remove the chain generator. We have a dedicated infrastructure to create an 'interesting' test chain with hivechain, and it's good to reuse. The chain generation code is also messy to maintain because it depends on go-ethereum core very deeply. Best to deal with this in one place only.

[fjl]

Damn, I based this on my outdated fork. This rebase is going to suck.

[fjl]

Most tests work. Just 6 more to fix.

[fjl]

This should be ready now.

[fjl]

I've merged the cancun PR into this one in order to be able to regenerate all tests - execution-apis already had these tests included.

[fjl]

Tests update with this PR: https://github.com/ethereum/execution-apis/pull/513/files

[lightclient]

Great work overall, just a few questions / comments

[lightclient]

Could you please take a look at this patch and apply if you agree? It should resolve all the speccheck issues:

diff --git a/cmd/speccheck/check.go b/cmd/speccheck/check.go
index 438cb28..586442f 100644
--- a/cmd/speccheck/check.go
+++ b/cmd/speccheck/check.go
@@ -37,14 +37,18 @@ func checkSpec(methods map[string]*methodSchema, rts []*roundTrip, re *regexp.Re
 				return fmt.Errorf("missing required parameter %s.param[%d]", rt.method, i)
 			}
 			if err := validate(&method.params[i].schema, rt.params[i], fmt.Sprintf("%s.param[%d]", rt.method, i)); err != nil {
-				return fmt.Errorf("unable to validate parameter: %s", err)
+				return fmt.Errorf("unable to validate parameter in %s: %s", rt.name, err)
 			}
 		}
-		if err := validate(&method.result.schema, rt.response, fmt.Sprintf("%s.result", rt.method)); err != nil {
+		if rt.response.Result == nil && rt.response.Error != nil {
+			// skip validation of errors, they haven't been standardized
+			continue
+		}
+		if err := validate(&method.result.schema, rt.response.Result, fmt.Sprintf("%s.result", rt.method)); err != nil {
 			// Print out the value and schema if there is an error to further debug.
 			buf, _ := json.Marshal(method.result.schema)
 			fmt.Println(string(buf))
-			fmt.Println(string(rt.response))
+			fmt.Println(string(rt.response.Result))
 			fmt.Println()
 			return fmt.Errorf("invalid result %s\n%#v", rt.name, err)
 		}
diff --git a/cmd/speccheck/roundtrips.go b/cmd/speccheck/roundtrips.go
index 57f1d39..eb7b473 100644
--- a/cmd/speccheck/roundtrips.go
+++ b/cmd/speccheck/roundtrips.go
@@ -30,7 +30,7 @@ type roundTrip struct {
 	method   string
 	name     string
 	params   [][]byte
-	response []byte
+	response *jsonrpcMessage
 }
 
 // readRtts walks a root directory and parses round trip HTTP exchanges
@@ -99,7 +99,7 @@ func readTest(testname string, filename string) ([]*roundTrip, error) {
 			if err != nil {
 				return nil, fmt.Errorf("unable to parse params: %s %v", err, req.Params)
 			}
-			rts = append(rts, &roundTrip{req.Method, testname, params, resp.Result})
+			rts = append(rts, &roundTrip{req.Method, testname, params, &resp})
 			req = nil
 		default:
 			return nil, fmt.Errorf("invalid line in test: %s", line)
diff --git a/testgen/generators.go b/testgen/generators.go
index 13df56e..374f6a7 100644
--- a/testgen/generators.go
+++ b/testgen/generators.go
@@ -60,8 +60,6 @@ var AllMethods = []MethodTests{
 	EthBlockNumber,
 	EthGetBlockByNumber,
 	EthGetBlockByHash,
-	EthGetHeaderByNumber,
-	EthGetHeaderByHash,
 	EthGetProof,
 	EthChainID,
 	EthGetBalance,
@@ -87,6 +85,10 @@ var AllMethods = []MethodTests{
 	DebugGetRawReceipts,
 	DebugGetRawTransaction,
 
+	// -- get header by xxx APIs are not currently in the standard spec
+	// EthGetHeaderByNumber,
+	// EthGetHeaderByHash,
+
 	// -- gas price tests are disabled because of non-determinism
 	// EthGasPrice,
 	// EthMaxPriorityFeePerGas,
@@ -688,7 +690,7 @@ var EthCreateAccessList = MethodTests{
 					"from":     sender,
 					"to":       emitContract,
 					"nonce":    hexutil.Uint64(nonce),
-					"gasLimit": hexutil.Uint64(60000),
+					"gas":      hexutil.Uint64(60000),
 					"gasPrice": (*hexutil.Big)(gasprice),
 					"input":    "0x010203040506",
 				}
@@ -722,7 +724,7 @@ This invocation uses EIP-1559 fields to specify the gas price.`,
 					"from":                 sender,
 					"to":                   emitContract,
 					"nonce":                hexutil.Uint64(nonce),
-					"gasLimit":             hexutil.Uint64(60000),
+					"gas":                  hexutil.Uint64(60000),
 					"maxFeePerGas":         (*hexutil.Big)(gasprice),
 					"maxPriorityFeePerGas": (*hexutil.Big)(big.NewInt(3)),
 					"input":                "0x010203040506",

[fjl]

I have applied the patch.

[lightclient]

Thanks so much!