feat/statetest: Implement StateTestOnly, Change Blob Tx 4844 Tests to StateTest, Use latest geth master

README.md

@@ -68,10 +68,11 @@ The following transition tools are supported by the framework:
 
 ### Upcoming EIP Development
 
-Generally, specific `t8n` implementations and branches must be used when developing tests for upcoming EIPs (last updated 2023-10-19):
+Generally, specific `t8n` implementations and branches must be used when developing tests for upcoming EIPs.
 
-- Cancun related EIPs (4844, 4788, 1153, 6780) - [lightclient/go-ethereum@devnet-10](https://github.com/lightclient/go-ethereum/tree/devnet-10)
-- EOF tests - [ethereum/evmone@master](https://github.com/ethereum/evmone)
+We use named reference tags to point to the specific version of the `t8n` implementation that needs to be used fill the tests.
+
+All current tags, their t8n implementation and branch they point to, are listed in [evm-config.yaml](evm-config.yaml).
 
 ## Getting Started
 

docs/CHANGELOG.md

@@ -8,6 +8,17 @@ Test fixtures for use by clients are available for each release on the [Github r
 
 ### 🧪 Test Cases
 
+- 🔀 BlockchainTests converted to StateTest (also automatically generate BlockchainTest) ([#368](https://github.com/ethereum/execution-spec-tests/pull/368)):
+  - tests/cancun/eip4844_blobs/test_blob_txs.py::test_invalid_normal_gas
+  - tests/cancun/eip4844_blobs/test_blob_txs.py::test_insufficient_balance_blob_tx
+  - tests/cancun/eip4844_blobs/test_blob_txs.py::test_invalid_tx_blob_count
+  - tests/cancun/eip4844_blobs/test_blob_txs.py::test_invalid_blob_hash_versioning_single_tx
+  - tests/cancun/eip4844_blobs/test_blob_txs.py::test_blob_tx_attribute_opcodes
+  - tests/cancun/eip4844_blobs/test_blob_txs.py::test_blob_tx_attribute_value_opcode
+  - tests/cancun/eip4844_blobs/test_blob_txs.py::test_blob_tx_attribute_calldata_opcodes
+  - tests/cancun/eip4844_blobs/test_blob_txs.py::test_blob_tx_attribute_gasprice_opcode
+  - tests/cancun/eip4844_blobs/test_blob_txs.py::test_blob_type_tx_pre_fork
+
 ### 🛠️ Framework
 
 - ✨ Add a `--single-fixture-per-file` flag to generate one fixture JSON file per test case ([#331](https://github.com/ethereum/execution-spec-tests/pull/331)).
@@ -16,12 +27,14 @@ Test fixtures for use by clients are available for each release on the [Github r
 - ✨ Fork objects used to write tests can now be compared using the `>`, `>=`, `<`, `<=` operators, to check for a fork being newer than, newer than or equal, older than, older than or equal, respectively when compared against other fork ([#367](https://github.com/ethereum/execution-spec-tests/pull/367))
 - 💥 Removed `--enable-hive` parameter, now all test types are generated by default ([#358](https://github.com/ethereum/execution-spec-tests/pull/358))
 - 💥 `StateTest`, spec format used to write tests, is now limited to a single transaction per test ([#361](https://github.com/ethereum/execution-spec-tests/pull/361))
+- ✨ `StateTestOnly`, spec format is now available and its only difference with `StateTest` is that it does not produce a `BlockchainTest` ([#368](https://github.com/ethereum/execution-spec-tests/pull/368))
 
 ### 🔧 EVM Tools
 
 ### 📋 Misc
 
 - 🔀 Docs: Update `t8n` tool branch to fill tests for development features in the [readme](https://github.com/ethereum/execution-spec-test) ([#338](https://github.com/ethereum/execution-spec-tests/pull/338)).
+- 🔀 Filling tool: Updated filling tool to go-ethereum@master ([#368](https://github.com/ethereum/execution-spec-tests/pull/368))
 
 ## Breaking Changes
 

evm-config.yaml

@@ -4,5 +4,5 @@ main:
   ref: master
 develop:
   impl: geth
-  repo: lightclient/go-ethereum
-  ref: devnet-10
+  repo: ethereum/go-ethereum
+  ref: master

src/ethereum_test_forks/base_fork.py

@@ -143,6 +143,14 @@ def header_beacon_root_required(cls, block_number: int, timestamp: int) -> bool:
         """
         pass
 
+    @classmethod
+    @abstractmethod
+    def blob_gas_per_blob(cls, block_number: int, timestamp: int) -> int:
+        """
+        Returns the amount of blob gas used per each blob for a given fork.
+        """
+        pass
+
     @classmethod
     @abstractmethod
     def get_reward(cls, block_number: int = 0, timestamp: int = 0) -> int:

src/ethereum_test_forks/forks/forks.py

@@ -72,6 +72,13 @@ def header_blob_gas_used_required(cls, block_number: int = 0, timestamp: int = 0
         """
         return False
 
+    @classmethod
+    def blob_gas_per_blob(cls, block_number: int, timestamp: int) -> int:
+        """
+        Returns the amount of blob gas used per each blob for a given fork.
+        """
+        return 0
+
     @classmethod
     def engine_new_payload_version(
         cls, block_number: int = 0, timestamp: int = 0
@@ -370,6 +377,13 @@ def header_beacon_root_required(cls, block_number: int = 0, timestamp: int = 0)
         """
         return True
 
+    @classmethod
+    def blob_gas_per_blob(cls, block_number: int, timestamp: int) -> int:
+        """
+        Blobs are enabled started from Cancun.
+        """
+        return 2**17
+
     @classmethod
     def tx_types(cls, block_number: int = 0, timestamp: int = 0) -> List[int]:
         """

src/ethereum_test_tools/spec/__init__.py

@@ -6,9 +6,9 @@
 from .base.base_test import BaseFixture, BaseTest, TestSpec, verify_post_alloc
 from .blockchain.blockchain_test import BlockchainTest, BlockchainTestFiller, BlockchainTestSpec
 from .fixture_collector import FixtureCollector, TestInfo
-from .state.state_test import StateTest, StateTestFiller, StateTestSpec
+from .state.state_test import StateTest, StateTestFiller, StateTestOnly, StateTestSpec
 
-SPEC_TYPES: List[Type[BaseTest]] = [BlockchainTest, StateTest]
+SPEC_TYPES: List[Type[BaseTest]] = [BlockchainTest, StateTest, StateTestOnly]
 
 __all__ = (
     "SPEC_TYPES",
@@ -20,6 +20,7 @@
     "FixtureCollector",
     "StateTest",
     "StateTestFiller",
+    "StateTestOnly",
     "StateTestSpec",
     "TestInfo",
     "TestSpec",

src/ethereum_test_tools/spec/blockchain/blockchain_test.py

@@ -79,6 +79,15 @@ def apply_new_parent(env: Environment, new_parent: FixtureHeader) -> "Environmen
     return env
 
 
+def count_blobs(txs: List[Transaction]) -> int:
+    """
+    Returns the number of blobs in a list of transactions.
+    """
+    return sum(
+        [len(tx.blob_versioned_hashes) for tx in txs if tx.blob_versioned_hashes is not None]
+    )
+
+
 @dataclass(kw_only=True)
 class BlockchainTest(BaseTest):
     """
@@ -229,6 +238,15 @@ def generate_block_data(
             environment=env,
         )
 
+        # One special case of the invalid transactions is the blob gas used, since this value
+        # is not included in the transition tool result, but it is included in the block header,
+        # and some clients check it before executing the block by simply counting the type-3 txs,
+        # we need to set the correct value by default.
+        if (
+            blob_gas_per_blob := fork.blob_gas_per_blob(Number(env.number), Number(env.timestamp))
+        ) > 0:
+            header.blob_gas_used = blob_gas_per_blob * count_blobs(txs)
+
         if block.header_verify is not None:
             # Verify the header after transition tool processing.
             header.verify(block.header_verify)

src/ethereum_test_tools/spec/blockchain/types.py

@@ -410,14 +410,19 @@ def verify(self, baseline: Header):
                 assert baseline_value is not Header.REMOVE_FIELD, "invalid baseline header"
                 value = getattr(self, field_name)
                 if baseline_value is Header.EMPTY_FIELD:
-                    assert value is None, f"invalid header field {header_field}"
+                    assert (
+                        value is None
+                    ), f"invalid header field {field_name}, got {value}, want None"
                     continue
                 metadata = header_field.metadata
                 field_metadata = metadata.get("source")
                 # type check is performed on collect()
                 if field_metadata.parse_type is not None:  # type: ignore
                     baseline_value = field_metadata.parse_type(baseline_value)  # type: ignore
-                assert value == baseline_value, f"invalid header field {header_field}"
+                assert value == baseline_value, (
+                    f"invalid header field ({field_name}) value, "
+                    + f"got {value}, want {baseline_value}"
+                )
 
     def build(
         self,

src/ethereum_test_tools/spec/state/state_test.py

@@ -13,10 +13,12 @@
 from ...common.json import to_json
 from ..base.base_test import BaseFixture, BaseTest, verify_post_alloc
 from ..blockchain.blockchain_test import Block, BlockchainTest
+from ..blockchain.types import Header
 from ..debugging import print_traces
 from .types import Fixture, FixtureForkPost
 
 BEACON_ROOTS_ADDRESS = Address(0x000F3DF6D732807EF1319FB7B8BB8522D0BEAC02)
+TARGET_BLOB_GAS_PER_BLOCK = 393216
 
 
 @dataclass(kw_only=True)
@@ -30,6 +32,8 @@ class StateTest(BaseTest):
     post: Mapping
     tx: Transaction
     engine_api_error_code: Optional[EngineAPIError] = None
+    blockchain_test_header_verify: Optional[Header] = None
+    blockchain_test_rlp_modifier: Optional[Header] = None
     tag: str = ""
     chain_id: int = 1
 
@@ -58,12 +62,22 @@ def _generate_blockchain_genesis_environment(self) -> Environment:
         genesis_env = copy(self.env)
 
         # Modify values to the proper values for the genesis block
+        # TODO: All of this can be moved to a new method in `Fork`
         genesis_env.withdrawals = None
         genesis_env.beacon_root = None
         genesis_env.number = Number(genesis_env.number) - 1
         assert (
             genesis_env.number >= 0
         ), "genesis block number cannot be negative, set state test env.number to 1"
+        if genesis_env.excess_blob_gas:
+            # The excess blob gas environment value means the value of the context (block header)
+            # where the transaction is executed. In a blockchain test, we need to indirectly
+            # set the excess blob gas by setting the excess blob gas of the genesis block
+            # to the expected value plus the TARGET_BLOB_GAS_PER_BLOCK, which is the value
+            # that will be subtracted from the excess blob gas when the first block is mined.
+            genesis_env.excess_blob_gas = (
+                Number(genesis_env.excess_blob_gas) + TARGET_BLOB_GAS_PER_BLOCK
+            )
 
         return genesis_env
 
@@ -83,6 +97,9 @@ def _generate_blockchain_blocks(self) -> List[Block]:
                 beacon_root=self.env.beacon_root,
                 txs=[self.tx],
                 ommers=[],
+                exception=self.tx.error,
+                header_verify=self.blockchain_test_header_verify,
+                rlp_modifier=self.blockchain_test_rlp_modifier,
             )
         ]
 
@@ -186,5 +203,25 @@ def generate(
         raise Exception(f"Unknown fixture format: {self.fixture_format}")
 
 
+class StateTestOnly(StateTest):
+    """
+    StateTest filler that only generates a state test fixture.
+    """
+
+    @classmethod
+    def pytest_parameter_name(cls) -> str:
+        """
+        Returns the parameter name used to identify this filler in a test.
+        """
+        return "state_test_only"
+
+    @classmethod
+    def fixture_formats(cls) -> List[FixtureFormats]:
+        """
+        Returns a list of fixture formats that can be output to the test spec.
+        """
+        return [FixtureFormats.STATE_TEST]
+
+
 StateTestSpec = Callable[[str], Generator[StateTest, None, None]]
 StateTestFiller = Type[StateTest]

src/ethereum_test_tools/spec/state/types.py

@@ -19,12 +19,86 @@
     Environment,
     Hash,
     HexNumber,
+    Number,
+    NumberConvertible,
     Transaction,
     ZeroPaddedHexNumber,
 )
 from ..base.base_test import BaseFixture
 
 
+@dataclass(kw_only=True)
+class FixtureEnvironment:
+    """
+    Type used to describe the environment of a state test.
+    """
+
+    coinbase: FixedSizeBytesConvertible = field(
+        default="0x2adc25665018aa1fe0e6bc666dac8fc2697ff9ba",
+        json_encoder=JSONEncoder.Field(
+            name="currentCoinbase",
+            cast_type=Address,
+        ),
+    )
+    gas_limit: NumberConvertible = field(
+        default=100000000000000000,
+        json_encoder=JSONEncoder.Field(
+            name="currentGasLimit",
+            cast_type=Number,
+        ),
+    )
+    number: NumberConvertible = field(
+        default=1,
+        json_encoder=JSONEncoder.Field(
+            name="currentNumber",
+            cast_type=Number,
+        ),
+    )
+    timestamp: NumberConvertible = field(
+        default=1000,
+        json_encoder=JSONEncoder.Field(
+            name="currentTimestamp",
+            cast_type=Number,
+        ),
+    )
+    prev_randao: Optional[NumberConvertible] = field(
+        default=None,
+        json_encoder=JSONEncoder.Field(
+            name="currentRandom",
+            cast_type=Number,
+        ),
+    )
+    difficulty: Optional[NumberConvertible] = field(
+        default=None,
+        json_encoder=JSONEncoder.Field(
+            name="currentDifficulty",
+            cast_type=Number,
+        ),
+    )
+    base_fee: Optional[NumberConvertible] = field(
+        default=None,
+        json_encoder=JSONEncoder.Field(
+            name="currentBaseFee",
+            cast_type=Number,
+        ),
+    )
+    excess_blob_gas: Optional[NumberConvertible] = field(
+        default=None,
+        json_encoder=JSONEncoder.Field(
+            name="currentExcessBlobGas",
+            cast_type=Number,
+        ),
+    )
+
+    @classmethod
+    def from_env(cls, env: Environment) -> "FixtureEnvironment":
+        """
+        Returns a FixtureEnvironment from an Environment.
+        """
+        kwargs = {field.name: getattr(env, field.name) for field in fields(cls)}
+        return cls(**kwargs)
+
+
 @dataclass(kw_only=True)
 class FixtureTransaction:
     """
@@ -195,6 +269,7 @@ def collect(
             state_root=state_root,
             logs_hash=logs_hash,
             tx_bytes=transaction.serialized_bytes(),
+            expected_exception=transaction.error,
             indexes=indexes,
         )
 
@@ -207,6 +282,7 @@ class Fixture(BaseFixture):
 
     env: Environment = field(
         json_encoder=JSONEncoder.Field(
+            cast_type=FixtureEnvironment.from_env,
             to_json=True,
         ),
     )