From 7f699bf5949de9d5ae21c9b8435d1da7dd06cb35 Mon Sep 17 00:00:00 2001 From: Etan Kissling Date: Thu, 8 Feb 2024 09:50:39 +0100 Subject: [PATCH 001/130] Rename `get_safe_execution_payload_hash` -> `*_block_hash` The name `get_safe_execution_payload_hash` is misleading, as it returns the execution block hash. There is no concept of a payload hash. Rename the function, and also update some documentation for clarity. --- fork_choice/safe-block.md | 6 +++--- specs/bellatrix/fork-choice.md | 2 +- specs/bellatrix/validator.md | 4 ++-- tests/README.md | 6 +++--- 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/fork_choice/safe-block.md b/fork_choice/safe-block.md index b76285b3a8..d4af9060d0 100644 --- a/fork_choice/safe-block.md +++ b/fork_choice/safe-block.md @@ -7,7 +7,7 @@ - [Introduction](#introduction) - [`get_safe_beacon_block_root`](#get_safe_beacon_block_root) -- [`get_safe_execution_payload_hash`](#get_safe_execution_payload_hash) +- [`get_safe_execution_block_hash`](#get_safe_execution_block_hash) @@ -31,10 +31,10 @@ def get_safe_beacon_block_root(store: Store) -> Root: *Note*: Currently safe block algorithm simply returns `store.justified_checkpoint.root` and is meant to be improved in the future. -## `get_safe_execution_payload_hash` +## `get_safe_execution_block_hash` ```python -def get_safe_execution_payload_hash(store: Store) -> Hash32: +def get_safe_execution_block_hash(store: Store) -> Hash32: safe_block_root = get_safe_beacon_block_root(store) safe_block = store.blocks[safe_block_root] diff --git a/specs/bellatrix/fork-choice.md b/specs/bellatrix/fork-choice.md index 7bf607d6e9..17fb8e024d 100644 --- a/specs/bellatrix/fork-choice.md +++ b/specs/bellatrix/fork-choice.md @@ -75,7 +75,7 @@ As per EIP-3675, before a post-transition block is finalized, `notify_forkchoice ##### `safe_block_hash` The `safe_block_hash` parameter MUST be set to return value of -[`get_safe_execution_payload_hash(store: Store)`](../../fork_choice/safe-block.md#get_safe_execution_payload_hash) function. +[`get_safe_execution_block_hash(store: Store)`](../../fork_choice/safe-block.md#get_safe_execution_block_hash) function. ##### `should_override_forkchoice_update` diff --git a/specs/bellatrix/validator.md b/specs/bellatrix/validator.md index 2900bd1f02..cb9dda05d6 100644 --- a/specs/bellatrix/validator.md +++ b/specs/bellatrix/validator.md @@ -120,8 +120,8 @@ To obtain an execution payload, a block proposer building a block on top of a `s 1. Set `payload_id = prepare_execution_payload(state, pow_chain, safe_block_hash, finalized_block_hash, suggested_fee_recipient, execution_engine)`, where: * `state` is the state object after applying `process_slots(state, slot)` transition to the resulting state of the parent block processing * `pow_chain` is a `Dict[Hash32, PowBlock]` dictionary that abstractly represents all blocks in the PoW chain with block hash as the dictionary key - * `safe_block_hash` is the return value of the `get_safe_execution_payload_hash(store: Store)` function call - * `finalized_block_hash` is the hash of the latest finalized execution payload (`Hash32()` if none yet finalized) + * `safe_block_hash` is the return value of the `get_safe_execution_block_hash(store: Store)` function call + * `finalized_block_hash` is the block hash of the latest finalized execution payload (`Hash32()` if none yet finalized) * `suggested_fee_recipient` is the value suggested to be used for the `fee_recipient` field of the execution payload diff --git a/tests/README.md b/tests/README.md index dbd2b31de2..8c281155c5 100644 --- a/tests/README.md +++ b/tests/README.md @@ -121,7 +121,7 @@ of return values. Here we add two values, the string `'pre'` and the initial sta ``` The state contains the last block, which is necessary for building up the next block (every block needs to -have the hash of the previous one in a blockchain). +have the root of the previous one in a blockchain). ```python signed_block = state_transition_and_sign_block(spec, state, block) @@ -291,8 +291,8 @@ not execute EVM programs or store user data. It exists to provide a secure sourc information about the latest verified block hash of the execution layer. For every slot a validator is randomly selected as the proposer. The proposer proposes a block -for the current head of the consensus layer chain (built on the previous block). That block -includes the hash of the proposed new head of the execution layer. +for the current head of the consensus layer chain (built on the previous block). That block +includes the block hash of the proposed new head of the execution layer. For every slot there is also a randomly selected committee of validators that needs to vote whether the new consensus layer block is valid, which requires the proposed head of the execution chain to From 94535a6a58172266f1c7ed5063427bbd6be5955a Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Mon, 13 May 2024 14:35:28 +0300 Subject: [PATCH 002/130] Add Electra merkle proof test vectors --- tests/generators/merkle_proof/main.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/generators/merkle_proof/main.py b/tests/generators/merkle_proof/main.py index b7d30fe9e4..69500137ab 100644 --- a/tests/generators/merkle_proof/main.py +++ b/tests/generators/merkle_proof/main.py @@ -1,4 +1,4 @@ -from eth2spec.test.helpers.constants import DENEB, EIP7594 +from eth2spec.test.helpers.constants import DENEB, ELECTRA, EIP7594 from eth2spec.gen_helpers.gen_from_tests.gen import run_state_test_generators, combine_mods @@ -9,10 +9,12 @@ _new_eip7594_mods = {key: 'eth2spec.test.eip7594.merkle_proof.test_' + key for key in [ 'single_merkle_proof', ]} + electra_mods = deneb_mods eip_7594_mods = combine_mods(_new_eip7594_mods, deneb_mods) all_mods = { DENEB: deneb_mods, + ELECTRA: electra_mods, EIP7594: eip_7594_mods, } From 01a2c7ba0755e79731b42ea1af59b34e898000c9 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Mon, 13 May 2024 15:27:52 +0300 Subject: [PATCH 003/130] Add `test_fork_has_compounding_withdrawal_credential` test case --- .../test/electra/fork/test_electra_fork_basic.py | 14 ++++++++++++++ .../pyspec/eth2spec/test/helpers/electra/fork.py | 2 ++ 2 files changed, 16 insertions(+) diff --git a/tests/core/pyspec/eth2spec/test/electra/fork/test_electra_fork_basic.py b/tests/core/pyspec/eth2spec/test/electra/fork/test_electra_fork_basic.py index 3bd6350b34..15961f40e1 100644 --- a/tests/core/pyspec/eth2spec/test/electra/fork/test_electra_fork_basic.py +++ b/tests/core/pyspec/eth2spec/test/electra/fork/test_electra_fork_basic.py @@ -80,3 +80,17 @@ def test_fork_random_misc_balances(spec, phases, state): @with_meta_tags(ELECTRA_FORK_TEST_META_TAGS) def test_fork_random_large_validator_set(spec, phases, state): yield from run_fork_test(phases[ELECTRA], state) + + +@with_phases(phases=[DENEB], other_phases=[ELECTRA]) +@spec_test +@with_state +@with_meta_tags(ELECTRA_FORK_TEST_META_TAGS) +def test_fork_has_compounding_withdrawal_credential(spec, phases, state): + post_spec = phases[ELECTRA] + validator = state.validators[0] + state.balances[0] = post_spec.MIN_ACTIVATION_BALANCE + 1 + validator.withdrawal_credentials = post_spec.COMPOUNDING_WITHDRAWAL_PREFIX + validator.withdrawal_credentials[1:] + post_state = yield from run_fork_test(post_spec, state) + + assert len(post_state.pending_balance_deposits) > 0 diff --git a/tests/core/pyspec/eth2spec/test/helpers/electra/fork.py b/tests/core/pyspec/eth2spec/test/helpers/electra/fork.py index 39a43a5233..0067a8cc07 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/electra/fork.py +++ b/tests/core/pyspec/eth2spec/test/helpers/electra/fork.py @@ -63,3 +63,5 @@ def run_fork_test(post_spec, pre_state): assert post_state.fork.epoch == post_spec.get_current_epoch(post_state) yield 'post', post_state + + return post_state From 8b264704aa937d2cf8ee3bd42ed32f48a4428c66 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Mon, 13 May 2024 15:37:32 +0300 Subject: [PATCH 004/130] Add `test_fork_pre_activation` test case --- .../test/electra/fork/test_electra_fork_basic.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/tests/core/pyspec/eth2spec/test/electra/fork/test_electra_fork_basic.py b/tests/core/pyspec/eth2spec/test/electra/fork/test_electra_fork_basic.py index 15961f40e1..852521a32b 100644 --- a/tests/core/pyspec/eth2spec/test/electra/fork/test_electra_fork_basic.py +++ b/tests/core/pyspec/eth2spec/test/electra/fork/test_electra_fork_basic.py @@ -82,6 +82,18 @@ def test_fork_random_large_validator_set(spec, phases, state): yield from run_fork_test(phases[ELECTRA], state) +@with_phases(phases=[DENEB], other_phases=[ELECTRA]) +@spec_test +@with_state +@with_meta_tags(ELECTRA_FORK_TEST_META_TAGS) +def test_fork_pre_activation(spec, phases, state): + post_spec = phases[ELECTRA] + state.validators[0].activation_epoch = spec.FAR_FUTURE_EPOCH + post_state = yield from run_fork_test(post_spec, state) + + assert len(post_state.pending_balance_deposits) > 0 + + @with_phases(phases=[DENEB], other_phases=[ELECTRA]) @spec_test @with_state From b848ca6dc7b5dc7d2b83e577ec13d79901067eff Mon Sep 17 00:00:00 2001 From: Csaba Kiraly Date: Tue, 28 May 2024 12:15:36 +0200 Subject: [PATCH 005/130] improved sampling description - describe sample selection - describe sample queries Signed-off-by: Csaba Kiraly --- specs/_features/eip7594/das-core.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/specs/_features/eip7594/das-core.md b/specs/_features/eip7594/das-core.md index dc50365b1e..df5ae81f59 100644 --- a/specs/_features/eip7594/das-core.md +++ b/specs/_features/eip7594/das-core.md @@ -239,7 +239,19 @@ To custody a particular column, a node joins the respective gossip subnet. Verif ## Peer sampling -A node SHOULD maintain a diverse set of peers for each column and each slot by verifying responsiveness to sample queries. At each slot, a node makes `SAMPLES_PER_SLOT` queries for samples from their peers via `DataColumnSidecarsByRoot` request. A node utilizes `get_custody_columns` helper to determine which peer(s) to request from. If a node has enough good/honest peers across all rows and columns, this has a high chance of success. +### Sample selection + +At each slot, a node SHOULD select at least `SAMPLES_PER_SLOT` column IDs for sampling. It is recommended to use uniform random selection without replacement based on local randomness. Sampling is considered successful if the node manages to retrieve all selected columns. + +### Sample queries + +A node SHOULD maintain a diverse set of peers for each column and each slot by verifying responsiveness to sample queries. + +A node SHOULD query for samples from their peers via `DataColumnSidecarsByRoot` request. A node utilizes `get_custody_columns` helper to determine which peer(s) it could request from. If more candidate peers are found, a node SHOULD randomize it's peer selection to distribute sample query load in the network. Nodes MAY use peer scoring to tune this selection (for example, by using weighted selection or by using a cut-off threshold). + +If a node already has a column because of custody, it is not required to send out queries for that column. + +If a node has enough good/honest peers across all columns, and the data is being made available, the above procedure has a high chance of success. ## Peer scoring From 8d332788b9aa70b4b307d58504ac02d0ced4862a Mon Sep 17 00:00:00 2001 From: Csaba Kiraly Date: Tue, 28 May 2024 12:24:25 +0200 Subject: [PATCH 006/130] clarify the use of LossyDAS Clarify that what matters is the false positive threshold, allowing different sampling strategies as protocol compliant behavior. Signed-off-by: Csaba Kiraly --- specs/_features/eip7594/das-core.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/specs/_features/eip7594/das-core.md b/specs/_features/eip7594/das-core.md index df5ae81f59..668b5095c5 100644 --- a/specs/_features/eip7594/das-core.md +++ b/specs/_features/eip7594/das-core.md @@ -243,6 +243,14 @@ To custody a particular column, a node joins the respective gossip subnet. Verif At each slot, a node SHOULD select at least `SAMPLES_PER_SLOT` column IDs for sampling. It is recommended to use uniform random selection without replacement based on local randomness. Sampling is considered successful if the node manages to retrieve all selected columns. +Alternatively, a node MAY use LossyDAS selecting more than `SAMPLES_PER_SLOT` columns while allowing some missing, respecting the same target false positive threshold (the probability of successful sampling of an unavailable block) as dictated by `SAMPLES_PER_SLOT`. The table below shows the number of samples and the number of allowed missing columns for this threshold. + +| Allowed missing (L) | 0| 1| 2| 3| 4| 5| 6| 7| 8| +|------------------------------- |--|--|--|--|--|--|--|--|--| +| Samples (S) for target threshold 5e-6 |16|20|23|26|29|32|34|37|39| + +Sampling is considered successful if any `S - L` columns are retrieved successfully. + ### Sample queries A node SHOULD maintain a diverse set of peers for each column and each slot by verifying responsiveness to sample queries. From 7b4d23c0ba4cbdc995ad673ead4d32e3eb242dd4 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 28 May 2024 22:57:52 +0800 Subject: [PATCH 007/130] fix toc --- specs/_features/eip7594/das-core.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/specs/_features/eip7594/das-core.md b/specs/_features/eip7594/das-core.md index 668b5095c5..a514358981 100644 --- a/specs/_features/eip7594/das-core.md +++ b/specs/_features/eip7594/das-core.md @@ -30,6 +30,8 @@ - [Column gossip](#column-gossip) - [Parameters](#parameters) - [Peer sampling](#peer-sampling) + - [Sample selection](#sample-selection) + - [Sample queries](#sample-queries) - [Peer scoring](#peer-scoring) - [Reconstruction and cross-seeding](#reconstruction-and-cross-seeding) - [DAS providers](#das-providers) From 4e1d566c43eb9175205348a58367fdc4de2a07c1 Mon Sep 17 00:00:00 2001 From: Csaba Kiraly Date: Wed, 29 May 2024 11:01:50 +0200 Subject: [PATCH 008/130] improve candidate peer text Signed-off-by: Csaba Kiraly --- specs/_features/eip7594/das-core.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/specs/_features/eip7594/das-core.md b/specs/_features/eip7594/das-core.md index a514358981..b99f2fc123 100644 --- a/specs/_features/eip7594/das-core.md +++ b/specs/_features/eip7594/das-core.md @@ -257,7 +257,9 @@ Sampling is considered successful if any `S - L` columns are retrieved successfu A node SHOULD maintain a diverse set of peers for each column and each slot by verifying responsiveness to sample queries. -A node SHOULD query for samples from their peers via `DataColumnSidecarsByRoot` request. A node utilizes `get_custody_columns` helper to determine which peer(s) it could request from. If more candidate peers are found, a node SHOULD randomize it's peer selection to distribute sample query load in the network. Nodes MAY use peer scoring to tune this selection (for example, by using weighted selection or by using a cut-off threshold). +A node SHOULD query for samples from selected peers via `DataColumnSidecarsByRoot` request. A node utilizes `get_custody_columns` helper to determine which peer(s) it could request from, identifying a list of candidate peers for each selected column. + +If more than one candidate peer is found for a given column, a node SHOULD randomize its peer selection to distribute sample query load in the network. Nodes MAY use peer scoring to tune this selection (for example, by using weighted selection or by using a cut-off threshold). If possible, it is also recommended to avoid requesting many columns from the same peer in order to avoid relying on and exposing the sample selection to a single peer. If a node already has a column because of custody, it is not required to send out queries for that column. From a04cd87c38b82c91d574deddfb41e242e53209d8 Mon Sep 17 00:00:00 2001 From: Csaba Kiraly Date: Wed, 29 May 2024 11:04:10 +0200 Subject: [PATCH 009/130] fix the (source-view) formatting of the table Signed-off-by: Csaba Kiraly --- specs/_features/eip7594/das-core.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/_features/eip7594/das-core.md b/specs/_features/eip7594/das-core.md index b99f2fc123..e07b1bae35 100644 --- a/specs/_features/eip7594/das-core.md +++ b/specs/_features/eip7594/das-core.md @@ -247,8 +247,8 @@ At each slot, a node SHOULD select at least `SAMPLES_PER_SLOT` column IDs for sa Alternatively, a node MAY use LossyDAS selecting more than `SAMPLES_PER_SLOT` columns while allowing some missing, respecting the same target false positive threshold (the probability of successful sampling of an unavailable block) as dictated by `SAMPLES_PER_SLOT`. The table below shows the number of samples and the number of allowed missing columns for this threshold. -| Allowed missing (L) | 0| 1| 2| 3| 4| 5| 6| 7| 8| -|------------------------------- |--|--|--|--|--|--|--|--|--| +| Allowed missing (L) | 0| 1| 2| 3| 4| 5| 6| 7| 8| +|----------------------------------------|--|--|--|--|--|--|--|--|--| | Samples (S) for target threshold 5e-6 |16|20|23|26|29|32|34|37|39| Sampling is considered successful if any `S - L` columns are retrieved successfully. From 5f3beca87121f6f121e41b9a61ea1a971dcb7cd7 Mon Sep 17 00:00:00 2001 From: Csaba Kiraly Date: Wed, 29 May 2024 11:08:52 +0200 Subject: [PATCH 010/130] remove LossyDAS naming from spec While the technique was introduced as LossyDAS, we don't need the name in the specification itself. Signed-off-by: Csaba Kiraly --- specs/_features/eip7594/das-core.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/_features/eip7594/das-core.md b/specs/_features/eip7594/das-core.md index e07b1bae35..b9674b177e 100644 --- a/specs/_features/eip7594/das-core.md +++ b/specs/_features/eip7594/das-core.md @@ -245,7 +245,7 @@ To custody a particular column, a node joins the respective gossip subnet. Verif At each slot, a node SHOULD select at least `SAMPLES_PER_SLOT` column IDs for sampling. It is recommended to use uniform random selection without replacement based on local randomness. Sampling is considered successful if the node manages to retrieve all selected columns. -Alternatively, a node MAY use LossyDAS selecting more than `SAMPLES_PER_SLOT` columns while allowing some missing, respecting the same target false positive threshold (the probability of successful sampling of an unavailable block) as dictated by `SAMPLES_PER_SLOT`. The table below shows the number of samples and the number of allowed missing columns for this threshold. +Alternatively, a node MAY use a method that selects more than `SAMPLES_PER_SLOT` columns while allowing some missing, respecting the same target false positive threshold (the probability of successful sampling of an unavailable block) as dictated by `SAMPLES_PER_SLOT`. The table below shows the number of samples and the number of allowed missing columns for this threshold. | Allowed missing (L) | 0| 1| 2| 3| 4| 5| 6| 7| 8| |----------------------------------------|--|--|--|--|--|--|--|--|--| From 436e58e3f8d2e3839e91864bde016f656cff31d1 Mon Sep 17 00:00:00 2001 From: Csaba Kiraly Date: Wed, 29 May 2024 14:25:33 +0200 Subject: [PATCH 011/130] add get_extended_sample_count helper function add LossyDAS sample count generation helper function Signed-off-by: Csaba Kiraly --- specs/_features/eip7594/das-core.md | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/specs/_features/eip7594/das-core.md b/specs/_features/eip7594/das-core.md index b9674b177e..7d5710a2d4 100644 --- a/specs/_features/eip7594/das-core.md +++ b/specs/_features/eip7594/das-core.md @@ -22,6 +22,7 @@ - [`compute_extended_matrix`](#compute_extended_matrix) - [`recover_matrix`](#recover_matrix) - [`get_data_column_sidecars`](#get_data_column_sidecars) + - [`get_extended_sample_count`](#get_extended_sample_count) - [Custody](#custody) - [Custody requirement](#custody-requirement) - [Public, deterministic selection](#public-deterministic-selection) @@ -199,6 +200,21 @@ def get_data_column_sidecars(signed_block: SignedBeaconBlock, return sidecars ``` +#### `get_extended_sample_count` + +```python +# from scipy.stats import hypergeom +def get_extended_sample_count(samples_per_slot: uint64, allowed_failures: uint64) -> uint64: + assert 0 <= allowed_failures <= NUMBER_OF_COLUMNS // 2 + + worst_case_missing = NUMBER_OF_COLUMNS // 2 + 1 + false_positive_threshold = hypergeom.cdf(0, NUMBER_OF_COLUMNS, worst_case_missing, samples_per_slot) + for sample_count in range(samples_per_slot, NUMBER_OF_COLUMNS + 1): + if hypergeom.cdf(allowed_failures, NUMBER_OF_COLUMNS, worst_case_missing, sample_count) <= false_positive_threshold: + break + return sample_count +``` + ## Custody ### Custody requirement @@ -245,13 +261,13 @@ To custody a particular column, a node joins the respective gossip subnet. Verif At each slot, a node SHOULD select at least `SAMPLES_PER_SLOT` column IDs for sampling. It is recommended to use uniform random selection without replacement based on local randomness. Sampling is considered successful if the node manages to retrieve all selected columns. -Alternatively, a node MAY use a method that selects more than `SAMPLES_PER_SLOT` columns while allowing some missing, respecting the same target false positive threshold (the probability of successful sampling of an unavailable block) as dictated by `SAMPLES_PER_SLOT`. The table below shows the number of samples and the number of allowed missing columns for this threshold. +Alternatively, a node MAY use a method that selects more than `SAMPLES_PER_SLOT` columns while allowing some missing, respecting the same target false positive threshold (the probability of successful sampling of an unavailable block) as dictated by the `SAMPLES_PER_SLOT` parameter. A node can use the `get_extended_sample_count(samples_per_slot, allowed_failures) -> sample_count` helper function to determine the sample count for any selected number of allowed failures. Sampling is then considered successful if any `sample_count - allowed_failures` columns are retrieved successfully. -| Allowed missing (L) | 0| 1| 2| 3| 4| 5| 6| 7| 8| -|----------------------------------------|--|--|--|--|--|--|--|--|--| -| Samples (S) for target threshold 5e-6 |16|20|23|26|29|32|34|37|39| +For reference, the table below shows the number of samples and the number of allowed missing columns assuming `NUMBER_OF_COLUMNS = 128` and `SAMPLES_PER_SLOT = 16`. -Sampling is considered successful if any `S - L` columns are retrieved successfully. +| Allowed missing | 0| 1| 2| 3| 4| 5| 6| 7| 8| +|-----------------|--|--|--|--|--|--|--|--|--| +| Sample count |16|20|24|27|29|32|35|37|40| ### Sample queries From 4c57399887e22aebffd325fe55db19b659263749 Mon Sep 17 00:00:00 2001 From: Csaba Kiraly Date: Tue, 4 Jun 2024 09:38:09 +0200 Subject: [PATCH 012/130] self-contained get_extended_sample_count Importing scipy is not preferred. This is a self-contained version. Eventually an import of math and use of math.comb makes it simpler. Solving other formatting issues as well. Signed-off-by: Csaba Kiraly --- specs/_features/eip7594/das-core.md | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/specs/_features/eip7594/das-core.md b/specs/_features/eip7594/das-core.md index 7d5710a2d4..f0f302a593 100644 --- a/specs/_features/eip7594/das-core.md +++ b/specs/_features/eip7594/das-core.md @@ -203,15 +203,28 @@ def get_data_column_sidecars(signed_block: SignedBeaconBlock, #### `get_extended_sample_count` ```python -# from scipy.stats import hypergeom def get_extended_sample_count(samples_per_slot: uint64, allowed_failures: uint64) -> uint64: assert 0 <= allowed_failures <= NUMBER_OF_COLUMNS // 2 + def math_comb(n, k): + if not 0 <= k <= n: + return 0 + r = 1 + for i in range(min(k, n - k)): + r = r * (n - i) // (i + 1) + return r + + def hypergeom_cdf(k, M, n, N): + return sum([math_comb(n, i) * math_comb(M - n, N - i) / math_comb(M, N) + for i in range(k + 1)]) + worst_case_missing = NUMBER_OF_COLUMNS // 2 + 1 - false_positive_threshold = hypergeom.cdf(0, NUMBER_OF_COLUMNS, worst_case_missing, samples_per_slot) + false_positive_threshold = hypergeom_cdf(0, NUMBER_OF_COLUMNS, + worst_case_missing, samples_per_slot) for sample_count in range(samples_per_slot, NUMBER_OF_COLUMNS + 1): - if hypergeom.cdf(allowed_failures, NUMBER_OF_COLUMNS, worst_case_missing, sample_count) <= false_positive_threshold: - break + if hypergeom_cdf(allowed_failures, NUMBER_OF_COLUMNS, + worst_case_missing, sample_count) <= false_positive_threshold: + break return sample_count ``` From 2ab4f1e12e9f59584326a475c3a1da9e98375802 Mon Sep 17 00:00:00 2001 From: Csaba Kiraly Date: Mon, 10 Jun 2024 11:16:06 +0200 Subject: [PATCH 013/130] get_extended_sample_count: use SAMPLES_PER_SLOT constant Signed-off-by: Csaba Kiraly --- specs/_features/eip7594/das-core.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/_features/eip7594/das-core.md b/specs/_features/eip7594/das-core.md index f0f302a593..442409ce4a 100644 --- a/specs/_features/eip7594/das-core.md +++ b/specs/_features/eip7594/das-core.md @@ -203,7 +203,7 @@ def get_data_column_sidecars(signed_block: SignedBeaconBlock, #### `get_extended_sample_count` ```python -def get_extended_sample_count(samples_per_slot: uint64, allowed_failures: uint64) -> uint64: +def get_extended_sample_count(allowed_failures: uint64) -> uint64: assert 0 <= allowed_failures <= NUMBER_OF_COLUMNS // 2 def math_comb(n, k): @@ -220,8 +220,8 @@ def get_extended_sample_count(samples_per_slot: uint64, allowed_failures: uint64 worst_case_missing = NUMBER_OF_COLUMNS // 2 + 1 false_positive_threshold = hypergeom_cdf(0, NUMBER_OF_COLUMNS, - worst_case_missing, samples_per_slot) - for sample_count in range(samples_per_slot, NUMBER_OF_COLUMNS + 1): + worst_case_missing, SAMPLES_PER_SLOT) + for sample_count in range(SAMPLES_PER_SLOT, NUMBER_OF_COLUMNS + 1): if hypergeom_cdf(allowed_failures, NUMBER_OF_COLUMNS, worst_case_missing, sample_count) <= false_positive_threshold: break From 6711acf3e295775273397f9d7968e33c9000bd45 Mon Sep 17 00:00:00 2001 From: Csaba Kiraly Date: Tue, 11 Jun 2024 02:31:22 +0200 Subject: [PATCH 014/130] clarifying reconstruction and cross-seeding - when should a node reconstruct - how to cross-seed for subscribed and unsubscribed topics Signed-off-by: Csaba Kiraly --- specs/_features/eip7594/das-core.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/_features/eip7594/das-core.md b/specs/_features/eip7594/das-core.md index 83d47e1856..e63ead25c7 100644 --- a/specs/_features/eip7594/das-core.md +++ b/specs/_features/eip7594/das-core.md @@ -235,11 +235,11 @@ Due to the deterministic custody functions, a node knows exactly what a peer sho ## Reconstruction and cross-seeding -If the node obtains 50%+ of all the columns, they can reconstruct the full data matrix via `recover_matrix` helper. +If a node fails to get a column on the column subnet, a node can also utilize the Req/Resp message to query the missing column from other peers. -If a node fails to sample a peer or fails to get a column on the column subnet, a node can utilize the Req/Resp message to query the missing column from other peers. +If the node obtains 50%+ of all the columns, it SHOULD reconstruct the full data matrix via `recover_matrix` helper. Nodes MAY delay this reconstruction allowing time for other columns to arrive over the network. If delaying reconstruction, nodes may use a random delay in order to desynchronize reconstruction among nodes, thus reducing overall CPU load. -Once the node obtain the column, the node SHOULD send the missing columns to the column subnets. +Once the node obtains a column through reconstruction, the node MUST expose the new column as if it had received it over the network. If the node is subscribed to the subnet corresponding to the column, it MUST send the reconstructed DataColumnSidecar to its topic mesh neighbors. If instead the node is not subscribed to the corresponding subnet, it SHOULD still expose the availability of the DataColumnSidecar as part of the gossip emission process. *Note*: A node always maintains a matrix view of the rows and columns they are following, able to cross-reference and cross-seed in either direction. From cf46b07ac4257f37849c322aeb5194128fd0cb48 Mon Sep 17 00:00:00 2001 From: Etan Kissling Date: Sun, 16 Jun 2024 10:04:41 +0200 Subject: [PATCH 015/130] Bump remerkleable to `v0.1.28` - https://github.com/protolambda/remerkleable/releases/tag/v0.1.28 --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index e5c348ada5..539db215b7 100644 --- a/setup.py +++ b/setup.py @@ -552,7 +552,7 @@ def run(self): "pycryptodome==3.15.0", "py_ecc==6.0.0", "milagro_bls_binding==1.9.0", - "remerkleable==0.1.27", + "remerkleable==0.1.28", "trie==2.0.2", RUAMEL_YAML_VERSION, "lru-dict==1.2.0", From fb020456cba20ffa76d5c3566c8240b8206a47f5 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 19 Jun 2024 02:18:29 +0800 Subject: [PATCH 016/130] Add `get_extended_sample_count` unit tests --- specs/_features/eip7594/das-core.md | 6 ++- .../test/eip7594/unittests/das/test_das.py | 52 +++++++++++++++++++ 2 files changed, 57 insertions(+), 1 deletion(-) diff --git a/specs/_features/eip7594/das-core.md b/specs/_features/eip7594/das-core.md index 442409ce4a..aec3a276e4 100644 --- a/specs/_features/eip7594/das-core.md +++ b/specs/_features/eip7594/das-core.md @@ -215,7 +215,11 @@ def get_extended_sample_count(allowed_failures: uint64) -> uint64: return r def hypergeom_cdf(k, M, n, N): - return sum([math_comb(n, i) * math_comb(M - n, N - i) / math_comb(M, N) + k = int(k) + M = int(M) + n = int(n) + N = int(N) + return sum([math_comb(n, i) * math_comb(M - n, N - i) // math_comb(M, N) for i in range(k + 1)]) worst_case_missing = NUMBER_OF_COLUMNS // 2 + 1 diff --git a/tests/core/pyspec/eth2spec/test/eip7594/unittests/das/test_das.py b/tests/core/pyspec/eth2spec/test/eip7594/unittests/das/test_das.py index cdbfad9ffe..d61f2e7cd9 100644 --- a/tests/core/pyspec/eth2spec/test/eip7594/unittests/das/test_das.py +++ b/tests/core/pyspec/eth2spec/test/eip7594/unittests/das/test_das.py @@ -1,5 +1,6 @@ import random from eth2spec.test.context import ( + expect_assertion_error, spec_test, single_phase, with_eip7594_and_later, @@ -67,3 +68,54 @@ def test_recover_matrix(spec): recovered_matrix = spec.recover_matrix(cells_dict, blob_count) flatten_original_cells = [cell for cells in original_cells for cell in cells] assert recovered_matrix == flatten_original_cells + + +@with_eip7594_and_later +@spec_test +@single_phase +def test_get_extended_sample_count__1(spec): + rng = random.Random(1111) + allowed_failures = rng.randint(0, spec.config.NUMBER_OF_COLUMNS // 2) + spec.get_extended_sample_count(allowed_failures) + + +@with_eip7594_and_later +@spec_test +@single_phase +def test_get_extended_sample_count__2(spec): + rng = random.Random(2222) + allowed_failures = rng.randint(0, spec.config.NUMBER_OF_COLUMNS // 2) + spec.get_extended_sample_count(allowed_failures) + + +@with_eip7594_and_later +@spec_test +@single_phase +def test_get_extended_sample_count__3(spec): + rng = random.Random(3333) + allowed_failures = rng.randint(0, spec.config.NUMBER_OF_COLUMNS // 2) + spec.get_extended_sample_count(allowed_failures) + + +@with_eip7594_and_later +@spec_test +@single_phase +def test_get_extended_sample_count__lower_bound(spec): + allowed_failures = 0 + spec.get_extended_sample_count(allowed_failures) + + +@with_eip7594_and_later +@spec_test +@single_phase +def test_get_extended_sample_count__upper_bound(spec): + allowed_failures = spec.config.NUMBER_OF_COLUMNS // 2 + spec.get_extended_sample_count(allowed_failures) + + +@with_eip7594_and_later +@spec_test +@single_phase +def test_get_extended_sample_count__upper_bound_exceed(spec): + allowed_failures = spec.config.NUMBER_OF_COLUMNS // 2 + 1 + expect_assertion_error(lambda: spec.get_extended_sample_count(allowed_failures)) From 2c384da09206b3d3c5f012228ce52558951097d1 Mon Sep 17 00:00:00 2001 From: Etan Kissling Date: Fri, 21 Jun 2024 10:51:18 +0200 Subject: [PATCH 017/130] Fix typing issue when `10e10` is interpreted as float On macOS Python, passing `10e10` to `rng.randint` results in an error when generating certain tests. Forcing `int` avoids the problem. --- .../core/pyspec/eth2spec/test/helpers/execution_payload.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py index dbfd29e671..b061f506a8 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py @@ -263,9 +263,9 @@ def build_randomized_execution_payload(spec, state, rng): execution_payload.logs_bloom = spec.ByteVector[spec.BYTES_PER_LOGS_BLOOM]( get_random_bytes_list(rng, spec.BYTES_PER_LOGS_BLOOM) ) - execution_payload.block_number = rng.randint(0, 10e10) - execution_payload.gas_limit = rng.randint(0, 10e10) - execution_payload.gas_used = rng.randint(0, 10e10) + execution_payload.block_number = rng.randint(0, int(10e10)) + execution_payload.gas_limit = rng.randint(0, int(10e10)) + execution_payload.gas_used = rng.randint(0, int(10e10)) extra_data_length = rng.randint(0, spec.MAX_EXTRA_DATA_BYTES) execution_payload.extra_data = spec.ByteList[spec.MAX_EXTRA_DATA_BYTES]( get_random_bytes_list(rng, extra_data_length) From 2035a9fcad479da965b1f9670ed959a53fcb42da Mon Sep 17 00:00:00 2001 From: Etan Kissling Date: Fri, 21 Jun 2024 10:59:13 +0200 Subject: [PATCH 018/130] Update light client specifications for Electra Electra introduces two changes that affect light client data handling: 1. The `ExecutionPayloadHeader` is extended with new fields. This is handled similarly as before with the Deneb fork. 2. The `BeaconState` generalized indices change due to lack of EIP-6493. This is handled by making the generalized index be fork dependent via a helper function that computes it dynamically. Furthermore, the case where pre-Electra light client data is consumed by an Electra based `LightClientStore` requires normalizing the shorter proof of the pre-Electra data to fit into the Electra data structure by prepending a zero hash. --- README.md | 2 +- pysetup/spec_builders/electra.py | 2 - specs/altair/light-client/sync-protocol.md | 59 ++++++- specs/capella/light-client/fork.md | 8 +- specs/deneb/light-client/fork.md | 8 +- specs/deneb/light-client/full-node.md | 2 +- specs/electra/light-client/fork.md | 133 ++++++++++++++ specs/electra/light-client/full-node.md | 80 +++++++++ specs/electra/light-client/p2p-interface.md | 111 ++++++++++++ specs/electra/light-client/sync-protocol.md | 165 ++++++++++++++++++ .../test/altair/light_client/test_sync.py | 95 ++++++++-- .../eth2spec/test/helpers/light_client.py | 99 +++++++++-- tests/generators/light_client/main.py | 4 +- 13 files changed, 714 insertions(+), 54 deletions(-) create mode 100644 specs/electra/light-client/fork.md create mode 100644 specs/electra/light-client/full-node.md create mode 100644 specs/electra/light-client/p2p-interface.md create mode 100644 specs/electra/light-client/sync-protocol.md diff --git a/README.md b/README.md index 58bff5b9e4..c62a4171dc 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ Features are researched and developed in parallel, and then consolidated into se ### In-development Specifications | Code Name or Topic | Specs | Notes | | - | - | - | -| Electra |
  • Core
    • [Beacon Chain changes](specs/electra/beacon-chain.md)
    • [EIP-6110 fork](specs/electra/fork.md)
  • Additions
    • [Honest validator guide changes](specs/electra/validator.md)
| +| Electra |
  • Core
    • [Beacon Chain changes](specs/electra/beacon-chain.md)
    • [EIP-6110 fork](specs/electra/fork.md)
  • Additions
    • [Light client sync protocol changes](specs/electra/light-client/sync-protocol.md) ([fork](specs/electra/light-client/fork.md), [full node](specs/electra/light-client/full-node.md), [networking](specs/electra/light-client/p2p-interface.md))
    • [Honest validator guide changes](specs/electra/validator.md)
| | Sharding (outdated) |
  • Core
    • [Beacon Chain changes](specs/_features/sharding/beacon-chain.md)
  • Additions
    • [P2P networking](specs/_features/sharding/p2p-interface.md)
| | Custody Game (outdated) |
  • Core
    • [Beacon Chain changes](specs/_features/custody_game/beacon-chain.md)
  • Additions
    • [Honest validator guide changes](specs/_features/custody_game/validator.md)
| Dependent on sharding | | Data Availability Sampling (outdated) |
  • Core
    • [Core types and functions](specs/_features/das/das-core.md)
    • [Fork choice changes](specs/_features/das/fork-choice.md)
  • Additions
    • [P2P Networking](specs/_features/das/p2p-interface.md)
    • [Sampling process](specs/_features/das/sampling.md)
|
  • Dependent on sharding
  • [Technical explainer](https://hackmd.io/@HWeNw8hNRimMm2m2GH56Cw/B1YJPGkpD)
| diff --git a/pysetup/spec_builders/electra.py b/pysetup/spec_builders/electra.py index 1f968a817d..2b8febd347 100644 --- a/pysetup/spec_builders/electra.py +++ b/pysetup/spec_builders/electra.py @@ -12,8 +12,6 @@ def imports(cls, preset_name: str): from eth2spec.deneb import {preset_name} as deneb ''' -## TODO: deal with changed gindices - @classmethod def hardcoded_ssz_dep_constants(cls) -> Dict[str, str]: return { diff --git a/specs/altair/light-client/sync-protocol.md b/specs/altair/light-client/sync-protocol.md index 2585889bf9..f7f38d1041 100644 --- a/specs/altair/light-client/sync-protocol.md +++ b/specs/altair/light-client/sync-protocol.md @@ -21,6 +21,9 @@ - [`LightClientOptimisticUpdate`](#lightclientoptimisticupdate) - [`LightClientStore`](#lightclientstore) - [Helper functions](#helper-functions) + - [`finalized_root_gindex_at_slot`](#finalized_root_gindex_at_slot) + - [`current_sync_committee_gindex_at_slot`](#current_sync_committee_gindex_at_slot) + - [`next_sync_committee_gindex_at_slot`](#next_sync_committee_gindex_at_slot) - [`is_valid_light_client_header`](#is_valid_light_client_header) - [`is_sync_committee_update`](#is_sync_committee_update) - [`is_finality_update`](#is_finality_update) @@ -28,6 +31,7 @@ - [`is_next_sync_committee_known`](#is_next_sync_committee_known) - [`get_safety_threshold`](#get_safety_threshold) - [`get_subtree_index`](#get_subtree_index) + - [`is_valid_normalized_merkle_branch`](#is_valid_normalized_merkle_branch) - [`compute_sync_committee_period_at_slot`](#compute_sync_committee_period_at_slot) - [Light client initialization](#light-client-initialization) - [`initialize_light_client_store`](#initialize_light_client_store) @@ -171,6 +175,30 @@ class LightClientStore(object): ## Helper functions +### `finalized_root_gindex_at_slot` + +```python +def finalized_root_gindex_at_slot(slot: Slot) -> GeneralizedIndex: + # pylint: disable=unused-argument + return FINALIZED_ROOT_GINDEX +``` + +### `current_sync_committee_gindex_at_slot` + +```python +def current_sync_committee_gindex_at_slot(slot: Slot) -> GeneralizedIndex: + # pylint: disable=unused-argument + return CURRENT_SYNC_COMMITTEE_GINDEX +``` + +### `next_sync_committee_gindex_at_slot` + +```python +def next_sync_committee_gindex_at_slot(slot: Slot) -> GeneralizedIndex: + # pylint: disable=unused-argument + return NEXT_SYNC_COMMITTEE_GINDEX +``` + ### `is_valid_light_client_header` ```python @@ -273,6 +301,22 @@ def get_subtree_index(generalized_index: GeneralizedIndex) -> uint64: return uint64(generalized_index % 2**(floorlog2(generalized_index))) ``` +### `is_valid_normalized_merkle_branch` + +```python +def is_valid_normalized_merkle_branch(leaf: Bytes32, + branch: Sequence[Bytes32], + gindex: GeneralizedIndex, + root: Root) -> bool: + depth = floorlog2(gindex) + index = get_subtree_index(gindex) + num_extra = len(branch) - depth + for i in range(num_extra): + if branch[i] != Bytes32(): + return False + return is_valid_merkle_branch(leaf, branch[num_extra:], depth, index, root) +``` + ### `compute_sync_committee_period_at_slot` ```python @@ -292,11 +336,10 @@ def initialize_light_client_store(trusted_block_root: Root, assert is_valid_light_client_header(bootstrap.header) assert hash_tree_root(bootstrap.header.beacon) == trusted_block_root - assert is_valid_merkle_branch( + assert is_valid_normalized_merkle_branch( leaf=hash_tree_root(bootstrap.current_sync_committee), branch=bootstrap.current_sync_committee_branch, - depth=floorlog2(CURRENT_SYNC_COMMITTEE_GINDEX), - index=get_subtree_index(CURRENT_SYNC_COMMITTEE_GINDEX), + gindex=current_sync_committee_gindex_at_slot(bootstrap.header.beacon.slot), root=bootstrap.header.beacon.state_root, ) @@ -364,11 +407,10 @@ def validate_light_client_update(store: LightClientStore, else: assert is_valid_light_client_header(update.finalized_header) finalized_root = hash_tree_root(update.finalized_header.beacon) - assert is_valid_merkle_branch( + assert is_valid_normalized_merkle_branch( leaf=finalized_root, branch=update.finality_branch, - depth=floorlog2(FINALIZED_ROOT_GINDEX), - index=get_subtree_index(FINALIZED_ROOT_GINDEX), + gindex=finalized_root_gindex_at_slot(update.attested_header.beacon.slot), root=update.attested_header.beacon.state_root, ) @@ -379,11 +421,10 @@ def validate_light_client_update(store: LightClientStore, else: if update_attested_period == store_period and is_next_sync_committee_known(store): assert update.next_sync_committee == store.next_sync_committee - assert is_valid_merkle_branch( + assert is_valid_normalized_merkle_branch( leaf=hash_tree_root(update.next_sync_committee), branch=update.next_sync_committee_branch, - depth=floorlog2(NEXT_SYNC_COMMITTEE_GINDEX), - index=get_subtree_index(NEXT_SYNC_COMMITTEE_GINDEX), + gindex=next_sync_committee_gindex_at_slot(update.attested_header.beacon.slot), root=update.attested_header.beacon.state_root, ) diff --git a/specs/capella/light-client/fork.md b/specs/capella/light-client/fork.md index 6dcc7578c2..6fcb6e3147 100644 --- a/specs/capella/light-client/fork.md +++ b/specs/capella/light-client/fork.md @@ -7,8 +7,8 @@ - [Introduction](#introduction) - - [Upgrading light client data](#upgrading-light-client-data) - - [Upgrading the store](#upgrading-the-store) +- [Upgrading light client data](#upgrading-light-client-data) +- [Upgrading the store](#upgrading-the-store) @@ -17,7 +17,7 @@ This document describes how to upgrade existing light client objects based on the [Altair specification](../../altair/light-client/sync-protocol.md) to Capella. This is necessary when processing pre-Capella data with a post-Capella `LightClientStore`. Note that the data being exchanged over the network protocols uses the original format. -### Upgrading light client data +## Upgrading light client data A Capella `LightClientStore` can still process earlier light client data. In order to do so, that pre-Capella data needs to be locally upgraded to Capella before processing. @@ -70,7 +70,7 @@ def upgrade_lc_optimistic_update_to_capella(pre: bellatrix.LightClientOptimistic ) ``` -### Upgrading the store +## Upgrading the store Existing `LightClientStore` objects based on Altair MUST be upgraded to Capella before Capella based light client data can be processed. The `LightClientStore` upgrade MAY be performed before `CAPELLA_FORK_EPOCH`. diff --git a/specs/deneb/light-client/fork.md b/specs/deneb/light-client/fork.md index 2dce4778ed..07230a21c7 100644 --- a/specs/deneb/light-client/fork.md +++ b/specs/deneb/light-client/fork.md @@ -7,8 +7,8 @@ - [Introduction](#introduction) - - [Upgrading light client data](#upgrading-light-client-data) - - [Upgrading the store](#upgrading-the-store) +- [Upgrading light client data](#upgrading-light-client-data) +- [Upgrading the store](#upgrading-the-store) @@ -17,7 +17,7 @@ This document describes how to upgrade existing light client objects based on the [Capella specification](../../capella/light-client/sync-protocol.md) to Deneb. This is necessary when processing pre-Deneb data with a post-Deneb `LightClientStore`. Note that the data being exchanged over the network protocols uses the original format. -### Upgrading light client data +## Upgrading light client data A Deneb `LightClientStore` can still process earlier light client data. In order to do so, that pre-Deneb data needs to be locally upgraded to Deneb before processing. @@ -90,7 +90,7 @@ def upgrade_lc_optimistic_update_to_deneb(pre: capella.LightClientOptimisticUpda ) ``` -### Upgrading the store +## Upgrading the store Existing `LightClientStore` objects based on Capella MUST be upgraded to Deneb before Deneb based light client data can be processed. The `LightClientStore` upgrade MAY be performed before `DENEB_FORK_EPOCH`. diff --git a/specs/deneb/light-client/full-node.md b/specs/deneb/light-client/full-node.md index db081b8e43..424723667c 100644 --- a/specs/deneb/light-client/full-node.md +++ b/specs/deneb/light-client/full-node.md @@ -17,7 +17,7 @@ ## Introduction -This upgrade adds information about the execution payload to light client data as part of the Deneb upgrade. +Execution payload data is updated to account for the Deneb upgrade. ## Helper functions diff --git a/specs/electra/light-client/fork.md b/specs/electra/light-client/fork.md new file mode 100644 index 0000000000..7bae9c3f4f --- /dev/null +++ b/specs/electra/light-client/fork.md @@ -0,0 +1,133 @@ +# Electra Light Client -- Fork Logic + +## Table of contents + + + + + +- [Introduction](#introduction) +- [Helper functions](#helper-functions) + - [`normalize_merkle_branch`](#normalize_merkle_branch) +- [Upgrading light client data](#upgrading-light-client-data) +- [Upgrading the store](#upgrading-the-store) + + + + +## Introduction + +This document describes how to upgrade existing light client objects based on the [Deneb specification](../../deneb/light-client/sync-protocol.md) to Electra. This is necessary when processing pre-Electra data with a post-Electra `LightClientStore`. Note that the data being exchanged over the network protocols uses the original format. + +## Helper functions + +### `normalize_merkle_branch` + +```python +def normalize_merkle_branch(branch: Sequence[Bytes32], + gindex: GeneralizedIndex) -> Sequence[Bytes32]: + depth = floorlog2(gindex) + num_extra = depth - len(branch) + return [Bytes32()] * num_extra + [*branch] +``` + +## Upgrading light client data + +A Electra `LightClientStore` can still process earlier light client data. In order to do so, that pre-Electra data needs to be locally upgraded to Electra before processing. + +```python +def upgrade_lc_header_to_electra(pre: deneb.LightClientHeader) -> LightClientHeader: + return LightClientHeader( + beacon=pre.beacon, + execution=ExecutionPayloadHeader( + parent_hash=pre.execution.parent_hash, + fee_recipient=pre.execution.fee_recipient, + state_root=pre.execution.state_root, + receipts_root=pre.execution.receipts_root, + logs_bloom=pre.execution.logs_bloom, + prev_randao=pre.execution.prev_randao, + block_number=pre.execution.block_number, + gas_limit=pre.execution.gas_limit, + gas_used=pre.execution.gas_used, + timestamp=pre.execution.timestamp, + extra_data=pre.execution.extra_data, + base_fee_per_gas=pre.execution.base_fee_per_gas, + block_hash=pre.execution.block_hash, + transactions_root=pre.execution.transactions_root, + withdrawals_root=pre.execution.withdrawals_root, + blob_gas_used=pre.execution.blob_gas_used, + excess_blob_gas=pre.execution.blob_gas_used, + deposit_requests_root=Root(), # [New in Electra:EIP6110] + withdrawal_requests_root=Root(), # [New in Electra:EIP7002:EIP7251] + consolidation_requests_root=Root(), # [New in Electra:EIP7251] + ), + execution_branch=pre.execution_branch, + ) +``` + +```python +def upgrade_lc_bootstrap_to_electra(pre: deneb.LightClientBootstrap) -> LightClientBootstrap: + return LightClientBootstrap( + header=upgrade_lc_header_to_electra(pre.header), + current_sync_committee=pre.current_sync_committee, + current_sync_committee_branch=normalize_merkle_branch( + pre.current_sync_committee_branch, CURRENT_SYNC_COMMITTEE_GINDEX), + ) +``` + +```python +def upgrade_lc_update_to_electra(pre: deneb.LightClientUpdate) -> LightClientUpdate: + return LightClientUpdate( + attested_header=upgrade_lc_header_to_electra(pre.attested_header), + next_sync_committee=pre.next_sync_committee, + next_sync_committee_branch=normalize_merkle_branch( + pre.next_sync_committee_branch, NEXT_SYNC_COMMITTEE_GINDEX), + finalized_header=upgrade_lc_header_to_electra(pre.finalized_header), + finality_branch=normalize_merkle_branch( + pre.finality_branch, FINALIZED_ROOT_GINDEX), + sync_aggregate=pre.sync_aggregate, + signature_slot=pre.signature_slot, + ) +``` + +```python +def upgrade_lc_finality_update_to_electra(pre: deneb.LightClientFinalityUpdate) -> LightClientFinalityUpdate: + return LightClientFinalityUpdate( + attested_header=upgrade_lc_header_to_electra(pre.attested_header), + finalized_header=upgrade_lc_header_to_electra(pre.finalized_header), + finality_branch=normalize_merkle_branch( + pre.finality_branch, FINALIZED_ROOT_GINDEX), + sync_aggregate=pre.sync_aggregate, + signature_slot=pre.signature_slot, + ) +``` + +```python +def upgrade_lc_optimistic_update_to_electra(pre: deneb.LightClientOptimisticUpdate) -> LightClientOptimisticUpdate: + return LightClientOptimisticUpdate( + attested_header=upgrade_lc_header_to_electra(pre.attested_header), + sync_aggregate=pre.sync_aggregate, + signature_slot=pre.signature_slot, + ) +``` + +## Upgrading the store + +Existing `LightClientStore` objects based on Deneb MUST be upgraded to Electra before Electra based light client data can be processed. The `LightClientStore` upgrade MAY be performed before `ELECTRA_FORK_EPOCH`. + +```python +def upgrade_lc_store_to_electra(pre: deneb.LightClientStore) -> LightClientStore: + if pre.best_valid_update is None: + best_valid_update = None + else: + best_valid_update = upgrade_lc_update_to_electra(pre.best_valid_update) + return LightClientStore( + finalized_header=upgrade_lc_header_to_electra(pre.finalized_header), + current_sync_committee=pre.current_sync_committee, + next_sync_committee=pre.next_sync_committee, + best_valid_update=best_valid_update, + optimistic_header=upgrade_lc_header_to_electra(pre.optimistic_header), + previous_max_active_participants=pre.previous_max_active_participants, + current_max_active_participants=pre.current_max_active_participants, + ) +``` diff --git a/specs/electra/light-client/full-node.md b/specs/electra/light-client/full-node.md new file mode 100644 index 0000000000..f08a2cc5ed --- /dev/null +++ b/specs/electra/light-client/full-node.md @@ -0,0 +1,80 @@ +# Electra Light Client -- Full Node + +**Notice**: This document is a work-in-progress for researchers and implementers. + +## Table of contents + + + + + +- [Introduction](#introduction) +- [Helper functions](#helper-functions) + - [Modified `block_to_light_client_header`](#modified-block_to_light_client_header) + + + + +## Introduction + +Execution payload data is updated to account for the Electra upgrade. + +## Helper functions + +### Modified `block_to_light_client_header` + +```python +def block_to_light_client_header(block: SignedBeaconBlock) -> LightClientHeader: + epoch = compute_epoch_at_slot(block.message.slot) + + if epoch >= CAPELLA_FORK_EPOCH: + payload = block.message.body.execution_payload + execution_header = ExecutionPayloadHeader( + parent_hash=payload.parent_hash, + fee_recipient=payload.fee_recipient, + state_root=payload.state_root, + receipts_root=payload.receipts_root, + logs_bloom=payload.logs_bloom, + prev_randao=payload.prev_randao, + block_number=payload.block_number, + gas_limit=payload.gas_limit, + gas_used=payload.gas_used, + timestamp=payload.timestamp, + extra_data=payload.extra_data, + base_fee_per_gas=payload.base_fee_per_gas, + block_hash=payload.block_hash, + transactions_root=hash_tree_root(payload.transactions), + withdrawals_root=hash_tree_root(payload.withdrawals), + ) + if epoch >= DENEB_FORK_EPOCH: + execution_header.blob_gas_used = payload.blob_gas_used + execution_header.excess_blob_gas = payload.excess_blob_gas + + # [New in Electra:EIP6110:EIP7002:EIP7251] + if epoch >= ELECTRA_FORK_EPOCH: + execution_header.deposit_requests_root = hash_tree_root(payload.deposit_requests) + execution_header.withdrawal_requests_root = hash_tree_root(payload.withdrawal_requests) + execution_header.consolidation_requests_root = hash_tree_root(payload.consolidation_requests) + + execution_branch = ExecutionBranch( + compute_merkle_proof(block.message.body, EXECUTION_PAYLOAD_GINDEX)) + else: + # Note that during fork transitions, `finalized_header` may still point to earlier forks. + # While Bellatrix blocks also contain an `ExecutionPayload` (minus `withdrawals_root`), + # it was not included in the corresponding light client data. To ensure compatibility + # with legacy data going through `upgrade_lc_header_to_capella`, leave out execution data. + execution_header = ExecutionPayloadHeader() + execution_branch = ExecutionBranch() + + return LightClientHeader( + beacon=BeaconBlockHeader( + slot=block.message.slot, + proposer_index=block.message.proposer_index, + parent_root=block.message.parent_root, + state_root=block.message.state_root, + body_root=hash_tree_root(block.message.body), + ), + execution=execution_header, + execution_branch=execution_branch, + ) +``` diff --git a/specs/electra/light-client/p2p-interface.md b/specs/electra/light-client/p2p-interface.md new file mode 100644 index 0000000000..3cbd5dd28f --- /dev/null +++ b/specs/electra/light-client/p2p-interface.md @@ -0,0 +1,111 @@ +# Electra Light Client -- Networking + +**Notice**: This document is a work-in-progress for researchers and implementers. + +## Table of contents + + + + + +- [Networking](#networking) + - [The gossip domain: gossipsub](#the-gossip-domain-gossipsub) + - [Topics and messages](#topics-and-messages) + - [Global topics](#global-topics) + - [`light_client_finality_update`](#light_client_finality_update) + - [`light_client_optimistic_update`](#light_client_optimistic_update) + - [The Req/Resp domain](#the-reqresp-domain) + - [Messages](#messages) + - [GetLightClientBootstrap](#getlightclientbootstrap) + - [LightClientUpdatesByRange](#lightclientupdatesbyrange) + - [GetLightClientFinalityUpdate](#getlightclientfinalityupdate) + - [GetLightClientOptimisticUpdate](#getlightclientoptimisticupdate) + + + + +## Networking + +The [Deneb light client networking specification](../../deneb/light-client/p2p-interface.md) is extended to exchange [Electra light client data](./sync-protocol.md). + +### The gossip domain: gossipsub + +#### Topics and messages + +##### Global topics + +###### `light_client_finality_update` + +[0]: # (eth2spec: skip) + +| `fork_version` | Message SSZ type | +|--------------------------------------------------------|-------------------------------------| +| `GENESIS_FORK_VERSION` | n/a | +| `ALTAIR_FORK_VERSION` through `BELLATRIX_FORK_VERSION` | `altair.LightClientFinalityUpdate` | +| `CAPELLA_FORK_VERSION` | `capella.LightClientFinalityUpdate` | +| `DENEB_FORK_VERSION` | `deneb.LightClientFinalityUpdate` | +| `ELECTRA_FORK_VERSION` and later | `electra.LightClientFinalityUpdate` | + +###### `light_client_optimistic_update` + +[0]: # (eth2spec: skip) + +| `fork_version` | Message SSZ type | +|--------------------------------------------------------|---------------------------------------| +| `GENESIS_FORK_VERSION` | n/a | +| `ALTAIR_FORK_VERSION` through `BELLATRIX_FORK_VERSION` | `altair.LightClientOptimisticUpdate` | +| `CAPELLA_FORK_VERSION` | `capella.LightClientOptimisticUpdate` | +| `DENEB_FORK_VERSION` | `deneb.LightClientOptimisticUpdate` | +| `ELECTRA_FORK_VERSION` and later | `electra.LightClientOptimisticUpdate` | + +### The Req/Resp domain + +#### Messages + +##### GetLightClientBootstrap + +[0]: # (eth2spec: skip) + +| `fork_version` | Response SSZ type | +|--------------------------------------------------------|------------------------------------| +| `GENESIS_FORK_VERSION` | n/a | +| `ALTAIR_FORK_VERSION` through `BELLATRIX_FORK_VERSION` | `altair.LightClientBootstrap` | +| `CAPELLA_FORK_VERSION` | `capella.LightClientBootstrap` | +| `DENEB_FORK_VERSION` | `deneb.LightClientBootstrap` | +| `ELECTRA_FORK_VERSION` and later | `electra.LightClientBootstrap` | + +##### LightClientUpdatesByRange + +[0]: # (eth2spec: skip) + +| `fork_version` | Response chunk SSZ type | +|--------------------------------------------------------|----------------------------------| +| `GENESIS_FORK_VERSION` | n/a | +| `ALTAIR_FORK_VERSION` through `BELLATRIX_FORK_VERSION` | `altair.LightClientUpdate` | +| `CAPELLA_FORK_VERSION` | `capella.LightClientUpdate` | +| `DENEB_FORK_VERSION` | `deneb.LightClientUpdate` | +| `ELECTRA_FORK_VERSION` and later | `electra.LightClientUpdate` | + +##### GetLightClientFinalityUpdate + +[0]: # (eth2spec: skip) + +| `fork_version` | Response SSZ type | +|--------------------------------------------------------|-------------------------------------| +| `GENESIS_FORK_VERSION` | n/a | +| `ALTAIR_FORK_VERSION` through `BELLATRIX_FORK_VERSION` | `altair.LightClientFinalityUpdate` | +| `CAPELLA_FORK_VERSION` | `capella.LightClientFinalityUpdate` | +| `DENEB_FORK_VERSION` | `deneb.LightClientFinalityUpdate` | +| `ELECTRA_FORK_VERSION` and later | `electra.LightClientFinalityUpdate` | + +##### GetLightClientOptimisticUpdate + +[0]: # (eth2spec: skip) + +| `fork_version` | Response SSZ type | +|--------------------------------------------------------|---------------------------------------| +| `GENESIS_FORK_VERSION` | n/a | +| `ALTAIR_FORK_VERSION` through `BELLATRIX_FORK_VERSION` | `altair.LightClientOptimisticUpdate` | +| `CAPELLA_FORK_VERSION` | `capella.LightClientOptimisticUpdate` | +| `DENEB_FORK_VERSION` | `deneb.LightClientOptimisticUpdate` | +| `ELECTRA_FORK_VERSION` and later | `electra.LightClientOptimisticUpdate` | diff --git a/specs/electra/light-client/sync-protocol.md b/specs/electra/light-client/sync-protocol.md new file mode 100644 index 0000000000..4bb597abaf --- /dev/null +++ b/specs/electra/light-client/sync-protocol.md @@ -0,0 +1,165 @@ +# Electra Light Client -- Sync Protocol + +**Notice**: This document is a work-in-progress for researchers and implementers. + +## Table of contents + + + + + +- [Introduction](#introduction) +- [Constants](#constants) +- [Helper functions](#helper-functions) + - [Modified `finalized_root_gindex_at_slot`](#modified-finalized_root_gindex_at_slot) + - [Modified `current_sync_committee_gindex_at_slot`](#modified-current_sync_committee_gindex_at_slot) + - [Modified `next_sync_committee_gindex_at_slot`](#modified-next_sync_committee_gindex_at_slot) + - [Modified `get_lc_execution_root`](#modified-get_lc_execution_root) + - [Modified `is_valid_light_client_header`](#modified-is_valid_light_client_header) + + + + +## Introduction + +This upgrade updates light client data to include the Electra changes to the [`ExecutionPayload`](../beacon-chain.md) structure and to the generalized indices of surrounding containers. It extends the [Deneb Light Client specifications](../../deneb/light-client/sync-protocol.md). The [fork document](./fork.md) explains how to upgrade existing Deneb based deployments to Electra. + +Additional documents describes the impact of the upgrade on certain roles: +- [Full node](./full-node.md) +- [Networking](./p2p-interface.md) + +## Constants + +| Name | Value | +| - | - | +| `FINALIZED_ROOT_GINDEX` | `get_generalized_index(BeaconState, 'finalized_checkpoint', 'root')` (= 169) | +| `CURRENT_SYNC_COMMITTEE_GINDEX` | `get_generalized_index(BeaconState, 'current_sync_committee')` (= 86) | +| `NEXT_SYNC_COMMITTEE_GINDEX` | `get_generalized_index(BeaconState, 'next_sync_committee')` (= 87) | + +## Helper functions + +### Modified `finalized_root_gindex_at_slot` + +```python +def finalized_root_gindex_at_slot(slot: Slot) -> GeneralizedIndex: + epoch = compute_epoch_at_slot(slot) + + # [Modified in Electra] + if epoch >= ELECTRA_FORK_EPOCH: + return FINALIZED_ROOT_GINDEX + return capella.FINALIZED_ROOT_GINDEX +``` + +### Modified `current_sync_committee_gindex_at_slot` + +```python +def current_sync_committee_gindex_at_slot(slot: Slot) -> GeneralizedIndex: + epoch = compute_epoch_at_slot(slot) + + # [Modified in Electra] + if epoch >= ELECTRA_FORK_EPOCH: + return CURRENT_SYNC_COMMITTEE_GINDEX + return capella.CURRENT_SYNC_COMMITTEE_GINDEX +``` + +### Modified `next_sync_committee_gindex_at_slot` + +```python +def next_sync_committee_gindex_at_slot(slot: Slot) -> GeneralizedIndex: + epoch = compute_epoch_at_slot(slot) + + # [Modified in Electra] + if epoch >= ELECTRA_FORK_EPOCH: + return NEXT_SYNC_COMMITTEE_GINDEX + return capella.NEXT_SYNC_COMMITTEE_GINDEX +``` + +### Modified `get_lc_execution_root` + +```python +def get_lc_execution_root(header: LightClientHeader) -> Root: + epoch = compute_epoch_at_slot(header.beacon.slot) + + # [New in Electra] + if epoch >= ELECTRA_FORK_EPOCH: + return hash_tree_root(header.execution) + + # [Modified in Electra] + if epoch >= DENEB_FORK_EPOCH: + execution_header = deneb.ExecutionPayloadHeader( + parent_hash=header.execution.parent_hash, + fee_recipient=header.execution.fee_recipient, + state_root=header.execution.state_root, + receipts_root=header.execution.receipts_root, + logs_bloom=header.execution.logs_bloom, + prev_randao=header.execution.prev_randao, + block_number=header.execution.block_number, + gas_limit=header.execution.gas_limit, + gas_used=header.execution.gas_used, + timestamp=header.execution.timestamp, + extra_data=header.execution.extra_data, + base_fee_per_gas=header.execution.base_fee_per_gas, + block_hash=header.execution.block_hash, + transactions_root=header.execution.transactions_root, + withdrawals_root=header.execution.withdrawals_root, + blob_gas_used=header.execution.blob_gas_used, + excess_blob_gas=header.execution.excess_blob_gas, + ) + return hash_tree_root(execution_header) + + if epoch >= CAPELLA_FORK_EPOCH: + execution_header = capella.ExecutionPayloadHeader( + parent_hash=header.execution.parent_hash, + fee_recipient=header.execution.fee_recipient, + state_root=header.execution.state_root, + receipts_root=header.execution.receipts_root, + logs_bloom=header.execution.logs_bloom, + prev_randao=header.execution.prev_randao, + block_number=header.execution.block_number, + gas_limit=header.execution.gas_limit, + gas_used=header.execution.gas_used, + timestamp=header.execution.timestamp, + extra_data=header.execution.extra_data, + base_fee_per_gas=header.execution.base_fee_per_gas, + block_hash=header.execution.block_hash, + transactions_root=header.execution.transactions_root, + withdrawals_root=header.execution.withdrawals_root, + ) + return hash_tree_root(execution_header) + + return Root() +``` + +### Modified `is_valid_light_client_header` + +```python +def is_valid_light_client_header(header: LightClientHeader) -> bool: + epoch = compute_epoch_at_slot(header.beacon.slot) + + # [New in Electra:EIP6110:EIP7002:EIP7251] + if epoch < ELECTRA_FORK_EPOCH: + if ( + header.execution.deposit_requests_root != Root() + or header.execution.withdrawal_requests_root != Root() + or header.execution.consolidation_requests_root != Root() + ): + return False + + if epoch < DENEB_FORK_EPOCH: + if header.execution.blob_gas_used != uint64(0) or header.execution.excess_blob_gas != uint64(0): + return False + + if epoch < CAPELLA_FORK_EPOCH: + return ( + header.execution == ExecutionPayloadHeader() + and header.execution_branch == ExecutionBranch() + ) + + return is_valid_merkle_branch( + leaf=get_lc_execution_root(header), + branch=header.execution_branch, + depth=floorlog2(EXECUTION_PAYLOAD_GINDEX), + index=get_subtree_index(EXECUTION_PAYLOAD_GINDEX), + root=header.beacon.body_root, + ) +``` diff --git a/tests/core/pyspec/eth2spec/test/altair/light_client/test_sync.py b/tests/core/pyspec/eth2spec/test/altair/light_client/test_sync.py index b1bb13ee93..acd64cf3bd 100644 --- a/tests/core/pyspec/eth2spec/test/altair/light_client/test_sync.py +++ b/tests/core/pyspec/eth2spec/test/altair/light_client/test_sync.py @@ -16,15 +16,16 @@ state_transition_with_full_block, ) from eth2spec.test.helpers.constants import ( - ALTAIR, BELLATRIX, CAPELLA, DENEB, + ALTAIR, BELLATRIX, CAPELLA, DENEB, ELECTRA, MINIMAL, ) from eth2spec.test.helpers.fork_transition import ( do_fork, + transition_across_forks, ) from eth2spec.test.helpers.forks import ( get_spec_for_fork_version, - is_post_capella, is_post_deneb, + is_post_capella, is_post_deneb, is_post_electra, ) from eth2spec.test.helpers.light_client import ( compute_start_slot_at_next_sync_committee_period, @@ -47,6 +48,8 @@ class LightClientSyncTest(object): def get_store_fork_version(s_spec): + if is_post_electra(s_spec): + return s_spec.config.ELECTRA_FORK_VERSION if is_post_deneb(s_spec): return s_spec.config.DENEB_FORK_VERSION if is_post_capella(s_spec): @@ -60,6 +63,11 @@ def setup_test(spec, state, s_spec=None, phases=None): if s_spec is None: s_spec = spec + if phases is None: + phases = { + spec.fork: spec, + s_spec.fork: s_spec, + } test.s_spec = s_spec yield "genesis_validators_root", "meta", "0x" + state.genesis_validators_root.hex() @@ -77,7 +85,7 @@ def setup_test(spec, state, s_spec=None, phases=None): yield "bootstrap_fork_digest", "meta", encode_hex(data_fork_digest) yield "bootstrap", data - upgraded = upgrade_lc_bootstrap_to_new_spec(d_spec, test.s_spec, data) + upgraded = upgrade_lc_bootstrap_to_new_spec(d_spec, test.s_spec, data, phases) test.store = test.s_spec.initialize_light_client_store(trusted_block_root, upgraded) store_fork_version = get_store_fork_version(test.s_spec) store_fork_digest = test.s_spec.compute_fork_digest(store_fork_version, test.genesis_validators_root) @@ -153,7 +161,7 @@ def emit_update(test, spec, state, block, attested_state, attested_block, finali [spec.Bytes32() for _ in range(spec.floorlog2(spec.NEXT_SYNC_COMMITTEE_GINDEX))] current_slot = state.slot - upgraded = upgrade_lc_update_to_new_spec(d_spec, test.s_spec, data) + upgraded = upgrade_lc_update_to_new_spec(d_spec, test.s_spec, data, phases) test.s_spec.process_light_client_update(test.store, upgraded, current_slot, test.genesis_validators_root) yield get_update_file_name(d_spec, data), data @@ -169,7 +177,7 @@ def emit_update(test, spec, state, block, attested_state, attested_block, finali def emit_upgrade_store(test, new_s_spec, phases=None): - test.store = upgrade_lc_store_to_new_spec(test.s_spec, new_s_spec, test.store) + test.store = upgrade_lc_store_to_new_spec(test.s_spec, new_s_spec, test.store, phases) test.s_spec = new_s_spec store_fork_version = get_store_fork_version(test.s_spec) store_fork_digest = test.s_spec.compute_fork_digest(store_fork_version, test.genesis_validators_root) @@ -561,7 +569,7 @@ def run_test_single_fork(spec, phases, state, fork): # Upgrade to post-fork spec, attested block is still before the fork attested_block = block.copy() attested_state = state.copy() - sync_aggregate, _ = get_sync_aggregate(phases[fork], state, phases=phases) + sync_aggregate, _ = get_sync_aggregate(spec, state, phases=phases) state, block = do_fork(state, spec, phases[fork], fork_epoch, sync_aggregate=sync_aggregate) spec = phases[fork] yield from emit_update(test, spec, state, block, attested_state, attested_block, finalized_block, phases=phases) @@ -635,6 +643,18 @@ def test_deneb_fork(spec, phases, state): yield from run_test_single_fork(spec, phases, state, DENEB) +@with_phases(phases=[DENEB], other_phases=[ELECTRA]) +@spec_test +@with_config_overrides({ + 'ELECTRA_FORK_EPOCH': 3, # `setup_test` advances to epoch 2 +}, emit=False) +@with_state +@with_matching_spec_config(emitted_fork=ELECTRA) +@with_presets([MINIMAL], reason="too slow") +def test_electra_fork(spec, phases, state): + yield from run_test_single_fork(spec, phases, state, ELECTRA) + + def run_test_multi_fork(spec, phases, state, fork_1, fork_2): # Start test test = yield from setup_test(spec, state, phases[fork_2], phases) @@ -646,17 +666,28 @@ def run_test_multi_fork(spec, phases, state, fork_1, fork_2): # ..., attested is from `fork_1`, ... fork_1_epoch = getattr(phases[fork_1].config, fork_1.upper() + '_FORK_EPOCH') - transition_to(spec, state, spec.compute_start_slot_at_epoch(fork_1_epoch) - 1) - state, attested_block = do_fork(state, spec, phases[fork_1], fork_1_epoch) - spec = phases[fork_1] + spec, state, attested_block = transition_across_forks( + spec, + state, + spec.compute_start_slot_at_epoch(fork_1_epoch), + phases, + with_block=True, + ) attested_state = state.copy() # ..., and signature is from `fork_2` fork_2_epoch = getattr(phases[fork_2].config, fork_2.upper() + '_FORK_EPOCH') - transition_to(spec, state, spec.compute_start_slot_at_epoch(fork_2_epoch) - 1) - sync_aggregate, _ = get_sync_aggregate(phases[fork_2], state) - state, block = do_fork(state, spec, phases[fork_2], fork_2_epoch, sync_aggregate=sync_aggregate) - spec = phases[fork_2] + spec, state, _ = transition_across_forks( + spec, state, spec.compute_start_slot_at_epoch(fork_2_epoch) - 1, phases) + sync_aggregate, _ = get_sync_aggregate(spec, state, phases=phases) + spec, state, block = transition_across_forks( + spec, + state, + spec.compute_start_slot_at_epoch(fork_2_epoch), + phases, + with_block=True, + sync_aggregate=sync_aggregate, + ) # Check that update applies yield from emit_update(test, spec, state, block, attested_state, attested_block, finalized_block, phases=phases) @@ -682,6 +713,33 @@ def test_capella_deneb_fork(spec, phases, state): yield from run_test_multi_fork(spec, phases, state, CAPELLA, DENEB) +@with_phases(phases=[BELLATRIX], other_phases=[CAPELLA, DENEB, ELECTRA]) +@spec_test +@with_config_overrides({ + 'CAPELLA_FORK_EPOCH': 3, # `setup_test` advances to epoch 2 + 'DENEB_FORK_EPOCH': 4, + 'ELECTRA_FORK_EPOCH': 5, +}, emit=False) +@with_state +@with_matching_spec_config(emitted_fork=ELECTRA) +@with_presets([MINIMAL], reason="too slow") +def test_capella_electra_fork(spec, phases, state): + yield from run_test_multi_fork(spec, phases, state, CAPELLA, ELECTRA) + + +@with_phases(phases=[CAPELLA], other_phases=[DENEB, ELECTRA]) +@spec_test +@with_config_overrides({ + 'DENEB_FORK_EPOCH': 3, # `setup_test` advances to epoch 2 + 'ELECTRA_FORK_EPOCH': 4, +}, emit=False) +@with_state +@with_matching_spec_config(emitted_fork=ELECTRA) +@with_presets([MINIMAL], reason="too slow") +def test_deneb_electra_fork(spec, phases, state): + yield from run_test_multi_fork(spec, phases, state, DENEB, ELECTRA) + + def run_test_upgraded_store_with_legacy_data(spec, phases, state, fork): # Start test (Legacy bootstrap with an upgraded store) test = yield from setup_test(spec, state, phases[fork], phases) @@ -713,10 +771,19 @@ def test_capella_store_with_legacy_data(spec, phases, state): yield from run_test_upgraded_store_with_legacy_data(spec, phases, state, CAPELLA) -@with_phases(phases=[ALTAIR, BELLATRIX, CAPELLA], other_phases=[DENEB]) +@with_phases(phases=[ALTAIR, BELLATRIX, CAPELLA], other_phases=[CAPELLA, DENEB]) @spec_test @with_state @with_matching_spec_config(emitted_fork=DENEB) @with_presets([MINIMAL], reason="too slow") def test_deneb_store_with_legacy_data(spec, phases, state): yield from run_test_upgraded_store_with_legacy_data(spec, phases, state, DENEB) + + +@with_phases(phases=[ALTAIR, BELLATRIX, CAPELLA, DENEB], other_phases=[CAPELLA, DENEB, ELECTRA]) +@spec_test +@with_state +@with_matching_spec_config(emitted_fork=ELECTRA) +@with_presets([MINIMAL], reason="too slow") +def test_electra_store_with_legacy_data(spec, phases, state): + yield from run_test_upgraded_store_with_legacy_data(spec, phases, state, ELECTRA) diff --git a/tests/core/pyspec/eth2spec/test/helpers/light_client.py b/tests/core/pyspec/eth2spec/test/helpers/light_client.py index 3bdaa46497..53544c5555 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/light_client.py +++ b/tests/core/pyspec/eth2spec/test/helpers/light_client.py @@ -1,8 +1,11 @@ +from eth2spec.test.helpers.constants import ( + CAPELLA, DENEB, ELECTRA, +) from eth2spec.test.helpers.fork_transition import ( transition_across_forks, ) from eth2spec.test.helpers.forks import ( - is_post_capella, is_post_deneb, + is_post_capella, is_post_deneb, is_post_electra ) from eth2spec.test.helpers.sync_committee import ( compute_aggregate_sync_committee_signature, @@ -88,6 +91,20 @@ def needs_upgrade_to_deneb(spec, new_spec): return is_post_deneb(new_spec) and not is_post_deneb(spec) +def needs_upgrade_to_electra(spec, new_spec): + return is_post_electra(new_spec) and not is_post_electra(spec) + + +def check_merkle_branch_equal(spec, new_spec, data, upgraded, gindex): + if is_post_electra(new_spec): + assert ( + new_spec.normalize_merkle_branch(upgraded, gindex) + == new_spec.normalize_merkle_branch(data, gindex) + ) + else: + assert upgraded == data + + def check_lc_header_equal(spec, new_spec, data, upgraded): assert upgraded.beacon.slot == data.beacon.slot assert upgraded.beacon.hash_tree_root() == data.beacon.hash_tree_root() @@ -98,15 +115,19 @@ def check_lc_header_equal(spec, new_spec, data, upgraded): assert new_spec.get_lc_execution_root(upgraded) == new_spec.Root() -def upgrade_lc_header_to_new_spec(spec, new_spec, data): +def upgrade_lc_header_to_new_spec(spec, new_spec, data, phases): upgraded = data if needs_upgrade_to_capella(spec, new_spec): - upgraded = new_spec.upgrade_lc_header_to_capella(upgraded) + upgraded = phases[CAPELLA].upgrade_lc_header_to_capella(upgraded) check_lc_header_equal(spec, new_spec, data, upgraded) if needs_upgrade_to_deneb(spec, new_spec): - upgraded = new_spec.upgrade_lc_header_to_deneb(upgraded) + upgraded = phases[DENEB].upgrade_lc_header_to_deneb(upgraded) + check_lc_header_equal(spec, new_spec, data, upgraded) + + if needs_upgrade_to_electra(spec, new_spec): + upgraded = phases[ELECTRA].upgrade_lc_header_to_electra(upgraded) check_lc_header_equal(spec, new_spec, data, upgraded) return upgraded @@ -115,18 +136,28 @@ def upgrade_lc_header_to_new_spec(spec, new_spec, data): def check_lc_bootstrap_equal(spec, new_spec, data, upgraded): check_lc_header_equal(spec, new_spec, data.header, upgraded.header) assert upgraded.current_sync_committee == data.current_sync_committee - assert upgraded.current_sync_committee_branch == data.current_sync_committee_branch + check_merkle_branch_equal( + spec, + new_spec, + data.current_sync_committee_branch, + upgraded.current_sync_committee_branch, + new_spec.CURRENT_SYNC_COMMITTEE_GINDEX, + ) -def upgrade_lc_bootstrap_to_new_spec(spec, new_spec, data): +def upgrade_lc_bootstrap_to_new_spec(spec, new_spec, data, phases): upgraded = data if needs_upgrade_to_capella(spec, new_spec): - upgraded = new_spec.upgrade_lc_bootstrap_to_capella(upgraded) + upgraded = phases[CAPELLA].upgrade_lc_bootstrap_to_capella(upgraded) check_lc_bootstrap_equal(spec, new_spec, data, upgraded) if needs_upgrade_to_deneb(spec, new_spec): - upgraded = new_spec.upgrade_lc_bootstrap_to_deneb(upgraded) + upgraded = phases[DENEB].upgrade_lc_bootstrap_to_deneb(upgraded) + check_lc_bootstrap_equal(spec, new_spec, data, upgraded) + + if needs_upgrade_to_electra(spec, new_spec): + upgraded = phases[ELECTRA].upgrade_lc_bootstrap_to_electra(upgraded) check_lc_bootstrap_equal(spec, new_spec, data, upgraded) return upgraded @@ -135,21 +166,38 @@ def upgrade_lc_bootstrap_to_new_spec(spec, new_spec, data): def check_lc_update_equal(spec, new_spec, data, upgraded): check_lc_header_equal(spec, new_spec, data.attested_header, upgraded.attested_header) assert upgraded.next_sync_committee == data.next_sync_committee - assert upgraded.next_sync_committee_branch == data.next_sync_committee_branch + check_merkle_branch_equal( + spec, + new_spec, + data.next_sync_committee_branch, + upgraded.next_sync_committee_branch, + new_spec.NEXT_SYNC_COMMITTEE_GINDEX, + ) check_lc_header_equal(spec, new_spec, data.finalized_header, upgraded.finalized_header) + check_merkle_branch_equal( + spec, + new_spec, + data.finality_branch, + upgraded.finality_branch, + new_spec.FINALIZED_ROOT_GINDEX, + ) assert upgraded.sync_aggregate == data.sync_aggregate assert upgraded.signature_slot == data.signature_slot -def upgrade_lc_update_to_new_spec(spec, new_spec, data): +def upgrade_lc_update_to_new_spec(spec, new_spec, data, phases): upgraded = data if needs_upgrade_to_capella(spec, new_spec): - upgraded = new_spec.upgrade_lc_update_to_capella(upgraded) + upgraded = phases[CAPELLA].upgrade_lc_update_to_capella(upgraded) check_lc_update_equal(spec, new_spec, data, upgraded) if needs_upgrade_to_deneb(spec, new_spec): - upgraded = new_spec.upgrade_lc_update_to_deneb(upgraded) + upgraded = phases[DENEB].upgrade_lc_update_to_deneb(upgraded) + check_lc_update_equal(spec, new_spec, data, upgraded) + + if needs_upgrade_to_electra(spec, new_spec): + upgraded = phases[ELECTRA].upgrade_lc_update_to_electra(upgraded) check_lc_update_equal(spec, new_spec, data, upgraded) return upgraded @@ -158,19 +206,30 @@ def upgrade_lc_update_to_new_spec(spec, new_spec, data): def check_lc_finality_update_equal(spec, new_spec, data, upgraded): check_lc_header_equal(spec, new_spec, data.attested_header, upgraded.attested_header) check_lc_header_equal(spec, new_spec, data.finalized_header, upgraded.finalized_header) + check_merkle_branch_equal( + spec, + new_spec, + data.finality_branch, + upgraded.finality_branch, + new_spec.FINALIZED_ROOT_GINDEX, + ) assert upgraded.sync_aggregate == data.sync_aggregate assert upgraded.signature_slot == data.signature_slot -def upgrade_lc_finality_update_to_new_spec(spec, new_spec, data): +def upgrade_lc_finality_update_to_new_spec(spec, new_spec, data, phases): upgraded = data if needs_upgrade_to_capella(spec, new_spec): - upgraded = new_spec.upgrade_lc_finality_update_to_capella(upgraded) + upgraded = phases[CAPELLA].upgrade_lc_finality_update_to_capella(upgraded) check_lc_finality_update_equal(spec, new_spec, data, upgraded) if needs_upgrade_to_deneb(spec, new_spec): - upgraded = new_spec.upgrade_lc_finality_update_to_deneb(upgraded) + upgraded = phases[DENEB].upgrade_lc_finality_update_to_deneb(upgraded) + check_lc_finality_update_equal(spec, new_spec, data, upgraded) + + if needs_upgrade_to_electra(spec, new_spec): + upgraded = phases[ELECTRA].upgrade_lc_finality_update_to_electra(upgraded) check_lc_finality_update_equal(spec, new_spec, data, upgraded) return upgraded @@ -189,15 +248,19 @@ def check_lc_store_equal(spec, new_spec, data, upgraded): assert upgraded.current_max_active_participants == data.current_max_active_participants -def upgrade_lc_store_to_new_spec(spec, new_spec, data): +def upgrade_lc_store_to_new_spec(spec, new_spec, data, phases): upgraded = data if needs_upgrade_to_capella(spec, new_spec): - upgraded = new_spec.upgrade_lc_store_to_capella(upgraded) + upgraded = phases[CAPELLA].upgrade_lc_store_to_capella(upgraded) check_lc_store_equal(spec, new_spec, data, upgraded) if needs_upgrade_to_deneb(spec, new_spec): - upgraded = new_spec.upgrade_lc_store_to_deneb(upgraded) + upgraded = phases[DENEB].upgrade_lc_store_to_deneb(upgraded) + check_lc_store_equal(spec, new_spec, data, upgraded) + + if needs_upgrade_to_electra(spec, new_spec): + upgraded = phases[ELECTRA].upgrade_lc_store_to_electra(upgraded) check_lc_store_equal(spec, new_spec, data, upgraded) return upgraded diff --git a/tests/generators/light_client/main.py b/tests/generators/light_client/main.py index cfe34aee4b..a3cdfd62fd 100644 --- a/tests/generators/light_client/main.py +++ b/tests/generators/light_client/main.py @@ -1,4 +1,4 @@ -from eth2spec.test.helpers.constants import ALTAIR, BELLATRIX, CAPELLA, DENEB +from eth2spec.test.helpers.constants import ALTAIR, BELLATRIX, CAPELLA, DENEB, ELECTRA from eth2spec.gen_helpers.gen_from_tests.gen import combine_mods, run_state_test_generators @@ -15,12 +15,14 @@ ]} capella_mods = combine_mods(_new_capella_mods, bellatrix_mods) deneb_mods = capella_mods + electra_mods = deneb_mods all_mods = { ALTAIR: altair_mods, BELLATRIX: bellatrix_mods, CAPELLA: capella_mods, DENEB: deneb_mods, + ELECTRA: electra_mods, } run_state_test_generators(runner_name="light_client", all_mods=all_mods) From ea04148a0d8aed539eafb681b58f09377db2f4d4 Mon Sep 17 00:00:00 2001 From: Etan Kissling Date: Fri, 21 Jun 2024 11:44:33 +0200 Subject: [PATCH 019/130] Lint --- specs/electra/light-client/sync-protocol.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/electra/light-client/sync-protocol.md b/specs/electra/light-client/sync-protocol.md index 4bb597abaf..6cfef2a5fe 100644 --- a/specs/electra/light-client/sync-protocol.md +++ b/specs/electra/light-client/sync-protocol.md @@ -47,7 +47,7 @@ def finalized_root_gindex_at_slot(slot: Slot) -> GeneralizedIndex: # [Modified in Electra] if epoch >= ELECTRA_FORK_EPOCH: return FINALIZED_ROOT_GINDEX - return capella.FINALIZED_ROOT_GINDEX + return GeneralizedIndex(capella.FINALIZED_ROOT_GINDEX) ``` ### Modified `current_sync_committee_gindex_at_slot` @@ -59,7 +59,7 @@ def current_sync_committee_gindex_at_slot(slot: Slot) -> GeneralizedIndex: # [Modified in Electra] if epoch >= ELECTRA_FORK_EPOCH: return CURRENT_SYNC_COMMITTEE_GINDEX - return capella.CURRENT_SYNC_COMMITTEE_GINDEX + return GeneralizedIndex(capella.CURRENT_SYNC_COMMITTEE_GINDEX) ``` ### Modified `next_sync_committee_gindex_at_slot` @@ -71,7 +71,7 @@ def next_sync_committee_gindex_at_slot(slot: Slot) -> GeneralizedIndex: # [Modified in Electra] if epoch >= ELECTRA_FORK_EPOCH: return NEXT_SYNC_COMMITTEE_GINDEX - return capella.NEXT_SYNC_COMMITTEE_GINDEX + return GeneralizedIndex(capella.NEXT_SYNC_COMMITTEE_GINDEX) ``` ### Modified `get_lc_execution_root` From 67a053d6ce53965aa4bd658a39a76e9bf93f0b69 Mon Sep 17 00:00:00 2001 From: Etan Kissling Date: Fri, 21 Jun 2024 14:39:01 +0200 Subject: [PATCH 020/130] Refer to Altair gindices as that's where they were introduced --- specs/electra/light-client/sync-protocol.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/electra/light-client/sync-protocol.md b/specs/electra/light-client/sync-protocol.md index 6cfef2a5fe..53a3bcb78b 100644 --- a/specs/electra/light-client/sync-protocol.md +++ b/specs/electra/light-client/sync-protocol.md @@ -47,7 +47,7 @@ def finalized_root_gindex_at_slot(slot: Slot) -> GeneralizedIndex: # [Modified in Electra] if epoch >= ELECTRA_FORK_EPOCH: return FINALIZED_ROOT_GINDEX - return GeneralizedIndex(capella.FINALIZED_ROOT_GINDEX) + return GeneralizedIndex(altair.FINALIZED_ROOT_GINDEX) ``` ### Modified `current_sync_committee_gindex_at_slot` @@ -59,7 +59,7 @@ def current_sync_committee_gindex_at_slot(slot: Slot) -> GeneralizedIndex: # [Modified in Electra] if epoch >= ELECTRA_FORK_EPOCH: return CURRENT_SYNC_COMMITTEE_GINDEX - return GeneralizedIndex(capella.CURRENT_SYNC_COMMITTEE_GINDEX) + return GeneralizedIndex(altair.CURRENT_SYNC_COMMITTEE_GINDEX) ``` ### Modified `next_sync_committee_gindex_at_slot` @@ -71,7 +71,7 @@ def next_sync_committee_gindex_at_slot(slot: Slot) -> GeneralizedIndex: # [Modified in Electra] if epoch >= ELECTRA_FORK_EPOCH: return NEXT_SYNC_COMMITTEE_GINDEX - return GeneralizedIndex(capella.NEXT_SYNC_COMMITTEE_GINDEX) + return GeneralizedIndex(altair.NEXT_SYNC_COMMITTEE_GINDEX) ``` ### Modified `get_lc_execution_root` From 835576a47c54cd6351278950118b9dfc7e757ea4 Mon Sep 17 00:00:00 2001 From: Etan Kissling Date: Fri, 21 Jun 2024 14:49:33 +0200 Subject: [PATCH 021/130] Enable light client data test generation for Electra --- tests/core/pyspec/eth2spec/test/helpers/constants.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/core/pyspec/eth2spec/test/helpers/constants.py b/tests/core/pyspec/eth2spec/test/helpers/constants.py index ed398516cd..6987ec9728 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/constants.py +++ b/tests/core/pyspec/eth2spec/test/helpers/constants.py @@ -39,7 +39,7 @@ EIP7594, ) # The forks that have light client specs -LIGHT_CLIENT_TESTING_FORKS = (*[item for item in MAINNET_FORKS if item != PHASE0],) +LIGHT_CLIENT_TESTING_FORKS = (*[item for item in MAINNET_FORKS if item != PHASE0], ELECTRA) # The forks that output to the test vectors. TESTGEN_FORKS = (*MAINNET_FORKS, ELECTRA, EIP7594, WHISK) # Forks allowed in the test runner `--fork` flag, to fail fast in case of typos From f30a3a3b055f9f1d8051bd92dda917a96597da05 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Mon, 24 Jun 2024 23:34:08 +0800 Subject: [PATCH 022/130] Update `test_incorrect_not_enough_consolidation_churn_available` and add assertions to test cases --- .../test_process_consolidation_request.py | 45 ++++++++++++++++++- 1 file changed, 43 insertions(+), 2 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/electra/block_processing/test_process_consolidation_request.py b/tests/core/pyspec/eth2spec/test/electra/block_processing/test_process_consolidation_request.py index b12438d0d1..b05c20f565 100644 --- a/tests/core/pyspec/eth2spec/test/electra/block_processing/test_process_consolidation_request.py +++ b/tests/core/pyspec/eth2spec/test/electra/block_processing/test_process_consolidation_request.py @@ -421,6 +421,9 @@ def test_incorrect_source_equals_target(spec, state): target_pubkey=state.validators[source_index].pubkey, ) + # Check the the return condition + assert consolidation.source_pubkey == consolidation.target_pubkey + yield from run_consolidation_processing( spec, state, consolidation, success=False ) @@ -454,6 +457,9 @@ def test_incorrect_exceed_pending_consolidations_limit(spec, state): ) set_eth1_withdrawal_credential_with_balance(spec, state, target_index) + # Check the the return condition + assert len(state.pending_consolidations) == spec.PENDING_CONSOLIDATIONS_LIMIT + yield from run_consolidation_processing( spec, state, consolidation, success=False ) @@ -463,7 +469,6 @@ def test_incorrect_exceed_pending_consolidations_limit(spec, state): @spec_state_test @single_phase def test_incorrect_not_enough_consolidation_churn_available(spec, state): - state.validators = state.validators[0:2] state.pending_consolidations = [ spec.PendingConsolidation(source_index=0, target_index=1) ] @@ -481,7 +486,12 @@ def test_incorrect_not_enough_consolidation_churn_available(spec, state): source_pubkey=state.validators[source_index].pubkey, target_pubkey=state.validators[target_index].pubkey, ) + set_eth1_withdrawal_credential_with_balance(spec, state, target_index) + + # Check the the return condition + assert spec.get_consolidation_churn_limit(state) <= spec.MIN_ACTIVATION_BALANCE + yield from run_consolidation_processing( spec, state, consolidation, success=False ) @@ -514,6 +524,9 @@ def test_incorrect_exited_source(spec, state): # exit source spec.initiate_validator_exit(state, source_index) + # Check the the return condition + assert state.validators[source_index].exit_epoch != spec.FAR_FUTURE_EPOCH + yield from run_consolidation_processing( spec, state, consolidation, success=False ) @@ -544,6 +557,10 @@ def test_incorrect_exited_target(spec, state): set_eth1_withdrawal_credential_with_balance(spec, state, target_index) # exit target spec.initiate_validator_exit(state, 1) + + # Check the the return condition + assert state.validators[target_index].exit_epoch != spec.FAR_FUTURE_EPOCH + yield from run_consolidation_processing( spec, state, consolidation, success=False ) @@ -576,6 +593,9 @@ def test_incorrect_inactive_source(spec, state): # set source validator as not yet activated state.validators[source_index].activation_epoch = spec.FAR_FUTURE_EPOCH + # Check the the return condition + assert not spec.is_active_validator(state.validators[source_index], current_epoch) + yield from run_consolidation_processing( spec, state, consolidation, success=False ) @@ -607,6 +627,10 @@ def test_incorrect_inactive_target(spec, state): # set target validator as not yet activated state.validators[1].activation_epoch = spec.FAR_FUTURE_EPOCH + + # Check the the return condition + assert not spec.is_active_validator(state.validators[target_index], current_epoch) + yield from run_consolidation_processing( spec, state, consolidation, success=False ) @@ -633,6 +657,10 @@ def test_incorrect_no_source_execution_withdrawal_credential(spec, state): target_pubkey=state.validators[target_index].pubkey, ) set_eth1_withdrawal_credential_with_balance(spec, state, target_index) + + # Check the the return condition + assert not spec.has_execution_withdrawal_credential(state.validators[source_index]) + yield from run_consolidation_processing( spec, state, consolidation, success=False ) @@ -661,6 +689,10 @@ def test_incorrect_no_target_execution_withdrawal_credential(spec, state): source_pubkey=state.validators[source_index].pubkey, target_pubkey=state.validators[target_index].pubkey, ) + + # Check the the return condition + assert not spec.has_execution_withdrawal_credential(state.validators[target_index]) + yield from run_consolidation_processing( spec, state, consolidation, success=False ) @@ -691,6 +723,9 @@ def test_incorrect_incorrect_source_address(spec, state): ) set_eth1_withdrawal_credential_with_balance(spec, state, target_index) + # Check the the return condition + assert not state.validators[source_index].withdrawal_credentials[12:] == consolidation.source_address + yield from run_consolidation_processing( spec, state, consolidation, success=False ) @@ -721,6 +756,9 @@ def test_incorrect_unknown_source_pubkey(spec, state): ) set_eth1_withdrawal_credential_with_balance(spec, state, target_index) + # Check the the return condition + assert not state.validators[source_index].pubkey == consolidation.source_pubkey + yield from run_consolidation_processing( spec, state, consolidation, success=False ) @@ -751,6 +789,9 @@ def test_incorrect_unknown_target_pubkey(spec, state): ) set_eth1_withdrawal_credential_with_balance(spec, state, target_index) + # Check the the return condition + assert not state.validators[target_index].pubkey == consolidation.target_pubkey + yield from run_consolidation_processing( spec, state, consolidation, success=False ) @@ -762,7 +803,7 @@ def run_consolidation_processing(spec, state, consolidation, success=True): - pre-state ('pre') - consolidation_request ('consolidation_request') - post-state ('post'). - If ``valid == False``, run expecting ``AssertionError`` + If ``success == False``, ``process_consolidation_request`` would return without any state change. """ if success: From beedf852cb81827772fcff7fc78137601864e204 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 25 Jun 2024 16:15:16 +0800 Subject: [PATCH 023/130] Revert division change and add comments --- specs/_features/eip7594/das-core.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/specs/_features/eip7594/das-core.md b/specs/_features/eip7594/das-core.md index aec3a276e4..fe01a5e867 100644 --- a/specs/_features/eip7594/das-core.md +++ b/specs/_features/eip7594/das-core.md @@ -206,7 +206,7 @@ def get_data_column_sidecars(signed_block: SignedBeaconBlock, def get_extended_sample_count(allowed_failures: uint64) -> uint64: assert 0 <= allowed_failures <= NUMBER_OF_COLUMNS // 2 - def math_comb(n, k): + def math_comb(n: int, k: int) -> int: if not 0 <= k <= n: return 0 r = 1 @@ -214,12 +214,14 @@ def get_extended_sample_count(allowed_failures: uint64) -> uint64: r = r * (n - i) // (i + 1) return r - def hypergeom_cdf(k, M, n, N): + def hypergeom_cdf(k: uint64, M: uint64, n: uint64, N: uint64) -> float: + # NOTE: It contains float-point computations. + # Convert uint64 to Python integers before computations. k = int(k) M = int(M) n = int(n) N = int(N) - return sum([math_comb(n, i) * math_comb(M - n, N - i) // math_comb(M, N) + return sum([math_comb(n, i) * math_comb(M - n, N - i) / math_comb(M, N) for i in range(k + 1)]) worst_case_missing = NUMBER_OF_COLUMNS // 2 + 1 From 4539265c5579d442154f2f9248072ad5bb511944 Mon Sep 17 00:00:00 2001 From: Csaba Kiraly Date: Tue, 25 Jun 2024 10:36:53 +0200 Subject: [PATCH 024/130] moving part of text in custody section Signed-off-by: Csaba Kiraly --- specs/_features/eip7594/das-core.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/specs/_features/eip7594/das-core.md b/specs/_features/eip7594/das-core.md index e63ead25c7..9de811efdf 100644 --- a/specs/_features/eip7594/das-core.md +++ b/specs/_features/eip7594/das-core.md @@ -223,7 +223,7 @@ In this construction, we extend the blobs using a one-dimensional erasure coding For each column -- use `data_column_sidecar_{subnet_id}` subnets, where `subnet_id` can be computed with the `compute_subnet_for_data_column_sidecar(column_index: ColumnIndex)` helper. The sidecars can be computed with the `get_data_column_sidecars(signed_block: SignedBeaconBlock, blobs: Sequence[Blob])` helper. -To custody a particular column, a node joins the respective gossip subnet. Verifiable samples from their respective column are gossiped on the assigned subnet. +Verifiable samples from their respective column are distributed on the assigned subnet. To custody a particular column, a node joins the respective gossipsub subnet. If a node fails to get a column on the column subnet, a node can also utilize the Req/Resp protocol to query the missing column from other peers. ## Peer sampling @@ -235,8 +235,6 @@ Due to the deterministic custody functions, a node knows exactly what a peer sho ## Reconstruction and cross-seeding -If a node fails to get a column on the column subnet, a node can also utilize the Req/Resp message to query the missing column from other peers. - If the node obtains 50%+ of all the columns, it SHOULD reconstruct the full data matrix via `recover_matrix` helper. Nodes MAY delay this reconstruction allowing time for other columns to arrive over the network. If delaying reconstruction, nodes may use a random delay in order to desynchronize reconstruction among nodes, thus reducing overall CPU load. Once the node obtains a column through reconstruction, the node MUST expose the new column as if it had received it over the network. If the node is subscribed to the subnet corresponding to the column, it MUST send the reconstructed DataColumnSidecar to its topic mesh neighbors. If instead the node is not subscribed to the corresponding subnet, it SHOULD still expose the availability of the DataColumnSidecar as part of the gossip emission process. From ee977381de036cee998a17f0b65e7ebdc8bd0413 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Tue, 25 Jun 2024 16:48:38 +0800 Subject: [PATCH 025/130] Add `test_get_extended_sample_count__table_in_spec` to verify the table content in the spec --- .../test/eip7594/unittests/das/test_das.py | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/tests/core/pyspec/eth2spec/test/eip7594/unittests/das/test_das.py b/tests/core/pyspec/eth2spec/test/eip7594/unittests/das/test_das.py index 283a1e4255..7110f2373e 100644 --- a/tests/core/pyspec/eth2spec/test/eip7594/unittests/das/test_das.py +++ b/tests/core/pyspec/eth2spec/test/eip7594/unittests/das/test_das.py @@ -3,6 +3,7 @@ expect_assertion_error, spec_test, single_phase, + with_config_overrides, with_eip7594_and_later, ) from eth2spec.test.helpers.sharding import ( @@ -116,3 +117,29 @@ def test_get_extended_sample_count__upper_bound(spec): def test_get_extended_sample_count__upper_bound_exceed(spec): allowed_failures = spec.config.NUMBER_OF_COLUMNS // 2 + 1 expect_assertion_error(lambda: spec.get_extended_sample_count(allowed_failures)) + + +@with_eip7594_and_later +@spec_test +@with_config_overrides({ + 'NUMBER_OF_COLUMNS': 128, + 'SAMPLES_PER_SLOT': 16, +}) +@single_phase +def test_get_extended_sample_count__table_in_spec(spec): + table = dict( + # (allowed_failures, expected_extended_sample_count) + { + 0: 16, + 1: 20, + 2: 24, + 3: 27, + 4: 29, + 5: 32, + 6: 35, + 7: 37, + 8: 40, + } + ) + for allowed_failures, expected_extended_sample_count in table.items(): + assert spec.get_extended_sample_count(allowed_failures=allowed_failures) == expected_extended_sample_count From ded072af400c53c9cd19cc426ff004b9db100bd2 Mon Sep 17 00:00:00 2001 From: dapplion <35266934+dapplion@users.noreply.github.com> Date: Tue, 25 Jun 2024 18:22:16 +0200 Subject: [PATCH 026/130] Make MAX_BLOBS_PER_BLOCK a config parameter --- configs/mainnet.yaml | 2 ++ configs/minimal.yaml | 2 ++ presets/mainnet/deneb.yaml | 2 -- presets/minimal/deneb.yaml | 2 -- specs/deneb/beacon-chain.md | 9 +++++++-- .../block_processing/test_process_execution_payload.py | 2 +- .../test/deneb/unittests/test_config_invariants.py | 8 ++++---- .../test/eip7594/unittests/test_config_invariants.py | 2 +- .../eth2spec/test/eip7594/unittests/test_security.py | 2 +- tests/core/pyspec/eth2spec/test/helpers/fork_choice.py | 2 +- .../pyspec/eth2spec/test/utils/randomized_block_tests.py | 2 +- 11 files changed, 20 insertions(+), 15 deletions(-) diff --git a/configs/mainnet.yaml b/configs/mainnet.yaml index 7051873ce9..39904616b2 100644 --- a/configs/mainnet.yaml +++ b/configs/mainnet.yaml @@ -147,6 +147,8 @@ MAX_REQUEST_BLOB_SIDECARS: 768 MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS: 4096 # `6` BLOB_SIDECAR_SUBNET_COUNT: 6 +## `uint64(6)` +MAX_BLOBS_PER_BLOCK: 6 # Whisk # `Epoch(2**8)` diff --git a/configs/minimal.yaml b/configs/minimal.yaml index 8e2a222d47..8ba65cec09 100644 --- a/configs/minimal.yaml +++ b/configs/minimal.yaml @@ -148,6 +148,8 @@ MAX_REQUEST_BLOB_SIDECARS: 768 MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS: 4096 # `6` BLOB_SIDECAR_SUBNET_COUNT: 6 +## `uint64(6)` +MAX_BLOBS_PER_BLOCK: 6 # Whisk WHISK_EPOCHS_PER_SHUFFLING_PHASE: 4 diff --git a/presets/mainnet/deneb.yaml b/presets/mainnet/deneb.yaml index 0f56b8bdfa..f426d3ae1a 100644 --- a/presets/mainnet/deneb.yaml +++ b/presets/mainnet/deneb.yaml @@ -6,7 +6,5 @@ FIELD_ELEMENTS_PER_BLOB: 4096 # `uint64(2**12)` (= 4096) MAX_BLOB_COMMITMENTS_PER_BLOCK: 4096 -# `uint64(6)` -MAX_BLOBS_PER_BLOCK: 6 # `floorlog2(get_generalized_index(BeaconBlockBody, 'blob_kzg_commitments')) + 1 + ceillog2(MAX_BLOB_COMMITMENTS_PER_BLOCK)` = 4 + 1 + 12 = 17 KZG_COMMITMENT_INCLUSION_PROOF_DEPTH: 17 diff --git a/presets/minimal/deneb.yaml b/presets/minimal/deneb.yaml index bc4fe4369a..9d0db086b8 100644 --- a/presets/minimal/deneb.yaml +++ b/presets/minimal/deneb.yaml @@ -6,7 +6,5 @@ FIELD_ELEMENTS_PER_BLOB: 4096 # [customized] MAX_BLOB_COMMITMENTS_PER_BLOCK: 16 -# `uint64(6)` -MAX_BLOBS_PER_BLOCK: 6 # [customized] `floorlog2(get_generalized_index(BeaconBlockBody, 'blob_kzg_commitments')) + 1 + ceillog2(MAX_BLOB_COMMITMENTS_PER_BLOCK)` = 4 + 1 + 4 = 9 KZG_COMMITMENT_INCLUSION_PROOF_DEPTH: 9 diff --git a/specs/deneb/beacon-chain.md b/specs/deneb/beacon-chain.md index 4f036d3c93..67307c7e7b 100644 --- a/specs/deneb/beacon-chain.md +++ b/specs/deneb/beacon-chain.md @@ -77,13 +77,18 @@ Deneb is a consensus-layer upgrade containing a number of features. Including: | Name | Value | Description | | - | - | - | | `MAX_BLOB_COMMITMENTS_PER_BLOCK` | `uint64(2**12)` (= 4096) | *[New in Deneb:EIP4844]* hardfork independent fixed theoretical limit same as `LIMIT_BLOBS_PER_TX` (see EIP 4844) | + +## Configuration + +### Execution + +| Name | Value | Description | +| - | - | - | | `MAX_BLOBS_PER_BLOCK` | `uint64(6)` | *[New in Deneb:EIP4844]* maximum number of blobs in a single block limited by `MAX_BLOB_COMMITMENTS_PER_BLOCK` | *Note*: The blob transactions are packed into the execution payload by the EL/builder with their corresponding blobs being independently transmitted and are limited by `MAX_BLOB_GAS_PER_BLOCK // GAS_PER_BLOB`. However the CL limit is independently defined by `MAX_BLOBS_PER_BLOCK`. -## Configuration - ### Validator cycle | Name | Value | diff --git a/tests/core/pyspec/eth2spec/test/deneb/block_processing/test_process_execution_payload.py b/tests/core/pyspec/eth2spec/test/deneb/block_processing/test_process_execution_payload.py index b0937aac96..c6a407e220 100644 --- a/tests/core/pyspec/eth2spec/test/deneb/block_processing/test_process_execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/deneb/block_processing/test_process_execution_payload.py @@ -254,7 +254,7 @@ def test_invalid_correct_input__execution_invalid(spec, state): def test_invalid_exceed_max_blobs_per_block(spec, state): execution_payload = build_empty_execution_payload(spec, state) - opaque_tx, _, blob_kzg_commitments, _ = get_sample_opaque_tx(spec, blob_count=spec.MAX_BLOBS_PER_BLOCK + 1) + opaque_tx, _, blob_kzg_commitments, _ = get_sample_opaque_tx(spec, blob_count=spec.config.MAX_BLOBS_PER_BLOCK + 1) execution_payload.transactions = [opaque_tx] execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) diff --git a/tests/core/pyspec/eth2spec/test/deneb/unittests/test_config_invariants.py b/tests/core/pyspec/eth2spec/test/deneb/unittests/test_config_invariants.py index f3fa956d0a..e190f1d449 100644 --- a/tests/core/pyspec/eth2spec/test/deneb/unittests/test_config_invariants.py +++ b/tests/core/pyspec/eth2spec/test/deneb/unittests/test_config_invariants.py @@ -9,17 +9,17 @@ @spec_test @single_phase def test_length(spec): - assert spec.MAX_BLOBS_PER_BLOCK < spec.MAX_BLOB_COMMITMENTS_PER_BLOCK + assert spec.config.MAX_BLOBS_PER_BLOCK < spec.MAX_BLOB_COMMITMENTS_PER_BLOCK @with_deneb_and_later @spec_test @single_phase def test_networking(spec): - assert spec.MAX_BLOBS_PER_BLOCK < spec.MAX_BLOB_COMMITMENTS_PER_BLOCK - assert spec.config.MAX_REQUEST_BLOB_SIDECARS == spec.config.MAX_REQUEST_BLOCKS_DENEB * spec.MAX_BLOBS_PER_BLOCK + assert spec.config.MAX_BLOBS_PER_BLOCK < spec.MAX_BLOB_COMMITMENTS_PER_BLOCK + assert spec.config.MAX_REQUEST_BLOB_SIDECARS == spec.config.MAX_REQUEST_BLOCKS_DENEB * spec.config.MAX_BLOBS_PER_BLOCK # Start with the same size, but `BLOB_SIDECAR_SUBNET_COUNT` could potentially increase later. - assert spec.config.BLOB_SIDECAR_SUBNET_COUNT == spec.MAX_BLOBS_PER_BLOCK + assert spec.config.BLOB_SIDECAR_SUBNET_COUNT == spec.config.MAX_BLOBS_PER_BLOCK for i in range(spec.MAX_BLOB_COMMITMENTS_PER_BLOCK): gindex = spec.get_generalized_index(spec.BeaconBlockBody, 'blob_kzg_commitments', i) assert spec.floorlog2(gindex) == spec.KZG_COMMITMENT_INCLUSION_PROOF_DEPTH diff --git a/tests/core/pyspec/eth2spec/test/eip7594/unittests/test_config_invariants.py b/tests/core/pyspec/eth2spec/test/eip7594/unittests/test_config_invariants.py index 998bf35128..776ea883aa 100644 --- a/tests/core/pyspec/eth2spec/test/eip7594/unittests/test_config_invariants.py +++ b/tests/core/pyspec/eth2spec/test/eip7594/unittests/test_config_invariants.py @@ -18,7 +18,7 @@ def test_invariants(spec): assert spec.config.MAX_REQUEST_DATA_COLUMN_SIDECARS == ( spec.config.MAX_REQUEST_BLOCKS_DENEB * spec.config.NUMBER_OF_COLUMNS ) - assert spec.config.MAX_CELLS_IN_EXTENDED_MATRIX == spec.MAX_BLOBS_PER_BLOCK * spec.config.NUMBER_OF_COLUMNS + assert spec.config.MAX_CELLS_IN_EXTENDED_MATRIX == spec.config.MAX_BLOBS_PER_BLOCK * spec.config.NUMBER_OF_COLUMNS @with_eip7594_and_later diff --git a/tests/core/pyspec/eth2spec/test/eip7594/unittests/test_security.py b/tests/core/pyspec/eth2spec/test/eip7594/unittests/test_security.py index dd85a673e5..282433b9c9 100644 --- a/tests/core/pyspec/eth2spec/test/eip7594/unittests/test_security.py +++ b/tests/core/pyspec/eth2spec/test/eip7594/unittests/test_security.py @@ -19,7 +19,7 @@ def test_sampling_config(spec): security_requirement = 0.01 assert probability_of_unavailable <= security_requirement - column_size_in_bytes = spec.FIELD_ELEMENTS_PER_CELL * spec.BYTES_PER_FIELD_ELEMENT * spec.MAX_BLOBS_PER_BLOCK + column_size_in_bytes = spec.FIELD_ELEMENTS_PER_CELL * spec.BYTES_PER_FIELD_ELEMENT * spec.config.MAX_BLOBS_PER_BLOCK bytes_per_slot = column_size_in_bytes * spec.SAMPLES_PER_SLOT # TODO: What is the bandwidth requirement? bandwidth_requirement = 10000 # bytes/s diff --git a/tests/core/pyspec/eth2spec/test/helpers/fork_choice.py b/tests/core/pyspec/eth2spec/test/helpers/fork_choice.py index 094e2e8a5c..8598870fb6 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/fork_choice.py +++ b/tests/core/pyspec/eth2spec/test/helpers/fork_choice.py @@ -178,7 +178,7 @@ def add_block(spec, # Check blob_data if blob_data is not None: - blobs = spec.List[spec.Blob, spec.MAX_BLOBS_PER_BLOCK](blob_data.blobs) + blobs = spec.List[spec.Blob, spec.config.MAX_BLOBS_PER_BLOCK](blob_data.blobs) blobs_root = blobs.hash_tree_root() yield get_blobs_file_name(blobs_root=blobs_root), blobs diff --git a/tests/core/pyspec/eth2spec/test/utils/randomized_block_tests.py b/tests/core/pyspec/eth2spec/test/utils/randomized_block_tests.py index a527731b14..08bc3c2494 100644 --- a/tests/core/pyspec/eth2spec/test/utils/randomized_block_tests.py +++ b/tests/core/pyspec/eth2spec/test/utils/randomized_block_tests.py @@ -251,7 +251,7 @@ def random_block_deneb(spec, state, signed_blocks, scenario_state, rng=Random(34 block = random_block_capella(spec, state, signed_blocks, scenario_state, rng=rng) # TODO: more commitments. blob_kzg_commitments: List[KZGCommitment, MAX_BLOBS_PER_BLOCK] opaque_tx, _, blob_kzg_commitments, _ = get_sample_opaque_tx( - spec, blob_count=rng.randint(0, spec.MAX_BLOBS_PER_BLOCK), rng=rng) + spec, blob_count=rng.randint(0, spec.config.MAX_BLOBS_PER_BLOCK), rng=rng) block.body.execution_payload.transactions.append(opaque_tx) block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) block.body.blob_kzg_commitments = blob_kzg_commitments From 25b810f5a4c3eea8544f4d793f5c5b411d3e59f6 Mon Sep 17 00:00:00 2001 From: dapplion <35266934+dapplion@users.noreply.github.com> Date: Wed, 26 Jun 2024 11:01:50 +0200 Subject: [PATCH 027/130] lint --- pysetup/spec_builders/deneb.py | 2 +- .../eth2spec/test/deneb/unittests/test_config_invariants.py | 5 ++++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/pysetup/spec_builders/deneb.py b/pysetup/spec_builders/deneb.py index dc3c175836..436ae70b1d 100644 --- a/pysetup/spec_builders/deneb.py +++ b/pysetup/spec_builders/deneb.py @@ -70,7 +70,7 @@ def hardcoded_custom_type_dep_constants(cls, spec_object) -> Dict[str, str]: return { 'BYTES_PER_FIELD_ELEMENT': spec_object.constant_vars['BYTES_PER_FIELD_ELEMENT'].value, 'FIELD_ELEMENTS_PER_BLOB': spec_object.preset_vars['FIELD_ELEMENTS_PER_BLOB'].value, - 'MAX_BLOBS_PER_BLOCK': spec_object.preset_vars['MAX_BLOBS_PER_BLOCK'].value, + 'MAX_BLOBS_PER_BLOCK': spec_object.config_vars['MAX_BLOBS_PER_BLOCK'].value, 'MAX_BLOB_COMMITMENTS_PER_BLOCK': spec_object.preset_vars['MAX_BLOB_COMMITMENTS_PER_BLOCK'].value, } diff --git a/tests/core/pyspec/eth2spec/test/deneb/unittests/test_config_invariants.py b/tests/core/pyspec/eth2spec/test/deneb/unittests/test_config_invariants.py index e190f1d449..1f44257856 100644 --- a/tests/core/pyspec/eth2spec/test/deneb/unittests/test_config_invariants.py +++ b/tests/core/pyspec/eth2spec/test/deneb/unittests/test_config_invariants.py @@ -17,7 +17,10 @@ def test_length(spec): @single_phase def test_networking(spec): assert spec.config.MAX_BLOBS_PER_BLOCK < spec.MAX_BLOB_COMMITMENTS_PER_BLOCK - assert spec.config.MAX_REQUEST_BLOB_SIDECARS == spec.config.MAX_REQUEST_BLOCKS_DENEB * spec.config.MAX_BLOBS_PER_BLOCK + assert ( + spec.config.MAX_REQUEST_BLOB_SIDECARS == + spec.config.MAX_REQUEST_BLOCKS_DENEB * spec.config.MAX_BLOBS_PER_BLOCK + ) # Start with the same size, but `BLOB_SIDECAR_SUBNET_COUNT` could potentially increase later. assert spec.config.BLOB_SIDECAR_SUBNET_COUNT == spec.config.MAX_BLOBS_PER_BLOCK for i in range(spec.MAX_BLOB_COMMITMENTS_PER_BLOCK): From 9280cc7be9100c37acf63876fcfcb974dafc6754 Mon Sep 17 00:00:00 2001 From: dapplion <35266934+dapplion@users.noreply.github.com> Date: Wed, 26 Jun 2024 12:46:16 +0200 Subject: [PATCH 028/130] Update doctoc --- specs/deneb/beacon-chain.md | 1 + 1 file changed, 1 insertion(+) diff --git a/specs/deneb/beacon-chain.md b/specs/deneb/beacon-chain.md index 67307c7e7b..c14116737e 100644 --- a/specs/deneb/beacon-chain.md +++ b/specs/deneb/beacon-chain.md @@ -13,6 +13,7 @@ - [Preset](#preset) - [Execution](#execution) - [Configuration](#configuration) + - [Execution](#execution-1) - [Validator cycle](#validator-cycle) - [Containers](#containers) - [Extended containers](#extended-containers) From 17dfb9ae5763b53101fc3fc9fba1e26b27de8750 Mon Sep 17 00:00:00 2001 From: Csaba Kiraly Date: Thu, 27 Jun 2024 09:30:39 +0200 Subject: [PATCH 029/130] fix get_extended_sample_count proc signature Co-authored-by: Pop Chunhapanya --- specs/_features/eip7594/das-core.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/_features/eip7594/das-core.md b/specs/_features/eip7594/das-core.md index bd86b80d85..9310a62eeb 100644 --- a/specs/_features/eip7594/das-core.md +++ b/specs/_features/eip7594/das-core.md @@ -304,7 +304,7 @@ To custody a particular column, a node joins the respective gossip subnet. Verif At each slot, a node SHOULD select at least `SAMPLES_PER_SLOT` column IDs for sampling. It is recommended to use uniform random selection without replacement based on local randomness. Sampling is considered successful if the node manages to retrieve all selected columns. -Alternatively, a node MAY use a method that selects more than `SAMPLES_PER_SLOT` columns while allowing some missing, respecting the same target false positive threshold (the probability of successful sampling of an unavailable block) as dictated by the `SAMPLES_PER_SLOT` parameter. A node can use the `get_extended_sample_count(samples_per_slot, allowed_failures) -> sample_count` helper function to determine the sample count for any selected number of allowed failures. Sampling is then considered successful if any `sample_count - allowed_failures` columns are retrieved successfully. +Alternatively, a node MAY use a method that selects more than `SAMPLES_PER_SLOT` columns while allowing some missing, respecting the same target false positive threshold (the probability of successful sampling of an unavailable block) as dictated by the `SAMPLES_PER_SLOT` parameter. A node can use the `get_extended_sample_count(allowed_failures) -> sample_count` helper function to determine the sample count for any selected number of allowed failures. Sampling is then considered successful if any `sample_count - allowed_failures` columns are retrieved successfully. For reference, the table below shows the number of samples and the number of allowed missing columns assuming `NUMBER_OF_COLUMNS = 128` and `SAMPLES_PER_SLOT = 16`. From 78b583d8b062aacec77b5e522f9e883e760e756a Mon Sep 17 00:00:00 2001 From: Csaba Kiraly Date: Thu, 27 Jun 2024 09:48:32 +0200 Subject: [PATCH 030/130] clarify use of get_extended_sample_count Here we assume uniform random selection without replacement. If other methods are used, the target false positive threshold is the main rule to follow. Signed-off-by: Csaba Kiraly --- specs/_features/eip7594/das-core.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/_features/eip7594/das-core.md b/specs/_features/eip7594/das-core.md index 9310a62eeb..2e62266466 100644 --- a/specs/_features/eip7594/das-core.md +++ b/specs/_features/eip7594/das-core.md @@ -304,7 +304,7 @@ To custody a particular column, a node joins the respective gossip subnet. Verif At each slot, a node SHOULD select at least `SAMPLES_PER_SLOT` column IDs for sampling. It is recommended to use uniform random selection without replacement based on local randomness. Sampling is considered successful if the node manages to retrieve all selected columns. -Alternatively, a node MAY use a method that selects more than `SAMPLES_PER_SLOT` columns while allowing some missing, respecting the same target false positive threshold (the probability of successful sampling of an unavailable block) as dictated by the `SAMPLES_PER_SLOT` parameter. A node can use the `get_extended_sample_count(allowed_failures) -> sample_count` helper function to determine the sample count for any selected number of allowed failures. Sampling is then considered successful if any `sample_count - allowed_failures` columns are retrieved successfully. +Alternatively, a node MAY use a method that selects more than `SAMPLES_PER_SLOT` columns while allowing some missing, respecting the same target false positive threshold (the probability of successful sampling of an unavailable block) as dictated by the `SAMPLES_PER_SLOT` parameter. If using uniform random selection without replacement, a node can use the `get_extended_sample_count(allowed_failures) -> sample_count` helper function to determine the sample count (number of unique column IDs) for any selected number of allowed failures. Sampling is then considered successful if any `sample_count - allowed_failures` columns are retrieved successfully. For reference, the table below shows the number of samples and the number of allowed missing columns assuming `NUMBER_OF_COLUMNS = 128` and `SAMPLES_PER_SLOT = 16`. From 1ad381dccb9c923e81d0c7c409677a9b02eb97d1 Mon Sep 17 00:00:00 2001 From: Csaba Kiraly Date: Thu, 27 Jun 2024 10:28:21 +0200 Subject: [PATCH 031/130] adding get_extended_sample_count docsstring Signed-off-by: Csaba Kiraly --- specs/_features/eip7594/das-core.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/specs/_features/eip7594/das-core.md b/specs/_features/eip7594/das-core.md index 2e62266466..293e85ad72 100644 --- a/specs/_features/eip7594/das-core.md +++ b/specs/_features/eip7594/das-core.md @@ -229,6 +229,14 @@ def get_data_column_sidecars(signed_block: SignedBeaconBlock, ```python def get_extended_sample_count(allowed_failures: uint64) -> uint64: assert 0 <= allowed_failures <= NUMBER_OF_COLUMNS // 2 + """ + Return the sample count if allowing failures. + + This helper demonstrates how to calculate the number of columns to query per slot when + allowing given number of failures, assuming uniform random selection without replacement. + Nested functions are direct replacements of Python library functions math.comb and + scipy.stats.hypergeom.cdf, with the same signatures. + """ def math_comb(n: int, k: int) -> int: if not 0 <= k <= n: From a3a6c916b236c9e8904090303f0c38ae49db1002 Mon Sep 17 00:00:00 2001 From: kevaundray Date: Thu, 27 Jun 2024 20:17:14 +0100 Subject: [PATCH 032/130] Remove proof parameter from `recover_cells_and_kzg_proofs` (#3819) --- specs/_features/eip7594/das-core.md | 3 +- .../polynomial-commitments-sampling.md | 20 ++-- .../test_polynomial_commitments.py | 5 +- .../kzg_7594/recover_cells_and_kzg_proofs.md | 2 +- tests/generators/kzg_7594/main.py | 108 ++++-------------- 5 files changed, 35 insertions(+), 103 deletions(-) diff --git a/specs/_features/eip7594/das-core.md b/specs/_features/eip7594/das-core.md index b2d21d1a0b..0d6530226d 100644 --- a/specs/_features/eip7594/das-core.md +++ b/specs/_features/eip7594/das-core.md @@ -179,9 +179,8 @@ def recover_matrix(partial_matrix: Sequence[MatrixEntry], for blob_index in range(blob_count): cell_indices = [e.column_index for e in partial_matrix if e.row_index == blob_index] cells = [e.cell for e in partial_matrix if e.row_index == blob_index] - proofs = [e.kzg_proof for e in partial_matrix if e.row_index == blob_index] - recovered_cells, recovered_proofs = recover_cells_and_kzg_proofs(cell_indices, cells, proofs) + recovered_cells, recovered_proofs = recover_cells_and_kzg_proofs(cell_indices, cells) for cell_index, (cell, proof) in enumerate(zip(recovered_cells, recovered_proofs)): extended_matrix.append(MatrixEntry( cell=cell, diff --git a/specs/_features/eip7594/polynomial-commitments-sampling.md b/specs/_features/eip7594/polynomial-commitments-sampling.md index 7be1a4a059..d7090c6731 100644 --- a/specs/_features/eip7594/polynomial-commitments-sampling.md +++ b/specs/_features/eip7594/polynomial-commitments-sampling.md @@ -645,8 +645,7 @@ def recover_data(cell_indices: Sequence[CellIndex], ```python def recover_cells_and_kzg_proofs(cell_indices: Sequence[CellIndex], - cells: Sequence[Cell], - proofs_bytes: Sequence[Bytes48]) -> Tuple[ + cells: Sequence[Cell]) -> Tuple[ Vector[Cell, CELLS_PER_EXT_BLOB], Vector[KZGProof, CELLS_PER_EXT_BLOB]]: """ @@ -660,7 +659,7 @@ def recover_cells_and_kzg_proofs(cell_indices: Sequence[CellIndex], Public method. """ - assert len(cell_indices) == len(cells) == len(proofs_bytes) + assert len(cell_indices) == len(cells) # Check we have enough cells to be able to perform the reconstruction assert CELLS_PER_EXT_BLOB / 2 <= len(cell_indices) <= CELLS_PER_EXT_BLOB # Check for duplicates @@ -671,9 +670,6 @@ def recover_cells_and_kzg_proofs(cell_indices: Sequence[CellIndex], # Check that each cell is the correct length for cell in cells: assert len(cell) == BYTES_PER_CELL - # Check that each proof is the correct length - for proof_bytes in proofs_bytes: - assert len(proof_bytes) == BYTES_PER_PROOF # Convert cells to coset evals cosets_evals = [cell_to_coset_evals(cell) for cell in cells] @@ -692,14 +688,12 @@ def recover_cells_and_kzg_proofs(cell_indices: Sequence[CellIndex], polynomial_eval = reconstructed_data[:FIELD_ELEMENTS_PER_BLOB] polynomial_coeff = polynomial_eval_to_coeff(polynomial_eval) recovered_proofs = [None] * CELLS_PER_EXT_BLOB - for i, cell_index in enumerate(cell_indices): - recovered_proofs[cell_index] = bytes_to_kzg_proof(proofs_bytes[i]) + for i in range(CELLS_PER_EXT_BLOB): - if recovered_proofs[i] is None: - coset = coset_for_cell(CellIndex(i)) - proof, ys = compute_kzg_proof_multi_impl(polynomial_coeff, coset) - assert coset_evals_to_cell(ys) == recovered_cells[i] - recovered_proofs[i] = proof + coset = coset_for_cell(CellIndex(i)) + proof, ys = compute_kzg_proof_multi_impl(polynomial_coeff, coset) + assert coset_evals_to_cell(ys) == recovered_cells[i] + recovered_proofs[i] = proof return recovered_cells, recovered_proofs ``` diff --git a/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py b/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py index 5bc3a4330a..e7f066501d 100644 --- a/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py +++ b/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py @@ -164,12 +164,11 @@ def test_recover_cells_and_kzg_proofs(spec): while j in cell_indices: j = rng.randint(0, spec.CELLS_PER_EXT_BLOB - 1) cell_indices.append(j) - # Now the cells/proofs themselves + # Now the cells themselves known_cells = [cells[cell_index] for cell_index in cell_indices] - known_proofs = [proofs[cell_index] for cell_index in cell_indices] # Recover the missing cells and proofs - recovered_cells, recovered_proofs = spec.recover_cells_and_kzg_proofs(cell_indices, known_cells, known_proofs) + recovered_cells, recovered_proofs = spec.recover_cells_and_kzg_proofs(cell_indices, known_cells) recovered_data = [x for xs in recovered_cells for x in xs] # Check that the original data match the non-extended portion of the recovered data diff --git a/tests/formats/kzg_7594/recover_cells_and_kzg_proofs.md b/tests/formats/kzg_7594/recover_cells_and_kzg_proofs.md index 4e839c8ff4..37c5950c55 100644 --- a/tests/formats/kzg_7594/recover_cells_and_kzg_proofs.md +++ b/tests/formats/kzg_7594/recover_cells_and_kzg_proofs.md @@ -21,4 +21,4 @@ All byte(s) fields are encoded as strings, hexadecimal encoding, prefixed with ` ## Condition -The `recover_cells_and_kzg_proofs` handler should recover missing cells and proofs, and the result should match the expected `output`. If any cell is invalid (e.g. incorrect length or one of the 32-byte blocks does not represent a BLS field element), any proof is invalid (e.g. not on the curve or not in the G1 subgroup of the BLS curve), or any `cell_index` is invalid (e.g. greater than the number of cells for an extended blob), it should error, i.e. the output should be `null`. +The `recover_cells_and_kzg_proofs` handler should recover missing cells and proofs, and the result should match the expected `output`. If any cell is invalid (e.g. incorrect length or one of the 32-byte blocks does not represent a BLS field element), or any `cell_index` is invalid (e.g. greater than the number of cells for an extended blob), it should error, i.e. the output should be `null`. diff --git a/tests/generators/kzg_7594/main.py b/tests/generators/kzg_7594/main.py index e39b9d64cc..428baa566c 100644 --- a/tests/generators/kzg_7594/main.py +++ b/tests/generators/kzg_7594/main.py @@ -13,7 +13,6 @@ from eth2spec.test.utils.kzg_tests import ( CELL_RANDOM_VALID1, CELL_RANDOM_VALID2, - G1, INVALID_BLOBS, INVALID_G1_POINTS, INVALID_INDIVIDUAL_CELL_BYTES, @@ -593,15 +592,14 @@ def case_recover_cells_and_kzg_proofs(): # Valid: No missing cells cells, proofs = VALID_CELLS_AND_PROOFS[0] cell_indices = list(range(spec.CELLS_PER_EXT_BLOB)) - recovered_cells, recovered_proofs = spec.recover_cells_and_kzg_proofs(cell_indices, cells, proofs) + recovered_cells, recovered_proofs = spec.recover_cells_and_kzg_proofs(cell_indices, cells) assert recovered_cells == cells assert recovered_proofs == proofs - identifier = make_id(cell_indices, cells, proofs) + identifier = make_id(cell_indices, cells) yield f'recover_cells_and_kzg_proofs_case_valid_no_missing_{identifier}', { 'input': { 'cell_indices': cell_indices, 'cells': encode_hex_list(cells), - 'proofs': encode_hex_list(proofs), }, 'output': (encode_hex_list(recovered_cells), encode_hex_list(recovered_proofs)) } @@ -610,16 +608,14 @@ def case_recover_cells_and_kzg_proofs(): cells, proofs = VALID_CELLS_AND_PROOFS[1] cell_indices = list(range(0, spec.CELLS_PER_EXT_BLOB, 2)) partial_cells = [cells[cell_index] for cell_index in cell_indices] - partial_proofs = [proofs[cell_index] for cell_index in cell_indices] - recovered_cells, recovered_proofs = spec.recover_cells_and_kzg_proofs(cell_indices, partial_cells, partial_proofs) + recovered_cells, recovered_proofs = spec.recover_cells_and_kzg_proofs(cell_indices, partial_cells) assert recovered_cells == cells assert recovered_proofs == proofs - identifier = make_id(cell_indices, partial_cells, partial_proofs) + identifier = make_id(cell_indices, partial_cells) yield f'recover_cells_and_kzg_proofs_case_valid_half_missing_every_other_cell_{identifier}', { 'input': { 'cell_indices': cell_indices, 'cells': encode_hex_list(partial_cells), - 'proofs': encode_hex_list(partial_proofs), }, 'output': (encode_hex_list(recovered_cells), encode_hex_list(recovered_proofs)) } @@ -628,8 +624,7 @@ def case_recover_cells_and_kzg_proofs(): cells, proofs = VALID_CELLS_AND_PROOFS[2] cell_indices = list(range(0, spec.CELLS_PER_EXT_BLOB // 2)) partial_cells = [cells[cell_index] for cell_index in cell_indices] - partial_proofs = [proofs[cell_index] for cell_index in cell_indices] - recovered_cells, recovered_proofs = spec.recover_cells_and_kzg_proofs(cell_indices, partial_cells, partial_proofs) + recovered_cells, recovered_proofs = spec.recover_cells_and_kzg_proofs(cell_indices, partial_cells) assert recovered_cells == cells assert recovered_proofs == proofs identifier = make_id(cell_indices, partial_cells) @@ -637,7 +632,6 @@ def case_recover_cells_and_kzg_proofs(): 'input': { 'cell_indices': cell_indices, 'cells': encode_hex_list(partial_cells), - 'proofs': encode_hex_list(partial_proofs), }, 'output': (encode_hex_list(recovered_cells), encode_hex_list(recovered_proofs)) } @@ -646,8 +640,7 @@ def case_recover_cells_and_kzg_proofs(): cells, proofs = VALID_CELLS_AND_PROOFS[3] cell_indices = list(range(spec.CELLS_PER_EXT_BLOB // 2, spec.CELLS_PER_EXT_BLOB)) partial_cells = [cells[cell_index] for cell_index in cell_indices] - partial_proofs = [proofs[cell_index] for cell_index in cell_indices] - recovered_cells, recovered_proofs = spec.recover_cells_and_kzg_proofs(cell_indices, partial_cells, partial_proofs) + recovered_cells, recovered_proofs = spec.recover_cells_and_kzg_proofs(cell_indices, partial_cells) assert recovered_cells == cells assert recovered_proofs == proofs identifier = make_id(cell_indices, partial_cells) @@ -655,7 +648,6 @@ def case_recover_cells_and_kzg_proofs(): 'input': { 'cell_indices': cell_indices, 'cells': encode_hex_list(partial_cells), - 'proofs': encode_hex_list(partial_proofs), }, 'output': (encode_hex_list(recovered_cells), encode_hex_list(recovered_proofs)) } @@ -668,95 +660,67 @@ def case_recover_cells_and_kzg_proofs(): 'input': { 'cell_indices': cell_indices, 'cells': encode_hex_list(partial_cells), - 'proofs': encode_hex_list(partial_proofs), }, 'output': None } # Edge case: More than half missing - cells, proofs = VALID_CELLS_AND_PROOFS[4] + cells, _ = VALID_CELLS_AND_PROOFS[4] cell_indices = list(range(spec.CELLS_PER_EXT_BLOB // 2 - 1)) partial_cells = [cells[cell_index] for cell_index in cell_indices] - partial_proofs = [proofs[cell_index] for cell_index in cell_indices] - expect_exception(spec.recover_cells_and_kzg_proofs, cell_indices, partial_cells, partial_proofs) - identifier = make_id(cell_indices, partial_cells, partial_proofs) + expect_exception(spec.recover_cells_and_kzg_proofs, cell_indices, partial_cells) + identifier = make_id(cell_indices, partial_cells) yield f'recover_cells_and_kzg_proofs_case_invalid_more_than_half_missing_{identifier}', { 'input': { 'cell_indices': cell_indices, 'cells': encode_hex_list(partial_cells), - 'proofs': encode_hex_list(partial_proofs), }, 'output': None } # Edge case: More cells provided than CELLS_PER_EXT_BLOB - cells, proofs = VALID_CELLS_AND_PROOFS[5] + cells, _ = VALID_CELLS_AND_PROOFS[5] cell_indices = list(range(spec.CELLS_PER_EXT_BLOB)) + [0] partial_cells = [cells[cell_index] for cell_index in cell_indices] - partial_proofs = [proofs[cell_index] for cell_index in cell_indices] - expect_exception(spec.recover_cells_and_kzg_proofs, cell_indices, partial_cells, partial_proofs) - identifier = make_id(cell_indices, partial_cells, partial_proofs) + expect_exception(spec.recover_cells_and_kzg_proofs, cell_indices, partial_cells) + identifier = make_id(cell_indices, partial_cells) yield f'recover_cells_and_kzg_proofs_case_invalid_more_cells_than_cells_per_ext_blob_{identifier}', { 'input': { 'cell_indices': cell_indices, 'cells': encode_hex_list(partial_cells), - 'proofs': encode_hex_list(partial_proofs), }, 'output': None } # Edge case: Invalid cell_index - cells, proofs = VALID_CELLS_AND_PROOFS[6] + cells, _ = VALID_CELLS_AND_PROOFS[6] cell_indices = list(range(spec.CELLS_PER_EXT_BLOB // 2)) partial_cells = [cells[cell_index] for cell_index in cell_indices] - partial_proofs = [proofs[cell_index] for cell_index in cell_indices] # Replace first cell_index with an invalid value cell_indices[0] = spec.CELLS_PER_EXT_BLOB - expect_exception(spec.recover_cells_and_kzg_proofs, cell_indices, partial_cells, partial_proofs) - identifier = make_id(cell_indices, partial_cells, partial_proofs) + expect_exception(spec.recover_cells_and_kzg_proofs, cell_indices, partial_cells) + identifier = make_id(cell_indices, partial_cells) yield f'recover_cells_and_kzg_proofs_case_invalid_cell_index_{identifier}', { 'input': { 'cell_indices': cell_indices, 'cells': encode_hex_list(partial_cells), - 'proofs': encode_hex_list(partial_proofs), }, 'output': None } # Edge case: Invalid cell for cell in INVALID_INDIVIDUAL_CELL_BYTES: - cells, proofs = VALID_CELLS_AND_PROOFS[6] + cells, _ = VALID_CELLS_AND_PROOFS[6] cell_indices = list(range(spec.CELLS_PER_EXT_BLOB // 2)) partial_cells = [cells[cell_index] for cell_index in cell_indices] - partial_proofs = [proofs[cell_index] for cell_index in cell_indices] # Replace first cell with an invalid value partial_cells[0] = cell - expect_exception(spec.recover_cells_and_kzg_proofs, cell_indices, partial_cells, partial_proofs) - identifier = make_id(cell_indices, partial_cells, partial_proofs) + expect_exception(spec.recover_cells_and_kzg_proofs, cell_indices, partial_cells) + identifier = make_id(cell_indices, partial_cells) yield f'recover_cells_and_kzg_proofs_case_invalid_cell_{identifier}', { 'input': { 'cell_indices': cell_indices, 'cells': encode_hex_list(partial_cells), - 'proofs': encode_hex_list(partial_proofs), - }, - 'output': None - } - - # Edge case: Invalid proof - for proof in INVALID_G1_POINTS: - cells, proofs = VALID_CELLS_AND_PROOFS[0] - cell_indices = list(range(spec.CELLS_PER_EXT_BLOB // 2)) - partial_cells = [cells[cell_index] for cell_index in cell_indices] - partial_proofs = [proofs[cell_index] for cell_index in cell_indices] - # Replace first proof with an invalid value - partial_proofs[0] = proof - expect_exception(spec.recover_cells_and_kzg_proofs, cell_indices, partial_cells, partial_proofs) - identifier = make_id(cell_indices, partial_cells, partial_proofs) - yield f'recover_cells_and_kzg_proofs_case_invalid_proof_{identifier}', { - 'input': { - 'cell_indices': cell_indices, - 'cells': encode_hex_list(partial_cells), - 'proofs': encode_hex_list(partial_proofs), }, 'output': None } @@ -765,16 +729,14 @@ def case_recover_cells_and_kzg_proofs(): cells, proofs = VALID_CELLS_AND_PROOFS[0] cell_indices = list(range(0, spec.CELLS_PER_EXT_BLOB, 2)) partial_cells = [cells[cell_index] for cell_index in cell_indices] - partial_proofs = [proofs[cell_index] for cell_index in cell_indices] # Add another cell_index cell_indices.append(spec.CELLS_PER_EXT_BLOB - 1) - expect_exception(spec.recover_cells_and_kzg_proofs, cell_indices, partial_cells, partial_proofs) - identifier = make_id(cell_indices, partial_cells, partial_proofs) + expect_exception(spec.recover_cells_and_kzg_proofs, cell_indices, partial_cells) + identifier = make_id(cell_indices, partial_cells) yield f'recover_cells_and_kzg_proofs_case_invalid_more_cell_indices_than_cells_{identifier}', { 'input': { 'cell_indices': cell_indices, 'cells': encode_hex_list(partial_cells), - 'proofs': encode_hex_list(partial_proofs), }, 'output': None } @@ -783,34 +745,14 @@ def case_recover_cells_and_kzg_proofs(): cells, proofs = VALID_CELLS_AND_PROOFS[1] cell_indices = list(range(0, spec.CELLS_PER_EXT_BLOB, 2)) partial_cells = [cells[cell_index] for cell_index in cell_indices] - partial_proofs = [proofs[cell_index] for cell_index in cell_indices] # Add another cell partial_cells.append(CELL_RANDOM_VALID1) - expect_exception(spec.recover_cells_and_kzg_proofs, cell_indices, partial_cells, partial_proofs) - identifier = make_id(cell_indices, partial_cells, partial_proofs) + expect_exception(spec.recover_cells_and_kzg_proofs, cell_indices, partial_cells) + identifier = make_id(cell_indices, partial_cells) yield f'recover_cells_and_kzg_proofs_case_invalid_more_cells_than_cell_indices_{identifier}', { 'input': { 'cell_indices': cell_indices, 'cells': encode_hex_list(partial_cells), - 'proofs': encode_hex_list(partial_proofs), - }, - 'output': None - } - - # Edge case: More proofs than cell_indices - cells, proofs = VALID_CELLS_AND_PROOFS[1] - cell_indices = list(range(0, spec.CELLS_PER_EXT_BLOB, 2)) - partial_cells = [cells[cell_index] for cell_index in cell_indices] - partial_proofs = [proofs[cell_index] for cell_index in cell_indices] - # Add another proof - partial_proofs.append(G1) - expect_exception(spec.recover_cells_and_kzg_proofs, cell_indices, partial_cells, partial_proofs) - identifier = make_id(cell_indices, partial_cells, partial_proofs) - yield f'recover_cells_and_kzg_proofs_case_invalid_more_proofs_than_cell_indices_{identifier}', { - 'input': { - 'cell_indices': cell_indices, - 'cells': encode_hex_list(partial_cells), - 'proofs': encode_hex_list(partial_proofs), }, 'output': None } @@ -824,16 +766,14 @@ def case_recover_cells_and_kzg_proofs(): # to insufficient cell count, not because of a duplicate cell. cell_indices = list(range(spec.CELLS_PER_EXT_BLOB // 2 + 1)) partial_cells = [cells[cell_index] for cell_index in cell_indices] - partial_proofs = [proofs[cell_index] for cell_index in cell_indices] # Replace first cell_index with the second cell_index cell_indices[0] = cell_indices[1] - expect_exception(spec.recover_cells_and_kzg_proofs, cell_indices, partial_cells, partial_proofs) - identifier = make_id(cell_indices, partial_cells, partial_proofs) + expect_exception(spec.recover_cells_and_kzg_proofs, cell_indices, partial_cells) + identifier = make_id(cell_indices, partial_cells) yield f'recover_cells_and_kzg_proofs_case_invalid_duplicate_cell_index_{identifier}', { 'input': { 'cell_indices': cell_indices, 'cells': encode_hex_list(partial_cells), - 'proofs': encode_hex_list(partial_proofs), }, 'output': None } From f25aac6adff95070431e520b5567c6a872939d2a Mon Sep 17 00:00:00 2001 From: dapplion <35266934+dapplion@users.noreply.github.com> Date: Fri, 28 Jun 2024 15:53:29 +0200 Subject: [PATCH 033/130] Add MetadataV3 with custody_subnet_count --- specs/electra/p2p-interface.md | 42 ++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/specs/electra/p2p-interface.md b/specs/electra/p2p-interface.md index ebdcaaa831..33d3b6592c 100644 --- a/specs/electra/p2p-interface.md +++ b/specs/electra/p2p-interface.md @@ -11,17 +11,39 @@ The specification of these changes continues in the same format as the network s - [Modifications in Electra](#modifications-in-electra) + - [MetaData](#metadata) - [The gossip domain: gossipsub](#the-gossip-domain-gossipsub) - [Topics and messages](#topics-and-messages) - [Global topics](#global-topics) - [`beacon_aggregate_and_proof`](#beacon_aggregate_and_proof) - [`beacon_attestation_{subnet_id}`](#beacon_attestation_subnet_id) + - [The Req/Resp domain](#the-reqresp-domain) + - [Messages](#messages) + - [GetMetaData v3](#getmetadata-v3) ## Modifications in Electra +### MetaData + +The `MetaData` stored locally by clients is updated with an additional field to communicate the custody subnet count. + +``` +( + seq_number: uint64 + attnets: Bitvector[ATTESTATION_SUBNET_COUNT] + syncnets: Bitvector[SYNC_COMMITTEE_SUBNET_COUNT] + custody_subnet_count: uint64 +) +``` + +Where + +- `seq_number`, `attnets`, and `syncnets` have the same meaning defined in the Altair document. +- `custody_subnet_count` represents the node's custody subnet count. Clients MAY reject ENRs with a value less than `CUSTODY_REQUIREMENT`. + ### The gossip domain: gossipsub Some gossip meshes are upgraded in the fork of Electra to support upgraded types. @@ -57,3 +79,23 @@ The following convenience variables are re-defined The following validations are added: * [REJECT] `len(committee_indices) == 1`, where `committee_indices = get_committee_indices(attestation)`. * [REJECT] `attestation.data.index == 0` + +### The Req/Resp domain + +#### Messages + +##### GetMetaData v3 + +**Protocol ID:** `/eth2/beacon_chain/req/metadata/3/` + +No Request Content. + +Response Content: + +``` +( + MetaData +) +``` + +Requests the MetaData of a peer, using the new `MetaData` definition given above that is extended from Altair. Other conditions for the `GetMetaData` protocol are unchanged from the Altair p2p networking document. From 83da3807abc132646493751310a21f7468e36af2 Mon Sep 17 00:00:00 2001 From: Benedikt Wagner <113296072+b-wagn@users.noreply.github.com> Date: Fri, 28 Jun 2024 17:02:20 +0200 Subject: [PATCH 034/130] EIP7594: Do universal verification in verify_cell_kzg_proof_batch() (#3812) * restructure verify_cell_kzg_proof_batch a bit * first draft of universal verification equation * add one more empty line to make linter happy * make linter happy * more testcases for verify_cell_kzg_proof_batch * verify_cell_kzg_proof_batch: derive coefficient via hash * rename verify_cell_kzg_proof_batch_challenge -> compute_verify_cell_kzg_proof_batch_challenge * verify_cell_kzg_proof_batch: editorial + some refactoring * Improve documentation and variable naming. * remove k_i from code and doc --------- Co-authored-by: Justin Traglia <95511699+jtraglia@users.noreply.github.com> --- .../polynomial-commitments-sampling.md | 219 +++++++++++++++--- .../test_polynomial_commitments.py | 90 +++++++ 2 files changed, 279 insertions(+), 30 deletions(-) diff --git a/specs/_features/eip7594/polynomial-commitments-sampling.md b/specs/_features/eip7594/polynomial-commitments-sampling.md index d7090c6731..10da2e58c3 100644 --- a/specs/_features/eip7594/polynomial-commitments-sampling.md +++ b/specs/_features/eip7594/polynomial-commitments-sampling.md @@ -22,6 +22,7 @@ - [`_fft_field`](#_fft_field) - [`fft_field`](#fft_field) - [`coset_fft_field`](#coset_fft_field) + - [`compute_verify_cell_kzg_proof_batch_challenge`](#compute_verify_cell_kzg_proof_batch_challenge) - [Polynomials in coefficient form](#polynomials-in-coefficient-form) - [`polynomial_eval_to_coeff`](#polynomial_eval_to_coeff) - [`add_polynomialcoeff`](#add_polynomialcoeff) @@ -34,7 +35,9 @@ - [KZG multiproofs](#kzg-multiproofs) - [`compute_kzg_proof_multi_impl`](#compute_kzg_proof_multi_impl) - [`verify_kzg_proof_multi_impl`](#verify_kzg_proof_multi_impl) + - [`verify_cell_kzg_proof_batch_impl`](#verify_cell_kzg_proof_batch_impl) - [Cell cosets](#cell-cosets) + - [`coset_shift_for_cell`](#coset_shift_for_cell) - [`coset_for_cell`](#coset_for_cell) - [Cells](#cells-1) - [Cell computation](#cell-computation) @@ -193,17 +196,17 @@ def coset_fft_field(vals: Sequence[BLSFieldElement], roots_of_unity: Sequence[BLSFieldElement], inv: bool=False) -> Sequence[BLSFieldElement]: """ - Computes an FFT/IFFT over a coset of the roots of unity. - This is useful for when one wants to divide by a polynomial which + Computes an FFT/IFFT over a coset of the roots of unity. + This is useful for when one wants to divide by a polynomial which vanishes on one or more elements in the domain. """ vals = vals.copy() - + def shift_vals(vals: Sequence[BLSFieldElement], factor: BLSFieldElement) -> Sequence[BLSFieldElement]: - """ - Multiply each entry in `vals` by succeeding powers of `factor` - i.e., [vals[0] * factor^0, vals[1] * factor^1, ..., vals[n] * factor^n] - """ + """ + Multiply each entry in `vals` by succeeding powers of `factor` + i.e., [vals[0] * factor^0, vals[1] * factor^1, ..., vals[n] * factor^n] + """ shift = 1 for i in range(len(vals)): vals[i] = BLSFieldElement((int(vals[i]) * shift) % BLS_MODULUS) @@ -222,6 +225,52 @@ def coset_fft_field(vals: Sequence[BLSFieldElement], return fft_field(vals, roots_of_unity, inv) ``` +#### `compute_verify_cell_kzg_proof_batch_challenge` + +```python +def compute_verify_cell_kzg_proof_batch_challenge(row_commitments: Sequence[KZGCommitment], + row_indices: Sequence[RowIndex], + column_indices: Sequence[ColumnIndex], + cosets_evals: Sequence[CosetEvals], + proofs: Sequence[KZGProof]) -> BLSFieldElement: + """ + Compute a random challenge r used in the universal verification equation. + This is used in verify_cell_kzg_proof_batch_impl. + + To compute the challenge, `RANDOM_CHALLENGE_KZG_CELL_BATCH_DOMAIN` is used as a hash prefix. + """ + # input the domain separator + hashinput = RANDOM_CHALLENGE_KZG_CELL_BATCH_DOMAIN + + # input the degree bound of the polynomial + hashinput += int.to_bytes(FIELD_ELEMENTS_PER_BLOB, 8, KZG_ENDIANNESS) + + # input the field elements per cell + hashinput += int.to_bytes(FIELD_ELEMENTS_PER_CELL, 8, KZG_ENDIANNESS) + + # input the number of commitments + num_commitments = len(row_commitments) + hashinput += int.to_bytes(num_commitments, 8, KZG_ENDIANNESS) + + # input the number of cells + num_cells = len(row_indices) + hashinput += int.to_bytes(num_cells, 8, KZG_ENDIANNESS) + + # input all commitments + for commitment in row_commitments: + hashinput += commitment + + # input each cell with its indices and proof + for k in range(num_cells): + hashinput += int.to_bytes(row_indices[k], 8, KZG_ENDIANNESS) + hashinput += int.to_bytes(column_indices[k], 8, KZG_ENDIANNESS) + for eval in cosets_evals[k]: + hashinput += bls_field_to_bytes(eval) + hashinput += proofs[k] + + return hash_to_bls_field(hashinput) +``` + ### Polynomials in coefficient form #### `polynomial_eval_to_coeff` @@ -362,12 +411,12 @@ def compute_kzg_proof_multi_impl( """ Compute a KZG multi-evaluation proof for a set of `k` points. - This is done by committing to the following quotient polynomial: + This is done by committing to the following quotient polynomial: Q(X) = f(X) - I(X) / Z(X) Where: - I(X) is the degree `k-1` polynomial that agrees with f(x) at all `k` points - Z(X) is the degree `k` polynomial that evaluates to zero on all `k` points - + We further note that since the degree of I(X) is less than the degree of Z(X), the computation can be simplified in monomial form to Q(X) = f(X) / Z(X) """ @@ -401,7 +450,7 @@ def verify_kzg_proof_multi_impl(commitment: KZGCommitment, Q(X) is the quotient polynomial computed by the prover I(X) is the degree k-1 polynomial that evaluates to `ys` at all `zs`` points Z(X) is the polynomial that evaluates to zero on all `k` points - + The verifier receives the commitments to Q(X) and f(X), so they check the equation holds by using the following pairing equation: e([Q(X)]_1, [Z(X)]_2) == e([f(X)]_1 - [I(X)]_1, [1]_2) @@ -423,14 +472,132 @@ def verify_kzg_proof_multi_impl(commitment: KZGCommitment, ])) ``` +#### `verify_cell_kzg_proof_batch_impl` + +```python +def verify_cell_kzg_proof_batch_impl(row_commitments: Sequence[KZGCommitment], + row_indices: Sequence[RowIndex], + column_indices: Sequence[ColumnIndex], + cosets_evals: Sequence[CosetEvals], + proofs: Sequence[KZGProof]) -> bool: + """ + Verify a set of cells, given their corresponding proofs and their coordinates (row_index, column_index) in the blob + matrix. The i-th cell is in row row_indices[i] and in column column_indices[i]. + The list of all commitments is provided in row_commitments_bytes. + + This function is the internal implementation of verify_cell_kzg_proof_batch. + """ + + # The verification equation that we will check is pairing (LL, LR) = pairing (RL, [1]), where + # LL = sum_k r^k proofs[k], + # LR = [s^n] + # RL = RLC - RLI + RLP, where + # RLC = sum_i weights[i] commitments[i] + # RLI = [sum_k r^k interpolation_poly_k(s)] + # RLP = sum_k (r^k * h_k^n) proofs[k] + # + # Here, the variables have the following meaning: + # - k < len(row_indices) is an index iterating over all cells in the input + # - r is a random coefficient, derived from hashing all data provided by the prover + # - s is the secret embedded in the KZG setup + # - n = FIELD_ELEMENTS_PER_CELL is the size of the evaluation domain + # - i ranges over all rows that are touched + # - weights[i] is a weight computed for row i. It depends on r and on which cells are in row i + # - interpolation_poly_k is the interpolation polynomial for the kth cell + # - h_k is the coset shift specifying the evaluation domain of the kth cell + + # Preparation + num_cells = len(row_indices) + n = FIELD_ELEMENTS_PER_CELL + num_rows = len(row_commitments) + + # Step 1: Compute a challenge r and its powers r^0, ..., r^{num_cells-1} + r = compute_verify_cell_kzg_proof_batch_challenge( + row_commitments, + row_indices, + column_indices, + cosets_evals, + proofs + ) + r_powers = compute_powers(r, num_cells) + + # Step 2: Compute LL = sum_k r^k proofs[k] + ll = bls.bytes48_to_G1(g1_lincomb(proofs, r_powers)) + + # Step 3: Compute LR = [s^n] + lr = bls.bytes96_to_G2(KZG_SETUP_G2_MONOMIAL[n]) + + # Step 4: Compute RL = RLC - RLI + RLP + # Step 4.1: Compute RLC = sum_i weights[i] commitments[i] + # Step 4.1a: Compute weights[i]: the sum of all r^k for which cell k is in row i. + # Note: we do that by iterating over all k and updating the correct weights[i] accordingly + weights = [0] * num_rows + for k in range(num_cells): + i = row_indices[k] + weights[i] = (weights[i] + int(r_powers[k])) % BLS_MODULUS + # Step 4.1b: Linearly combine the weights with the commitments to get RLC + rlc = bls.bytes48_to_G1(g1_lincomb(row_commitments, weights)) + + # Step 4.2: Compute RLI = [sum_k r^k interpolation_poly_k(s)] + # Note: an efficient implementation would use the IDFT based method explained in the blog post + sum_interp_polys_coeff = [0] + for k in range(num_cells): + interp_poly_coeff = interpolate_polynomialcoeff(coset_for_cell(column_indices[k]), cosets_evals[k]) + interp_poly_scaled_coeff = multiply_polynomialcoeff([r_powers[k]], interp_poly_coeff) + sum_interp_polys_coeff = add_polynomialcoeff(sum_interp_polys_coeff, interp_poly_scaled_coeff) + rli = bls.bytes48_to_G1(g1_lincomb(KZG_SETUP_G1_MONOMIAL[:n], sum_interp_polys_coeff)) + + # Step 4.3: Compute RLP = sum_k (r^k * h_k^n) proofs[k] + weighted_r_powers = [] + for k in range(num_cells): + h_k = int(coset_shift_for_cell(column_indices[k])) + h_k_pow = pow(h_k, n, BLS_MODULUS) + wrp = (int(r_powers[k]) * h_k_pow) % BLS_MODULUS + weighted_r_powers.append(wrp) + rlp = bls.bytes48_to_G1(g1_lincomb(proofs, weighted_r_powers)) + + # Step 4.4: Compute RL = RLC - RLI + RLP + rl = bls.add(rlc, bls.neg(rli)) + rl = bls.add(rl, rlp) + + # Step 5: Check pairing (LL, LR) = pairing (RL, [1]) + return (bls.pairing_check([ + [ll, lr], + [rl, bls.neg(bls.bytes96_to_G2(KZG_SETUP_G2_MONOMIAL[0]))], + ])) +``` + + ### Cell cosets +#### `coset_shift_for_cell` + +```python +def coset_shift_for_cell(cell_index: CellIndex) -> BLSFieldElement: + """ + Get the shift that determines the coset for a given ``cell_index``. + Precisely, consider the group of roots of unity of order FIELD_ELEMENTS_PER_CELL * CELLS_PER_EXT_BLOB. + Let G = {1, g, g^2, ...} denote its subgroup of order FIELD_ELEMENTS_PER_CELL. + Then, the coset is defined as h * G = {h, hg, hg^2, ...} for an element h. + This function returns h. + """ + assert cell_index < CELLS_PER_EXT_BLOB + roots_of_unity_brp = bit_reversal_permutation( + compute_roots_of_unity(FIELD_ELEMENTS_PER_EXT_BLOB) + ) + return roots_of_unity_brp[FIELD_ELEMENTS_PER_CELL * cell_index] +``` + #### `coset_for_cell` ```python def coset_for_cell(cell_index: CellIndex) -> Coset: """ Get the coset for a given ``cell_index``. + Precisely, consider the group of roots of unity of order FIELD_ELEMENTS_PER_CELL * CELLS_PER_EXT_BLOB. + Let G = {1, g, g^2, ...} denote its subgroup of order FIELD_ELEMENTS_PER_CELL. + Then, the coset is defined as h * G = {h, hg, hg^2, ...}. + This function, returns the coset. """ assert cell_index < CELLS_PER_EXT_BLOB roots_of_unity_brp = bit_reversal_permutation( @@ -457,7 +624,7 @@ def compute_cells_and_kzg_proofs(blob: Blob) -> Tuple[ Public method. """ assert len(blob) == BYTES_PER_BLOB - + polynomial = blob_to_polynomial(blob) polynomial_coeff = polynomial_eval_to_coeff(polynomial) @@ -491,7 +658,7 @@ def verify_cell_kzg_proof(commitment_bytes: Bytes48, assert cell_index < CELLS_PER_EXT_BLOB assert len(cell) == BYTES_PER_CELL assert len(proof_bytes) == BYTES_PER_PROOF - + coset = coset_for_cell(cell_index) return verify_kzg_proof_multi_impl( @@ -511,18 +678,15 @@ def verify_cell_kzg_proof_batch(row_commitments_bytes: Sequence[Bytes48], proofs_bytes: Sequence[Bytes48]) -> bool: """ Verify a set of cells, given their corresponding proofs and their coordinates (row_index, column_index) in the blob - matrix. The list of all commitments is also provided in row_commitments_bytes. + matrix. The i-th cell is in row = row_indices[i] and in column = column_indices[i]. + The list of all commitments is provided in row_commitments_bytes. - This function implements the naive algorithm of checking every cell - individually; an efficient algorithm can be found here: + This function implements the universal verification equation that has been introduced here: https://ethresear.ch/t/a-universal-verification-equation-for-data-availability-sampling/13240 - This implementation does not require randomness, but for the algorithm that - requires it, `RANDOM_CHALLENGE_KZG_CELL_BATCH_DOMAIN` should be used to compute - the challenge value. - Public method. """ + assert len(cells) == len(proofs_bytes) == len(row_indices) == len(column_indices) for commitment_bytes in row_commitments_bytes: assert len(commitment_bytes) == BYTES_PER_COMMITMENT @@ -535,18 +699,13 @@ def verify_cell_kzg_proof_batch(row_commitments_bytes: Sequence[Bytes48], for proof_bytes in proofs_bytes: assert len(proof_bytes) == BYTES_PER_PROOF - # Get commitments via row indices - commitments_bytes = [row_commitments_bytes[row_index] for row_index in row_indices] - # Get objects from bytes - commitments = [bytes_to_kzg_commitment(commitment_bytes) for commitment_bytes in commitments_bytes] + row_commitments = [bytes_to_kzg_commitment(commitment_bytes) for commitment_bytes in row_commitments_bytes] cosets_evals = [cell_to_coset_evals(cell) for cell in cells] proofs = [bytes_to_kzg_proof(proof_bytes) for proof_bytes in proofs_bytes] - return all( - verify_kzg_proof_multi_impl(commitment, coset_for_cell(column_index), coset_evals, proof) - for commitment, column_index, coset_evals, proof in zip(commitments, column_indices, cosets_evals, proofs) - ) + # Do the actual verification + return verify_cell_kzg_proof_batch_impl(row_commitments, row_indices, column_indices, cosets_evals, proofs) ``` ## Reconstruction @@ -558,7 +717,7 @@ def construct_vanishing_polynomial(missing_cell_indices: Sequence[CellIndex]) -> """ Given the cells indices that are missing from the data, compute the polynomial that vanishes at every point that corresponds to a missing field element. - + This method assumes that all of the cells cannot be missing. In this case the vanishing polynomial could be computed as Z(x) = x^n - 1, where `n` is FIELD_ELEMENTS_PER_EXT_BLOB. @@ -617,7 +776,7 @@ def recover_data(cell_indices: Sequence[CellIndex], extended_evaluation_times_zero = [BLSFieldElement(int(a) * int(b) % BLS_MODULUS) for a, b in zip(zero_poly_eval, extended_evaluation)] - # Convert (E*Z)(x) to monomial form + # Convert (E*Z)(x) to monomial form extended_evaluation_times_zero_coeffs = fft_field(extended_evaluation_times_zero, roots_of_unity_extended, inv=True) # Convert (E*Z)(x) to evaluation form over a coset of the FFT domain @@ -684,7 +843,7 @@ def recover_cells_and_kzg_proofs(cell_indices: Sequence[CellIndex], recovered_cells = [ coset_evals_to_cell(reconstructed_data[i * FIELD_ELEMENTS_PER_CELL:(i + 1) * FIELD_ELEMENTS_PER_CELL]) for i in range(CELLS_PER_EXT_BLOB)] - + polynomial_eval = reconstructed_data[:FIELD_ELEMENTS_PER_BLOB] polynomial_coeff = polynomial_eval_to_coeff(polynomial_eval) recovered_proofs = [None] * CELLS_PER_EXT_BLOB diff --git a/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py b/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py index e7f066501d..dd5aece3f9 100644 --- a/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py +++ b/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py @@ -126,6 +126,8 @@ def test_verify_cell_kzg_proof(spec): @spec_test @single_phase def test_verify_cell_kzg_proof_batch(spec): + + # test with a single blob / commitment blob = get_sample_blob(spec) commitment = spec.blob_to_kzg_commitment(blob) cells, proofs = spec.compute_cells_and_kzg_proofs(blob) @@ -140,6 +142,94 @@ def test_verify_cell_kzg_proof_batch(spec): proofs_bytes=[proofs[0], proofs[4]], ) + # now test with three blobs / commitments + all_blobs = [] + all_commitments = [] + all_cells = [] + all_proofs = [] + for _ in range(3): + blob = get_sample_blob(spec) + commitment = spec.blob_to_kzg_commitment(blob) + cells, proofs = spec.compute_cells_and_kzg_proofs(blob) + + assert len(cells) == len(proofs) + + all_blobs.append(blob) + all_commitments.append(commitment) + all_cells.append(cells) + all_proofs.append(proofs) + + # the cells of interest + row_indices = [0, 0, 1, 2, 1] + column_indices = [0, 4, 0, 1, 2] + cells = [all_cells[i][j] for (i, j) in zip(row_indices, column_indices)] + proofs = [all_proofs[i][j] for (i, j) in zip(row_indices, column_indices)] + + # do the check + assert spec.verify_cell_kzg_proof_batch( + row_commitments_bytes=all_commitments, + row_indices=row_indices, + column_indices=column_indices, + cells=cells, + proofs_bytes=proofs, + ) + + +@with_eip7594_and_later +@spec_test +@single_phase +def test_verify_cell_kzg_proof_batch_invalid(spec): + + # test with a single blob / commitment + blob = get_sample_blob(spec) + commitment = spec.blob_to_kzg_commitment(blob) + cells, proofs = spec.compute_cells_and_kzg_proofs(blob) + + assert len(cells) == len(proofs) + + assert not spec.verify_cell_kzg_proof_batch( + row_commitments_bytes=[commitment], + row_indices=[0, 0], + column_indices=[0, 4], + cells=[cells[0], cells[5]], # Note: this is where it should go wrong + proofs_bytes=[proofs[0], proofs[4]], + ) + + # now test with three blobs / commitments + all_blobs = [] + all_commitments = [] + all_cells = [] + all_proofs = [] + for _ in range(3): + blob = get_sample_blob(spec) + commitment = spec.blob_to_kzg_commitment(blob) + cells, proofs = spec.compute_cells_and_kzg_proofs(blob) + + assert len(cells) == len(proofs) + + all_blobs.append(blob) + all_commitments.append(commitment) + all_cells.append(cells) + all_proofs.append(proofs) + + # the cells of interest + row_indices = [0, 0, 1, 2, 1] + column_indices = [0, 4, 0, 1, 2] + cells = [all_cells[i][j] for (i, j) in zip(row_indices, column_indices)] + proofs = [all_proofs[i][j] for (i, j) in zip(row_indices, column_indices)] + + # let's change one of the cells. Then it should not verify + cells[1] = all_cells[1][3] + + # do the check + assert not spec.verify_cell_kzg_proof_batch( + row_commitments_bytes=all_commitments, + row_indices=row_indices, + column_indices=column_indices, + cells=cells, + proofs_bytes=proofs, + ) + @with_eip7594_and_later @spec_test From 0e03d8c08dde46cdb4bf4aee871d6ca50a572b08 Mon Sep 17 00:00:00 2001 From: Justin Traglia Date: Fri, 28 Jun 2024 14:24:49 -0500 Subject: [PATCH 035/130] Fix beacon-chain.md section header issues --- specs/electra/beacon-chain.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/specs/electra/beacon-chain.md b/specs/electra/beacon-chain.md index 351f8634fd..405ccf3120 100644 --- a/specs/electra/beacon-chain.md +++ b/specs/electra/beacon-chain.md @@ -49,8 +49,8 @@ - [Updated `is_fully_withdrawable_validator`](#updated-is_fully_withdrawable_validator) - [Updated `is_partially_withdrawable_validator`](#updated-is_partially_withdrawable_validator) - [Misc](#misc-1) - - [`get_committee_indices`](#get_committee_indices) - - [`get_validator_max_effective_balance`](#get_validator_max_effective_balance) + - [New `get_committee_indices`](#new-get_committee_indices) + - [New `get_validator_max_effective_balance`](#new-get_validator_max_effective_balance) - [Beacon state accessors](#beacon-state-accessors) - [New `get_balance_churn_limit`](#new-get_balance_churn_limit) - [New `get_activation_exit_churn_limit`](#new-get_activation_exit_churn_limit) @@ -60,7 +60,7 @@ - [Modified `get_attesting_indices`](#modified-get_attesting_indices) - [Modified `get_next_sync_committee_indices`](#modified-get_next_sync_committee_indices) - [Beacon state mutators](#beacon-state-mutators) - - [Updated `initiate_validator_exit`](#updated--initiate_validator_exit) + - [Updated `initiate_validator_exit`](#updated-initiate_validator_exit) - [New `switch_to_compounding_validator`](#new-switch_to_compounding_validator) - [New `queue_excess_active_balance`](#new-queue_excess_active_balance) - [New `queue_entire_balance_and_reset_validator`](#new-queue_entire_balance_and_reset_validator) @@ -70,7 +70,7 @@ - [Beacon chain state transition function](#beacon-chain-state-transition-function) - [Epoch processing](#epoch-processing) - [Updated `process_epoch`](#updated-process_epoch) - - [Updated `process_registry_updates`](#updated--process_registry_updates) + - [Updated `process_registry_updates`](#updated-process_registry_updates) - [New `process_pending_balance_deposits`](#new-process_pending_balance_deposits) - [New `process_pending_consolidations`](#new-process_pending_consolidations) - [Updated `process_effective_balance_updates`](#updated-process_effective_balance_updates) @@ -85,7 +85,7 @@ - [Attestations](#attestations) - [Modified `process_attestation`](#modified-process_attestation) - [Deposits](#deposits) - - [Updated `apply_deposit`](#updated--apply_deposit) + - [Updated `apply_deposit`](#updated-apply_deposit) - [New `is_valid_deposit_signature`](#new-is_valid_deposit_signature) - [Modified `add_validator_to_registry`](#modified-add_validator_to_registry) - [Updated `get_validator_from_deposit`](#updated-get_validator_from_deposit) @@ -525,14 +525,14 @@ def is_partially_withdrawable_validator(validator: Validator, balance: Gwei) -> ### Misc -#### `get_committee_indices` +#### New `get_committee_indices` ```python def get_committee_indices(committee_bits: Bitvector) -> Sequence[CommitteeIndex]: return [CommitteeIndex(index) for index, bit in enumerate(committee_bits) if bit] ``` -#### `get_validator_max_effective_balance` +#### New `get_validator_max_effective_balance` ```python def get_validator_max_effective_balance(validator: Validator) -> Gwei: @@ -648,7 +648,7 @@ def get_next_sync_committee_indices(state: BeaconState) -> Sequence[ValidatorInd ### Beacon state mutators -#### Updated `initiate_validator_exit` +#### Updated `initiate_validator_exit` ```python def initiate_validator_exit(state: BeaconState, index: ValidatorIndex) -> None: @@ -810,7 +810,7 @@ def process_epoch(state: BeaconState) -> None: process_sync_committee_updates(state) ``` -#### Updated `process_registry_updates` +#### Updated `process_registry_updates` `process_registry_updates` uses the updated definition of `initiate_validator_exit` and changes how the activation epochs are computed for eligible validators. @@ -1160,7 +1160,7 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: ##### Deposits -###### Updated `apply_deposit` +###### Updated `apply_deposit` *NOTE*: `process_deposit` is updated with a new definition of `apply_deposit`. From 77c371b22a294538df33ae0f2e899d5297b9f382 Mon Sep 17 00:00:00 2001 From: Justin Traglia <95511699+jtraglia@users.noreply.github.com> Date: Mon, 1 Jul 2024 15:36:17 -0500 Subject: [PATCH 036/130] Replace "updated" with "modified" in Electra spec (#3823) --- specs/electra/beacon-chain.md | 101 +++++++++++++++++++++------------- 1 file changed, 62 insertions(+), 39 deletions(-) diff --git a/specs/electra/beacon-chain.md b/specs/electra/beacon-chain.md index 405ccf3120..358355e7ef 100644 --- a/specs/electra/beacon-chain.md +++ b/specs/electra/beacon-chain.md @@ -41,13 +41,13 @@ - [`BeaconState`](#beaconstate) - [Helper functions](#helper-functions) - [Predicates](#predicates) - - [Updated `compute_proposer_index`](#updated-compute_proposer_index) - - [Updated `is_eligible_for_activation_queue`](#updated-is_eligible_for_activation_queue) + - [Modified `compute_proposer_index`](#modified-compute_proposer_index) + - [Modified `is_eligible_for_activation_queue`](#modified-is_eligible_for_activation_queue) - [New `is_compounding_withdrawal_credential`](#new-is_compounding_withdrawal_credential) - [New `has_compounding_withdrawal_credential`](#new-has_compounding_withdrawal_credential) - [New `has_execution_withdrawal_credential`](#new-has_execution_withdrawal_credential) - - [Updated `is_fully_withdrawable_validator`](#updated-is_fully_withdrawable_validator) - - [Updated `is_partially_withdrawable_validator`](#updated-is_partially_withdrawable_validator) + - [Modified `is_fully_withdrawable_validator`](#modified-is_fully_withdrawable_validator) + - [Modified `is_partially_withdrawable_validator`](#modified-is_partially_withdrawable_validator) - [Misc](#misc-1) - [New `get_committee_indices`](#new-get_committee_indices) - [New `get_validator_max_effective_balance`](#new-get_validator_max_effective_balance) @@ -60,24 +60,24 @@ - [Modified `get_attesting_indices`](#modified-get_attesting_indices) - [Modified `get_next_sync_committee_indices`](#modified-get_next_sync_committee_indices) - [Beacon state mutators](#beacon-state-mutators) - - [Updated `initiate_validator_exit`](#updated-initiate_validator_exit) + - [Modified `initiate_validator_exit`](#modified-initiate_validator_exit) - [New `switch_to_compounding_validator`](#new-switch_to_compounding_validator) - [New `queue_excess_active_balance`](#new-queue_excess_active_balance) - [New `queue_entire_balance_and_reset_validator`](#new-queue_entire_balance_and_reset_validator) - [New `compute_exit_epoch_and_update_churn`](#new-compute_exit_epoch_and_update_churn) - [New `compute_consolidation_epoch_and_update_churn`](#new-compute_consolidation_epoch_and_update_churn) - - [Updated `slash_validator`](#updated-slash_validator) + - [Modified `slash_validator`](#modified-slash_validator) - [Beacon chain state transition function](#beacon-chain-state-transition-function) - [Epoch processing](#epoch-processing) - - [Updated `process_epoch`](#updated-process_epoch) - - [Updated `process_registry_updates`](#updated-process_registry_updates) + - [Modified `process_epoch`](#modified-process_epoch) + - [Modified `process_registry_updates`](#modified-process_registry_updates) - [New `process_pending_balance_deposits`](#new-process_pending_balance_deposits) - [New `process_pending_consolidations`](#new-process_pending_consolidations) - - [Updated `process_effective_balance_updates`](#updated-process_effective_balance_updates) + - [Modified `process_effective_balance_updates`](#modified-process_effective_balance_updates) - [Block processing](#block-processing) - [Withdrawals](#withdrawals) - - [Updated `get_expected_withdrawals`](#updated-get_expected_withdrawals) - - [Updated `process_withdrawals`](#updated-process_withdrawals) + - [Modified `get_expected_withdrawals`](#modified-get_expected_withdrawals) + - [Modified `process_withdrawals`](#modified-process_withdrawals) - [Execution payload](#execution-payload) - [Modified `process_execution_payload`](#modified-process_execution_payload) - [Operations](#operations) @@ -85,12 +85,12 @@ - [Attestations](#attestations) - [Modified `process_attestation`](#modified-process_attestation) - [Deposits](#deposits) - - [Updated `apply_deposit`](#updated-apply_deposit) + - [Modified `apply_deposit`](#modified-apply_deposit) - [New `is_valid_deposit_signature`](#new-is_valid_deposit_signature) - [Modified `add_validator_to_registry`](#modified-add_validator_to_registry) - - [Updated `get_validator_from_deposit`](#updated-get_validator_from_deposit) + - [Modified `get_validator_from_deposit`](#modified-get_validator_from_deposit) - [Voluntary exits](#voluntary-exits) - - [Updated `process_voluntary_exit`](#updated-process_voluntary_exit) + - [Modified `process_voluntary_exit`](#modified-process_voluntary_exit) - [Execution layer withdrawal requests](#execution-layer-withdrawal-requests) - [New `process_withdrawal_request`](#new-process_withdrawal_request) - [Deposit requests](#deposit-requests) @@ -429,9 +429,9 @@ class BeaconState(Container): ### Predicates -#### Updated `compute_proposer_index` +#### Modified `compute_proposer_index` -*Note*: The function is modified to use `MAX_EFFECTIVE_BALANCE_ELECTRA` preset. +*Note*: The function `compute_proposer_index` is modified to use `MAX_EFFECTIVE_BALANCE_ELECTRA`. ```python def compute_proposer_index(state: BeaconState, indices: Sequence[ValidatorIndex], seed: Bytes32) -> ValidatorIndex: @@ -452,7 +452,9 @@ def compute_proposer_index(state: BeaconState, indices: Sequence[ValidatorIndex] i += 1 ``` -#### Updated `is_eligible_for_activation_queue` +#### Modified `is_eligible_for_activation_queue` + +*Note*: The function `is_eligible_for_activation_queue` is modified to use `MIN_ACTIVATION_BALANCE` instead of `MAX_EFFECTIVE_BALANCE`. ```python def is_eligible_for_activation_queue(validator: Validator) -> bool: @@ -492,7 +494,9 @@ def has_execution_withdrawal_credential(validator: Validator) -> bool: return has_compounding_withdrawal_credential(validator) or has_eth1_withdrawal_credential(validator) ``` -#### Updated `is_fully_withdrawable_validator` +#### Modified `is_fully_withdrawable_validator` + +*Note*: The function `is_fully_withdrawable_validator` is modified to use `has_execution_withdrawal_credential` instead of `has_eth1_withdrawal_credential`. ```python def is_fully_withdrawable_validator(validator: Validator, balance: Gwei, epoch: Epoch) -> bool: @@ -506,7 +510,9 @@ def is_fully_withdrawable_validator(validator: Validator, balance: Gwei, epoch: ) ``` -#### Updated `is_partially_withdrawable_validator` +#### Modified `is_partially_withdrawable_validator` + +*Note*: The function `is_partially_withdrawable_validator` is modified to use `get_validator_max_effective_balance` instead of `MAX_EFFECTIVE_BALANCE` and `has_execution_withdrawal_credential` instead of `has_eth1_withdrawal_credential`. ```python def is_partially_withdrawable_validator(validator: Validator, balance: Gwei) -> bool: @@ -597,6 +603,8 @@ def get_pending_balance_to_withdraw(state: BeaconState, validator_index: Validat #### Modified `get_attesting_indices` +*Note*: The function `get_attesting_indices` is modified to support EIP7549. + ```python def get_attesting_indices(state: BeaconState, attestation: Attestation) -> Set[ValidatorIndex]: """ @@ -618,7 +626,7 @@ def get_attesting_indices(state: BeaconState, attestation: Attestation) -> Set[V #### Modified `get_next_sync_committee_indices` -*Note*: The function is modified to use `MAX_EFFECTIVE_BALANCE_ELECTRA` preset. +*Note*: The function `get_next_sync_committee_indices` is modified to use `MAX_EFFECTIVE_BALANCE_ELECTRA`. ```python def get_next_sync_committee_indices(state: BeaconState) -> Sequence[ValidatorIndex]: @@ -645,10 +653,11 @@ def get_next_sync_committee_indices(state: BeaconState) -> Sequence[ValidatorInd return sync_committee_indices ``` - ### Beacon state mutators -#### Updated `initiate_validator_exit` +#### Modified `initiate_validator_exit` + +*Note*: The function `initiate_validator_exit` is modified to use the new `compute_exit_epoch_and_update_churn` function. ```python def initiate_validator_exit(state: BeaconState, index: ValidatorIndex) -> None: @@ -692,6 +701,7 @@ def queue_excess_active_balance(state: BeaconState, index: ValidatorIndex) -> No ``` #### New `queue_entire_balance_and_reset_validator` + ```python def queue_entire_balance_and_reset_validator(state: BeaconState, index: ValidatorIndex) -> None: balance = state.balances[index] @@ -757,7 +767,9 @@ def compute_consolidation_epoch_and_update_churn(state: BeaconState, consolidati return state.earliest_consolidation_epoch ``` -#### Updated `slash_validator` +#### Modified `slash_validator` + +*Note*: The function `slash_validator` is modified to change how the slashing penalty and proposer/whistleblower rewards are calculated in accordance with EIP7251. ```python def slash_validator(state: BeaconState, @@ -791,7 +803,10 @@ def slash_validator(state: BeaconState, ### Epoch processing -#### Updated `process_epoch` +#### Modified `process_epoch` + +*Note*: The function `process_epoch` is modified to call updated functions and to process pending balance deposits and pending consolidations which are new in Electra. + ```python def process_epoch(state: BeaconState) -> None: process_justification_and_finalization(state) @@ -810,9 +825,9 @@ def process_epoch(state: BeaconState) -> None: process_sync_committee_updates(state) ``` -#### Updated `process_registry_updates` +#### Modified `process_registry_updates` -`process_registry_updates` uses the updated definition of `initiate_validator_exit` +*Note*: The function `process_registry_updates` is modified to use the updated definition of `initiate_validator_exit` and changes how the activation epochs are computed for eligible validators. ```python @@ -899,9 +914,9 @@ def process_pending_consolidations(state: BeaconState) -> None: state.pending_consolidations = state.pending_consolidations[next_pending_consolidation:] ``` -#### Updated `process_effective_balance_updates` +#### Modified `process_effective_balance_updates` -`process_effective_balance_updates` is updated with a new limit for the maximum effective balance. +*Note*: The function `process_effective_balance_updates` is modified to use the new limit for the maximum effective balance. ```python def process_effective_balance_updates(state: BeaconState) -> None: @@ -911,6 +926,7 @@ def process_effective_balance_updates(state: BeaconState) -> None: HYSTERESIS_INCREMENT = uint64(EFFECTIVE_BALANCE_INCREMENT // HYSTERESIS_QUOTIENT) DOWNWARD_THRESHOLD = HYSTERESIS_INCREMENT * HYSTERESIS_DOWNWARD_MULTIPLIER UPWARD_THRESHOLD = HYSTERESIS_INCREMENT * HYSTERESIS_UPWARD_MULTIPLIER + # [Modified in Electra:EIP7251] EFFECTIVE_BALANCE_LIMIT = ( MAX_EFFECTIVE_BALANCE_ELECTRA if has_compounding_withdrawal_credential(validator) else MIN_ACTIVATION_BALANCE @@ -938,7 +954,9 @@ def process_block(state: BeaconState, block: BeaconBlock) -> None: #### Withdrawals -##### Updated `get_expected_withdrawals` +##### Modified `get_expected_withdrawals` + +*Note*: The function `get_expected_withdrawals` is modified to support EIP7251. ```python def get_expected_withdrawals(state: BeaconState) -> Tuple[Sequence[Withdrawal], uint64]: @@ -994,7 +1012,9 @@ def get_expected_withdrawals(state: BeaconState) -> Tuple[Sequence[Withdrawal], return withdrawals, partial_withdrawals_count ``` -##### Updated `process_withdrawals` +##### Modified `process_withdrawals` + +*Note*: The function `process_withdrawals` is modified to support EIP7251. ```python def process_withdrawals(state: BeaconState, payload: ExecutionPayload) -> None: @@ -1160,9 +1180,9 @@ def process_attestation(state: BeaconState, attestation: Attestation) -> None: ##### Deposits -###### Updated `apply_deposit` +###### Modified `apply_deposit` -*NOTE*: `process_deposit` is updated with a new definition of `apply_deposit`. +*Note*: The function `process_deposit` is modified to support EIP7251. ```python def apply_deposit(state: BeaconState, @@ -1180,7 +1200,7 @@ def apply_deposit(state: BeaconState, index = ValidatorIndex(validator_pubkeys.index(pubkey)) state.pending_balance_deposits.append( PendingBalanceDeposit(index=index, amount=amount) - ) # [Modified in Electra:EIP-7251] + ) # [Modified in Electra:EIP7251] # Check if valid deposit switch to compounding credentials if ( is_compounding_withdrawal_credential(withdrawal_credentials) @@ -1210,6 +1230,8 @@ def is_valid_deposit_signature(pubkey: BLSPubkey, ###### Modified `add_validator_to_registry` +*Note*: The function `add_validator_to_registry` is modified to initialize the validator with a balance of zero and add a pending balance deposit to the queue. + ```python def add_validator_to_registry(state: BeaconState, pubkey: BLSPubkey, @@ -1225,7 +1247,9 @@ def add_validator_to_registry(state: BeaconState, state.pending_balance_deposits.append(PendingBalanceDeposit(index=index, amount=amount)) # [New in Electra:EIP7251] ``` -###### Updated `get_validator_from_deposit` +###### Modified `get_validator_from_deposit` + +*Note*: The function `get_validator_from_deposit` is modified to initialize the validator with an effective balance of zero. ```python def get_validator_from_deposit(pubkey: BLSPubkey, withdrawal_credentials: Bytes32) -> Validator: @@ -1241,7 +1265,10 @@ def get_validator_from_deposit(pubkey: BLSPubkey, withdrawal_credentials: Bytes3 ``` ##### Voluntary exits -###### Updated `process_voluntary_exit` + +###### Modified `process_voluntary_exit` + +*Note*: The function `process_voluntary_exit` is modified to ensure the validator has no pending withdrawals in the queue. ```python def process_voluntary_exit(state: BeaconState, signed_voluntary_exit: SignedVoluntaryExit) -> None: @@ -1269,8 +1296,6 @@ def process_voluntary_exit(state: BeaconState, signed_voluntary_exit: SignedVolu ###### New `process_withdrawal_request` -*Note*: This function is new in Electra following EIP-7002 and EIP-7251. - ```python def process_withdrawal_request( state: BeaconState, @@ -1338,8 +1363,6 @@ def process_withdrawal_request( ###### New `process_deposit_request` -*Note*: This function is new in Electra:EIP6110. - ```python def process_deposit_request(state: BeaconState, deposit_request: DepositRequest) -> None: # Set deposit request start index From 7b72cbdfad48d577d48c62a466bdf949816f55b9 Mon Sep 17 00:00:00 2001 From: Justin Traglia Date: Mon, 1 Jul 2024 18:42:32 -0500 Subject: [PATCH 037/130] Fix link to Deneb spec --- specs/electra/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/electra/beacon-chain.md b/specs/electra/beacon-chain.md index 358355e7ef..d9185b2030 100644 --- a/specs/electra/beacon-chain.md +++ b/specs/electra/beacon-chain.md @@ -110,7 +110,7 @@ Electra is a consensus-layer upgrade containing a number of features. Including: * [EIP-7251](https://eips.ethereum.org/EIPS/eip-7251): Increase the MAX_EFFECTIVE_BALANCE * [EIP-7549](https://eips.ethereum.org/EIPS/eip-7549): Move committee index outside Attestation -*Note:* This specification is built upon [Deneb](../../deneb/beacon_chain.md) and is under active development. +*Note:* This specification is built upon [Deneb](../deneb/beacon_chain.md) and is under active development. ## Constants From b4f44caef3a5dadde2f99683b0bbb12ebe7ece17 Mon Sep 17 00:00:00 2001 From: Etan Kissling Date: Tue, 2 Jul 2024 16:26:04 +0200 Subject: [PATCH 038/130] Fix EL block hash computation for Deneb The parent beacon block hash was incorrectly set to zero in #3778. Passing the state to the computation function allows correct hash computation. --- .../test_process_execution_payload.py | 16 ++++---- .../fork_choice/test_on_merge_block.py | 8 ++-- .../test/bellatrix/sync/test_optimistic.py | 6 +-- .../unittests/test_validate_merge_block.py | 14 +++---- .../test_process_execution_payload.py | 2 +- .../test_process_withdrawals.py | 38 +++++++++---------- .../test/deneb/fork_choice/test_on_block.py | 2 +- .../merkle_proof/test_single_merkle_proof.py | 2 +- .../eth2spec/test/deneb/sanity/test_blocks.py | 2 +- .../unittests/validator/test_validator.py | 2 +- .../merkle_proof/test_single_merkle_proof.py | 2 +- .../test/electra/sanity/blocks/test_blocks.py | 8 ++-- .../sanity/blocks/test_deposit_transition.py | 4 +- .../test/helpers/execution_payload.py | 14 ++++--- .../pyspec/eth2spec/test/helpers/genesis.py | 6 ++- .../test/utils/randomized_block_tests.py | 2 +- 16 files changed, 68 insertions(+), 60 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/bellatrix/block_processing/test_process_execution_payload.py b/tests/core/pyspec/eth2spec/test/bellatrix/block_processing/test_process_execution_payload.py index a55a6e12a4..325431e7a9 100644 --- a/tests/core/pyspec/eth2spec/test/bellatrix/block_processing/test_process_execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/bellatrix/block_processing/test_process_execution_payload.py @@ -133,7 +133,7 @@ def test_bad_parent_hash_first_payload(spec, state): execution_payload = build_empty_execution_payload(spec, state) execution_payload.parent_hash = b'\x55' * 32 - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_execution_payload_processing(spec, state, execution_payload) @@ -146,7 +146,7 @@ def test_invalid_bad_parent_hash_regular_payload(spec, state): execution_payload = build_empty_execution_payload(spec, state) execution_payload.parent_hash = spec.Hash32() - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) @@ -156,7 +156,7 @@ def run_bad_prev_randao_test(spec, state): execution_payload = build_empty_execution_payload(spec, state) execution_payload.prev_randao = b'\x42' * 32 - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) @@ -182,7 +182,7 @@ def run_bad_everything_test(spec, state): execution_payload.parent_hash = spec.Hash32() execution_payload.prev_randao = spec.Bytes32() execution_payload.timestamp = 0 - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) @@ -211,7 +211,7 @@ def run_bad_timestamp_test(spec, state, is_future): else: timestamp = execution_payload.timestamp - 1 execution_payload.timestamp = timestamp - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) @@ -249,7 +249,7 @@ def run_non_empty_extra_data_test(spec, state): execution_payload = build_empty_execution_payload(spec, state) execution_payload.extra_data = b'\x45' * 12 - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_execution_payload_processing(spec, state, execution_payload) assert state.latest_execution_payload_header.extra_data == execution_payload.extra_data @@ -278,7 +278,7 @@ def run_non_empty_transactions_test(spec, state): spec.Transaction(b'\x99' * 128) for _ in range(num_transactions) ] - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_execution_payload_processing(spec, state, execution_payload) assert state.latest_execution_payload_header.transactions_root == execution_payload.transactions.hash_tree_root() @@ -304,7 +304,7 @@ def run_zero_length_transaction_test(spec, state): execution_payload = build_empty_execution_payload(spec, state) execution_payload.transactions = [spec.Transaction(b'')] assert len(execution_payload.transactions[0]) == 0 - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_execution_payload_processing(spec, state, execution_payload) assert state.latest_execution_payload_header.transactions_root == execution_payload.transactions.hash_tree_root() diff --git a/tests/core/pyspec/eth2spec/test/bellatrix/fork_choice/test_on_merge_block.py b/tests/core/pyspec/eth2spec/test/bellatrix/fork_choice/test_on_merge_block.py index 86071c225b..1520d85c5b 100644 --- a/tests/core/pyspec/eth2spec/test/bellatrix/fork_choice/test_on_merge_block.py +++ b/tests/core/pyspec/eth2spec/test/bellatrix/fork_choice/test_on_merge_block.py @@ -75,7 +75,7 @@ def test_all_valid(spec, state): def run_func(): block = build_empty_block_for_next_slot(spec, state) block.body.execution_payload.parent_hash = pow_block.block_hash - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) signed_block = state_transition_and_sign_block(spec, state, block) yield from tick_and_add_block(spec, store, signed_block, test_steps, merge_block=True) # valid @@ -107,7 +107,7 @@ def test_block_lookup_failed(spec, state): def run_func(): block = build_empty_block_for_next_slot(spec, state) block.body.execution_payload.parent_hash = pow_block.block_hash - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) signed_block = state_transition_and_sign_block(spec, state, block) yield from tick_and_add_block(spec, store, signed_block, test_steps, valid=False, merge_block=True, block_not_found=True) @@ -141,7 +141,7 @@ def test_too_early_for_merge(spec, state): def run_func(): block = build_empty_block_for_next_slot(spec, state) block.body.execution_payload.parent_hash = pow_block.block_hash - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) signed_block = state_transition_and_sign_block(spec, state, block) yield from tick_and_add_block(spec, store, signed_block, test_steps, valid=False, merge_block=True) @@ -174,7 +174,7 @@ def test_too_late_for_merge(spec, state): def run_func(): block = build_empty_block_for_next_slot(spec, state) block.body.execution_payload.parent_hash = pow_block.block_hash - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) signed_block = state_transition_and_sign_block(spec, state, block) yield from tick_and_add_block(spec, store, signed_block, test_steps, valid=False, merge_block=True) diff --git a/tests/core/pyspec/eth2spec/test/bellatrix/sync/test_optimistic.py b/tests/core/pyspec/eth2spec/test/bellatrix/sync/test_optimistic.py index eb56e368a8..2d1aaced91 100644 --- a/tests/core/pyspec/eth2spec/test/bellatrix/sync/test_optimistic.py +++ b/tests/core/pyspec/eth2spec/test/bellatrix/sync/test_optimistic.py @@ -65,7 +65,7 @@ def test_from_syncing_to_invalid(spec, state): block_hashes[f'chain_a_{i - 1}'] if i != 0 else block_hashes['block_0'] ) block.body.execution_payload.extra_data = spec.hash(bytes(f'chain_a_{i}', 'UTF-8')) - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) block_hashes[f'chain_a_{i}'] = block.body.execution_payload.block_hash signed_block = state_transition_and_sign_block(spec, state, block) @@ -82,7 +82,7 @@ def test_from_syncing_to_invalid(spec, state): block_hashes[f'chain_b_{i - 1}'] if i != 0 else block_hashes['block_0'] ) block.body.execution_payload.extra_data = spec.hash(bytes(f'chain_b_{i}', 'UTF-8')) - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) block_hashes[f'chain_b_{i}'] = block.body.execution_payload.block_hash signed_block = state_transition_with_full_block(spec, state, True, True, block=block) @@ -95,7 +95,7 @@ def test_from_syncing_to_invalid(spec, state): block = build_empty_block_for_next_slot(spec, state) block.body.execution_payload.parent_hash = signed_blocks_b[-1].message.body.execution_payload.block_hash block.body.execution_payload.extra_data = spec.hash(bytes(f'chain_b_{i}', 'UTF-8')) - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) block_hashes['chain_b_3'] = block.body.execution_payload.block_hash # Ensure that no duplicate block hashes diff --git a/tests/core/pyspec/eth2spec/test/bellatrix/unittests/test_validate_merge_block.py b/tests/core/pyspec/eth2spec/test/bellatrix/unittests/test_validate_merge_block.py index df9792a43d..d519dfac0f 100644 --- a/tests/core/pyspec/eth2spec/test/bellatrix/unittests/test_validate_merge_block.py +++ b/tests/core/pyspec/eth2spec/test/bellatrix/unittests/test_validate_merge_block.py @@ -60,7 +60,7 @@ def test_validate_merge_block_success(spec, state): pow_chain.head().total_difficulty = spec.config.TERMINAL_TOTAL_DIFFICULTY block = build_empty_block_for_next_slot(spec, state) block.body.execution_payload.parent_hash = pow_chain.head().block_hash - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) run_validate_merge_block(spec, pow_chain, block) @@ -81,7 +81,7 @@ def test_validate_merge_block_fail_parent_block_lookup(spec, state): pow_chain.head().total_difficulty = spec.config.TERMINAL_TOTAL_DIFFICULTY block = build_empty_block_for_next_slot(spec, state) block.body.execution_payload.parent_hash = pow_chain.head().block_hash - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) run_validate_merge_block(spec, pow_chain, block, valid=False) @@ -93,7 +93,7 @@ def test_validate_merge_block_fail_after_terminal(spec, state): pow_chain.head().total_difficulty = spec.config.TERMINAL_TOTAL_DIFFICULTY + uint256(1) block = build_empty_block_for_next_slot(spec, state) block.body.execution_payload.parent_hash = pow_chain.head().block_hash - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) run_validate_merge_block(spec, pow_chain, block, valid=False) @@ -110,7 +110,7 @@ def test_validate_merge_block_tbh_override_success(spec, state): pow_chain.head().block_hash = TERMINAL_BLOCK_HASH block = build_empty_block_for_next_slot(spec, state) block.body.execution_payload.parent_hash = pow_chain.head().block_hash - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) run_validate_merge_block(spec, pow_chain, block) @@ -126,7 +126,7 @@ def test_validate_merge_block_fail_parent_hash_is_not_tbh(spec, state): pow_chain.head().total_difficulty = spec.config.TERMINAL_TOTAL_DIFFICULTY block = build_empty_block_for_next_slot(spec, state) block.body.execution_payload.parent_hash = pow_chain.head().block_hash - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) run_validate_merge_block(spec, pow_chain, block, valid=False) @@ -143,7 +143,7 @@ def test_validate_merge_block_terminal_block_hash_fail_activation_not_reached(sp pow_chain.head().block_hash = TERMINAL_BLOCK_HASH block = build_empty_block_for_next_slot(spec, state) block.body.execution_payload.parent_hash = pow_chain.head().block_hash - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) run_validate_merge_block(spec, pow_chain, block, valid=False) @@ -159,5 +159,5 @@ def test_validate_merge_block_fail_activation_not_reached_parent_hash_is_not_tbh pow_chain.head().total_difficulty = spec.config.TERMINAL_TOTAL_DIFFICULTY block = build_empty_block_for_next_slot(spec, state) block.body.execution_payload.parent_hash = pow_chain.head().block_hash - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) run_validate_merge_block(spec, pow_chain, block, valid=False) diff --git a/tests/core/pyspec/eth2spec/test/capella/block_processing/test_process_execution_payload.py b/tests/core/pyspec/eth2spec/test/capella/block_processing/test_process_execution_payload.py index 1a2c1771a2..62e171b1f2 100644 --- a/tests/core/pyspec/eth2spec/test/capella/block_processing/test_process_execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/capella/block_processing/test_process_execution_payload.py @@ -19,6 +19,6 @@ def test_invalid_bad_parent_hash_first_payload(spec, state): execution_payload = build_empty_execution_payload(spec, state) execution_payload.parent_hash = b'\x55' * 32 - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_execution_payload_processing(spec, state, execution_payload, valid=False) diff --git a/tests/core/pyspec/eth2spec/test/capella/block_processing/test_process_withdrawals.py b/tests/core/pyspec/eth2spec/test/capella/block_processing/test_process_withdrawals.py index 1fe05c18df..891ad35ed6 100644 --- a/tests/core/pyspec/eth2spec/test/capella/block_processing/test_process_withdrawals.py +++ b/tests/core/pyspec/eth2spec/test/capella/block_processing/test_process_withdrawals.py @@ -259,7 +259,7 @@ def test_invalid_non_withdrawable_non_empty_withdrawals(spec, state): amount=420, ) execution_payload.withdrawals.append(withdrawal) - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_withdrawals_processing(spec, state, execution_payload, valid=False) @@ -272,7 +272,7 @@ def test_invalid_one_expected_full_withdrawal_and_none_in_withdrawals(spec, stat next_slot(spec, state) execution_payload = build_empty_execution_payload(spec, state) execution_payload.withdrawals = [] - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_withdrawals_processing(spec, state, execution_payload, valid=False) @@ -285,7 +285,7 @@ def test_invalid_one_expected_partial_withdrawal_and_none_in_withdrawals(spec, s next_slot(spec, state) execution_payload = build_empty_execution_payload(spec, state) execution_payload.withdrawals = [] - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_withdrawals_processing(spec, state, execution_payload, valid=False) @@ -298,7 +298,7 @@ def test_invalid_one_expected_full_withdrawal_and_duplicate_in_withdrawals(spec, next_slot(spec, state) execution_payload = build_empty_execution_payload(spec, state) execution_payload.withdrawals.append(execution_payload.withdrawals[0].copy()) - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_withdrawals_processing(spec, state, execution_payload, valid=False) @@ -311,7 +311,7 @@ def test_invalid_two_expected_partial_withdrawal_and_duplicate_in_withdrawals(sp next_slot(spec, state) execution_payload = build_empty_execution_payload(spec, state) execution_payload.withdrawals.append(execution_payload.withdrawals[0].copy()) - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_withdrawals_processing(spec, state, execution_payload, valid=False) @@ -325,7 +325,7 @@ def test_invalid_max_per_slot_full_withdrawals_and_one_less_in_withdrawals(spec, next_slot(spec, state) execution_payload = build_empty_execution_payload(spec, state) execution_payload.withdrawals = execution_payload.withdrawals[:-1] - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_withdrawals_processing(spec, state, execution_payload, valid=False) @@ -339,7 +339,7 @@ def test_invalid_max_per_slot_partial_withdrawals_and_one_less_in_withdrawals(sp next_slot(spec, state) execution_payload = build_empty_execution_payload(spec, state) execution_payload.withdrawals = execution_payload.withdrawals[:-1] - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_withdrawals_processing(spec, state, execution_payload, valid=False) @@ -353,7 +353,7 @@ def test_invalid_a_lot_fully_withdrawable_too_few_in_withdrawals(spec, state): next_slot(spec, state) execution_payload = build_empty_execution_payload(spec, state) execution_payload.withdrawals = execution_payload.withdrawals[:-1] - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_withdrawals_processing(spec, state, execution_payload, valid=False) @@ -367,7 +367,7 @@ def test_invalid_a_lot_partially_withdrawable_too_few_in_withdrawals(spec, state next_slot(spec, state) execution_payload = build_empty_execution_payload(spec, state) execution_payload.withdrawals = execution_payload.withdrawals[:-1] - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_withdrawals_processing(spec, state, execution_payload, valid=False) @@ -382,7 +382,7 @@ def test_invalid_a_lot_mixed_withdrawable_in_queue_too_few_in_withdrawals(spec, next_slot(spec, state) execution_payload = build_empty_execution_payload(spec, state) execution_payload.withdrawals = execution_payload.withdrawals[:-1] - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_withdrawals_processing(spec, state, execution_payload, valid=False) @@ -400,7 +400,7 @@ def test_invalid_incorrect_withdrawal_index(spec, state): next_slot(spec, state) execution_payload = build_empty_execution_payload(spec, state) execution_payload.withdrawals[0].index += 1 - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_withdrawals_processing(spec, state, execution_payload, valid=False) @@ -414,7 +414,7 @@ def test_invalid_incorrect_address_full(spec, state): next_slot(spec, state) execution_payload = build_empty_execution_payload(spec, state) execution_payload.withdrawals[0].address = b'\xff' * 20 - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_withdrawals_processing(spec, state, execution_payload, valid=False) @@ -428,7 +428,7 @@ def test_invalid_incorrect_address_partial(spec, state): next_slot(spec, state) execution_payload = build_empty_execution_payload(spec, state) execution_payload.withdrawals[0].address = b'\xff' * 20 - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_withdrawals_processing(spec, state, execution_payload, valid=False) @@ -441,7 +441,7 @@ def test_invalid_incorrect_amount_full(spec, state): next_slot(spec, state) execution_payload = build_empty_execution_payload(spec, state) execution_payload.withdrawals[0].amount += 1 - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_withdrawals_processing(spec, state, execution_payload, valid=False) @@ -454,7 +454,7 @@ def test_invalid_incorrect_amount_partial(spec, state): next_slot(spec, state) execution_payload = build_empty_execution_payload(spec, state) execution_payload.withdrawals[0].amount += 1 - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_withdrawals_processing(spec, state, execution_payload, valid=False) @@ -474,7 +474,7 @@ def test_invalid_one_of_many_incorrectly_full(spec, state): withdrawal.index += 1 withdrawal.address = b'\x99' * 20 withdrawal.amount += 4000000 - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_withdrawals_processing(spec, state, execution_payload, valid=False) @@ -494,7 +494,7 @@ def test_invalid_one_of_many_incorrectly_partial(spec, state): withdrawal.index += 1 withdrawal.address = b'\x99' * 20 withdrawal.amount += 4000000 - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_withdrawals_processing(spec, state, execution_payload, valid=False) @@ -514,7 +514,7 @@ def test_invalid_many_incorrectly_full(spec, state): withdrawal.address = i.to_bytes(20, 'big') else: withdrawal.amount += 1 - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_withdrawals_processing(spec, state, execution_payload, valid=False) @@ -534,7 +534,7 @@ def test_invalid_many_incorrectly_partial(spec, state): withdrawal.address = i.to_bytes(20, 'big') else: withdrawal.amount += 1 - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_withdrawals_processing(spec, state, execution_payload, valid=False) diff --git a/tests/core/pyspec/eth2spec/test/deneb/fork_choice/test_on_block.py b/tests/core/pyspec/eth2spec/test/deneb/fork_choice/test_on_block.py index 4ab7d819a9..f8ffa62346 100644 --- a/tests/core/pyspec/eth2spec/test/deneb/fork_choice/test_on_block.py +++ b/tests/core/pyspec/eth2spec/test/deneb/fork_choice/test_on_block.py @@ -29,7 +29,7 @@ def get_block_with_blob(spec, state, rng=None): block = build_empty_block_for_next_slot(spec, state) opaque_tx, blobs, blob_kzg_commitments, blob_kzg_proofs = get_sample_opaque_tx(spec, blob_count=1, rng=rng) block.body.execution_payload.transactions = [opaque_tx] - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) block.body.blob_kzg_commitments = blob_kzg_commitments return block, blobs, blob_kzg_proofs diff --git a/tests/core/pyspec/eth2spec/test/deneb/merkle_proof/test_single_merkle_proof.py b/tests/core/pyspec/eth2spec/test/deneb/merkle_proof/test_single_merkle_proof.py index 8ec6a1d927..1dda310123 100644 --- a/tests/core/pyspec/eth2spec/test/deneb/merkle_proof/test_single_merkle_proof.py +++ b/tests/core/pyspec/eth2spec/test/deneb/merkle_proof/test_single_merkle_proof.py @@ -36,7 +36,7 @@ def _run_blob_kzg_commitment_merkle_proof_test(spec, state, rng=None): ) block.body.blob_kzg_commitments = blob_kzg_commitments block.body.execution_payload.transactions = [opaque_tx] - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) signed_block = sign_block(spec, state, block, proposer_index=0) blob_sidecars = spec.get_blob_sidecars(signed_block, blobs, proofs) blob_index = 0 diff --git a/tests/core/pyspec/eth2spec/test/deneb/sanity/test_blocks.py b/tests/core/pyspec/eth2spec/test/deneb/sanity/test_blocks.py index 8598530d9a..94db2c34fc 100644 --- a/tests/core/pyspec/eth2spec/test/deneb/sanity/test_blocks.py +++ b/tests/core/pyspec/eth2spec/test/deneb/sanity/test_blocks.py @@ -40,7 +40,7 @@ def run_block_with_blobs(spec, state, blob_count, tx_count=1, blob_gas_used=1, e block.body.execution_payload.transactions = txs block.body.execution_payload.blob_gas_used = blob_gas_used block.body.execution_payload.excess_blob_gas = excess_blob_gas - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) if valid: signed_block = state_transition_and_sign_block(spec, state, block) diff --git a/tests/core/pyspec/eth2spec/test/deneb/unittests/validator/test_validator.py b/tests/core/pyspec/eth2spec/test/deneb/unittests/validator/test_validator.py index 7ed88cba45..6724e8304a 100644 --- a/tests/core/pyspec/eth2spec/test/deneb/unittests/validator/test_validator.py +++ b/tests/core/pyspec/eth2spec/test/deneb/unittests/validator/test_validator.py @@ -26,7 +26,7 @@ def _get_sample_sidecars(spec, state, rng): block.body.blob_kzg_commitments = blob_kzg_commitments_1 + blob_kzg_commitments_2 block.body.execution_payload.transactions = [opaque_tx_1, opaque_tx_2] - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) blobs = blobs_1 + blobs_2 proofs = proofs_1 + proofs_2 diff --git a/tests/core/pyspec/eth2spec/test/eip7594/merkle_proof/test_single_merkle_proof.py b/tests/core/pyspec/eth2spec/test/eip7594/merkle_proof/test_single_merkle_proof.py index 8e67b3d0bf..3cdf3685e9 100644 --- a/tests/core/pyspec/eth2spec/test/eip7594/merkle_proof/test_single_merkle_proof.py +++ b/tests/core/pyspec/eth2spec/test/eip7594/merkle_proof/test_single_merkle_proof.py @@ -36,7 +36,7 @@ def _run_blob_kzg_commitments_merkle_proof_test(spec, state, rng=None): ) block.body.blob_kzg_commitments = blob_kzg_commitments block.body.execution_payload.transactions = [opaque_tx] - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) signed_block = sign_block(spec, state, block, proposer_index=0) column_sidcars = spec.get_data_column_sidecars(signed_block, blobs) column_sidcar = column_sidcars[0] diff --git a/tests/core/pyspec/eth2spec/test/electra/sanity/blocks/test_blocks.py b/tests/core/pyspec/eth2spec/test/electra/sanity/blocks/test_blocks.py index 4fc76e9c93..9fa4d609fc 100644 --- a/tests/core/pyspec/eth2spec/test/electra/sanity/blocks/test_blocks.py +++ b/tests/core/pyspec/eth2spec/test/electra/sanity/blocks/test_blocks.py @@ -42,7 +42,7 @@ def test_basic_el_withdrawal_request(spec, state): ) block = build_empty_block_for_next_slot(spec, state) block.body.execution_payload.withdrawal_requests = [withdrawal_request] - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) signed_block = state_transition_and_sign_block(spec, state, block) yield 'blocks', [signed_block] @@ -79,7 +79,7 @@ def test_basic_btec_and_el_withdrawal_request_in_same_block(spec, state): ) block.body.execution_payload.withdrawal_requests = [withdrawal_request] - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) signed_block = state_transition_and_sign_block(spec, state, block) yield 'blocks', [signed_block] @@ -131,7 +131,7 @@ def test_basic_btec_before_el_withdrawal_request(spec, state): ) block_2 = build_empty_block_for_next_slot(spec, state) block_2.body.execution_payload.withdrawal_requests = [withdrawal_request] - block_2.body.execution_payload.block_hash = compute_el_block_hash(spec, block_2.body.execution_payload) + block_2.body.execution_payload.block_hash = compute_el_block_hash(spec, block_2.body.execution_payload, state) signed_block_2 = state_transition_and_sign_block(spec, state, block_2) yield 'blocks', [signed_block_1, signed_block_2] @@ -164,7 +164,7 @@ def test_cl_exit_and_el_withdrawal_request_in_same_block(spec, state): block = build_empty_block_for_next_slot(spec, state) block.body.voluntary_exits = signed_voluntary_exits block.body.execution_payload.withdrawal_requests = [withdrawal_request] - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) signed_block = state_transition_and_sign_block(spec, state, block) yield 'blocks', [signed_block] diff --git a/tests/core/pyspec/eth2spec/test/electra/sanity/blocks/test_deposit_transition.py b/tests/core/pyspec/eth2spec/test/electra/sanity/blocks/test_deposit_transition.py index f253b6c60d..4b8c1cd4f8 100644 --- a/tests/core/pyspec/eth2spec/test/electra/sanity/blocks/test_deposit_transition.py +++ b/tests/core/pyspec/eth2spec/test/electra/sanity/blocks/test_deposit_transition.py @@ -103,7 +103,7 @@ def prepare_state_and_block(spec, # Assign deposits and deposit requests block.body.deposits = deposits block.body.execution_payload.deposit_requests = deposit_requests - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) return state, block @@ -220,7 +220,7 @@ def test_deposit_transition__deposit_and_top_up_same_block(spec, state): # Artificially assign deposit's pubkey to a deposit request of the same block top_up_keys = [block.body.deposits[0].data.pubkey] block.body.execution_payload.deposit_requests[0].pubkey = top_up_keys[0] - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) pre_pending_deposits = len(state.pending_balance_deposits) diff --git a/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py index b061f506a8..d8be4a92ae 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py @@ -59,6 +59,7 @@ def compute_el_header_block_hash(spec, payload_header, transactions_trie_root, withdrawals_trie_root=None, + parent_beacon_block_root=None, requests_trie_root=None): """ Computes the RLP execution block hash described by an `ExecutionPayloadHeader`. @@ -106,8 +107,7 @@ def compute_el_header_block_hash(spec, # excess_blob_gas execution_payload_header_rlp.append((big_endian_int, payload_header.excess_blob_gas)) # parent_beacon_root - empty_root = bytes.fromhex("0000000000000000000000000000000000000000000000000000000000000000") - execution_payload_header_rlp.append((Binary(32, 32), empty_root)) + execution_payload_header_rlp.append((Binary(32, 32), parent_beacon_block_root)) if is_post_electra(spec): # requests_root execution_payload_header_rlp.append((Binary(32, 32), requests_trie_root)) @@ -186,15 +186,18 @@ def get_consolidation_request_rlp_bytes(consolidation_request): return b"\x02" + encode(values, sedes) -def compute_el_block_hash(spec, payload): +def compute_el_block_hash(spec, payload, pre_state): transactions_trie_root = compute_trie_root_from_indexed_data(payload.transactions) withdrawals_trie_root = None + parent_beacon_block_root = None requests_trie_root = None if is_post_capella(spec): withdrawals_encoded = [get_withdrawal_rlp(withdrawal) for withdrawal in payload.withdrawals] withdrawals_trie_root = compute_trie_root_from_indexed_data(withdrawals_encoded) + if is_post_deneb(spec): + parent_beacon_block_root = pre_state.latest_block_header.hash_tree_root() if is_post_electra(spec): requests_encoded = [] requests_encoded += [get_deposit_request_rlp_bytes(request) for request in payload.deposit_requests] @@ -210,6 +213,7 @@ def compute_el_block_hash(spec, payload): payload_header, transactions_trie_root, withdrawals_trie_root, + parent_beacon_block_root, requests_trie_root, ) @@ -250,7 +254,7 @@ def build_empty_execution_payload(spec, state, randao_mix=None): payload.withdrawal_requests = [] payload.consolidation_requests = [] - payload.block_hash = compute_el_block_hash(spec, payload) + payload.block_hash = compute_el_block_hash(spec, payload, state) return payload @@ -278,7 +282,7 @@ def build_randomized_execution_payload(spec, state, rng): for _ in range(num_transactions) ] - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) return execution_payload diff --git a/tests/core/pyspec/eth2spec/test/helpers/genesis.py b/tests/core/pyspec/eth2spec/test/helpers/genesis.py index 4013b67bf2..95c1e97e55 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/genesis.py +++ b/tests/core/pyspec/eth2spec/test/helpers/genesis.py @@ -6,7 +6,7 @@ compute_el_header_block_hash, ) from eth2spec.test.helpers.forks import ( - is_post_altair, is_post_bellatrix, is_post_capella, is_post_electra, is_post_whisk, + is_post_altair, is_post_bellatrix, is_post_capella, is_post_deneb, is_post_electra, is_post_whisk, ) from eth2spec.test.helpers.keys import pubkeys from eth2spec.test.helpers.whisk import compute_whisk_initial_tracker_cached, compute_whisk_initial_k_commitment_cached @@ -65,10 +65,13 @@ def get_sample_genesis_execution_payload_header(spec, transactions_trie_root = bytes.fromhex("56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421") withdrawals_trie_root = None + parent_beacon_block_root = None requests_trie_root = None if is_post_capella(spec): withdrawals_trie_root = bytes.fromhex("56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421") + if is_post_deneb(spec): + parent_beacon_block_root = bytes.fromhex("0000000000000000000000000000000000000000000000000000000000000000") if is_post_electra(spec): requests_trie_root = bytes.fromhex("56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421") @@ -77,6 +80,7 @@ def get_sample_genesis_execution_payload_header(spec, payload_header, transactions_trie_root, withdrawals_trie_root, + parent_beacon_block_root, requests_trie_root, ) return payload_header diff --git a/tests/core/pyspec/eth2spec/test/utils/randomized_block_tests.py b/tests/core/pyspec/eth2spec/test/utils/randomized_block_tests.py index a527731b14..f3497de45d 100644 --- a/tests/core/pyspec/eth2spec/test/utils/randomized_block_tests.py +++ b/tests/core/pyspec/eth2spec/test/utils/randomized_block_tests.py @@ -253,7 +253,7 @@ def random_block_deneb(spec, state, signed_blocks, scenario_state, rng=Random(34 opaque_tx, _, blob_kzg_commitments, _ = get_sample_opaque_tx( spec, blob_count=rng.randint(0, spec.MAX_BLOBS_PER_BLOCK), rng=rng) block.body.execution_payload.transactions.append(opaque_tx) - block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload) + block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) block.body.blob_kzg_commitments = blob_kzg_commitments return block From a63bad0c093d31e394be9430c776d28bdbec68e5 Mon Sep 17 00:00:00 2001 From: Etan Kissling Date: Tue, 2 Jul 2024 19:56:14 +0200 Subject: [PATCH 039/130] Fix Deneb `test_process_execution_payload` --- .../test_process_execution_payload.py | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/deneb/block_processing/test_process_execution_payload.py b/tests/core/pyspec/eth2spec/test/deneb/block_processing/test_process_execution_payload.py index b0937aac96..b8bd39a04d 100644 --- a/tests/core/pyspec/eth2spec/test/deneb/block_processing/test_process_execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/deneb/block_processing/test_process_execution_payload.py @@ -78,7 +78,7 @@ def test_incorrect_blob_tx_type(spec, state): opaque_tx = b'\x04' + opaque_tx[1:] # incorrect tx type execution_payload.transactions = [opaque_tx] - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_execution_payload_processing(spec, state, execution_payload, blob_kzg_commitments) @@ -95,7 +95,7 @@ def test_incorrect_transaction_length_1_extra_byte(spec, state): opaque_tx = opaque_tx + b'\x12' # incorrect tx length, longer execution_payload.transactions = [opaque_tx] - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_execution_payload_processing(spec, state, execution_payload, blob_kzg_commitments) @@ -112,7 +112,7 @@ def test_incorrect_transaction_length_1_byte_short(spec, state): opaque_tx = opaque_tx[:-1] # incorrect tx length, shorter execution_payload.transactions = [opaque_tx] - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_execution_payload_processing(spec, state, execution_payload, blob_kzg_commitments) @@ -129,7 +129,7 @@ def test_incorrect_transaction_length_empty(spec, state): opaque_tx = opaque_tx[0:0] # incorrect tx length, empty execution_payload.transactions = [opaque_tx] - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_execution_payload_processing(spec, state, execution_payload, blob_kzg_commitments) @@ -146,7 +146,7 @@ def test_incorrect_transaction_length_32_extra_bytes(spec, state): opaque_tx = opaque_tx + b'\x12' * 32 # incorrect tx length execution_payload.transactions = [opaque_tx] - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_execution_payload_processing(spec, state, execution_payload, blob_kzg_commitments) @@ -162,7 +162,7 @@ def test_no_transactions_with_commitments(spec, state): _, _, blob_kzg_commitments, _ = get_sample_opaque_tx(spec) execution_payload.transactions = [] - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_execution_payload_processing(spec, state, execution_payload, blob_kzg_commitments) @@ -179,7 +179,7 @@ def test_incorrect_commitment(spec, state): blob_kzg_commitments[0] = b'\x12' * 48 # incorrect commitment execution_payload.transactions = [opaque_tx] - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_execution_payload_processing(spec, state, execution_payload, blob_kzg_commitments) @@ -196,7 +196,7 @@ def test_incorrect_commitments_order(spec, state): blob_kzg_commitments = [blob_kzg_commitments[1], blob_kzg_commitments[0]] # incorrect order execution_payload.transactions = [opaque_tx] - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_execution_payload_processing(spec, state, execution_payload, blob_kzg_commitments) @@ -227,7 +227,7 @@ def test_zeroed_commitment(spec, state): assert all(commitment == b'\x00' * 48 for commitment in blob_kzg_commitments) execution_payload.transactions = [opaque_tx] - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_execution_payload_processing(spec, state, execution_payload, blob_kzg_commitments) @@ -243,7 +243,7 @@ def test_invalid_correct_input__execution_invalid(spec, state): opaque_tx, _, blob_kzg_commitments, _ = get_sample_opaque_tx(spec) execution_payload.transactions = [opaque_tx] - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_execution_payload_processing(spec, state, execution_payload, blob_kzg_commitments, valid=False, execution_valid=False) @@ -257,6 +257,6 @@ def test_invalid_exceed_max_blobs_per_block(spec, state): opaque_tx, _, blob_kzg_commitments, _ = get_sample_opaque_tx(spec, blob_count=spec.MAX_BLOBS_PER_BLOCK + 1) execution_payload.transactions = [opaque_tx] - execution_payload.block_hash = compute_el_block_hash(spec, execution_payload) + execution_payload.block_hash = compute_el_block_hash(spec, execution_payload, state) yield from run_execution_payload_processing(spec, state, execution_payload, blob_kzg_commitments, valid=False) From c078a97f0c3e33043bd6863f94552e2ae002ebd4 Mon Sep 17 00:00:00 2001 From: Justin Traglia Date: Tue, 2 Jul 2024 15:38:15 -0500 Subject: [PATCH 040/130] Remove verify_cell_kzg_proof (non-batch) --- .../polynomial-commitments-sampling.md | 27 --- .../test_polynomial_commitments.py | 14 -- tests/formats/kzg_7594/README.md | 1 - .../formats/kzg_7594/verify_cell_kzg_proof.md | 26 --- tests/generators/kzg_7594/main.py | 155 ------------------ 5 files changed, 223 deletions(-) delete mode 100644 tests/formats/kzg_7594/verify_cell_kzg_proof.md diff --git a/specs/_features/eip7594/polynomial-commitments-sampling.md b/specs/_features/eip7594/polynomial-commitments-sampling.md index 10da2e58c3..de04557820 100644 --- a/specs/_features/eip7594/polynomial-commitments-sampling.md +++ b/specs/_features/eip7594/polynomial-commitments-sampling.md @@ -43,7 +43,6 @@ - [Cell computation](#cell-computation) - [`compute_cells_and_kzg_proofs`](#compute_cells_and_kzg_proofs) - [Cell verification](#cell-verification) - - [`verify_cell_kzg_proof`](#verify_cell_kzg_proof) - [`verify_cell_kzg_proof_batch`](#verify_cell_kzg_proof_batch) - [Reconstruction](#reconstruction) - [`construct_vanishing_polynomial`](#construct_vanishing_polynomial) @@ -642,32 +641,6 @@ def compute_cells_and_kzg_proofs(blob: Blob) -> Tuple[ ### Cell verification -#### `verify_cell_kzg_proof` - -```python -def verify_cell_kzg_proof(commitment_bytes: Bytes48, - cell_index: CellIndex, - cell: Cell, - proof_bytes: Bytes48) -> bool: - """ - Check a cell proof - - Public method. - """ - assert len(commitment_bytes) == BYTES_PER_COMMITMENT - assert cell_index < CELLS_PER_EXT_BLOB - assert len(cell) == BYTES_PER_CELL - assert len(proof_bytes) == BYTES_PER_PROOF - - coset = coset_for_cell(cell_index) - - return verify_kzg_proof_multi_impl( - bytes_to_kzg_commitment(commitment_bytes), - coset, - cell_to_coset_evals(cell), - bytes_to_kzg_proof(proof_bytes)) -``` - #### `verify_cell_kzg_proof_batch` ```python diff --git a/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py b/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py index dd5aece3f9..2331dda96a 100644 --- a/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py +++ b/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py @@ -108,20 +108,6 @@ def test_construct_vanishing_polynomial(spec): assert all(a != 0 for a in zero_poly_eval_brp[start:end]) -@with_eip7594_and_later -@spec_test -@single_phase -def test_verify_cell_kzg_proof(spec): - blob = get_sample_blob(spec) - commitment = spec.blob_to_kzg_commitment(blob) - cells, proofs = spec.compute_cells_and_kzg_proofs(blob) - - cell_index = 0 - assert spec.verify_cell_kzg_proof(commitment, cell_index, cells[cell_index], proofs[cell_index]) - cell_index = 1 - assert spec.verify_cell_kzg_proof(commitment, cell_index, cells[cell_index], proofs[cell_index]) - - @with_eip7594_and_later @spec_test @single_phase diff --git a/tests/formats/kzg_7594/README.md b/tests/formats/kzg_7594/README.md index dbd95dd3dc..f03bc3707c 100644 --- a/tests/formats/kzg_7594/README.md +++ b/tests/formats/kzg_7594/README.md @@ -8,6 +8,5 @@ The KZG test suite runner has the following handlers: - [`compute_cells`](./compute_cells.md) - [`compute_cells_and_kzg_proofs`](./compute_cells_and_kzg_proofs.md) -- [`verify_cell_kzg_proof`](./verify_cell_kzg_proof.md) - [`verify_cell_kzg_proof_batch`](./verify_cell_kzg_proof_batch.md) - [`recover_all_cells`](./recover_all_cells.md) diff --git a/tests/formats/kzg_7594/verify_cell_kzg_proof.md b/tests/formats/kzg_7594/verify_cell_kzg_proof.md deleted file mode 100644 index 3d5f33b524..0000000000 --- a/tests/formats/kzg_7594/verify_cell_kzg_proof.md +++ /dev/null @@ -1,26 +0,0 @@ -# Test format: Verify cell KZG proof - -Use the cell KZG `proof` to verify that the KZG `commitment` for a given `cell` is correct. - -## Test case format - -The test data is declared in a `data.yaml` file: - -```yaml -input: - commitment: Bytes48 -- the KZG commitment - cell_index: CellIndex -- the cell index - cell: Cell -- the cell - proof: Bytes48 -- the KZG proof for the cell -output: bool -- true (correct proof) or false (incorrect proof) -``` - -- `Bytes48` is a 48-byte hexadecimal string, prefixed with `0x`. -- `CellIndex` is an unsigned 64-bit integer. -- `Cell` is a 2048-byte hexadecimal string, prefixed with `0x`. - -All byte(s) fields are encoded as strings, hexadecimal encoding, prefixed with `0x`. - -## Condition - -The `verify_cell_kzg_proof` handler should verify that `commitment` is a correct KZG commitment to `cell` by using the cell KZG proof `proof`, and the result should match the expected `output`. If the commitment or proof is invalid (e.g. not on the curve or not in the G1 subgroup of the BLS curve), `cell` is invalid (e.g. incorrect length or one of the 32-byte blocks does not represent a BLS field element), or `cell_index` is invalid (e.g. greater than the number of cells for an extended blob), it should error, i.e. the output should be `null`. diff --git a/tests/generators/kzg_7594/main.py b/tests/generators/kzg_7594/main.py index 428baa566c..54bc02dd5b 100644 --- a/tests/generators/kzg_7594/main.py +++ b/tests/generators/kzg_7594/main.py @@ -58,160 +58,6 @@ def case_compute_cells_and_kzg_proofs(): } -############################################################################### -# Test cases for verify_cell_kzg_proof -############################################################################### - -def case_verify_cell_kzg_proof(): - # Valid cases - for i in range(len(VALID_BLOBS)): - cells, proofs = VALID_CELLS_AND_PROOFS[i] - commitment = VALID_COMMITMENTS[i] - cell_index = (2 ** i - 1) % spec.CELLS_PER_EXT_BLOB - cell = cells[cell_index] - proof = proofs[cell_index] - assert spec.verify_cell_kzg_proof(commitment, cell_index, cell, proof) - identifier = make_id(commitment, cell_index, cell, proof) - yield f'verify_cell_kzg_proof_case_valid_{identifier}', { - 'input': { - 'commitment': encode_hex(commitment), - 'cell_index': cell_index, - 'cell': encode_hex(cell), - 'proof': encode_hex(proof), - }, - 'output': True - } - - # Incorrect commitment - for i in range(len(VALID_BLOBS)): - cells, proofs = VALID_CELLS_AND_PROOFS[i] - commitment = bls_add_one(VALID_COMMITMENTS[i]) - cell_index = 99 % spec.CELLS_PER_EXT_BLOB - cell = cells[cell_index] - proof = proofs[cell_index] - assert not spec.verify_cell_kzg_proof(commitment, cell_index, cell, proof) - identifier = make_id(commitment, cell_index, cell, proof) - yield f'verify_cell_kzg_proof_case_incorrect_commitment_{identifier}', { - 'input': { - 'commitment': encode_hex(commitment), - 'cell_index': cell_index, - 'cell': encode_hex(cell), - 'proof': encode_hex(proof), - }, - 'output': False - } - - # Incorrect cell - for i in range(len(VALID_INDIVIDUAL_RANDOM_CELL_BYTES)): - cell_index = 16 % spec.CELLS_PER_EXT_BLOB - commitment = VALID_COMMITMENTS[i] - cells, proofs = VALID_CELLS_AND_PROOFS[i] - cell = VALID_INDIVIDUAL_RANDOM_CELL_BYTES[i] - proof = proofs[cell_index] - assert not spec.verify_cell_kzg_proof(commitment, cell_index, cell, proof) - identifier = make_id(commitment, cell_index, cell, proof) - yield f'verify_cell_kzg_proof_case_incorrect_cell_{identifier}', { - 'input': { - 'commitment': encode_hex(commitment), - 'cell_index': cell_index, - 'cell': encode_hex(cell), - 'proof': encode_hex(proof), - }, - 'output': False - } - - # Incorrect proof - for i in range(len(VALID_BLOBS)): - cell_index = 91 % spec.CELLS_PER_EXT_BLOB - commitment = VALID_COMMITMENTS[i] - cells, proofs = VALID_CELLS_AND_PROOFS[i] - cell = cells[cell_index] - proof = bls_add_one(proofs[cell_index]) - assert not spec.verify_cell_kzg_proof(commitment, cell_index, cell, proof) - identifier = make_id(commitment, cell_index, cell, proof) - yield f'verify_cell_kzg_proof_case_incorrect_proof_{identifier}', { - 'input': { - 'commitment': encode_hex(commitment), - 'cell_index': cell_index, - 'cell': encode_hex(cell), - 'proof': encode_hex(proof), - }, - 'output': False - } - - # Edge case: Invalid commitment - for commitment in INVALID_G1_POINTS: - cells, proofs = VALID_CELLS_AND_PROOFS[0] - cell_index = 81 % spec.CELLS_PER_EXT_BLOB - cell = cells[cell_index] - proof = proofs[cell_index] - expect_exception(spec.verify_cell_kzg_proof, commitment, cell_index, cell, proof) - identifier = make_id(commitment, cell_index, cell, proof) - yield f'verify_cell_kzg_proof_case_invalid_commitment_{identifier}', { - 'input': { - 'commitment': encode_hex(commitment), - 'cell_index': cell_index, - 'cell': encode_hex(cell), - 'proof': encode_hex(proof), - }, - 'output': None - } - - # Edge case: Invalid cell_index - for cell_index in [spec.CELLS_PER_EXT_BLOB, spec.CELLS_PER_EXT_BLOB + 1]: - cells, proofs = VALID_CELLS_AND_PROOFS[1] - commitment = VALID_COMMITMENTS[1] - cell = cells[0] - proof = proofs[0] - expect_exception(spec.verify_cell_kzg_proof, commitment, cell_index, cell, proof) - identifier = make_id(commitment, cell_index, cell, proof) - yield f'verify_cell_kzg_proof_case_invalid_cell_index_{identifier}', { - 'input': { - 'commitment': encode_hex(commitment), - 'cell_index': cell_index, - 'cell': encode_hex(cell), - 'proof': encode_hex(proof), - }, - 'output': None - } - - # Edge case: Invalid cell - for cell in INVALID_INDIVIDUAL_CELL_BYTES: - cell_index = 32 % spec.CELLS_PER_EXT_BLOB - commitment = VALID_COMMITMENTS[2] - cells, proofs = VALID_CELLS_AND_PROOFS[2] - proof = proofs[cell_index] - expect_exception(spec.verify_cell_kzg_proof, commitment, cell_index, cell, proof) - identifier = make_id(commitment, cell_index, cell, proof) - yield f'verify_cell_kzg_proof_case_invalid_cell_{identifier}', { - 'input': { - 'commitment': encode_hex(commitment), - 'cell_index': cell_index, - 'cell': encode_hex(cell), - 'proof': encode_hex(proof), - }, - 'output': None - } - - # Edge case: Invalid proof - for proof in INVALID_G1_POINTS: - cells, _ = VALID_CELLS_AND_PROOFS[3] - commitment = VALID_COMMITMENTS[3] - cell_index = 36 % spec.CELLS_PER_EXT_BLOB - cell = cells[cell_index] - expect_exception(spec.verify_cell_kzg_proof, commitment, cell_index, cell, proof) - identifier = make_id(commitment, cell_index, cell, proof) - yield f'verify_cell_kzg_proof_case_invalid_proof_{identifier}', { - 'input': { - 'commitment': encode_hex(commitment), - 'cell_index': cell_index, - 'cell': encode_hex(cell), - 'proof': encode_hex(proof), - }, - 'output': None - } - - ############################################################################### # Test cases for verify_cell_kzg_proof_batch ############################################################################### @@ -812,7 +658,6 @@ def cases_fn() -> Iterable[gen_typing.TestCase]: gen_runner.run_generator("kzg_7594", [ # EIP-7594 create_provider(EIP7594, 'compute_cells_and_kzg_proofs', case_compute_cells_and_kzg_proofs), - create_provider(EIP7594, 'verify_cell_kzg_proof', case_verify_cell_kzg_proof), create_provider(EIP7594, 'verify_cell_kzg_proof_batch', case_verify_cell_kzg_proof_batch), create_provider(EIP7594, 'recover_cells_and_kzg_proofs', case_recover_cells_and_kzg_proofs), ]) From 1189d52526d8f02218f3fe22c57a9bdddd97bdf5 Mon Sep 17 00:00:00 2001 From: Justin Traglia Date: Tue, 2 Jul 2024 16:35:28 -0500 Subject: [PATCH 041/130] Remove VALID_INDIVIDUAL_RANDOM_CELL_BYTES import --- tests/generators/kzg_7594/main.py | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/generators/kzg_7594/main.py b/tests/generators/kzg_7594/main.py index 54bc02dd5b..b3b922c54c 100644 --- a/tests/generators/kzg_7594/main.py +++ b/tests/generators/kzg_7594/main.py @@ -19,7 +19,6 @@ VALID_BLOBS, VALID_CELLS_AND_PROOFS, VALID_COMMITMENTS, - VALID_INDIVIDUAL_RANDOM_CELL_BYTES, bls_add_one, encode_hex_list, expect_exception, From 0daa2acdff87d4ca1fe549a29fc2eff15c0e33a1 Mon Sep 17 00:00:00 2001 From: Justin Traglia Date: Tue, 2 Jul 2024 17:05:40 -0500 Subject: [PATCH 042/130] Remove verify_kzg_proof_multi_impl too --- .../polynomial-commitments-sampling.md | 40 ------------------- 1 file changed, 40 deletions(-) diff --git a/specs/_features/eip7594/polynomial-commitments-sampling.md b/specs/_features/eip7594/polynomial-commitments-sampling.md index de04557820..9625ea3d72 100644 --- a/specs/_features/eip7594/polynomial-commitments-sampling.md +++ b/specs/_features/eip7594/polynomial-commitments-sampling.md @@ -34,7 +34,6 @@ - [`evaluate_polynomialcoeff`](#evaluate_polynomialcoeff) - [KZG multiproofs](#kzg-multiproofs) - [`compute_kzg_proof_multi_impl`](#compute_kzg_proof_multi_impl) - - [`verify_kzg_proof_multi_impl`](#verify_kzg_proof_multi_impl) - [`verify_cell_kzg_proof_batch_impl`](#verify_cell_kzg_proof_batch_impl) - [Cell cosets](#cell-cosets) - [`coset_shift_for_cell`](#coset_shift_for_cell) @@ -432,45 +431,6 @@ def compute_kzg_proof_multi_impl( return KZGProof(g1_lincomb(KZG_SETUP_G1_MONOMIAL[:len(quotient_polynomial)], quotient_polynomial)), ys ``` -#### `verify_kzg_proof_multi_impl` - -```python -def verify_kzg_proof_multi_impl(commitment: KZGCommitment, - zs: Coset, - ys: CosetEvals, - proof: KZGProof) -> bool: - """ - Verify a KZG multi-evaluation proof for a set of `k` points. - - This is done by checking if the following equation holds: - Q(x) Z(x) = f(X) - I(X) - Where: - f(X) is the polynomial that we want to verify opens at `k` points to `k` values - Q(X) is the quotient polynomial computed by the prover - I(X) is the degree k-1 polynomial that evaluates to `ys` at all `zs`` points - Z(X) is the polynomial that evaluates to zero on all `k` points - - The verifier receives the commitments to Q(X) and f(X), so they check the equation - holds by using the following pairing equation: - e([Q(X)]_1, [Z(X)]_2) == e([f(X)]_1 - [I(X)]_1, [1]_2) - """ - - assert len(zs) == len(ys) - - # Compute [Z(X)]_2 - zero_poly = g2_lincomb(KZG_SETUP_G2_MONOMIAL[:len(zs) + 1], vanishing_polynomialcoeff(zs)) - # Compute [I(X)]_1 - interpolated_poly = g1_lincomb(KZG_SETUP_G1_MONOMIAL[:len(zs)], interpolate_polynomialcoeff(zs, ys)) - - return (bls.pairing_check([ - [bls.bytes48_to_G1(proof), bls.bytes96_to_G2(zero_poly)], - [ - bls.add(bls.bytes48_to_G1(commitment), bls.neg(bls.bytes48_to_G1(interpolated_poly))), - bls.neg(bls.bytes96_to_G2(KZG_SETUP_G2_MONOMIAL[0])), - ], - ])) -``` - #### `verify_cell_kzg_proof_batch_impl` ```python From 228752c0d9600cb1eef2a68e58cc13a16d097b59 Mon Sep 17 00:00:00 2001 From: Justin Traglia Date: Fri, 5 Jul 2024 15:31:07 -0500 Subject: [PATCH 043/130] In batch cell verification, check if there are zero cells --- specs/_features/eip7594/polynomial-commitments-sampling.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/specs/_features/eip7594/polynomial-commitments-sampling.md b/specs/_features/eip7594/polynomial-commitments-sampling.md index 10da2e58c3..b2c34a4842 100644 --- a/specs/_features/eip7594/polynomial-commitments-sampling.md +++ b/specs/_features/eip7594/polynomial-commitments-sampling.md @@ -511,6 +511,10 @@ def verify_cell_kzg_proof_batch_impl(row_commitments: Sequence[KZGCommitment], n = FIELD_ELEMENTS_PER_CELL num_rows = len(row_commitments) + # Given zero cells, the result is true + if num_cells == 0: + return True + # Step 1: Compute a challenge r and its powers r^0, ..., r^{num_cells-1} r = compute_verify_cell_kzg_proof_batch_challenge( row_commitments, @@ -567,7 +571,6 @@ def verify_cell_kzg_proof_batch_impl(row_commitments: Sequence[KZGCommitment], ])) ``` - ### Cell cosets #### `coset_shift_for_cell` From b9e7b031b5f2c18d76143007ea779a32b5505155 Mon Sep 17 00:00:00 2001 From: Justin Traglia Date: Mon, 8 Jul 2024 09:00:14 -0500 Subject: [PATCH 044/130] Update algorithm & remove check --- .../eip7594/polynomial-commitments-sampling.md | 6 +----- .../test_polynomial_commitments.py | 14 ++++++++++++++ 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/specs/_features/eip7594/polynomial-commitments-sampling.md b/specs/_features/eip7594/polynomial-commitments-sampling.md index b2c34a4842..999e5ef343 100644 --- a/specs/_features/eip7594/polynomial-commitments-sampling.md +++ b/specs/_features/eip7594/polynomial-commitments-sampling.md @@ -511,10 +511,6 @@ def verify_cell_kzg_proof_batch_impl(row_commitments: Sequence[KZGCommitment], n = FIELD_ELEMENTS_PER_CELL num_rows = len(row_commitments) - # Given zero cells, the result is true - if num_cells == 0: - return True - # Step 1: Compute a challenge r and its powers r^0, ..., r^{num_cells-1} r = compute_verify_cell_kzg_proof_batch_challenge( row_commitments, @@ -544,7 +540,7 @@ def verify_cell_kzg_proof_batch_impl(row_commitments: Sequence[KZGCommitment], # Step 4.2: Compute RLI = [sum_k r^k interpolation_poly_k(s)] # Note: an efficient implementation would use the IDFT based method explained in the blog post - sum_interp_polys_coeff = [0] + sum_interp_polys_coeff = [0] * n for k in range(num_cells): interp_poly_coeff = interpolate_polynomialcoeff(coset_for_cell(column_indices[k]), cosets_evals[k]) interp_poly_scaled_coeff = multiply_polynomialcoeff([r_powers[k]], interp_poly_coeff) diff --git a/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py b/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py index dd5aece3f9..dcb43f4fa0 100644 --- a/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py +++ b/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py @@ -122,6 +122,20 @@ def test_verify_cell_kzg_proof(spec): assert spec.verify_cell_kzg_proof(commitment, cell_index, cells[cell_index], proofs[cell_index]) +@with_eip7594_and_later +@spec_test +@single_phase +def test_verify_cell_kzg_proof_batch_zero_cells(spec): + # Verify with zero cells should return true + assert spec.verify_cell_kzg_proof_batch( + row_commitments_bytes=[], + row_indices=[], + column_indices=[], + cells=[], + proofs_bytes=[], + ) + + @with_eip7594_and_later @spec_test @single_phase From 104fbc737285ac27246035903145878587b142f1 Mon Sep 17 00:00:00 2001 From: Potuz Date: Mon, 24 Jun 2024 18:29:49 +0200 Subject: [PATCH 045/130] beacon chain changes --- specs/_features/epbs/beacon-chain.md | 627 +++++++++++++++++++++++++++ 1 file changed, 627 insertions(+) create mode 100644 specs/_features/epbs/beacon-chain.md diff --git a/specs/_features/epbs/beacon-chain.md b/specs/_features/epbs/beacon-chain.md new file mode 100644 index 0000000000..870744ac18 --- /dev/null +++ b/specs/_features/epbs/beacon-chain.md @@ -0,0 +1,627 @@ +# EIP-XXXX -- The Beacon Chain + +## Table of contents + + + + + +- [Introduction](#introduction) +- [Constants](#constants) + - [Payload status](#payload-status) +- [Preset](#preset) + - [Misc](#misc) + - [Domain types](#domain-types) + - [Max operations per block](#max-operations-per-block) +- [Containers](#containers) + - [New containers](#new-containers) + - [`PayloadAttestationData`](#payloadattestationdata) + - [`PayloadAttestation`](#payloadattestation) + - [`PayloadAttestationMessage`](#payloadattestationmessage) + - [`IndexedPayloadAttestation`](#indexedpayloadattestation) + - [`SignedExecutionPayloadHeader`](#signedexecutionpayloadheader) + - [`ExecutionPayloadEnvelope`](#executionpayloadenvelope) + - [`SignedExecutionPayloadEnvelope`](#signedexecutionpayloadenvelope) + - [Modified containers](#modified-containers) + - [`BeaconBlockBody`](#beaconblockbody) + - [`ExecutionPayloadHeader`](#executionpayloadheader) + - [`BeaconState`](#beaconstate) +- [Helper functions](#helper-functions) + - [Math](#math) + - [`bit_floor`](#bit_floor) + - [Predicates](#predicates) + - [`is_valid_indexed_payload_attestation`](#is_valid_indexed_payload_attestation) + - [`is_parent_block_full`](#is_parent_block_full) + - [Beacon State accessors](#beacon-state-accessors) + - [`get_ptc`](#get_ptc) + - [Modified `get_attesting_indices`](#modified-get_attesting_indices) + - [`get_payload_attesting_indices`](#get_payload_attesting_indices) + - [`get_indexed_payload_attestation`](#get_indexed_payload_attestation) +- [Beacon chain state transition function](#beacon-chain-state-transition-function) + - [Block processing](#block-processing) + - [Modified `process_withdrawals`](#modified-process_withdrawals) + - [New `verify_execution_payload_header_signature`](#new-verify_execution_payload_header_signature) + - [New `process_execution_payload_header`](#new-process_execution_payload_header) + - [Modified `process_operations`](#modified-process_operations) + - [Payload Attestations](#payload-attestations) + - [New `verify_execution_payload_envelope_signature`](#new-verify_execution_payload_envelope_signature) + - [Modified `process_execution_payload`](#modified-process_execution_payload) + + + + +## Introduction + +This is the beacon chain specification of the enshrined proposer builder separation feature. + +*Note:* This specification is built upon [Electra](../../electra/beacon-chain.md) and is under active development. + +This feature adds new staked consensus participants called *Builders* and new honest validators duties called *payload timeliness attestations*. The slot is divided in **four** intervals. Honest validators gather *signed bids* (a `SignedExecutionPayloadHeader`) from builders and submit their consensus blocks (a `SignedBeaconBlock`) including these bids at the beginning of the slot. At the start of the second interval, honest validators submit attestations just as they do previous to this feature). At the start of the third interval, aggregators aggregate these attestations and the builder broadcasts either a full payload or a message indicating that they are withholding the payload (a `SignedExecutionPayloadEnvelope`). At the start of the fourth interval, some validators selected to be members of the new **Payload Timeliness Committee** (PTC) attest to the presence and timeliness of the builder's payload. + +At any given slot, the status of the blockchain's head may be either +- A block from a previous slot (e.g. the current slot's proposer did not submit its block). +- An *empty* block from the current slot (e.g. the proposer submitted a timely block, but the builder did not reveal the payload on time). +- A full block for the current slot (both the proposer and the builder revealed on time). + +## Constants + +### Payload status + +| Name | Value | +| - | - | +| `PAYLOAD_ABSENT` | `uint8(0)` | +| `PAYLOAD_PRESENT` | `uint8(1)` | +| `PAYLOAD_WITHHELD` | `uint8(2)` | +| `PAYLOAD_INVALID_STATUS` | `uint8(3)` | + +## Preset + +### Misc + +| Name | Value | +| - | - | +| `PTC_SIZE` | `uint64(2**9)` (=512) # (New in EIP-XXXX) | + +### Domain types + +| Name | Value | +| - | - | +| `DOMAIN_BEACON_BUILDER` | `DomainType('0x1B000000')` # (New in EIP-XXXX)| +| `DOMAIN_PTC_ATTESTER` | `DomainType('0x0C000000')` # (New in EIP-XXXX)| + +### Max operations per block + +| Name | Value | +| - | - | +| `MAX_PAYLOAD_ATTESTATIONS` | `2**2` (= 4) # (New in EIP-XXXX) | + +## Containers + +### New containers + +#### `PayloadAttestationData` + +```python +class PayloadAttestationData(Container): + beacon_block_root: Root + slot: Slot + payload_status: uint8 +``` + +#### `PayloadAttestation` + +```python +class PayloadAttestation(Container): + aggregation_bits: BitVector[PTC_SIZE] + data: PayloadAttestationData + signature: BLSSignature +``` + +#### `PayloadAttestationMessage` + +```python +class PayloadAttestationMessage(Container): + validator_index: ValidatorIndex + data: PayloadAttestationData + signature: BLSSignature +``` + +#### `IndexedPayloadAttestation` + +```python +class IndexedPayloadAttestation(Container): + attesting_indices: List[ValidatorIndex, PTC_SIZE] + data: PayloadAttestationData + signature: BLSSignature +``` + +#### `SignedExecutionPayloadHeader` + +```python +class SignedExecutionPayloadHeader(Container): + message: ExecutionPayloadHeader + signature: BLSSignature +``` + +#### `ExecutionPayloadEnvelope` + +```python +class ExecutionPayloadEnvelope(Container): + payload: ExecutionPayload + builder_index: ValidatorIndex + beacon_block_root: Root + blob_kzg_commitments: List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK] + payload_withheld: bool + state_root: Root +``` + +#### `SignedExecutionPayloadEnvelope` + +```python +class SignedExecutionPayloadEnvelope(Container): + message: ExecutionPayloadEnvelope + signature: BLSSignature +``` + +### Modified containers + + +#### `BeaconBlockBody` +**Note:** The Beacon Block body is modified to contain a `Signed ExecutionPayloadHeader`. The containers `BeaconBlock` and `SignedBeaconBlock` are modified indirectly. + +```python +class BeaconBlockBody(Container): + randao_reveal: BLSSignature + eth1_data: Eth1Data # Eth1 data vote + graffiti: Bytes32 # Arbitrary data + # Operations + proposer_slashings: List[ProposerSlashing, MAX_PROPOSER_SLASHINGS] + attester_slashings: List[AttesterSlashing, MAX_ATTESTER_SLASHINGS] + attestations: List[Attestation, MAX_ATTESTATIONS] + deposits: List[Deposit, MAX_DEPOSITS] + voluntary_exits: List[SignedVoluntaryExit, MAX_VOLUNTARY_EXITS] + sync_aggregate: SyncAggregate + # Execution + # Removed execution_payload [Removed in EIP-XXXX] + # Removed blob_kzg_commitments [Removed in EIP-XXXX] + bls_to_execution_changes: List[SignedBLSToExecutionChange, MAX_BLS_TO_EXECUTION_CHANGES] + # PBS + signed_execution_payload_header: SignedExecutionPayloadHeader # [New in EIP-XXXX] + payload_attestations: List[PayloadAttestation, MAX_PAYLOAD_ATTESTATIONS] # [New in EIP-XXXX] +``` + +#### `ExecutionPayloadHeader` + +**Note:** The `ExecutionPayloadHeader` is modified to only contain the block hash of the committed `ExecutionPayload` in addition to the builder's payment information and KZG commitments root to verify the inclusion proofs. + +```python +class ExecutionPayloadHeader(Container): + parent_block_hash: Hash32 + parent_block_root: Root + block_hash: Hash32 + builder_index: ValidatorIndex + slot: Slot + value: Gwei + blob_kzg_commitments_root: Root +``` + +#### `BeaconState` +*Note*: The `BeaconState` is modified to track the last withdrawals honored in the CL. The `latest_execution_payload_header` is modified semantically to refer not to a past committed `ExecutionPayload` but instead it corresponds to the state's slot builder's bid. Another addition is to track the last committed block hash and the last slot that was full, that is in which there were both consensus and execution blocks included. + +```python +class BeaconState(Container): + # Versioning + genesis_time: uint64 + genesis_validators_root: Root + slot: Slot + fork: Fork + # History + latest_block_header: BeaconBlockHeader + block_roots: Vector[Root, SLOTS_PER_HISTORICAL_ROOT] + state_roots: Vector[Root, SLOTS_PER_HISTORICAL_ROOT] + historical_roots: List[Root, HISTORICAL_ROOTS_LIMIT] # Frozen in Capella, replaced by historical_summaries + # Eth1 + eth1_data: Eth1Data + eth1_data_votes: List[Eth1Data, EPOCHS_PER_ETH1_VOTING_PERIOD * SLOTS_PER_EPOCH] + eth1_deposit_index: uint64 + # Registry + validators: List[Validator, VALIDATOR_REGISTRY_LIMIT] + balances: List[Gwei, VALIDATOR_REGISTRY_LIMIT] + # Randomness + randao_mixes: Vector[Bytes32, EPOCHS_PER_HISTORICAL_VECTOR] + # Slashings + slashings: Vector[Gwei, EPOCHS_PER_SLASHINGS_VECTOR] # Per-epoch sums of slashed effective balances + # Participation + previous_epoch_participation: List[ParticipationFlags, VALIDATOR_REGISTRY_LIMIT] + current_epoch_participation: List[ParticipationFlags, VALIDATOR_REGISTRY_LIMIT] + # Finality + justification_bits: Bitvector[JUSTIFICATION_BITS_LENGTH] # Bit set for every recent justified epoch + previous_justified_checkpoint: Checkpoint + current_justified_checkpoint: Checkpoint + finalized_checkpoint: Checkpoint + # Inactivity + inactivity_scores: List[uint64, VALIDATOR_REGISTRY_LIMIT] + # Sync + current_sync_committee: SyncCommittee + next_sync_committee: SyncCommittee + # Execution + latest_execution_payload_header: ExecutionPayloadHeader + # Withdrawals + next_withdrawal_index: WithdrawalIndex + next_withdrawal_validator_index: ValidatorIndex + # Deep history valid from Capella onwards + historical_summaries: List[HistoricalSummary, HISTORICAL_ROOTS_LIMIT] + deposit_requests_start_index: uint64 + deposit_balance_to_consume: Gwei + exit_balance_to_consume: Gwei + earliest_exit_epoch: Epoch + consolidation_balance_to_consume: Gwei + earliest_consolidation_epoch: Epoch + pending_balance_deposits: List[PendingBalanceDeposit, PENDING_BALANCE_DEPOSITS_LIMIT] + pending_partial_withdrawals: List[PendingPartialWithdrawal, PENDING_PARTIAL_WITHDRAWALS_LIMIT] + pending_consolidations: List[PendingConsolidation, PENDING_CONSOLIDATIONS_LIMIT] + # PBS + latest_block_hash: Hash32 # [New in EIP-XXXX] + latest_full_slot: Slot # [New in EIP-XXXX] + latest_withdrawals_root: Root # [New in EIP-XXXX] +``` + +## Helper functions + +### Math + +#### `bit_floor` + +```python +def bit_floor(n: uint64) -> uint64: + """ + if ``n`` is not zero, returns the largest power of `2` that is not greater than `n`. + """ + if n == 0: + return 0 + return uint64(1) << (n.bit_length() - 1) +``` + +### Predicates + +#### `is_valid_indexed_payload_attestation` + +```python +def is_valid_indexed_payload_attestation(state: BeaconState, indexed_payload_attestation: IndexedPayloadAttestation) -> bool: + """ + Check if ``indexed_payload_attestation`` is not empty, has sorted and unique indices and has a valid aggregate signature. + """ + # Verify the data is valid + if indexed_payload_attestation.data.payload_status >= PAYLOAD_INVALID_STATUS: + return False + + # Verify indices are sorted and unique + indices = indexed_payload_attestation.attesting_indices + if len(indices) == 0 or not indices == sorted(set(indices)): + return False + # Verify aggregate signature + pubkeys = [state.validators[i].pubkey for i in indices] + domain = get_domain(state, DOMAIN_PTC_ATTESTER, None) + signing_root = compute_signing_root(indexed_payload_attestation.data, domain) + return bls.FastAggregateVerify(pubkeys, signing_root, indexed_payload_attestation.signature) +``` + +#### `is_parent_block_full` + +This function returns true if the last committed payload header was fulfilled with a payload, this can only happen when both beacon block and payload were present. This function must be called on a beacon state before processing the execution payload header in the block. + +```python +def is_parent_block_full(state: BeaconState) -> bool: + return state.latest_execution_payload_header.block_hash == state.latest_block_hash +``` + +### Beacon State accessors + +#### `get_ptc` + +```python +def get_ptc(state: BeaconState, slot: Slot) -> Vector[ValidatorIndex, PTC_SIZE]: + """ + Get the payload timeliness committee for the given ``slot`` + """ + epoch = compute_epoch_at_slot(slot) + committees_per_slot = bit_floor(min(get_committee_count_per_slot(state, epoch), PTC_SIZE)) + members_per_committee = PTC_SIZE // committees_per_slot + + validator_indices = [] + for idx in range(committees_per_slot): + beacon_committee = get_beacon_committee(state, slot, idx) + validator_indices += beacon_committee[:members_per_committee] + return validator_indices +``` + +#### Modified `get_attesting_indices` +`get_attesting_indices` is modified to ignore PTC votes + +```python +def get_attesting_indices(state: BeaconState, attestation: Attestation) -> Set[ValidatorIndex]: + """ + Return the set of attesting indices corresponding to ``aggregation_bits`` and ``committee_bits``. + """ + output: Set[ValidatorIndex] = set() + committee_indices = get_committee_indices(attestation.committee_bits) + committee_offset = 0 + for index in committee_indices: + committee = get_beacon_committee(state, attestation.data.slot, index) + committee_attesters = set( + index for i, index in enumerate(committee) if attestation.aggregation_bits[committee_offset + i]) + output = output.union(committee_attesters) + + committee_offset += len(committee) + + ptc = get_ptc(state, attestation.data.slot) + return [i for i in output if i not in ptc] +``` + +#### `get_payload_attesting_indices` + +```python +def get_payload_attesting_indices(state: BeaconState, slot: Slot, + payload_attestation: PayloadAttestation) -> Set[ValidatorIndex]: + """ + Return the set of attesting indices corresponding to ``payload_attestation``. + """ + ptc = get_ptc(state, slot) + return set(index for i, index in enumerate(ptc) if payload_attestation.aggregation_bits[i]) +``` + + +#### `get_indexed_payload_attestation` + +```python +def get_indexed_payload_attestation(state: BeaconState, slot: Slot, + payload_attestation: PayloadAttestation) -> IndexedPayloadAttestation: + """ + Return the indexed payload attestation corresponding to ``payload_attestation``. + """ + attesting_indices = get_payload_attesting_indices(state, slot, payload_attestation) + + return IndexedPayloadAttestation( + attesting_indices=sorted(attesting_indices), + data=payload_attestation.data, + signature=payload_attestation.signature, + ) +``` + +## Beacon chain state transition function + +*Note*: state transition is fundamentally modified in EIP-XXXX. The full state transition is broken in two parts, first importing a signed block and then importing an execution payload. + +The post-state corresponding to a pre-state `state` and a signed beacon block `signed_block` is defined as `state_transition(state, signed_block)`. State transitions that trigger an unhandled exception (e.g. a failed `assert` or an out-of-range list access) are considered invalid. State transitions that cause a `uint64` overflow or underflow are also considered invalid. + +The post-state corresponding to a pre-state `state` and a signed execution payload envelope `signed_envelope` is defined as `process_execution_payload(state, signed_envelope)`. State transitions that trigger an unhandled exception (e.g. a failed `assert` or an out-of-range list access) are considered invalid. State transitions that cause a `uint64` overflow or underflow are also considered invalid. + +### Block processing + +```python +def process_block(state: BeaconState, block: BeaconBlock) -> None: + process_block_header(state, block) + process_withdrawals(state) [Modified in EIP-XXXX] + process_execution_payload_header(state, block) # [Modified in EIP-XXXX, removed process_execution_payload] + process_randao(state, block.body) + process_eth1_data(state, block.body) + process_operations(state, block.body) # [Modified in EIP-XXXX] + process_sync_aggregate(state, block.body.sync_aggregate) +``` + +#### Modified `process_withdrawals` +**Note:** This is modified to take only the `state` as parameter. Withdrawals are deterministic given the beacon state, any execution payload that has the corresponding block as parent beacon block is required to honor these withdrawals in the execution layer. This function must be called before `process_execution_payload_header` as this latter function affects validator balances. + +```python +def process_withdrawals(state: BeaconState) -> None: + ## return early if the parent block was empty + if not is_parent_block_full(state): + return + + withdrawals, partial_withdrawals_count = get_expected_withdrawals(state) + state.latest_withdrawals_root = hash_tree_root(withdrawals) + for withdrawal in withdrawals: + decrease_balance(state, withdrawal.validator_index, withdrawal.amount) + + # Update pending partial withdrawals + state.pending_partial_withdrawals = state.pending_partial_withdrawals[partial_withdrawals_count:] + + # Update the next withdrawal index if this block contained withdrawals + if len(withdrawals) != 0: + latest_withdrawal = withdrawals[-1] + state.next_withdrawal_index = WithdrawalIndex(latest_withdrawal.index + 1) + + # Update the next validator index to start the next withdrawal sweep + if len(withdrawals) == MAX_WITHDRAWALS_PER_PAYLOAD: + # Next sweep starts after the latest withdrawal's validator index + next_validator_index = ValidatorIndex((withdrawals[-1].validator_index + 1) % len(state.validators)) + state.next_withdrawal_validator_index = next_validator_index + else: + # Advance sweep by the max length of the sweep if there was not a full set of withdrawals + next_index = state.next_withdrawal_validator_index + MAX_VALIDATORS_PER_WITHDRAWALS_SWEEP + next_validator_index = ValidatorIndex(next_index % len(state.validators)) + state.next_withdrawal_validator_index = next_validator_index +``` + +#### New `verify_execution_payload_header_signature` + +```python +def verify_execution_payload_header_signature(state: BeaconState, + signed_header: SignedExecutionPayloadHeader) -> bool: + # Check the signature + builder = state.validators[signed_header.message.builder_index] + signing_root = compute_signing_root(signed_header.message, get_domain(state, DOMAIN_BEACON_BUILDER)) + return bls.Verify(builder.pubkey, signing_root, signed_header.signature) +``` + +#### New `process_execution_payload_header` + +```python +def process_execution_payload_header(state: BeaconState, block: BeaconBlock) -> None: + # Verify the header signature + signed_header = block.body.signed_execution_payload_header + assert verify_execution_payload_header_signature(state, signed_header) + + # Check that the builder has funds to cover the bid + header = signed_header.message + builder_index = header.builder_index + amount = header.value + assert state.balances[builder_index] >= amount + + # Verify that the bid is for the current slot + assert header.slot == block.slot + # Verify that the bid is for the right parent block + assert header.parent_block_hash == state.latest_block_hash + assert header.parent_block_root == block.parent_root + + # Transfer the funds from the builder to the proposer + decrease_balance(state, builder_index, amount) + increase_balance(state, block.proposer_index, amount) + + # Cache the signed execution payload header + state.latest_execution_payload_header = header +``` + +#### Modified `process_operations` + +**Note:** `process_operations` is modified to process PTC attestations + +```python +def process_operations(state: BeaconState, body: BeaconBlockBody) -> None: + # Verify that outstanding deposits are processed up to the maximum number of deposits + assert len(body.deposits) == min(MAX_DEPOSITS, state.eth1_data.deposit_count - state.eth1_deposit_index) + + def for_ops(operations: Sequence[Any], fn: Callable[[BeaconState, Any], None]) -> None: + for operation in operations: + fn(state, operation) + + for_ops(body.proposer_slashings, process_proposer_slashing) + for_ops(body.attester_slashings, process_attester_slashing) + for_ops(body.attestations, process_attestation) + for_ops(body.deposits, process_deposit) + for_ops(body.voluntary_exits, process_voluntary_exit) + for_ops(body.bls_to_execution_changes, process_bls_to_execution_change) + # Removed `process_deposit_request` in EIP-XXXX + # Removed `process_withdrawal_request` in EIP-XXXX + # Removed `process_consolidation_request` in EIP-XXXX + for_ops(body.payload_attestations, process_payload_attestation) # [New in EIP-XXXX] +``` + +##### Payload Attestations + +```python +def remove_flag(flags: ParticipationFlags, flag_index: int) -> ParticipationFlags: + flag = ParticipationFlags(2**flag_index) + return flags & ~flag +``` + +```python +def process_payload_attestation(state: BeaconState, payload_attestation: PayloadAttestation) -> None: + ## Check that the attestation is for the parent beacon block + data = payload_attestation.data + assert data.beacon_block_root == state.latest_block_header.parent_root + ## Check that the attestation is for the previous slot + assert data.slot + 1 == state.slot + + #Verify signature + indexed_payload_attestation = get_indexed_payload_attestation(state, data.slot, payload_attestation) + assert is_valid_indexed_payload_attestation(state, indexed_payload_attestation) + + ptc = get_ptc(state, data.slot) + if state.slot % SLOTS_PER_EPOCH == 0: + epoch_participation = state.previous_epoch_participation + else: + epoch_participation = state.current_epoch_participation + + # Return early if the attestation is for the wrong payload status + payload_was_present = data.slot == state.latest_full_slot + voted_present = data.payload_status == PAYLOAD_PRESENT + proposer_reward_denominator = (WEIGHT_DENOMINATOR - PROPOSER_WEIGHT) * WEIGHT_DENOMINATOR // PROPOSER_WEIGHT + proposer_index = get_beacon_proposer_index(state) + if voted_present != payload_was_present: + # Unset the flags in case they were set by an equivocating ptc attestation + proposer_penalty_numerator = 0 + for index in indexed_payload_attestation.attesting_indices: + for flag_index, weight in enumerate(PARTICIPATION_FLAG_WEIGHTS): + if has_flag(epoch_participation[index], flag_index): + epoch_participation[index] = remove_flag(epoch_participation[index], flag_index) + proposer_penalty_numerator += get_base_reward(state, index) * weight + # Penalize the proposer + proposer_penalty = Gwei(2*proposer_penalty_numerator // proposer_reward_denominator) + decrease_balance(state, proposer_index, proposer_penalty) + return + + # Reward the proposer and set all the participation flags in case of correct attestations + proposer_reward_numerator = 0 + for index in indexed_payload_attestation.attesting_indices: + for flag_index, weight in enumerate(PARTICIPATION_FLAG_WEIGHTS): + if not has_flag(epoch_participation[index], flag_index): + epoch_participation[index] = add_flag(epoch_participation[index], flag_index) + proposer_reward_numerator += get_base_reward(state, index) * weight + + # Reward proposer + proposer_reward = Gwei(proposer_reward_numerator // proposer_reward_denominator) + increase_balance(state, proposer_index, proposer_reward) +``` + +#### New `verify_execution_payload_envelope_signature` + +```python +def verify_execution_payload_envelope_signature(state: BeaconState, signed_envelope: SignedExecutionPayloadEnvelope) -> bool: + builder = state.validators[signed_envelope.message.builder_index] + signing_root = compute_signing_root(signed_envelope.message, get_domain(state, DOMAIN_BEACON_BUILDER)) + return bls.Verify(builder.pubkey, signing_root, signed_envelope.signature) +``` + +#### Modified `process_execution_payload` +*Note*: `process_execution_payload` is now an independent check in state transition. It is called when importing a signed execution payload proposed by the builder of the current slot. + +```python +def process_execution_payload(state: BeaconState, signed_envelope: SignedExecutionPayloadEnvelope, execution_engine: ExecutionEngine, verify = True) -> None: + # Verify signature + if verify: + assert verify_execution_payload_envelope_signature(state, signed_envelope) + envelope = signed_envelope.message + payload = envelope.payload + # Verify consistency with the beacon block + assert envelope.beacon_block_root == hash_tree_root(state.latest_block_header) + + # Verify the withdrawals root + assert hash_tree_root(payload.withdrawals) == state.last_withdrawals_root + + # Verify consistency with the committed header + committed_header = state.execution_payload_header + assert envelope.builder_index == committed_header.builder_index + assert committed_header.blob_kzg_commitments_root == hash_tree_root(envelope.blob_kzg_commitments) + + if not envelope.payload_withheld: + assert committed_header.block_hash == payload.block_hash + # Verify consistency of the parent hash with respect to the previous execution payload + assert payload.parent_hash == state.latest_block_hash + # Verify prev_randao + assert payload.prev_randao == get_randao_mix(state, get_current_epoch(state)) + # Verify timestamp + assert payload.timestamp == compute_timestamp_at_slot(state, state.slot) + # Verify the execution payload is valid + versioned_hashes = [kzg_commitment_to_versioned_hash(commitment) for commitment in envelope.blob_kzg_commitments] + assert execution_engine.verify_and_notify_new_payload( + NewPayloadRequest( + execution_payload=payload, + versioned_hashes=versioned_hashes, + parent_beacon_block_root=state.latest_block_header.parent_root, + ) + ) + + # Process Electra operations + for_ops(payload.deposit_requests, process_deposit_request) + for_ops(payload.withdrawal_requests, process_withdrawal_request) + for_ops(payload, process_consolidation_request) + + # Cache the execution payload header and proposer + state.latest_block_hash = payload.block_hash + state.latest_full_slot = state.slot + + # Verify the state root + if verify: + assert envelope.state_root == hash_tree_root(state) +``` From 673c75cab3157380675e3f2746a5cb7c83ec5062 Mon Sep 17 00:00:00 2001 From: Potuz Date: Tue, 25 Jun 2024 11:42:30 +0200 Subject: [PATCH 046/130] Forkchoice changes --- specs/_features/epbs/fork-choice.md | 500 ++++++++++++++++++++++++++++ 1 file changed, 500 insertions(+) create mode 100644 specs/_features/epbs/fork-choice.md diff --git a/specs/_features/epbs/fork-choice.md b/specs/_features/epbs/fork-choice.md new file mode 100644 index 0000000000..9d9786b701 --- /dev/null +++ b/specs/_features/epbs/fork-choice.md @@ -0,0 +1,500 @@ +# ePBS -- Fork Choice + +## Table of contents + + + + +- [Introduction](#introduction) +- [Constants](#constants) +- [Containers](#containers) + - [New `ChildNode`](#new-childnode) +- [Helpers](#helpers) + - [Modified `LatestMessage`](#modified-latestmessage) + - [Modified `update_latest_messages`](#modified-update_latest_messages) + - [Modified `Store`](#modified-store) + - [`notify_ptc_messages`](#notify_ptc_messages) + - [`is_payload_present`](#is_payload_present) + - [`is_parent_node_full`](#is_parent_node_full) + - [Modified `get_ancestor`](#modified-get_ancestor) + - [Modified `get_checkpoint_block`](#modified-get_checkpoint_block) + - [`is_supporting_vote`](#is_supporting_vote) + - [New `compute_proposer_boost`](#new-compute_proposer_boost) + - [New `compute_withhold_boost`](#new-compute_withhold_boost) + - [New `compute_reveal_boost`](#new-compute_reveal_boost) + - [Modified `get_weight`](#modified-get_weight) + - [Modified `get_head`](#modified-get_head) +- [Updated fork-choice handlers](#updated-fork-choice-handlers) + - [Modified `on_block`](#modified-on_block) +- [New fork-choice handlers](#new-fork-choice-handlers) + - [New `on_execution_payload`](#new-on_execution_payload) + - [`seconds_into_slot`](#seconds_into_slot) + - [Modified `on_tick_per_slot`](#modified-on_tick_per_slot) + - [`on_payload_attestation_message`](#on_payload_attestation_message) + + + + +## Introduction + +This is the modification of the fork choice accompanying the ePBS upgrade. + +## Constants + +| Name | Value | +| -------------------- | ----------- | +| `PAYLOAD_TIMELY_THRESHOLD` | `PTC_SIZE/2` (=`uint64(256)`) | +| `PROPOSER_SCORE_BOOST` | `20` # [modified in EIP-XXXX] | +| `PAYLOAD_WITHHOLD_BOOST` | `40` | +| `PAYLOAD_REVEAL_BOOST` | `40` | + +## Containers + +### New `ChildNode` +Auxiliary class to consider `(block, slot, bool)` LMD voting + +```python +class ChildNode(Container): + root: Root + slot: Slot + is_payload_present: bool +``` + +## Helpers + +### Modified `LatestMessage` +**Note:** The class is modified to keep track of the slot instead of the epoch. + +```python +@dataclass(eq=True, frozen=True) +class LatestMessage(object): + slot: Slot + root: Root +``` + +### Modified `update_latest_messages` +**Note:** the function `update_latest_messages` is updated to use the attestation slot instead of target. Notice that this function is only called on validated attestations and validators cannot attest twice in the same epoch without equivocating. Notice also that target epoch number and slot number are validated on `validate_on_attestation`. + +```python +def update_latest_messages(store: Store, attesting_indices: Sequence[ValidatorIndex], attestation: Attestation) -> None: + slot = attestation.data.slot + beacon_block_root = attestation.data.beacon_block_root + non_equivocating_attesting_indices = [i for i in attesting_indices if i not in store.equivocating_indices] + for i in non_equivocating_attesting_indices: + if i not in store.latest_messages or slot > store.latest_messages[i].slot: + store.latest_messages[i] = LatestMessage(slot=slot, root=beacon_block_root) +``` + +### Modified `Store` +**Note:** `Store` is modified to track the intermediate states of "empty" consensus blocks, that is, those consensus blocks for which the corresponding execution payload has not been revealed or has not been included on chain. + +```python +@dataclass +class Store(object): + time: uint64 + genesis_time: uint64 + justified_checkpoint: Checkpoint + finalized_checkpoint: Checkpoint + unrealized_justified_checkpoint: Checkpoint + unrealized_finalized_checkpoint: Checkpoint + proposer_boost_root: Root + equivocating_indices: Set[ValidatorIndex] + blocks: Dict[Root, BeaconBlock] = field(default_factory=dict) + block_states: Dict[Root, BeaconState] = field(default_factory=dict) + checkpoint_states: Dict[Checkpoint, BeaconState] = field(default_factory=dict) + latest_messages: Dict[ValidatorIndex, LatestMessage] = field(default_factory=dict) + unrealized_justifications: Dict[Root, Checkpoint] = field(default_factory=dict) + execution_payload_states: Dict[Root, BeaconState] = field(default_factory=dict) # [New in EIP-XXXX] + ptc_vote: Dict[Root, Vector[uint8, PTC_SIZE]] = field(default_factory=dict) # [New in EIP-XXXX] + payload_withhold_boost_root: Root # [New in EIP-XXXX] + payload_withhold_boost_full: bool # [New in EIP-XXXX] + payload_reveal_boost_root: Root # [New in EIP-XXXX] +``` + +### `notify_ptc_messages` + +```python +def notify_ptc_messages(store: Store, state: BeaconState, payload_attestations: Sequence[PayloadAttestation]) -> None: + """ + Extracts a list of ``PayloadAttestationMessage`` from ``payload_attestations`` and updates the store with them + These Payload attestations are assumed to be in the beacon block hence signature verification is not needed + """ + if state.slot == 0: + return + for payload_attestation in payload_attestations: + indexed_payload_attestation = get_indexed_payload_attestation(state, state.slot - 1, payload_attestation) + for idx in indexed_payload_attestation.attesting_indices: + store.on_payload_attestation_message(PayloadAttestationMessage(validator_index=idx, + data=payload_attestation.data, signature= BLSSignature(), is_from_block=true)) +``` + +### `is_payload_present` + +```python +def is_payload_present(store: Store, beacon_block_root: Root) -> bool: + """ + Return whether the execution payload for the beacon block with root ``beacon_block_root`` was voted as present + by the PTC + """ + # The beacon block root must be known + assert beacon_block_root in store.ptc_vote + return store.ptc_vote[beacon_block_root].count(PAYLOAD_PRESENT) > PAYLOAD_TIMELY_THRESHOLD +``` + +### `is_parent_node_full` + +```python +def is_parent_node_full(store: Store, block: BeaconBlock) -> bool: + parent = store.blocks[block.parent_root] + return block.body.signed_execution_payload_header.message.parent_block_hash == parent.body.signed_execution_payload_header.message.block_hash +``` + +### Modified `get_ancestor` +**Note:** `get_ancestor` is modified to return whether the chain is based on an *empty* or *full* block. + +```python +def get_ancestor(store: Store, root: Root, slot: Slot) -> ChildNode: + """ + Returns the beacon block root, the slot and the payload status of the ancestor of the beacon block + with ``root`` at ``slot``. If the beacon block with ``root`` is already at ``slot`` or we are + requesting an ancestor "in the future" it returns its PTC status instead of the actual payload content. + """ + block = store.blocks[root] + if block.slot <= slot: + return ChildNode(root=root, slot=slot, is_payload_present=is_payload_present(store, root)) + + parent = store.blocks[block.parent_root] + if parent.slot > slot: + return get_ancestor(store, block.parent_root, slot) + return ChildNode(root=block.parent_root, slot=parent.slot, is_payload_present=is_parent_node_full(block)) +``` + +### Modified `get_checkpoint_block` +**Note:** `get_checkpoint_block` is modified to use the new `get_ancestor` + +```python +def get_checkpoint_block(store: Store, root: Root, epoch: Epoch) -> Root: + """ + Compute the checkpoint block for epoch ``epoch`` in the chain of block ``root`` + """ + epoch_first_slot = compute_start_slot_at_epoch(epoch) + return get_ancestor(store, root, epoch_first_slot).root +``` + + +### `is_supporting_vote` + +```python +def is_supporting_vote(store: Store, node: ChildNode, message: LatestMessage) -> bool: + """ + Returns whether a vote for ``message.root`` supports the chain containing the beacon block ``node.root`` with the + payload contents indicated by ``node.is_payload_present`` as head during slot ``node.slot``. + """ + if node.root == message.root: + # an attestation for a given root always counts for that root regardless if full or empty + # as long as the attestation happened after the requested slot. + return node.slot <= message.slot + message_block = store.blocks[message.root] + if node.slot >= message_block.slot: + return False + ancestor = get_ancestor(store, message.root, node.slot) + return (node.root == ancestor.root) and (node.is_payload_present == ancestor.is_payload_present) +``` + +### New `compute_proposer_boost` +This is a helper to compute the proposer boost. It applies the proposer boost to any ancestor of the proposer boost root taking into account the payload presence. There is one exception: if the requested node has the same root and slot as the block with the proposer boost root, then the proposer boost is applied to both empty and full versions of the node. +```python +def compute_proposer_boost(store: Store, state: State, node: ChildNode) -> Gwei: + if store.proposer_boost_root == Root(): + return Gwei(0) + ancestor = get_ancestor(store, store.proposer_boost_root, node.slot) + if ancestor.root != node.root: + return Gwei(0) + proposer_boost_slot = store.blocks[store.proposer_boost_root].slot + # Proposer boost is not applied after skipped slots + if node.slot > proposer_boost_slot: + return Gwei(0) + if (node.slot < proposer_boost_slot) and (ancestor.is_payload_present != node.is_payload_present): + return Gwei(0) + committee_weight = get_total_active_balance(state) // SLOTS_PER_EPOCH + return (committee_weight * PROPOSER_SCORE_BOOST) // 100 +``` + +### New `compute_withhold_boost` +This is a similar helper that applies for the withhold boost. In this case this always takes into account the reveal status. + +```python +def compute_withhold_boost(store: Store, state: State, node: ChildNode) -> Gwei: + if store.payload_withhold_boost_root == Root(): + return Gwei(0) + ancestor = get_ancestor(store, store.payload_withold_boost_root, node.slot) + if ancestor.root != node.root: + return Gwei(0) + if node.slot >= store.blocks[store.payload_withhold_boost_root].slot: + ancestor.is_payload_present = store.payload_withhold_boost_full + if ancestor.is_payload_present != node.is_payload_present: + return Gwei(0) + + committee_weight = get_total_active_balance(state) // SLOTS_PER_EPOCH + return (committee_weight * PAYLOAD_WITHHOLD_BOOST) // 100 +``` + +### New `compute_reveal_boost` +This is a similar helper to the last two, the only difference is that the reveal boost is only applied to the full version of the node when querying for the same slot as the revealed payload. + +```python +def compute_reveal_boost(store: Store, state: State, node: ChildNode) -> Gwei: + if store.payload_reveal_boost_root == Root(): + return Gwei(0) + ancestor = get_ancestor(store, store.payload_reveal_boost_root, node.slot) + if ancestor.root != node.root: + return Gwei(0) + if node.slot >= store.blocks[store.payload_reveal_boost_root].slot: + ancestor.is_payload_present = True + if is_ancestor_full != node.is_payload_present: + return Gwei(0) + committee_weight = get_total_active_balance(state) // SLOTS_PER_EPOCH + return (committee_weight * PAYLOAD_REVEAL_BOOST) // 100 +``` + +### Modified `get_weight` + +**Note:** `get_weight` is modified to only count votes for descending chains that support the status of a triple `Root, Slot, bool`, where the `bool` indicates if the block was full or not. `Slot` is needed for a correct implementation of `(Block, Slot)` voting. + +```python +def get_weight(store: Store, node: ChildNode) -> Gwei: + state = store.checkpoint_states[store.justified_checkpoint] + unslashed_and_active_indices = [ + i for i in get_active_validator_indices(state, get_current_epoch(state)) + if not is_slashed_attester(state.validators[i]) + ] + attestation_score = Gwei(sum( + state.validators[i].effective_balance for i in unslashed_and_active_indices + if (i in store.latest_messages + and i not in store.equivocating_indices + and is_supporting_vote(store, node, store.latest_messages[i])) + )) + + # Compute boosts + proposer_score = compute_boost(store, state, node) + builder_reveal_score = compute_reveal_boost(store, state, node) + builder_withhold_score = compute_withhold_boost(store, state, node) + + return attestation_score + proposer_score + builder_reveal_score + builder_withhold_score +``` + +### Modified `get_head` + +**Note:** `get_head` is a modified to use the new `get_weight` function. It returns the `ChildNode` object corresponidng to the head block. + +```python +def get_head(store: Store) -> ChildNode: + # Get filtered block tree that only includes viable branches + blocks = get_filtered_block_tree(store) + # Execute the LMD-GHOST fork choice + justified_root = store.justified_checkpoint.root + justified_block = store.blocks[justified_root] + justified_slot = justified_block.slot + justified_full = is_payload_present(store, justified_root) + best_child = ChildNode(root=head_root, slot=head_slot, is_payload_present=head_full) + while True: + children = [ + ChildNode(root=root, slot=block.slot, is_payload_present=present) for (root, block) in blocks.items() + if block.parent_root == best_child.root and + is_parent_node_full(store, block) == best_child.is_payload_present if root != store.justified_root + for present in (True, False) if root in store.execution_payload_states or present == False + ] + if len(children) == 0: + return best_child + # if we have children we consider the current head advanced as a possible head + children += [ChildNode(root=best_child.root, slot=best_child.slot + 1, best_child.is_payload_present)] + # Sort by latest attesting balance with ties broken lexicographically + # Ties broken by favoring full blocks according to the PTC vote + # Ties are then broken by favoring full blocks + # Ties broken then by favoring higher slot numbers + # Ties then broken by favoring block with lexicographically higher root + new_best_child = max(children, key=lambda child: (get_weight(store, child), is_payload_present(store, child.root), child.is_payload_present, child.slot, child.root)) + if new_best_child.root == best_child.root: + return new_best_child + best_child = new_best_child +``` + +## Updated fork-choice handlers + +### Modified `on_block` + +*Note*: The handler `on_block` is modified to consider the pre `state` of the given consensus beacon block depending not only on the parent block root, but also on the parent blockhash. In addition we delay the checking of blob data availability until the processing of the execution payload. + +```python +def on_block(store: Store, signed_block: SignedBeaconBlock) -> None: + """ + Run ``on_block`` upon receiving a new block. + """ + block = signed_block.message + # Parent block must be known + assert block.parent_root in store.block_states + + # Check if this blocks builds on empty or full parent block + parent_block = store.blocks[block.parent_root] + header = block.body.signed_execution_payload_header.message + parent_header = parent_block.body.signed_execution_payload_header.message + # Make a copy of the state to avoid mutability issues + if is_parent_node_full(store, block): + assert block.parent_root in store.execution_payload_states + state = copy(store.execution_payload_states[block.parent_root]) + else: + assert header.parent_block_hash == parent_header.parent_block_hash + state = copy(store.block_states[block.parent_root]) + + # Blocks cannot be in the future. If they are, their consideration must be delayed until they are in the past. + current_slot = get_current_slot(store) + assert current_slot >= block.slot + + # Check that block is later than the finalized epoch slot (optimization to reduce calls to get_ancestor) + finalized_slot = compute_start_slot_at_epoch(store.finalized_checkpoint.epoch) + assert block.slot > finalized_slot + # Check block is a descendant of the finalized block at the checkpoint finalized slot + finalized_checkpoint_block = get_checkpoint_block( + store, + block.parent_root, + store.finalized_checkpoint.epoch, + ) + assert store.finalized_checkpoint.root == finalized_checkpoint_block + + # Check the block is valid and compute the post-state + block_root = hash_tree_root(block) + state_transition(state, signed_block, True) + + # Add new block to the store + store.blocks[block_root] = block + # Add new state for this block to the store + store.block_states[block_root] = state + # Add a new PTC voting for this block to the store + store.ptc_vote[block_root] = [PAYLOAD_ABSENT]*PTC_SIZE + + # Notify the store about the payload_attestations in the block + store.notify_ptc_messages(state, block.body.payload_attestations) + # Add proposer score boost if the block is timely + time_into_slot = (store.time - store.genesis_time) % SECONDS_PER_SLOT + is_before_attesting_interval = time_into_slot < SECONDS_PER_SLOT // INTERVALS_PER_SLOT + if get_current_slot(store) == block.slot and is_before_attesting_interval: + store.proposer_boost_root = hash_tree_root(block) + + # Update checkpoints in store if necessary + update_checkpoints(store, state.current_justified_checkpoint, state.finalized_checkpoint) + + # Eagerly compute unrealized justification and finality. + compute_pulled_up_tip(store, block_root) +``` + +## New fork-choice handlers + +### New `on_execution_payload` + +The handler `on_execution_payload` is called when the node receives a `SignedExecutionPayloadEnvelope` to sync. + +```python +def on_execution_payload(store: Store, signed_envelope: SignedExecutionPayloadEnvelope) -> None: + """ + Run ``on_execution_payload`` upon receiving a new execution payload. + """ + envelope = signed_envelope.message + # The corresponding beacon block root needs to be known + assert envelope.beacon_block_root in store.block_states + + # Check if blob data is available + # If not, this payload MAY be queued and subsequently considered when blob data becomes available + assert is_data_available(envelope.beacon_block_root, envelope.blob_kzg_commitments) + + # Make a copy of the state to avoid mutability issues + state = copy(store.block_states[envelope.beacon_block_root]) + + # Process the execution payload + process_execution_payload(state, signed_envelope, EXECUTION_ENGINE) + + # Add new state for this payload to the store + store.execution_payload_states[envelope.beacon_block_root] = state +``` + +### `seconds_into_slot` + +```python +def seconds_into_slot(store: Store) -> uint64: + return (store.time - store.genesis_time) % SECONDS_PER_SLOT +``` + +### Modified `on_tick_per_slot` + +Modified to reset the payload boost roots + +```python +def on_tick_per_slot(store: Store, time: uint64) -> None: + previous_slot = get_current_slot(store) + + # Update store time + store.time = time + + current_slot = get_current_slot(store) + + # If this is a new slot, reset store.proposer_boost_root + if current_slot > previous_slot: + store.proposer_boost_root = Root() + else: + # Reset the payload boost if this is the attestation time + if seconds_into_slot(store) >= SECONDS_PER_SLOT // INTERVALS_PER_SLOT: + store.payload_withhold_boost_root = Root() + store.payload_withhold_boost_full = False + store.payload_reveal_boost_root = Root() + + # If a new epoch, pull-up justification and finalization from previous epoch + if current_slot > previous_slot and compute_slots_since_epoch_start(current_slot) == 0: + update_checkpoints(store, store.unrealized_justified_checkpoint, store.unrealized_finalized_checkpoint) +``` + +### `on_payload_attestation_message` + +```python +def on_payload_attestation_message(store: Store, + ptc_message: PayloadAttestationMessage, is_from_block: bool=False) -> None: + """ + Run ``on_payload_attestation_message`` upon receiving a new ``ptc_message`` directly on the wire. + """ + # The beacon block root must be known + data = ptc_message.data + # PTC attestation must be for a known block. If block is unknown, delay consideration until the block is found + state = store.block_states[data.beacon_block_root] + ptc = get_ptc(state, data.slot) + # PTC votes can only change the vote for their assigned beacon block, return early otherwise + if data.slot != state.slot: + return + # Check that the attester is from the PTC + assert ptc_message.validator_index in ptc + + # Verify the signature and check that its for the current slot if it is coming from the wire + if not is_from_block: + # Check that the attestation is for the current slot + assert data.slot == get_current_slot(store) + # Verify the signature + assert is_valid_indexed_payload_attestation(state, + IndexedPayloadAttestation(attesting_indices = [ptc_message.validator_index], data = data, + signature = ptc_message.signature)) + # Update the ptc vote for the block + ptc_index = ptc.index(ptc_message.validator_index) + ptc_vote = store.ptc_vote[data.beacon_block_root] + ptc_vote[ptc_index] = data.payload_status + + # Only update payload boosts with attestations from a block if the block is for the current slot and it's early + if is_from_block && data.slot + 1 != get_current_slot(store): + return + time_into_slot = (store.time - store.genesis_time) % SECONDS_PER_SLOT + if is_from_block and time_into_slot >= SECONDS_PER_SLOT // INTERVALS_PER_SLOT: + return + + # Update the payload boosts if threshold has been achieved + if ptc_vote.count(PAYLOAD_PRESENT) > PAYLOAD_TIMELY_THRESHOLD: + store.payload_reveal_boost_root = data.beacon_block_root + if ptc_vote.count(PAYLOAD_WITHHELD) > PAYLOAD_TIMELY_THRESHOLD: + block = store.blocks[data.beacon_block_root] + store.payload_withhold_boost_root = block.parent_root + store.payload_withhold_boost_full = is_parent_node_full(block) +``` From 971c57c8ea6a1aec5d76b7304afa0d27ab823f38 Mon Sep 17 00:00:00 2001 From: Potuz Date: Tue, 25 Jun 2024 12:13:59 +0200 Subject: [PATCH 047/130] Honest validator guide --- specs/_features/epbs/validator.md | 123 ++++++++++++++++++++++++++++++ 1 file changed, 123 insertions(+) create mode 100644 specs/_features/epbs/validator.md diff --git a/specs/_features/epbs/validator.md b/specs/_features/epbs/validator.md new file mode 100644 index 0000000000..20ab108a62 --- /dev/null +++ b/specs/_features/epbs/validator.md @@ -0,0 +1,123 @@ +# ePBS -- Honest Validator + +This document represents the changes and additions to the Honest validator guide included in the ePBS fork. + + + +**Table of Contents** + +- [Validator assignment](#validator-assignment) + - [Lookahead](#lookahead) +- [Beacon chain responsibilities](#beacon-chain-responsibilities) + - [Block proposal](#block-proposal) + - [Constructing the new `signed_execution_payload_header_envelope` field in `BeaconBlockBody`](#constructing-the-new-signed_execution_payload_header_envelope-field-in--beaconblockbody) + - [Constructing the new `payload_attestations` field in `BeaconBlockBody`](#constructing-the-new-payload_attestations-field-in--beaconblockbody) + - [Blob sidecars](#blob-sidecars) + - [Payload timeliness attestation](#payload-timeliness-attestation) + - [Constructing a payload attestation](#constructing-a-payload-attestation) + + + +## Validator assignment + +A validator may be a member of the new Payload Timeliness Committee (PTC) for a given slot. To check for PTC assignments the validator uses the helper `get_ptc_assignment(state, epoch, validator_index)` where `epoch <= next_epoch`. + +PTC committee selection is only stable within the context of the current and next epoch. + +```python +def get_ptc_assignment(state: BeaconState, + epoch: Epoch, + validator_index: ValidatorIndex + ) -> Optional[Slot]: + """ + Returns the slot during the requested epoch in which the validator with index `validator_index` + is a member of the PTC. Returns None if no assignment is found. + """ + next_epoch = Epoch(get_current_epoch(state) + 1) + assert epoch <= next_epoch + + start_slot = compute_start_slot_at_epoch(epoch) + for slot in range(start_slot, start_slot + SLOTS_PER_EPOCH): + if validator_index in get_ptc(state, Slot(slot)): + return Slot(slot) + return None +``` + +### Lookahead + +[New in EIP-XXXX] + +`get_ptc_assignment` should be called at the start of each epoch to get the assignment for the next epoch (`current_epoch + 1`). A validator should plan for future assignments by noting their assigned PTC slot. + +## Beacon chain responsibilities + +All validator responsibilities remain unchanged other than the following: + +- Proposers are no longer required to broadcast `BlobSidecar` objects, as this becomes a builder's duty. +- Some validators are selected per slot to become PTC members, these validators must broadcat `PayloadAttestationMessage` objects during the assigned slot before the deadline of `3 * SECONDS_PER_SLOT // INTERVALS_PER_SLOT` seconds into the slot. + + +### Block proposal + +Validators are still expected to propose `SignedBeaconBlock` at the beginning of any slot during which `is_proposer(state, validator_index)` returns `true`. The mechanism to prepare this beacon block and related sidecars differs from previous forks as follows + +#### Constructing the new `signed_execution_payload_header_envelope` field in `BeaconBlockBody` + +To obtain `signed_execution_payload_header`, a block proposer building a block on top of a `state` must take the following actions: +* Listen to the `execution_payload_header` gossip global topic and save an accepted `signed_execution_payload_header` from a builder. Proposer MAY obtain these signed messages by other off-protocol means. +* The `signed_execution_payload_header` must satisfy the verification conditions found in `process_execution_payload_header`, that is + - The header signature must be valid + - The builder balance can cover the header value + - The header slot is for the proposal block slot + - The header parent block hash equals the state's `latest_block_hash`. + - The header parent block root equals the current block's `parent_root`. +* Select one bid and set `body.signed_execution_payload_header = signed_execution_payload_header` + +#### Constructing the new `payload_attestations` field in `BeaconBlockBody` + +Up to `MAX_PAYLOAD_ATTESTATIONS`, aggregate payload attestations can be included in the block. The validator will have to +* Listen to the `payload_attestation_message` gossip global topic +* The payload attestations added must satisfy the verification conditions found in payload attestation gossip validation and payload attestation processing. This means + - The `data.beacon_block_root` corresponds to `block.parent_root`. + - The slot of the parent block is exactly one slot before the proposing slot. + - The signature of the payload attestation data message verifies correctly. +* The proposer needs to aggregate all payload attestations with the same data into a given `PayloadAttestation` object. For this it needs to fill the `aggregation_bits` field by using the relative position of the validator indices with respect to the PTC that is obtained from `get_ptc(state, block_slot - 1)`. +* The proposer should only include payload attestations that are consistent with the current block they are proposing. That is, if the previous block had a payload, they should only include attestations with `payload_status = PAYLOAD_PRESENT`. Proposers are penalized for attestations that are not-consistent with their view. + +#### Blob sidecars +The blob sidecars are no longer broadcast by the validator, and thus their construction is not necessary. This deprecates the corresponding sections from the honest validator guide in the Electra fork, moving them, albeit with some modifications, to the [honest Builder guide](./builder.md) + +### Payload timeliness attestation + +Some validators are selected to submit payload timeliness attestations. Validators should call `get_ptc_assignment` at the beginning of an epoch to be prepared to submit their PTC attestations during the next epoch. + +A validator should create and broadcast the `payload_attestation_message` to the global execution attestation subnet not after `SECONDS_PER_SLOT * 3 / INTERVALS_PER_SLOT` seconds since the start of `slot` + +#### Constructing a payload attestation + +If a validator is in the payload attestation committee for the current slot (as obtained from `get_ptc_assignment` above) then the validator should prepare a `PayloadAttestationMessage` for the current slot, +according to the logic in `get_payload_attestation_message` below and broadcast it not after `SECONDS_PER_SLOT * 3 / INTERVALS_PER_SLOT` seconds since the start of the slot, to the global `payload_attestation_message` pubsub topic. + +The validator creates `payload_attestation_message` as follows: +* If the validator has not seen any beacon block for the assigned slot, do not submit a payload attestation. It will be ignored anyway. +* Set `data.beacon_block_root` be the HTR of the beacon block seen for the assigned slot +* Set `data.slot` to be the assigned slot. +* Set `data.payload_status` as follows + - If a `SignedExecutionPayloadEnvelope` has been seen referencing the block `data.beacon_block_root` and the envelope has `payload_withheld = False`, set to `PAYLOAD_PRESENT`. + - If a `SignedExecutionPayloadEnvelope` has been seen referencing the block `data.beacon_block_root` and the envelope has `payload_withheld = True`, set to `PAYLOAD_WITHHELD`. + - If no `SignedExecutionPayloadEnvelope` has been seen referencing the block `data.beacon_block_root` set to `PAYLOAD_ABSENT`. +* Set `payload_attestation_message.validator_index = validator_index` where `validator_index` is the validator chosen to submit. The private key mapping to `state.validators[validator_index].pubkey` is used to sign the payload timeliness attestation. +* Sign the `payload_attestation_message.data` using the helper `get_payload_attestation_message_signature`. + +Notice that the attester only signs the `PayloadAttestationData` and not the `validator_index` field in the message. Proposers need to aggregate these attestations as described above. + +```python +def get_payload_attestation_message_signature(state: BeaconState, attestation: PayloadAttestationMessage, privkey: int) -> BLSSignature: + domain = get_domain(state, DOMAIN_PTC_ATTESTER, compute_epoch_at_slot(attestation.data.slot)) + signing_root = compute_signing_root(attestation.data, domain) + return bls.Sign(privkey, signing_root) +``` + +**Remark** Validators do not need to check the full validity of the `ExecutionPayload` contained in within the envelope, but the checks in the [P2P guide](./p2p-interface.md) should pass for the `SignedExecutionPayloadEnvelope`. + + From 614e14ed8a89742c878052fd767e2b56b1b0cc8d Mon Sep 17 00:00:00 2001 From: Potuz Date: Tue, 25 Jun 2024 13:46:43 +0200 Subject: [PATCH 048/130] Honest Builder Guide --- specs/_features/epbs/builder.md | 112 ++++++++++++++++++++++++++++++++ 1 file changed, 112 insertions(+) create mode 100644 specs/_features/epbs/builder.md diff --git a/specs/_features/epbs/builder.md b/specs/_features/epbs/builder.md new file mode 100644 index 0000000000..3cd14d990b --- /dev/null +++ b/specs/_features/epbs/builder.md @@ -0,0 +1,112 @@ +# ePBS -- Honest Builder + +This is an accompanying document which describes the expected actions of a "builder" participating in the Ethereum proof-of-stake protocol. + + + + +- [Introduction](#introduction) +- [Builders attributions](#builders-attributions) + - [Constructing the payload bid](#constructing-the-payload-bid) + - [Constructing the `BlobSidecar`s](#constructing-the-blobsidecars) + - [Constructing the execution payload envelope](#constructing-the-execution-payload-envelope) + - [Honest payload withheld messages](#honest-payload-withheld-messages) + + + +## Introduction + +With the ePBS Fork, the protocol includes new staked participants of the protocol called *Builders*. While Builders are a subset of the validator set, they have extra attributions that are optional. Validators may opt to not be builders and as such we collect the set of guidelines for those validators that want to act as builders in this document. + +## Builders attributions + +Builders can submit bids to produce execution payloads. They can broadcast these bids in the form of `SignedExecutionPayloadHeader` objects, these objects encode a commitment to reveal an execution payload in exchange for a payment. When their bids are chosen by the corresponding proposer, builders are expected to broadcast an accompanying `SignedExecutionPayloadEnvelope` object honoring the commitment. + +Thus, builders tasks are divided in two, submitting bids, and submitting payloads. + +### Constructing the payload bid + +Builders can broadcast a payload bid for the current or the next slot's proposer to include. They produce a `SignedExecutionPayloadHeader` as follows. + +1. Set `header.parent_block_hash` to the current head of the execution chain (this can be obtained from the beacon state as `state.last_block_hash`). +2. Set `header.parent_block_root` to be the head of the consensus chain (this can be obtained from the beacon state as `hash_tree_root(state.latest_block_header)`. The `parent_block_root` and `parent_block_hash` must be compatible, in the sense that they both should come from the same `state` by the method described in this and the previous point. +3. Construct an execution payload. This can be performed with an external execution engine with a call to `engine_getPayloadV4`. +4. Set `header.block_hash` to be the block hash of the constructed payload, that is `payload.block_hash` +5. Set `header.builder_index` to be the validator index of the builder performing these actions. +6. Set `header.slot` to be the slot for which this bid is aimed. This slot **MUST** be either the current slot or the next slot. +7. Set `header.value` to be the value that the builder will pay the proposer if the bid is accepted. The builder **MUST** have balance enough to fulfill this bid. +8. Set `header.kzg_commitments_root` to be the `hash_tree_root` of the `blobsbundle.commitments` field returned by `engine_getPayloadV4`. + +After building the `header`, the builder obtains a `signature` of the header by using + +```python +def get_execution_payload_header_signature(state: BeaconState, header: ExecutionPayloadHeader, privkey: int) -> BLSSignature: + domain = get_domain(state, DOMAIN_BEACON_BUILDER, compute_epoch_at_slot(header.slot)) + signing_root = compute_signing_root(header, domain) + return bls.Sign(privkey, signing_root) +``` + +The builder assembles then `signed_exceution_payload_header = SignedExecutionPayloadHeader(message=header, signature=signature)` and broadcasts it on the `execution_payload_header` global gossip topic. + +### Constructing the `BlobSidecar`s + +[Modified in EIP-XXXX] + +The `BlobSidecar` container is modified indirectly because the constant `KZG_COMMITMENT_INCLUSION_PROOF_DEPTH` is modified. Each sidecar is obtained from the modified + +```python +def get_blob_sidecars(signed_block: SignedBeaconBlock, + blobs: Sequence[Blob], + blob_kzg_proofs: Sequence[KZGProof]) -> Sequence[BlobSidecar]: + block = signed_block.message + block_header = BeaconBlockHeader( + slot=block.slot, + proposer_index=block.proposer_index, + parent_root=block.parent_root, + state_root=block.state_root, + body_root=hash_tree_root(block.body), + ) + signed_block_header = SignedBeaconBlockHeader(message=block_header, signature=signed_block.signature) + return [ + BlobSidecar( + index=index, + blob=blob, + kzg_commitment=block.body.blob_kzg_commitments[index], + kzg_proof=blob_kzg_proofs[index], + signed_block_header=signed_block_header, + kzg_commitment_inclusion_proof=compute_merkle_proof( + block.body, + GeneralizedIndex(KZG_GENERALIZED_INDEX_PREFIX + index), + ), + ) + for index, blob in enumerate(blobs) + ] +``` + +### Constructing the execution payload envelope + +When the proposer publishes a valid `SignedBeaconBlock` containing a signed commitment by the builder, the builder is later expected to broadcast the corresponding `SignedExecutionPayloadEnvelope` that fulfills this commitment. See below for a special case of an *honestly withheld payload*. + +To construct the `execution_payload_envelope` the builder must perform the following steps, we alias `header` to be the committed `ExecutionPayloadHeader` in the beacon block. + +1. Set the `payload` field to be the `ExecutionPayload` constructed when creating the corresponding bid. This payload **MUST** have the same block hash as `header.block_hash`. +2. Set the `builder_index` field to be the validator index of the builder performing these steps. This field **MUST** be `header.builder_index`. +3. Set `beacon_block_root` to be the `hash_tree_root` of the corresponding beacon block. +4. Set `blob_kzg_commitments` to be the `commitments` field of the blobs bundle constructed when constructing the bid. This field **MUST** have a `hash_tree_root` equal to `header.blob_kzg_commitments_root`. +5. Set `payload_witheld` to `False`. + +After setting these parameters, the builder should run `process_execution_payload(state, signed_envelope, verify=False)` and this function should not trigger an exception. + +6. Set `state_root` to `hash_tree_root(state)`. +After preparing the `envelope` the builder should sign the envelope using: +```python +def get_execution_payload_envelope_signature(state: BeaconState, envelope: ExecutionPayloadEnvelope, privkey: int) -> BLSSignature: + domain = get_domain(state, DOMAIN_BEACON_BUILDER, compute_epoch_at_slot(state.slot)) + signing_root = compute_signing_root(envelope, domain) + return bls.Sign(privkey, signing_root) +``` +The builder assembles then `signed_exceution_payload_envelope = SignedExecutionPayloadEnvelope(message=envelope, signature=signature)` and broadcasts it on the `execution_payload` global gossip topic. + +### Honest payload withheld messages + +An honest builder that has seen a `SignedBeaconBlock` referencing his signed bid, but that block was not timely and thus it is not the head of the builder's chain, may choose to withhold their execution payload. For this the builder should simply act as if it were building an empty payload, without any transactions, withdrawals, etc. The `payload.block_hash` may not be equal to `header.block_hash`. The builder may then sets `payload_withheld` to `True`. If the PTC sees this message and votes for it, validators will attribute a *withholding boost* to the builder, which would increase the forkchoice weight of the parent block, favoring it and preventing the builder from being charged for the bid by not revealing. From cd7dbef16f246b0bc2edff14f9d8284210de343d Mon Sep 17 00:00:00 2001 From: Potuz Date: Tue, 25 Jun 2024 13:58:41 +0200 Subject: [PATCH 049/130] P2P changes --- specs/_features/epbs/p2p-interface.md | 258 ++++++++++++++++++++++++++ 1 file changed, 258 insertions(+) create mode 100644 specs/_features/epbs/p2p-interface.md diff --git a/specs/_features/epbs/p2p-interface.md b/specs/_features/epbs/p2p-interface.md new file mode 100644 index 0000000000..9d89b0dc5d --- /dev/null +++ b/specs/_features/epbs/p2p-interface.md @@ -0,0 +1,258 @@ +# ePBS -- Networking + +This document contains the consensus-layer networking specification for ePBS. + + + + +- [Modification in EIP-XXXX](#modification-in-eip-xxxx) + - [Preset](#preset) + - [Containers](#containers) + - [`BlobSidecar`](#blobsidecar) + - [Helpers](#helpers) + - [Modified `verify_blob_sidecar_inclusion_proof`](#modified-verify_blob_sidecar_inclusion_proof) + - [The gossip domain: gossipsub](#the-gossip-domain-gossipsub) + - [Topics and messages](#topics-and-messages) + - [Global topics](#global-topics) + - [`beacon_block`](#beacon_block) + - [`execution_payload`](#execution_payload) + - [`payload_attestation_message`](#payload_attestation_message) + - [`execution_payload_header`](#execution_payload_header) + - [The Req/Resp domain](#the-reqresp-domain) + - [Messages](#messages) + - [BeaconBlocksByRange v3](#beaconblocksbyrange-v3) + - [BeaconBlocksByRoot v3](#beaconblocksbyroot-v3) + - [BlobSidecarsByRoot v2](#blobsidecarsbyroot-v2) + - [ExecutionPayloadEnvelopeByRoot v1](#executionpayloadenvelopebyroot-v1) + + + +## Modification in EIP-XXXX + +### Preset + +*[Modified in EIP-XXXX]* + +| Name | Value | Description | +|------------------------------------------|-----------------------------------|---------------------------------------------------------------------| +| `KZG_COMMITMENT_INCLUSION_PROOF_DEPTH` | TODO: Compute it when the spec stabilizes | Merkle proof depth for the `blob_kzg_commitments` list item | +| `KZG_GENERALIZED_INDEX_PREFIX` | TODO: Compute it when the spec stabilizes | Generalized index for the first item in the `blob_kzg_commitments` list | + +### Containers + +#### `BlobSidecar` + +The `BlobSidecar` container is modified indirectly because the constant `KZG_COMMITMENT_INCLUSION_PROOF_DEPTH` is modified. + +```python +class BlobSidecar(Container): + index: BlobIndex # Index of blob in block + blob: Blob + kzg_commitment: KZGCommitment + kzg_proof: KZGProof # Allows for quick verification of kzg_commitment + signed_block_header: SignedBeaconBlockHeader + kzg_commitment_inclusion_proof: Vector[Bytes32, KZG_COMMITMENT_INCLUSION_PROOF_DEPTH] +``` + +#### Helpers + +##### Modified `verify_blob_sidecar_inclusion_proof` + +`verify_blob_sidecar_inclusion_proof` is modified in EIP-XXXX to account for the fact that the KZG commitments are included in the `ExecutionPayloadEnvelope` and no longer in the beacon block body. + +```python +def verify_blob_sidecar_inclusion_proof(blob_sidecar: BlobSidecar) -> bool: + # hardcoded here because the block does not include the commitments but only their root. + gindex = GeneralizedIndex(KZG_GENERALIZED_INDEX_PREFIX + blob_sidecar.index) + + return is_valid_merkle_branch( + leaf=blob_sidecar.kzg_commitment.hash_tree_root(), + branch=blob_sidecar.kzg_commitment_inclusion_proof, + depth=KZG_COMMITMENT_INCLUSION_PROOF_DEPTH, + index=gindex, + root=blob_sidecar.signed_block_header.message.body_root, + ) +``` + +### The gossip domain: gossipsub + +Some gossip meshes are upgraded in the fork of EIP-XXXX to support upgraded types. + +#### Topics and messages + +Topics follow the same specification as in prior upgrades. + +The `beacon_block` topic is updated to support the modified type +| Name | Message Type | +| --- | --- | +| `beacon_block` | `SignedBeaconBlock` [modified in EIP-XXXX] | + +The new topics along with the type of the `data` field of a gossipsub message are given in this table: + +| Name | Message Type | +|-------------------------------|------------------------------------------------------| +| `execution_payload_header` | `SignedExecutionPayloadHeader` [New in EIP-XXXX] | +| `execution_payload` | `SignedExecutionPayloadEnvelope` [New in EIP-XXXX] | +| `payload_attestation_message` | `PayloadAttestationMessage` [New in EIP-XXXX] | + +##### Global topics + +EIP-XXXX introduces new global topics for execution header, execution payload and payload attestation. + +###### `beacon_block` + +[Modified in EIP-XXXX] + +The *type* of the payload of this topic changes to the (modified) `SignedBeaconBlock` found in [the Beacon Chain changes](./beacon-chain.md). + +There are no new validations for this topic. However, all validations with regards to the `ExecutionPayload` are removed: + +- _[REJECT]_ The length of KZG commitments is less than or equal to the limitation defined in Consensus Layer -- i.e. validate that len(body.signed_beacon_block.message.blob_kzg_commitments) <= MAX_BLOBS_PER_BLOCK +- _[REJECT]_ The block's execution payload timestamp is correct with respect to the slot + -- i.e. `execution_payload.timestamp == compute_timestamp_at_slot(state, block.slot)`. +- If `execution_payload` verification of block's parent by an execution node is *not* complete: + - [REJECT] The block's parent (defined by `block.parent_root`) passes all validation (excluding execution node verification of the `block.body.execution_payload`). +- otherwise: + - [IGNORE] The block's parent (defined by `block.parent_root`) passes all validation (including execution node verification of the `block.body.execution_payload`). +- [REJECT] The block's parent (defined by `block.parent_root`) passes validation. + +And instead the following validations are set in place with the alias `header = signed_execution_payload_header.message`: + +- If `execution_payload` verification of block's execution payload parent by an execution node **is complete**: + - [REJECT] The block's execution payload parent (defined by `header.parent_block_hash`) passes all validation. +- [REJECT] The block's parent (defined by `block.parent_root`) passes validation. + +###### `execution_payload` + +This topic is used to propagate execution payload messages as `SignedExecutionPayloadEnvelope`. + +The following validations MUST pass before forwarding the `signed_execution_payload_envelope` on the network, assuming the alias `envelope = signed_execution_payload_envelope.message`, `payload = payload_envelope.payload`: + +- _[IGNORE]_ The envelope's block root `envelope.block_root` has been seen (via both gossip and non-gossip sources) (a client MAY queue payload for processing once the block is retrieved). +- _[IGNORE]_ The node has not seen another valid `SignedExecutionPayloadEnvelope` for this block root from this builder. + +Let `block` be the block with `envelope.beacon_block_root`. +Let `header` alias `block.body.signed_execution_payload_header.message` (notice that this can be obtained from the `state.signed_execution_payload_header`) +- _[REJECT]_ `block` passes validation. +- _[REJECT]_ `envelope.builder_index == header.builder_index` +- if `envelope.payload_withheld == False` then + - _[REJECT]_ `payload.block_hash == header.block_hash` +- _[REJECT]_ The builder signature, `signed_execution_payload_envelope.signature`, is valid with respect to the builder's public key. + +###### `payload_attestation_message` + +This topic is used to propagate signed payload attestation message. + +The following validations MUST pass before forwarding the `payload_attestation_message` on the network, assuming the alias `data = payload_attestation_message.data`: + +- _[IGNORE]_ `data.slot` is the current slot. +- _[REJECT]_ `data.payload_status < PAYLOAD_INVALID_STATUS` +- _[IGNORE]_ the `payload_attestation_message` is the first valid payload attestation message received from the validator index. +- _[IGNORE]_ The attestation's `data.beacon_block_root` has been seen (via both gossip and non-gossip sources) (a client MAY queue attestation for processing once the block is retrieved. Note a client might want to request payload after). +_ _[REJECT]_ The beacon block with root `data.beacon_block_root` passes validation. +- _[REJECT]_ The validator index is within the payload committee in `get_ptc(state, data.slot)`. For the current's slot head state. +- _[REJECT]_ The signature of `payload_attestation_message.signature` is valid with respect to the validator index. + +###### `execution_payload_header` + +This topic is used to propagate signed bids as `SignedExecutionPayloadHeader`. + +The following validations MUST pass before forwarding the `signed_execution_payload_header` on the network, assuming the alias `header = signed_execution_payload_header.message`: + +- _[IGNORE]_ this is the first signed bid seen with a valid signature from the given builder for this slot. +- _[IGNORE]_ this bid is the highest value bid seen for the pair of the corresponding slot and the given parent block hash. +- _[REJECT]_ The signed builder bid, `header.builder_index` is a valid and non-slashed builder index in state. +- _[IGNORE]_ The signed builder bid value, `header.value`, is less or equal than the builder's balance in state. i.e. `MIN_BUILDER_BALANCE + header.value < state.builder_balances[header.builder_index]`. +- _[IGNORE]_ `header.parent_block_hash` is the block hash of a known execution payload in fork choice. +- _[IGNORE]_ `header.slot` is the current slot or the next slot. +- _[REJECT]_ The builder signature, `signed_execution_payload_header_envelope.signature`, is valid with respect to the `header_envelope.builder_index`. + +### The Req/Resp domain + +#### Messages + +##### BeaconBlocksByRange v3 + +**Protocol ID:** `/eth2/beacon_chain/req/beacon_blocks_by_range/3/` + +[0]: # (eth2spec: skip) + +| `fork_version` | Chunk SSZ type | +|--------------------------|-------------------------------| +| `GENESIS_FORK_VERSION` | `phase0.SignedBeaconBlock` | +| `ALTAIR_FORK_VERSION` | `altair.SignedBeaconBlock` | +| `BELLATRIX_FORK_VERSION` | `bellatrix.SignedBeaconBlock` | +| `CAPELLA_FORK_VERSION` | `capella.SignedBeaconBlock` | +| `DENEB_FORK_VERSION` | `deneb.SignedBeaconBlock` | +| `EPBS_FORK_VERSION` | `epbs.SignedBeaconBlock` | + +##### BeaconBlocksByRoot v3 + +**Protocol ID:** `/eth2/beacon_chain/req/beacon_blocks_by_root/3/` + +Per `context = compute_fork_digest(fork_version, genesis_validators_root)`: + +[1]: # (eth2spec: skip) + +| `fork_version` | Chunk SSZ type | +|--------------------------|-------------------------------| +| `GENESIS_FORK_VERSION` | `phase0.SignedBeaconBlock` | +| `ALTAIR_FORK_VERSION` | `altair.SignedBeaconBlock` | +| `BELLATRIX_FORK_VERSION` | `bellatrix.SignedBeaconBlock` | +| `CAPELLA_FORK_VERSION` | `capella.SignedBeaconBlock` | +| `DENEB_FORK_VERSION` | `deneb.SignedBeaconBlock` | +| `EPBS_FORK_VERSION` | `epbs.SignedBeaconBlock` | + + +##### BlobSidecarsByRoot v2 + +**Protocol ID:** `/eth2/beacon_chain/req/blob_sidecars_by_root/2/` + +[1]: # (eth2spec: skip) + +| `fork_version` | Chunk SSZ type | +|--------------------------|-------------------------------| +| `DENEB_FORK_VERSION` | `deneb.BlobSidecar` | +| `EPBS_FORK_VERSION` | `epbs.BlobSidecar` | + + +##### ExecutionPayloadEnvelopeByRoot v1 + +**Protocol ID:** `/eth2/beacon_chain/req/execution_payload_envelope_by_root/1/` + +The `` field is calculated as `context = compute_fork_digest(fork_version, genesis_validators_root)`: + +[1]: # (eth2spec: skip) + +| `fork_version` | Chunk SSZ type | +|---------------------|---------------------------------------| +| `EPBS_FORK_VERSION` | `epbs.SignedExecutionPayloadEnvelope` | + +Request Content: + +``` +( + List[Root, MAX_REQUEST_PAYLOAD] +) +``` + +Response Content: + +``` +( + List[SignedExecutionPayloadEnvelope, MAX_REQUEST_PAYLOAD] +) +``` +Requests execution payload envelope by `signed_execution_payload_envelope.message.block_root`. The response is a list of `SignedExecutionPayloadEnvelope` whose length is less than or equal to the number of requested execution payload envelopes. It may be less in the case that the responding peer is missing payload envelopes. + +No more than `MAX_REQUEST_PAYLOAD` may be requested at a time. + +ExecutionPayloadEnvelopeByRoot is primarily used to recover recent execution payload envelope (e.g. when receiving a payload attestation with revealed status as true but never received a payload). + +The request MUST be encoded as an SSZ-field. + +The response MUST consist of zero or more `response_chunk`. Each successful `response_chunk` MUST contain a single `SignedExecutionPayloadEnvelope` payload. + +Clients MUST support requesting payload envelopes since the latest finalized epoch. + +Clients MUST respond with at least one payload envelope, if they have it. Clients MAY limit the number of payload envelopes in the response. From 6d6ef7baa4524bcd93ce1eef823642a3e184ef17 Mon Sep 17 00:00:00 2001 From: Potuz Date: Tue, 25 Jun 2024 14:09:56 +0200 Subject: [PATCH 050/130] Fork logic --- specs/_features/epbs/fork.md | 130 +++++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100644 specs/_features/epbs/fork.md diff --git a/specs/_features/epbs/fork.md b/specs/_features/epbs/fork.md new file mode 100644 index 0000000000..f08f5da7c0 --- /dev/null +++ b/specs/_features/epbs/fork.md @@ -0,0 +1,130 @@ +# ePBS -- Fork Logic + +**Notice**: This document is a work-in-progress for researchers and implementers. + +## Table of contents + + + + +- [Introduction](#introduction) +- [Configuration](#configuration) +- [Helper functions](#helper-functions) + - [Misc](#misc) + - [Modified `compute_fork_version`](#modified-compute_fork_version) +- [Fork to EIP-XXXX](#fork-to-eip-xxxx) + - [Fork trigger](#fork-trigger) + - [Upgrading the state](#upgrading-the-state) + + + +## Introduction + +This document describes the process of the EIP-XXXX upgrade. + +## Configuration + +Warning: this configuration is not definitive. + +| Name | Value | +|---------------------| - | +| `EPBS_FORK_VERSION` | `Version('0x06000000')` | +| `EPBS_FORK_EPOCH` | `Epoch(18446744073709551615)` **TBD** | + +## Helper functions + +### Misc + +#### Modified `compute_fork_version` + +```python +def compute_fork_version(epoch: Epoch) -> Version: + """ + Return the fork version at the given ``epoch``. + """ + if epoch >= EPBS_FORK_EPOCH: + return EPBS_FORK_VERSION + if epoch >= ELECTRA_FORK_EPOCH: + return ELECTRA_FORK_VERSION + if epoch >= DENEB_FORK_EPOCH: + return DENEB_FORK_VERSION + if epoch >= CAPELLA_FORK_EPOCH: + return CAPELLA_FORK_VERSION + if epoch >= BELLATRIX_FORK_EPOCH: + return BELLATRIX_FORK_VERSION + if epoch >= ALTAIR_FORK_EPOCH: + return ALTAIR_FORK_VERSION + return GENESIS_FORK_VERSION +``` + +## Fork to EIP-XXXX + +### Fork trigger + +TBD. This fork is defined for testing purposes, the EIP may be combined with other +consensus-layer upgrade. +For now, we assume the condition will be triggered at epoch `EPBS_FORK_EPOCH`. + +### Upgrading the state + +If `state.slot % SLOTS_PER_EPOCH == 0` and `compute_epoch_at_slot(state.slot) == EPBS_FORK_EPOCH`, +an irregular state change is made to upgrade to ePBS. + +```python +def upgrade_to_epbs(pre: deneb.BeaconState) -> BeaconState: + epoch = deneb.get_current_epoch(pre) + + post = BeaconState( + # Versioning + genesis_time=pre.genesis_time, + genesis_validators_root=pre.genesis_validators_root, + slot=pre.slot, + fork=Fork( + previous_version=pre.fork.current_version, + current_version=EPBS_FORK_EPOCH, # [Modified in ePBS] + epoch=epoch, + ), + # History + latest_block_header=pre.latest_block_header, + block_roots=pre.block_roots, + state_roots=pre.state_roots, + historical_roots=pre.historical_roots, + # Eth1 + eth1_data=pre.eth1_data, + eth1_data_votes=pre.eth1_data_votes, + eth1_deposit_index=pre.eth1_deposit_index, + # Registry + validators=pre.validators, + balances=pre.balances, + # Randomness + randao_mixes=pre.randao_mixes, + # Slashings + slashings=pre.slashings, + # Participation + previous_epoch_participation=pre.previous_epoch_participation, + current_epoch_participation=pre.current_epoch_participation, + # Finality + justification_bits=pre.justification_bits, + previous_justified_checkpoint=pre.previous_justified_checkpoint, + current_justified_checkpoint=pre.current_justified_checkpoint, + finalized_checkpoint=pre.finalized_checkpoint, + # Inactivity + inactivity_scores=pre.inactivity_scores, + # Sync + current_sync_committee=pre.current_sync_committee, + next_sync_committee=pre.next_sync_committee, + # Execution-layer + latest_execution_payload_header=ExecutionPayloadHeader(), # [Modified in EIP-XXXX] + # Withdrawals + next_withdrawal_index=pre.next_withdrawal_index, + next_withdrawal_validator_index=pre.next_withdrawal_validator_index, + # Deep history valid from Capella onwards + historical_summaries=pre.historical_summaries, + # ePBS + latest_block_hash=pre.latest_execution_payload_header.block_hash, # [New in EIP-XXXX] + latest_full_slot=pre.slot, # [New in EIP-XXXX] + last_withdrawals_root=Root(), # [New in EIP-XXXX] + ) + + return post +``` From 0d082fdc0e6c2b0e9aa1079ec410229497f26ad9 Mon Sep 17 00:00:00 2001 From: Potuz Date: Fri, 28 Jun 2024 10:32:46 -0300 Subject: [PATCH 051/130] Add comment about attestations and sync committee duties --- specs/_features/epbs/validator.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/specs/_features/epbs/validator.md b/specs/_features/epbs/validator.md index 20ab108a62..50b4d13087 100644 --- a/specs/_features/epbs/validator.md +++ b/specs/_features/epbs/validator.md @@ -56,6 +56,14 @@ All validator responsibilities remain unchanged other than the following: - Proposers are no longer required to broadcast `BlobSidecar` objects, as this becomes a builder's duty. - Some validators are selected per slot to become PTC members, these validators must broadcat `PayloadAttestationMessage` objects during the assigned slot before the deadline of `3 * SECONDS_PER_SLOT // INTERVALS_PER_SLOT` seconds into the slot. +### Attestation + +Attestation duties are not changed for validators, however the attestation deadline is implicitly changed by the change in `INTERVALS_PER_SLOT`. + +### Sync Committee participations + +Sync committee duties are not changed for validators, however the submission deadline is implicitly changed by the change in `INTERVALS_PER_SLOT`. + ### Block proposal From 6e02cb4ea49d2293b0c56e2a431198f031d07ea3 Mon Sep 17 00:00:00 2001 From: terence Date: Fri, 28 Jun 2024 06:51:20 -0700 Subject: [PATCH 052/130] Fix epbs consensus spec to be executable (#10) Fix epbs consensus spec to be executable --- presets/mainnet/eip-xxxx.yaml | 8 +++++++ presets/minimal/eip-xxxx.yaml | 8 +++++++ pysetup/constants.py | 1 + pysetup/md_doc_paths.py | 2 ++ pysetup/spec_builders/__init__.py | 3 ++- pysetup/spec_builders/eipxxxx.py | 21 +++++++++++++++++++ .../{epbs => eipxxxx}/beacon-chain.md | 4 ++-- specs/_features/{epbs => eipxxxx}/builder.md | 0 .../{epbs => eipxxxx}/fork-choice.md | 6 +++--- specs/_features/{epbs => eipxxxx}/fork.md | 0 .../{epbs => eipxxxx}/p2p-interface.md | 0 .../_features/{epbs => eipxxxx}/validator.md | 0 .../core/pyspec/eth2spec/eipxxxx/__init__.py | 1 + .../pyspec/eth2spec/test/helpers/constants.py | 6 +++++- 14 files changed, 53 insertions(+), 7 deletions(-) create mode 100644 presets/mainnet/eip-xxxx.yaml create mode 100644 presets/minimal/eip-xxxx.yaml create mode 100644 pysetup/spec_builders/eipxxxx.py rename specs/_features/{epbs => eipxxxx}/beacon-chain.md (99%) rename specs/_features/{epbs => eipxxxx}/builder.md (100%) rename specs/_features/{epbs => eipxxxx}/fork-choice.md (98%) rename specs/_features/{epbs => eipxxxx}/fork.md (100%) rename specs/_features/{epbs => eipxxxx}/p2p-interface.md (100%) rename specs/_features/{epbs => eipxxxx}/validator.md (100%) create mode 100644 tests/core/pyspec/eth2spec/eipxxxx/__init__.py diff --git a/presets/mainnet/eip-xxxx.yaml b/presets/mainnet/eip-xxxx.yaml new file mode 100644 index 0000000000..6b081bbf34 --- /dev/null +++ b/presets/mainnet/eip-xxxx.yaml @@ -0,0 +1,8 @@ +# Mainnet preset - EIPXXXX + +# Execution +# --------------------------------------------------------------- +# 2**9 (= 512) +PTC_SIZE: 512 +# 2**2 (= 4) +MAX_PAYLOAD_ATTESTATIONS: 4 \ No newline at end of file diff --git a/presets/minimal/eip-xxxx.yaml b/presets/minimal/eip-xxxx.yaml new file mode 100644 index 0000000000..d596ed8971 --- /dev/null +++ b/presets/minimal/eip-xxxx.yaml @@ -0,0 +1,8 @@ +# Minimal preset - EIPXXXX + +# Execution +# --------------------------------------------------------------- +# 2**1(= 2) +PTC_SIZE: 2 +# 2**2 (= 4) +MAX_PAYLOAD_ATTESTATIONS: 4 diff --git a/pysetup/constants.py b/pysetup/constants.py index e26efb8e06..0f91c981d9 100644 --- a/pysetup/constants.py +++ b/pysetup/constants.py @@ -8,6 +8,7 @@ EIP7594 = 'eip7594' EIP6800 = 'eip6800' WHISK = 'whisk' +EIPXXXX = 'eipxxxx' # The helper functions that are used when defining constants diff --git a/pysetup/md_doc_paths.py b/pysetup/md_doc_paths.py index 28ebc71379..96d27a74e5 100644 --- a/pysetup/md_doc_paths.py +++ b/pysetup/md_doc_paths.py @@ -10,6 +10,7 @@ WHISK, EIP7594, EIP6800, + EIPXXXX, ) @@ -23,6 +24,7 @@ WHISK: CAPELLA, EIP7594: DENEB, EIP6800: DENEB, + EIPXXXX: ELECTRA, } ALL_FORKS = list(PREVIOUS_FORK_OF.keys()) diff --git a/pysetup/spec_builders/__init__.py b/pysetup/spec_builders/__init__.py index 922cee18b2..3c8be80b16 100644 --- a/pysetup/spec_builders/__init__.py +++ b/pysetup/spec_builders/__init__.py @@ -7,12 +7,13 @@ from .whisk import WhiskSpecBuilder from .eip7594 import EIP7594SpecBuilder from .eip6800 import EIP6800SpecBuilder +from .eipxxxx import EIPXXXXSpecBuilder spec_builders = { builder.fork: builder for builder in ( Phase0SpecBuilder, AltairSpecBuilder, BellatrixSpecBuilder, CapellaSpecBuilder, DenebSpecBuilder, - ElectraSpecBuilder, WhiskSpecBuilder, EIP7594SpecBuilder, EIP6800SpecBuilder, + ElectraSpecBuilder, WhiskSpecBuilder, EIP7594SpecBuilder, EIP6800SpecBuilder, EIPXXXXSpecBuilder, ) } diff --git a/pysetup/spec_builders/eipxxxx.py b/pysetup/spec_builders/eipxxxx.py new file mode 100644 index 0000000000..dcb267d736 --- /dev/null +++ b/pysetup/spec_builders/eipxxxx.py @@ -0,0 +1,21 @@ +from typing import Dict + +from .base import BaseSpecBuilder +from ..constants import EIPXXXX + + +class EIPXXXXSpecBuilder(BaseSpecBuilder): + fork: str = EIPXXXX + + @classmethod + def imports(cls, preset_name: str): + return f''' +from eth2spec.eipxxxx import {preset_name} as eipxxxx +''' + + @classmethod + def hardcoded_custom_type_dep_constants(cls, spec_object) -> Dict[str, str]: + return { + 'PTC_SIZE': spec_object.preset_vars['PTC_SIZE'].value, + 'MAX_PAYLOAD_ATTESTATIONS': spec_object.preset_vars['MAX_PAYLOAD_ATTESTATIONS'].value, + } \ No newline at end of file diff --git a/specs/_features/epbs/beacon-chain.md b/specs/_features/eipxxxx/beacon-chain.md similarity index 99% rename from specs/_features/epbs/beacon-chain.md rename to specs/_features/eipxxxx/beacon-chain.md index 870744ac18..1dd381c98c 100644 --- a/specs/_features/epbs/beacon-chain.md +++ b/specs/_features/eipxxxx/beacon-chain.md @@ -112,7 +112,7 @@ class PayloadAttestationData(Container): ```python class PayloadAttestation(Container): - aggregation_bits: BitVector[PTC_SIZE] + aggregation_bits: Bitvector[PTC_SIZE] data: PayloadAttestationData signature: BLSSignature ``` @@ -151,7 +151,7 @@ class ExecutionPayloadEnvelope(Container): builder_index: ValidatorIndex beacon_block_root: Root blob_kzg_commitments: List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK] - payload_withheld: bool + payload_withheld: boolean state_root: Root ``` diff --git a/specs/_features/epbs/builder.md b/specs/_features/eipxxxx/builder.md similarity index 100% rename from specs/_features/epbs/builder.md rename to specs/_features/eipxxxx/builder.md diff --git a/specs/_features/epbs/fork-choice.md b/specs/_features/eipxxxx/fork-choice.md similarity index 98% rename from specs/_features/epbs/fork-choice.md rename to specs/_features/eipxxxx/fork-choice.md index 9d9786b701..02f95aeef2 100644 --- a/specs/_features/epbs/fork-choice.md +++ b/specs/_features/eipxxxx/fork-choice.md @@ -296,7 +296,7 @@ def get_head(store: Store) -> ChildNode: justified_block = store.blocks[justified_root] justified_slot = justified_block.slot justified_full = is_payload_present(store, justified_root) - best_child = ChildNode(root=head_root, slot=head_slot, is_payload_present=head_full) + best_child = ChildNode(root=justified_root, slot=justified_slot, is_payload_present=justified_full) while True: children = [ ChildNode(root=root, slot=block.slot, is_payload_present=present) for (root, block) in blocks.items() @@ -307,7 +307,7 @@ def get_head(store: Store) -> ChildNode: if len(children) == 0: return best_child # if we have children we consider the current head advanced as a possible head - children += [ChildNode(root=best_child.root, slot=best_child.slot + 1, best_child.is_payload_present)] + children += [ChildNode(root=best_child.root, slot=best_child.slot + 1, is_payload_present=best_child.is_payload_present)] # Sort by latest attesting balance with ties broken lexicographically # Ties broken by favoring full blocks according to the PTC vote # Ties are then broken by favoring full blocks @@ -484,7 +484,7 @@ def on_payload_attestation_message(store: Store, ptc_vote[ptc_index] = data.payload_status # Only update payload boosts with attestations from a block if the block is for the current slot and it's early - if is_from_block && data.slot + 1 != get_current_slot(store): + if is_from_block and data.slot + 1 != get_current_slot(store): return time_into_slot = (store.time - store.genesis_time) % SECONDS_PER_SLOT if is_from_block and time_into_slot >= SECONDS_PER_SLOT // INTERVALS_PER_SLOT: diff --git a/specs/_features/epbs/fork.md b/specs/_features/eipxxxx/fork.md similarity index 100% rename from specs/_features/epbs/fork.md rename to specs/_features/eipxxxx/fork.md diff --git a/specs/_features/epbs/p2p-interface.md b/specs/_features/eipxxxx/p2p-interface.md similarity index 100% rename from specs/_features/epbs/p2p-interface.md rename to specs/_features/eipxxxx/p2p-interface.md diff --git a/specs/_features/epbs/validator.md b/specs/_features/eipxxxx/validator.md similarity index 100% rename from specs/_features/epbs/validator.md rename to specs/_features/eipxxxx/validator.md diff --git a/tests/core/pyspec/eth2spec/eipxxxx/__init__.py b/tests/core/pyspec/eth2spec/eipxxxx/__init__.py new file mode 100644 index 0000000000..91b4a68537 --- /dev/null +++ b/tests/core/pyspec/eth2spec/eipxxxx/__init__.py @@ -0,0 +1 @@ +from . import mainnet as spec # noqa:F401 diff --git a/tests/core/pyspec/eth2spec/test/helpers/constants.py b/tests/core/pyspec/eth2spec/test/helpers/constants.py index ed398516cd..2fbbfb90d6 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/constants.py +++ b/tests/core/pyspec/eth2spec/test/helpers/constants.py @@ -20,6 +20,7 @@ ELECTRA = SpecForkName('electra') WHISK = SpecForkName('whisk') EIP7594 = SpecForkName('eip7594') +EIPXXXX = SpecForkName('eipxxxx') # # SpecFork settings @@ -37,11 +38,12 @@ ELECTRA, # Experimental patches EIP7594, + EIPXXXX, ) # The forks that have light client specs LIGHT_CLIENT_TESTING_FORKS = (*[item for item in MAINNET_FORKS if item != PHASE0],) # The forks that output to the test vectors. -TESTGEN_FORKS = (*MAINNET_FORKS, ELECTRA, EIP7594, WHISK) +TESTGEN_FORKS = (*MAINNET_FORKS, ELECTRA, EIP7594, WHISK, EIPXXXX) # Forks allowed in the test runner `--fork` flag, to fail fast in case of typos ALLOWED_TEST_RUNNER_FORKS = (*ALL_PHASES, WHISK) @@ -57,6 +59,7 @@ # Experimental patches WHISK: CAPELLA, EIP7594: DENEB, + EIPXXXX: ELECTRA, } # For fork transition tests @@ -67,6 +70,7 @@ BELLATRIX: CAPELLA, CAPELLA: DENEB, DENEB: ELECTRA, + ELECTRA: EIPXXXX, } ALL_PRE_POST_FORKS = POST_FORK_OF.items() From a01f124286a204e4e2d16237f6682d32cd2499dd Mon Sep 17 00:00:00 2001 From: Potuz Date: Fri, 28 Jun 2024 11:43:30 -0300 Subject: [PATCH 053/130] modify INTEVALS_PER_SLOT --- specs/_features/eipxxxx/fork-choice.md | 1 + 1 file changed, 1 insertion(+) diff --git a/specs/_features/eipxxxx/fork-choice.md b/specs/_features/eipxxxx/fork-choice.md index 02f95aeef2..2988a1798e 100644 --- a/specs/_features/eipxxxx/fork-choice.md +++ b/specs/_features/eipxxxx/fork-choice.md @@ -44,6 +44,7 @@ This is the modification of the fork choice accompanying the ePBS upgrade. | Name | Value | | -------------------- | ----------- | | `PAYLOAD_TIMELY_THRESHOLD` | `PTC_SIZE/2` (=`uint64(256)`) | +| `INTERVALS_PER_SLOT` | `4` # [modified in EIP-XXXX] | | `PROPOSER_SCORE_BOOST` | `20` # [modified in EIP-XXXX] | | `PAYLOAD_WITHHOLD_BOOST` | `40` | | `PAYLOAD_REVEAL_BOOST` | `40` | From b7b1de93e9d5d5081dbdbbfb2b9bf16324860e44 Mon Sep 17 00:00:00 2001 From: Potuz Date: Sun, 30 Jun 2024 16:48:09 -0300 Subject: [PATCH 054/130] Verify withdrawals only when present --- specs/_features/eipxxxx/beacon-chain.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/_features/eipxxxx/beacon-chain.md b/specs/_features/eipxxxx/beacon-chain.md index 1dd381c98c..63c77c3387 100644 --- a/specs/_features/eipxxxx/beacon-chain.md +++ b/specs/_features/eipxxxx/beacon-chain.md @@ -586,15 +586,15 @@ def process_execution_payload(state: BeaconState, signed_envelope: SignedExecuti # Verify consistency with the beacon block assert envelope.beacon_block_root == hash_tree_root(state.latest_block_header) - # Verify the withdrawals root - assert hash_tree_root(payload.withdrawals) == state.last_withdrawals_root - # Verify consistency with the committed header committed_header = state.execution_payload_header assert envelope.builder_index == committed_header.builder_index assert committed_header.blob_kzg_commitments_root == hash_tree_root(envelope.blob_kzg_commitments) if not envelope.payload_withheld: + # Verify the withdrawals root + assert hash_tree_root(payload.withdrawals) == state.last_withdrawals_root + assert committed_header.block_hash == payload.block_hash # Verify consistency of the parent hash with respect to the previous execution payload assert payload.parent_hash == state.latest_block_hash From 51e47ff6e4374dcb92649512da9bcd9218ec8bb3 Mon Sep 17 00:00:00 2001 From: DanGoron <97026899+gorondan@users.noreply.github.com> Date: Mon, 1 Jul 2024 16:02:26 +0300 Subject: [PATCH 055/130] small typo fixes (#12) --- specs/_features/eipxxxx/builder.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/specs/_features/eipxxxx/builder.md b/specs/_features/eipxxxx/builder.md index 3cd14d990b..f6acaf961b 100644 --- a/specs/_features/eipxxxx/builder.md +++ b/specs/_features/eipxxxx/builder.md @@ -5,12 +5,13 @@ This is an accompanying document which describes the expected actions of a "buil -- [Introduction](#introduction) -- [Builders attributions](#builders-attributions) - - [Constructing the payload bid](#constructing-the-payload-bid) - - [Constructing the `BlobSidecar`s](#constructing-the-blobsidecars) - - [Constructing the execution payload envelope](#constructing-the-execution-payload-envelope) - - [Honest payload withheld messages](#honest-payload-withheld-messages) +- [ePBS -- Honest Builder](#epbs----honest-builder) + - [Introduction](#introduction) + - [Builders attributions](#builders-attributions) + - [Constructing the payload bid](#constructing-the-payload-bid) + - [Constructing the `BlobSidecar`s](#constructing-the-blobsidecars) + - [Constructing the execution payload envelope](#constructing-the-execution-payload-envelope) + - [Honest payload withheld messages](#honest-payload-withheld-messages) @@ -46,7 +47,7 @@ def get_execution_payload_header_signature(state: BeaconState, header: Execution return bls.Sign(privkey, signing_root) ``` -The builder assembles then `signed_exceution_payload_header = SignedExecutionPayloadHeader(message=header, signature=signature)` and broadcasts it on the `execution_payload_header` global gossip topic. +The builder assembles then `signed_execution_payload_header = SignedExecutionPayloadHeader(message=header, signature=signature)` and broadcasts it on the `execution_payload_header` global gossip topic. ### Constructing the `BlobSidecar`s @@ -105,7 +106,7 @@ def get_execution_payload_envelope_signature(state: BeaconState, envelope: Execu signing_root = compute_signing_root(envelope, domain) return bls.Sign(privkey, signing_root) ``` -The builder assembles then `signed_exceution_payload_envelope = SignedExecutionPayloadEnvelope(message=envelope, signature=signature)` and broadcasts it on the `execution_payload` global gossip topic. +The builder assembles then `signed_execution_payload_envelope = SignedExecutionPayloadEnvelope(message=envelope, signature=signature)` and broadcasts it on the `execution_payload` global gossip topic. ### Honest payload withheld messages From 25a0fdc9917a8b1e01f118c7415a1e7eb39c39e8 Mon Sep 17 00:00:00 2001 From: Potuz Date: Mon, 1 Jul 2024 10:03:12 -0300 Subject: [PATCH 056/130] doctoc --- specs/_features/eipxxxx/builder.md | 13 ++++++------- specs/_features/eipxxxx/validator.md | 2 ++ 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/specs/_features/eipxxxx/builder.md b/specs/_features/eipxxxx/builder.md index f6acaf961b..4c5cde6525 100644 --- a/specs/_features/eipxxxx/builder.md +++ b/specs/_features/eipxxxx/builder.md @@ -5,13 +5,12 @@ This is an accompanying document which describes the expected actions of a "buil -- [ePBS -- Honest Builder](#epbs----honest-builder) - - [Introduction](#introduction) - - [Builders attributions](#builders-attributions) - - [Constructing the payload bid](#constructing-the-payload-bid) - - [Constructing the `BlobSidecar`s](#constructing-the-blobsidecars) - - [Constructing the execution payload envelope](#constructing-the-execution-payload-envelope) - - [Honest payload withheld messages](#honest-payload-withheld-messages) +- [Introduction](#introduction) +- [Builders attributions](#builders-attributions) + - [Constructing the payload bid](#constructing-the-payload-bid) + - [Constructing the `BlobSidecar`s](#constructing-the-blobsidecars) + - [Constructing the execution payload envelope](#constructing-the-execution-payload-envelope) + - [Honest payload withheld messages](#honest-payload-withheld-messages) diff --git a/specs/_features/eipxxxx/validator.md b/specs/_features/eipxxxx/validator.md index 50b4d13087..86db0072db 100644 --- a/specs/_features/eipxxxx/validator.md +++ b/specs/_features/eipxxxx/validator.md @@ -9,6 +9,8 @@ This document represents the changes and additions to the Honest validator guide - [Validator assignment](#validator-assignment) - [Lookahead](#lookahead) - [Beacon chain responsibilities](#beacon-chain-responsibilities) + - [Attestation](#attestation) + - [Sync Committee participations](#sync-committee-participations) - [Block proposal](#block-proposal) - [Constructing the new `signed_execution_payload_header_envelope` field in `BeaconBlockBody`](#constructing-the-new-signed_execution_payload_header_envelope-field-in--beaconblockbody) - [Constructing the new `payload_attestations` field in `BeaconBlockBody`](#constructing-the-new-payload_attestations-field-in--beaconblockbody) From ed7f17e0bffa5656851189ba7755d4534a2583bf Mon Sep 17 00:00:00 2001 From: Potuz Date: Mon, 1 Jul 2024 13:11:43 -0300 Subject: [PATCH 057/130] Add gas_limit to the execution payload header --- specs/_features/eipxxxx/beacon-chain.md | 6 +++++- specs/_features/eipxxxx/builder.md | 15 ++++++++------- 2 files changed, 13 insertions(+), 8 deletions(-) diff --git a/specs/_features/eipxxxx/beacon-chain.md b/specs/_features/eipxxxx/beacon-chain.md index 63c77c3387..3e4857c30a 100644 --- a/specs/_features/eipxxxx/beacon-chain.md +++ b/specs/_features/eipxxxx/beacon-chain.md @@ -192,13 +192,14 @@ class BeaconBlockBody(Container): #### `ExecutionPayloadHeader` -**Note:** The `ExecutionPayloadHeader` is modified to only contain the block hash of the committed `ExecutionPayload` in addition to the builder's payment information and KZG commitments root to verify the inclusion proofs. +**Note:** The `ExecutionPayloadHeader` is modified to only contain the block hash of the committed `ExecutionPayload` in addition to the builder's payment information, gas limit and KZG commitments root to verify the inclusion proofs. ```python class ExecutionPayloadHeader(Container): parent_block_hash: Hash32 parent_block_root: Root block_hash: Hash32 + gas_limit: uint64 builder_index: ValidatorIndex slot: Slot value: Gwei @@ -595,6 +596,9 @@ def process_execution_payload(state: BeaconState, signed_envelope: SignedExecuti # Verify the withdrawals root assert hash_tree_root(payload.withdrawals) == state.last_withdrawals_root + # Verify the gas_limit + assert commited_header.gas_limit == payload.gas_limit + assert committed_header.block_hash == payload.block_hash # Verify consistency of the parent hash with respect to the previous execution payload assert payload.parent_hash == state.latest_block_hash diff --git a/specs/_features/eipxxxx/builder.md b/specs/_features/eipxxxx/builder.md index 4c5cde6525..e44111bd90 100644 --- a/specs/_features/eipxxxx/builder.md +++ b/specs/_features/eipxxxx/builder.md @@ -28,14 +28,15 @@ Thus, builders tasks are divided in two, submitting bids, and submitting payload Builders can broadcast a payload bid for the current or the next slot's proposer to include. They produce a `SignedExecutionPayloadHeader` as follows. -1. Set `header.parent_block_hash` to the current head of the execution chain (this can be obtained from the beacon state as `state.last_block_hash`). +1. Set `header.parent_block_hash` to the current head of the execution chain (this can be obtained from the beacon state as `state.last_block_hash`). 2. Set `header.parent_block_root` to be the head of the consensus chain (this can be obtained from the beacon state as `hash_tree_root(state.latest_block_header)`. The `parent_block_root` and `parent_block_hash` must be compatible, in the sense that they both should come from the same `state` by the method described in this and the previous point. -3. Construct an execution payload. This can be performed with an external execution engine with a call to `engine_getPayloadV4`. -4. Set `header.block_hash` to be the block hash of the constructed payload, that is `payload.block_hash` -5. Set `header.builder_index` to be the validator index of the builder performing these actions. -6. Set `header.slot` to be the slot for which this bid is aimed. This slot **MUST** be either the current slot or the next slot. -7. Set `header.value` to be the value that the builder will pay the proposer if the bid is accepted. The builder **MUST** have balance enough to fulfill this bid. -8. Set `header.kzg_commitments_root` to be the `hash_tree_root` of the `blobsbundle.commitments` field returned by `engine_getPayloadV4`. +3. Construct an execution payload. This can be performed with an external execution engine with a call to `engine_getPayloadV4`. +4. Set `header.block_hash` to be the block hash of the constructed payload, that is `payload.block_hash`. +5. Set `header.gas_limit` to be the gas limit of the constructed payload, that is `payload.gas_limit`. +6. Set `header.builder_index` to be the validator index of the builder performing these actions. +7. Set `header.slot` to be the slot for which this bid is aimed. This slot **MUST** be either the current slot or the next slot. +8. Set `header.value` to be the value that the builder will pay the proposer if the bid is accepted. The builder **MUST** have balance enough to fulfill this bid. +9. Set `header.kzg_commitments_root` to be the `hash_tree_root` of the `blobsbundle.commitments` field returned by `engine_getPayloadV4`. After building the `header`, the builder obtains a `signature` of the header by using From 95dd081357800593ab8f4906d2fceafc67e2e6ba Mon Sep 17 00:00:00 2001 From: Potuz Date: Tue, 2 Jul 2024 09:15:43 -0300 Subject: [PATCH 058/130] codespell --- specs/_features/eipxxxx/validator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/_features/eipxxxx/validator.md b/specs/_features/eipxxxx/validator.md index 86db0072db..78c9f178b2 100644 --- a/specs/_features/eipxxxx/validator.md +++ b/specs/_features/eipxxxx/validator.md @@ -56,7 +56,7 @@ def get_ptc_assignment(state: BeaconState, All validator responsibilities remain unchanged other than the following: - Proposers are no longer required to broadcast `BlobSidecar` objects, as this becomes a builder's duty. -- Some validators are selected per slot to become PTC members, these validators must broadcat `PayloadAttestationMessage` objects during the assigned slot before the deadline of `3 * SECONDS_PER_SLOT // INTERVALS_PER_SLOT` seconds into the slot. +- Some validators are selected per slot to become PTC members, these validators must broadcast `PayloadAttestationMessage` objects during the assigned slot before the deadline of `3 * SECONDS_PER_SLOT // INTERVALS_PER_SLOT` seconds into the slot. ### Attestation From 752a804db62cd9e3ff41d92c1e4e0dc830661781 Mon Sep 17 00:00:00 2001 From: Potuz Date: Tue, 2 Jul 2024 10:11:55 -0300 Subject: [PATCH 059/130] use boolean instead of bool in forkchoice --- Makefile | 2 +- specs/_features/eipxxxx/fork-choice.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index bdf4bdde7a..07dc301735 100644 --- a/Makefile +++ b/Makefile @@ -35,7 +35,7 @@ MARKDOWN_FILES = $(wildcard $(SPEC_DIR)/*/*.md) \ $(wildcard $(SPEC_DIR)/_features/*/*/*.md) \ $(wildcard $(SSZ_DIR)/*.md) -ALL_EXECUTABLE_SPEC_NAMES = phase0 altair bellatrix capella deneb electra whisk eip6800 +ALL_EXECUTABLE_SPEC_NAMES = phase0 altair bellatrix capella deneb electra whisk eip6800 eipxxxx # The parameters for commands. Use `foreach` to avoid listing specs again. COVERAGE_SCOPE := $(foreach S,$(ALL_EXECUTABLE_SPEC_NAMES), --cov=eth2spec.$S.$(TEST_PRESET_TYPE)) PYLINT_SCOPE := $(foreach S,$(ALL_EXECUTABLE_SPEC_NAMES), ./eth2spec/$S) diff --git a/specs/_features/eipxxxx/fork-choice.md b/specs/_features/eipxxxx/fork-choice.md index 2988a1798e..ae6113cb61 100644 --- a/specs/_features/eipxxxx/fork-choice.md +++ b/specs/_features/eipxxxx/fork-choice.md @@ -58,7 +58,7 @@ Auxiliary class to consider `(block, slot, bool)` LMD voting class ChildNode(Container): root: Root slot: Slot - is_payload_present: bool + is_payload_present: boolean ``` ## Helpers @@ -108,7 +108,7 @@ class Store(object): execution_payload_states: Dict[Root, BeaconState] = field(default_factory=dict) # [New in EIP-XXXX] ptc_vote: Dict[Root, Vector[uint8, PTC_SIZE]] = field(default_factory=dict) # [New in EIP-XXXX] payload_withhold_boost_root: Root # [New in EIP-XXXX] - payload_withhold_boost_full: bool # [New in EIP-XXXX] + payload_withhold_boost_full: boolean # [New in EIP-XXXX] payload_reveal_boost_root: Root # [New in EIP-XXXX] ``` From 49975887932f87df340634c522f484d5df46abe7 Mon Sep 17 00:00:00 2001 From: Potuz Date: Tue, 2 Jul 2024 10:29:12 -0300 Subject: [PATCH 060/130] add fake value to KZG indices --- specs/_features/eipxxxx/p2p-interface.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/_features/eipxxxx/p2p-interface.md b/specs/_features/eipxxxx/p2p-interface.md index 9d89b0dc5d..eae9552fdc 100644 --- a/specs/_features/eipxxxx/p2p-interface.md +++ b/specs/_features/eipxxxx/p2p-interface.md @@ -35,8 +35,8 @@ This document contains the consensus-layer networking specification for ePBS. | Name | Value | Description | |------------------------------------------|-----------------------------------|---------------------------------------------------------------------| -| `KZG_COMMITMENT_INCLUSION_PROOF_DEPTH` | TODO: Compute it when the spec stabilizes | Merkle proof depth for the `blob_kzg_commitments` list item | -| `KZG_GENERALIZED_INDEX_PREFIX` | TODO: Compute it when the spec stabilizes | Generalized index for the first item in the `blob_kzg_commitments` list | +| `KZG_COMMITMENT_INCLUSION_PROOF_DEPTH` | `0` # TODO: Compute it when the spec stabilizes | Merkle proof depth for the `blob_kzg_commitments` list item | +| `KZG_GENERALIZED_INDEX_PREFIX` | `0` # TODO: Compute it when the spec stabilizes | Generalized index for the first item in the `blob_kzg_commitments` list | ### Containers From ad8d71dbdbd1a0dc02a0f7a85b77a6a3302efdb9 Mon Sep 17 00:00:00 2001 From: Potuz Date: Tue, 2 Jul 2024 14:17:53 -0300 Subject: [PATCH 061/130] Fix python execution 1 --- specs/_features/eipxxxx/beacon-chain.md | 14 ++++-- specs/_features/eipxxxx/fork-choice.md | 66 ++++++++++++++++++------- 2 files changed, 57 insertions(+), 23 deletions(-) diff --git a/specs/_features/eipxxxx/beacon-chain.md b/specs/_features/eipxxxx/beacon-chain.md index 3e4857c30a..b4d25e8a76 100644 --- a/specs/_features/eipxxxx/beacon-chain.md +++ b/specs/_features/eipxxxx/beacon-chain.md @@ -329,9 +329,9 @@ def get_ptc(state: BeaconState, slot: Slot) -> Vector[ValidatorIndex, PTC_SIZE]: committees_per_slot = bit_floor(min(get_committee_count_per_slot(state, epoch), PTC_SIZE)) members_per_committee = PTC_SIZE // committees_per_slot - validator_indices = [] + validator_indices: List[ValidatorIndex] = [] for idx in range(committees_per_slot): - beacon_committee = get_beacon_committee(state, slot, idx) + beacon_committee = get_beacon_committee(state, slot, CommitteeIndex(idx)) validator_indices += beacon_committee[:members_per_committee] return validator_indices ``` @@ -356,7 +356,7 @@ def get_attesting_indices(state: BeaconState, attestation: Attestation) -> Set[V committee_offset += len(committee) ptc = get_ptc(state, attestation.data.slot) - return [i for i in output if i not in ptc] + return Set([i for i in output if i not in ptc]) ``` #### `get_payload_attesting_indices` @@ -402,7 +402,7 @@ The post-state corresponding to a pre-state `state` and a signed execution paylo ```python def process_block(state: BeaconState, block: BeaconBlock) -> None: process_block_header(state, block) - process_withdrawals(state) [Modified in EIP-XXXX] + process_withdrawals(state) # [Modified in EIP-XXXX] process_execution_payload_header(state, block) # [Modified in EIP-XXXX, removed process_execution_payload] process_randao(state, block.body) process_eth1_data(state, block.body) @@ -597,7 +597,7 @@ def process_execution_payload(state: BeaconState, signed_envelope: SignedExecuti assert hash_tree_root(payload.withdrawals) == state.last_withdrawals_root # Verify the gas_limit - assert commited_header.gas_limit == payload.gas_limit + assert committed_header.gas_limit == payload.gas_limit assert committed_header.block_hash == payload.block_hash # Verify consistency of the parent hash with respect to the previous execution payload @@ -617,6 +617,10 @@ def process_execution_payload(state: BeaconState, signed_envelope: SignedExecuti ) # Process Electra operations + def for_ops(operations: Sequence[Any], fn: Callable[[BeaconState, Any], None]) -> None: + for operation in operations: + fn(state, operation) + for_ops(payload.deposit_requests, process_deposit_request) for_ops(payload.withdrawal_requests, process_withdrawal_request) for_ops(payload, process_consolidation_request) diff --git a/specs/_features/eipxxxx/fork-choice.md b/specs/_features/eipxxxx/fork-choice.md index ae6113cb61..ca360cd7eb 100644 --- a/specs/_features/eipxxxx/fork-choice.md +++ b/specs/_features/eipxxxx/fork-choice.md @@ -99,19 +99,49 @@ class Store(object): unrealized_justified_checkpoint: Checkpoint unrealized_finalized_checkpoint: Checkpoint proposer_boost_root: Root + payload_withhold_boost_root: Root # [New in EIP-XXXX] + payload_withhold_boost_full: boolean # [New in EIP-XXXX] + payload_reveal_boost_root: Root # [New in EIP-XXXX] equivocating_indices: Set[ValidatorIndex] blocks: Dict[Root, BeaconBlock] = field(default_factory=dict) block_states: Dict[Root, BeaconState] = field(default_factory=dict) + block_timeliness: Dict[Root, boolean] = field(default_factory=dict) checkpoint_states: Dict[Checkpoint, BeaconState] = field(default_factory=dict) latest_messages: Dict[ValidatorIndex, LatestMessage] = field(default_factory=dict) unrealized_justifications: Dict[Root, Checkpoint] = field(default_factory=dict) execution_payload_states: Dict[Root, BeaconState] = field(default_factory=dict) # [New in EIP-XXXX] ptc_vote: Dict[Root, Vector[uint8, PTC_SIZE]] = field(default_factory=dict) # [New in EIP-XXXX] - payload_withhold_boost_root: Root # [New in EIP-XXXX] - payload_withhold_boost_full: boolean # [New in EIP-XXXX] - payload_reveal_boost_root: Root # [New in EIP-XXXX] ``` - + +### Modified `get_forkchoice_store` + +```python +def get_forkchoice_store(anchor_state: BeaconState, anchor_block: BeaconBlock) -> Store: + assert anchor_block.state_root == hash_tree_root(anchor_state) + anchor_root = hash_tree_root(anchor_block) + anchor_epoch = get_current_epoch(anchor_state) + justified_checkpoint = Checkpoint(epoch=anchor_epoch, root=anchor_root) + finalized_checkpoint = Checkpoint(epoch=anchor_epoch, root=anchor_root) + proposer_boost_root = Root() + return Store( + time=uint64(anchor_state.genesis_time + SECONDS_PER_SLOT * anchor_state.slot), + genesis_time=anchor_state.genesis_time, + justified_checkpoint=justified_checkpoint, + finalized_checkpoint=finalized_checkpoint, + unrealized_justified_checkpoint=justified_checkpoint, + unrealized_finalized_checkpoint=finalized_checkpoint, + proposer_boost_root=proposer_boost_root, + payload_withhold_boost_root=proposer_boost_root, # [New in EIP-XXXX] + payload_withhold_boost_full=True, # [New in EIP-XXXX] + payload_reveal_boost_root=proposer_boost_root, # [New in EIP-XXXX] + equivocating_indices=set(), + blocks={anchor_root: copy(anchor_block)}, + block_states={anchor_root: copy(anchor_state)}, + checkpoint_states={justified_checkpoint: copy(anchor_state)}, + unrealized_justifications={anchor_root: justified_checkpoint}, + ) +``` + ### `notify_ptc_messages` ```python @@ -123,10 +153,10 @@ def notify_ptc_messages(store: Store, state: BeaconState, payload_attestations: if state.slot == 0: return for payload_attestation in payload_attestations: - indexed_payload_attestation = get_indexed_payload_attestation(state, state.slot - 1, payload_attestation) + indexed_payload_attestation = get_indexed_payload_attestation(state, Slot(state.slot - 1), payload_attestation) for idx in indexed_payload_attestation.attesting_indices: - store.on_payload_attestation_message(PayloadAttestationMessage(validator_index=idx, - data=payload_attestation.data, signature= BLSSignature(), is_from_block=true)) + on_payload_attestation_message(store, PayloadAttestationMessage(validator_index=idx, + data=payload_attestation.data, signature= BLSSignature(), is_from_block=True)) ``` ### `is_payload_present` @@ -167,7 +197,7 @@ def get_ancestor(store: Store, root: Root, slot: Slot) -> ChildNode: parent = store.blocks[block.parent_root] if parent.slot > slot: return get_ancestor(store, block.parent_root, slot) - return ChildNode(root=block.parent_root, slot=parent.slot, is_payload_present=is_parent_node_full(block)) + return ChildNode(root=block.parent_root, slot=parent.slot, is_payload_present=is_parent_node_full(store, block)) ``` ### Modified `get_checkpoint_block` @@ -205,7 +235,7 @@ def is_supporting_vote(store: Store, node: ChildNode, message: LatestMessage) -> ### New `compute_proposer_boost` This is a helper to compute the proposer boost. It applies the proposer boost to any ancestor of the proposer boost root taking into account the payload presence. There is one exception: if the requested node has the same root and slot as the block with the proposer boost root, then the proposer boost is applied to both empty and full versions of the node. ```python -def compute_proposer_boost(store: Store, state: State, node: ChildNode) -> Gwei: +def compute_proposer_boost(store: Store, state: BeaconState, node: ChildNode) -> Gwei: if store.proposer_boost_root == Root(): return Gwei(0) ancestor = get_ancestor(store, store.proposer_boost_root, node.slot) @@ -225,10 +255,10 @@ def compute_proposer_boost(store: Store, state: State, node: ChildNode) -> Gwei: This is a similar helper that applies for the withhold boost. In this case this always takes into account the reveal status. ```python -def compute_withhold_boost(store: Store, state: State, node: ChildNode) -> Gwei: +def compute_withhold_boost(store: Store, state: BeaconState, node: ChildNode) -> Gwei: if store.payload_withhold_boost_root == Root(): return Gwei(0) - ancestor = get_ancestor(store, store.payload_withold_boost_root, node.slot) + ancestor = get_ancestor(store, store.payload_withhold_boost_root, node.slot) if ancestor.root != node.root: return Gwei(0) if node.slot >= store.blocks[store.payload_withhold_boost_root].slot: @@ -244,7 +274,7 @@ def compute_withhold_boost(store: Store, state: State, node: ChildNode) -> Gwei: This is a similar helper to the last two, the only difference is that the reveal boost is only applied to the full version of the node when querying for the same slot as the revealed payload. ```python -def compute_reveal_boost(store: Store, state: State, node: ChildNode) -> Gwei: +def compute_reveal_boost(store: Store, state: BeaconState, node: ChildNode) -> Gwei: if store.payload_reveal_boost_root == Root(): return Gwei(0) ancestor = get_ancestor(store, store.payload_reveal_boost_root, node.slot) @@ -252,7 +282,7 @@ def compute_reveal_boost(store: Store, state: State, node: ChildNode) -> Gwei: return Gwei(0) if node.slot >= store.blocks[store.payload_reveal_boost_root].slot: ancestor.is_payload_present = True - if is_ancestor_full != node.is_payload_present: + if ancestor.is_payload_present != node.is_payload_present: return Gwei(0) committee_weight = get_total_active_balance(state) // SLOTS_PER_EPOCH return (committee_weight * PAYLOAD_REVEAL_BOOST) // 100 @@ -267,7 +297,7 @@ def get_weight(store: Store, node: ChildNode) -> Gwei: state = store.checkpoint_states[store.justified_checkpoint] unslashed_and_active_indices = [ i for i in get_active_validator_indices(state, get_current_epoch(state)) - if not is_slashed_attester(state.validators[i]) + if not state.validators[i].slashed ] attestation_score = Gwei(sum( state.validators[i].effective_balance for i in unslashed_and_active_indices @@ -277,7 +307,7 @@ def get_weight(store: Store, node: ChildNode) -> Gwei: )) # Compute boosts - proposer_score = compute_boost(store, state, node) + proposer_score = compute_proposer_boost(store, state, node) builder_reveal_score = compute_reveal_boost(store, state, node) builder_withhold_score = compute_withhold_boost(store, state, node) @@ -302,7 +332,7 @@ def get_head(store: Store) -> ChildNode: children = [ ChildNode(root=root, slot=block.slot, is_payload_present=present) for (root, block) in blocks.items() if block.parent_root == best_child.root and - is_parent_node_full(store, block) == best_child.is_payload_present if root != store.justified_root + is_parent_node_full(store, block) == best_child.is_payload_present if root != store.justified_checkpoint.root for present in (True, False) if root in store.execution_payload_states or present == False ] if len(children) == 0: @@ -374,7 +404,7 @@ def on_block(store: Store, signed_block: SignedBeaconBlock) -> None: store.ptc_vote[block_root] = [PAYLOAD_ABSENT]*PTC_SIZE # Notify the store about the payload_attestations in the block - store.notify_ptc_messages(state, block.body.payload_attestations) + notify_ptc_messages(store, state, block.body.payload_attestations) # Add proposer score boost if the block is timely time_into_slot = (store.time - store.genesis_time) % SECONDS_PER_SLOT is_before_attesting_interval = time_into_slot < SECONDS_PER_SLOT // INTERVALS_PER_SLOT @@ -497,5 +527,5 @@ def on_payload_attestation_message(store: Store, if ptc_vote.count(PAYLOAD_WITHHELD) > PAYLOAD_TIMELY_THRESHOLD: block = store.blocks[data.beacon_block_root] store.payload_withhold_boost_root = block.parent_root - store.payload_withhold_boost_full = is_parent_node_full(block) + store.payload_withhold_boost_full = is_parent_node_full(store, block) ``` From 2a43ce54d6e8dc9131fe6a58527cc6b13e5171e2 Mon Sep 17 00:00:00 2001 From: Potuz Date: Tue, 2 Jul 2024 14:36:30 -0300 Subject: [PATCH 062/130] doctoc --- specs/_features/eipxxxx/fork-choice.md | 1 + 1 file changed, 1 insertion(+) diff --git a/specs/_features/eipxxxx/fork-choice.md b/specs/_features/eipxxxx/fork-choice.md index ca360cd7eb..52b7de0371 100644 --- a/specs/_features/eipxxxx/fork-choice.md +++ b/specs/_features/eipxxxx/fork-choice.md @@ -13,6 +13,7 @@ - [Modified `LatestMessage`](#modified-latestmessage) - [Modified `update_latest_messages`](#modified-update_latest_messages) - [Modified `Store`](#modified-store) + - [Modified `get_forkchoice_store`](#modified-get_forkchoice_store) - [`notify_ptc_messages`](#notify_ptc_messages) - [`is_payload_present`](#is_payload_present) - [`is_parent_node_full`](#is_parent_node_full) From 4500eabf8cc70b925e68133d4793336f92f9ce7c Mon Sep 17 00:00:00 2001 From: Potuz Date: Tue, 2 Jul 2024 17:30:59 -0300 Subject: [PATCH 063/130] Fix python execution 2 --- specs/_features/eipxxxx/beacon-chain.md | 54 ++++++++++++++----------- specs/_features/eipxxxx/builder.md | 6 ++- specs/_features/eipxxxx/fork-choice.md | 43 ++++++++++++-------- 3 files changed, 61 insertions(+), 42 deletions(-) diff --git a/specs/_features/eipxxxx/beacon-chain.md b/specs/_features/eipxxxx/beacon-chain.md index b4d25e8a76..88075df5a9 100644 --- a/specs/_features/eipxxxx/beacon-chain.md +++ b/specs/_features/eipxxxx/beacon-chain.md @@ -80,20 +80,20 @@ At any given slot, the status of the blockchain's head may be either | Name | Value | | - | - | -| `PTC_SIZE` | `uint64(2**9)` (=512) # (New in EIP-XXXX) | +| `PTC_SIZE` | `uint64(2**9)` (=512) # (New in EIP-XXXX) | ### Domain types | Name | Value | | - | - | -| `DOMAIN_BEACON_BUILDER` | `DomainType('0x1B000000')` # (New in EIP-XXXX)| -| `DOMAIN_PTC_ATTESTER` | `DomainType('0x0C000000')` # (New in EIP-XXXX)| +| `DOMAIN_BEACON_BUILDER` | `DomainType('0x1B000000')` # (New in EIP-XXXX)| +| `DOMAIN_PTC_ATTESTER` | `DomainType('0x0C000000')` # (New in EIP-XXXX)| ### Max operations per block | Name | Value | | - | - | -| `MAX_PAYLOAD_ATTESTATIONS` | `2**2` (= 4) # (New in EIP-XXXX) | +| `MAX_PAYLOAD_ATTESTATIONS` | `2**2` (= 4) # (New in EIP-XXXX) | ## Containers @@ -186,8 +186,8 @@ class BeaconBlockBody(Container): # Removed blob_kzg_commitments [Removed in EIP-XXXX] bls_to_execution_changes: List[SignedBLSToExecutionChange, MAX_BLS_TO_EXECUTION_CHANGES] # PBS - signed_execution_payload_header: SignedExecutionPayloadHeader # [New in EIP-XXXX] - payload_attestations: List[PayloadAttestation, MAX_PAYLOAD_ATTESTATIONS] # [New in EIP-XXXX] + signed_execution_payload_header: SignedExecutionPayloadHeader # [New in EIP-XXXX] + payload_attestations: List[PayloadAttestation, MAX_PAYLOAD_ATTESTATIONS] # [New in EIP-XXXX] ``` #### `ExecutionPayloadHeader` @@ -262,9 +262,9 @@ class BeaconState(Container): pending_partial_withdrawals: List[PendingPartialWithdrawal, PENDING_PARTIAL_WITHDRAWALS_LIMIT] pending_consolidations: List[PendingConsolidation, PENDING_CONSOLIDATIONS_LIMIT] # PBS - latest_block_hash: Hash32 # [New in EIP-XXXX] - latest_full_slot: Slot # [New in EIP-XXXX] - latest_withdrawals_root: Root # [New in EIP-XXXX] + latest_block_hash: Hash32 # [New in EIP-XXXX] + latest_full_slot: Slot # [New in EIP-XXXX] + latest_withdrawals_root: Root # [New in EIP-XXXX] ``` ## Helper functions @@ -288,9 +288,12 @@ def bit_floor(n: uint64) -> uint64: #### `is_valid_indexed_payload_attestation` ```python -def is_valid_indexed_payload_attestation(state: BeaconState, indexed_payload_attestation: IndexedPayloadAttestation) -> bool: +def is_valid_indexed_payload_attestation( + state: BeaconState, + indexed_payload_attestation: IndexedPayloadAttestation) -> bool: """ - Check if ``indexed_payload_attestation`` is not empty, has sorted and unique indices and has a valid aggregate signature. + Check if ``indexed_payload_attestation`` is not empty, has sorted and unique indices and has + a valid aggregate signature. """ # Verify the data is valid if indexed_payload_attestation.data.payload_status >= PAYLOAD_INVALID_STATUS: @@ -402,8 +405,8 @@ The post-state corresponding to a pre-state `state` and a signed execution paylo ```python def process_block(state: BeaconState, block: BeaconBlock) -> None: process_block_header(state, block) - process_withdrawals(state) # [Modified in EIP-XXXX] - process_execution_payload_header(state, block) # [Modified in EIP-XXXX, removed process_execution_payload] + process_withdrawals(state) # [Modified in EIP-XXXX] + process_execution_payload_header(state, block) # [Modified in EIP-XXXX, removed process_execution_payload] process_randao(state, block.body) process_eth1_data(state, block.body) process_operations(state, block.body) # [Modified in EIP-XXXX] @@ -415,7 +418,7 @@ def process_block(state: BeaconState, block: BeaconBlock) -> None: ```python def process_withdrawals(state: BeaconState) -> None: - ## return early if the parent block was empty + # return early if the parent block was empty if not is_parent_block_full(state): return @@ -448,7 +451,7 @@ def process_withdrawals(state: BeaconState) -> None: ```python def verify_execution_payload_header_signature(state: BeaconState, - signed_header: SignedExecutionPayloadHeader) -> bool: + signed_header: SignedExecutionPayloadHeader) -> bool: # Check the signature builder = state.validators[signed_header.message.builder_index] signing_root = compute_signing_root(signed_header.message, get_domain(state, DOMAIN_BEACON_BUILDER)) @@ -505,7 +508,7 @@ def process_operations(state: BeaconState, body: BeaconBlockBody) -> None: # Removed `process_deposit_request` in EIP-XXXX # Removed `process_withdrawal_request` in EIP-XXXX # Removed `process_consolidation_request` in EIP-XXXX - for_ops(body.payload_attestations, process_payload_attestation) # [New in EIP-XXXX] + for_ops(body.payload_attestations, process_payload_attestation) # [New in EIP-XXXX] ``` ##### Payload Attestations @@ -518,17 +521,16 @@ def remove_flag(flags: ParticipationFlags, flag_index: int) -> ParticipationFlag ```python def process_payload_attestation(state: BeaconState, payload_attestation: PayloadAttestation) -> None: - ## Check that the attestation is for the parent beacon block + # Check that the attestation is for the parent beacon block data = payload_attestation.data assert data.beacon_block_root == state.latest_block_header.parent_root - ## Check that the attestation is for the previous slot + # Check that the attestation is for the previous slot assert data.slot + 1 == state.slot - #Verify signature + # Verify signature indexed_payload_attestation = get_indexed_payload_attestation(state, data.slot, payload_attestation) assert is_valid_indexed_payload_attestation(state, indexed_payload_attestation) - ptc = get_ptc(state, data.slot) if state.slot % SLOTS_PER_EPOCH == 0: epoch_participation = state.previous_epoch_participation else: @@ -548,7 +550,7 @@ def process_payload_attestation(state: BeaconState, payload_attestation: Payload epoch_participation[index] = remove_flag(epoch_participation[index], flag_index) proposer_penalty_numerator += get_base_reward(state, index) * weight # Penalize the proposer - proposer_penalty = Gwei(2*proposer_penalty_numerator // proposer_reward_denominator) + proposer_penalty = Gwei(2 * proposer_penalty_numerator // proposer_reward_denominator) decrease_balance(state, proposer_index, proposer_penalty) return @@ -568,7 +570,8 @@ def process_payload_attestation(state: BeaconState, payload_attestation: Payload #### New `verify_execution_payload_envelope_signature` ```python -def verify_execution_payload_envelope_signature(state: BeaconState, signed_envelope: SignedExecutionPayloadEnvelope) -> bool: +def verify_execution_payload_envelope_signature( + state: BeaconState, signed_envelope: SignedExecutionPayloadEnvelope) -> bool: builder = state.validators[signed_envelope.message.builder_index] signing_root = compute_signing_root(signed_envelope.message, get_domain(state, DOMAIN_BEACON_BUILDER)) return bls.Verify(builder.pubkey, signing_root, signed_envelope.signature) @@ -578,7 +581,9 @@ def verify_execution_payload_envelope_signature(state: BeaconState, signed_envel *Note*: `process_execution_payload` is now an independent check in state transition. It is called when importing a signed execution payload proposed by the builder of the current slot. ```python -def process_execution_payload(state: BeaconState, signed_envelope: SignedExecutionPayloadEnvelope, execution_engine: ExecutionEngine, verify = True) -> None: +def process_execution_payload(state: BeaconState, + signed_envelope: SignedExecutionPayloadEnvelope, + execution_engine: ExecutionEngine, verify=True) -> None: # Verify signature if verify: assert verify_execution_payload_envelope_signature(state, signed_envelope) @@ -607,7 +612,8 @@ def process_execution_payload(state: BeaconState, signed_envelope: SignedExecuti # Verify timestamp assert payload.timestamp == compute_timestamp_at_slot(state, state.slot) # Verify the execution payload is valid - versioned_hashes = [kzg_commitment_to_versioned_hash(commitment) for commitment in envelope.blob_kzg_commitments] + versioned_hashes = [kzg_commitment_to_versioned_hash(commitment) + for commitment in envelope.blob_kzg_commitments] assert execution_engine.verify_and_notify_new_payload( NewPayloadRequest( execution_payload=payload, diff --git a/specs/_features/eipxxxx/builder.md b/specs/_features/eipxxxx/builder.md index e44111bd90..35cf296039 100644 --- a/specs/_features/eipxxxx/builder.md +++ b/specs/_features/eipxxxx/builder.md @@ -41,7 +41,8 @@ Builders can broadcast a payload bid for the current or the next slot's proposer After building the `header`, the builder obtains a `signature` of the header by using ```python -def get_execution_payload_header_signature(state: BeaconState, header: ExecutionPayloadHeader, privkey: int) -> BLSSignature: +def get_execution_payload_header_signature( + state: BeaconState, header: ExecutionPayloadHeader, privkey: int) -> BLSSignature: domain = get_domain(state, DOMAIN_BEACON_BUILDER, compute_epoch_at_slot(header.slot)) signing_root = compute_signing_root(header, domain) return bls.Sign(privkey, signing_root) @@ -101,7 +102,8 @@ After setting these parameters, the builder should run `process_execution_payloa 6. Set `state_root` to `hash_tree_root(state)`. After preparing the `envelope` the builder should sign the envelope using: ```python -def get_execution_payload_envelope_signature(state: BeaconState, envelope: ExecutionPayloadEnvelope, privkey: int) -> BLSSignature: +def get_execution_payload_envelope_signature( + state: BeaconState, envelope: ExecutionPayloadEnvelope, privkey: int) -> BLSSignature: domain = get_domain(state, DOMAIN_BEACON_BUILDER, compute_epoch_at_slot(state.slot)) signing_root = compute_signing_root(envelope, domain) return bls.Sign(privkey, signing_root) diff --git a/specs/_features/eipxxxx/fork-choice.md b/specs/_features/eipxxxx/fork-choice.md index 52b7de0371..2ddc79a98f 100644 --- a/specs/_features/eipxxxx/fork-choice.md +++ b/specs/_features/eipxxxx/fork-choice.md @@ -44,7 +44,7 @@ This is the modification of the fork choice accompanying the ePBS upgrade. | Name | Value | | -------------------- | ----------- | -| `PAYLOAD_TIMELY_THRESHOLD` | `PTC_SIZE/2` (=`uint64(256)`) | +| `PAYLOAD_TIMELY_THRESHOLD` | `PTC_SIZE / 2` (=`uint64(256)`) | | `INTERVALS_PER_SLOT` | `4` # [modified in EIP-XXXX] | | `PROPOSER_SCORE_BOOST` | `20` # [modified in EIP-XXXX] | | `PAYLOAD_WITHHOLD_BOOST` | `40` | @@ -100,9 +100,9 @@ class Store(object): unrealized_justified_checkpoint: Checkpoint unrealized_finalized_checkpoint: Checkpoint proposer_boost_root: Root - payload_withhold_boost_root: Root # [New in EIP-XXXX] - payload_withhold_boost_full: boolean # [New in EIP-XXXX] - payload_reveal_boost_root: Root # [New in EIP-XXXX] + payload_withhold_boost_root: Root # [New in EIP-XXXX] + payload_withhold_boost_full: boolean # [New in EIP-XXXX] + payload_reveal_boost_root: Root # [New in EIP-XXXX] equivocating_indices: Set[ValidatorIndex] blocks: Dict[Root, BeaconBlock] = field(default_factory=dict) block_states: Dict[Root, BeaconState] = field(default_factory=dict) @@ -110,8 +110,8 @@ class Store(object): checkpoint_states: Dict[Checkpoint, BeaconState] = field(default_factory=dict) latest_messages: Dict[ValidatorIndex, LatestMessage] = field(default_factory=dict) unrealized_justifications: Dict[Root, Checkpoint] = field(default_factory=dict) - execution_payload_states: Dict[Root, BeaconState] = field(default_factory=dict) # [New in EIP-XXXX] - ptc_vote: Dict[Root, Vector[uint8, PTC_SIZE]] = field(default_factory=dict) # [New in EIP-XXXX] + execution_payload_states: Dict[Root, BeaconState] = field(default_factory=dict) # [New in EIP-XXXX] + ptc_vote: Dict[Root, Vector[uint8, PTC_SIZE]] = field(default_factory=dict) # [New in EIP-XXXX] ``` ### Modified `get_forkchoice_store` @@ -132,9 +132,9 @@ def get_forkchoice_store(anchor_state: BeaconState, anchor_block: BeaconBlock) - unrealized_justified_checkpoint=justified_checkpoint, unrealized_finalized_checkpoint=finalized_checkpoint, proposer_boost_root=proposer_boost_root, - payload_withhold_boost_root=proposer_boost_root, # [New in EIP-XXXX] - payload_withhold_boost_full=True, # [New in EIP-XXXX] - payload_reveal_boost_root=proposer_boost_root, # [New in EIP-XXXX] + payload_withhold_boost_root=proposer_boost_root, # [New in EIP-XXXX] + payload_withhold_boost_full=True, # [New in EIP-XXXX] + payload_reveal_boost_root=proposer_boost_root, # [New in EIP-XXXX] equivocating_indices=set(), blocks={anchor_root: copy(anchor_block)}, block_states={anchor_root: copy(anchor_state)}, @@ -156,8 +156,9 @@ def notify_ptc_messages(store: Store, state: BeaconState, payload_attestations: for payload_attestation in payload_attestations: indexed_payload_attestation = get_indexed_payload_attestation(state, Slot(state.slot - 1), payload_attestation) for idx in indexed_payload_attestation.attesting_indices: - on_payload_attestation_message(store, PayloadAttestationMessage(validator_index=idx, - data=payload_attestation.data, signature= BLSSignature(), is_from_block=True)) + on_payload_attestation_message(store, + PayloadAttestationMessage(validator_index=idx, + data=payload_attestation.data, signature=BLSSignature(), is_from_block=True)) ``` ### `is_payload_present` @@ -333,19 +334,29 @@ def get_head(store: Store) -> ChildNode: children = [ ChildNode(root=root, slot=block.slot, is_payload_present=present) for (root, block) in blocks.items() if block.parent_root == best_child.root and - is_parent_node_full(store, block) == best_child.is_payload_present if root != store.justified_checkpoint.root - for present in (True, False) if root in store.execution_payload_states or present == False + is_parent_node_full(store, block) == best_child.is_payload_present if + root != store.justified_checkpoint.root + for present in (True, False) if root in store.execution_payload_states or not present ] if len(children) == 0: return best_child # if we have children we consider the current head advanced as a possible head - children += [ChildNode(root=best_child.root, slot=best_child.slot + 1, is_payload_present=best_child.is_payload_present)] + children += [ + ChildNode(root=best_child.root, slot=best_child.slot + 1, is_payload_present=best_child.is_payload_present) + ] # Sort by latest attesting balance with ties broken lexicographically # Ties broken by favoring full blocks according to the PTC vote # Ties are then broken by favoring full blocks # Ties broken then by favoring higher slot numbers # Ties then broken by favoring block with lexicographically higher root - new_best_child = max(children, key=lambda child: (get_weight(store, child), is_payload_present(store, child.root), child.is_payload_present, child.slot, child.root)) + new_best_child = max(children, key=lambda child: ( + get_weight(store, child), + is_payload_present(store, child.root), + child.is_payload_present, + child.slot, + child.root + ) + ) if new_best_child.root == best_child.root: return new_best_child best_child = new_best_child @@ -402,7 +413,7 @@ def on_block(store: Store, signed_block: SignedBeaconBlock) -> None: # Add new state for this block to the store store.block_states[block_root] = state # Add a new PTC voting for this block to the store - store.ptc_vote[block_root] = [PAYLOAD_ABSENT]*PTC_SIZE + store.ptc_vote[block_root] = [PAYLOAD_ABSENT] * PTC_SIZE # Notify the store about the payload_attestations in the block notify_ptc_messages(store, state, block.body.payload_attestations) From d7a199fe0948b483e1502edc3500f6cb3d145f48 Mon Sep 17 00:00:00 2001 From: Potuz Date: Tue, 2 Jul 2024 21:04:01 -0300 Subject: [PATCH 064/130] Fix python execution 3 --- specs/_features/eipxxxx/fork-choice.md | 39 +++++++++++++++++--------- specs/_features/eipxxxx/fork.md | 8 +++--- specs/_features/eipxxxx/validator.md | 15 +++++----- 3 files changed, 38 insertions(+), 24 deletions(-) diff --git a/specs/_features/eipxxxx/fork-choice.md b/specs/_features/eipxxxx/fork-choice.md index 2ddc79a98f..4ee870ffef 100644 --- a/specs/_features/eipxxxx/fork-choice.md +++ b/specs/_features/eipxxxx/fork-choice.md @@ -156,9 +156,15 @@ def notify_ptc_messages(store: Store, state: BeaconState, payload_attestations: for payload_attestation in payload_attestations: indexed_payload_attestation = get_indexed_payload_attestation(state, Slot(state.slot - 1), payload_attestation) for idx in indexed_payload_attestation.attesting_indices: - on_payload_attestation_message(store, - PayloadAttestationMessage(validator_index=idx, - data=payload_attestation.data, signature=BLSSignature(), is_from_block=True)) + on_payload_attestation_message( + store, + PayloadAttestationMessage( + validator_index=idx, + data=payload_attestation.data, + signature=BLSSignature(), + is_from_block=True + ) + ) ``` ### `is_payload_present` @@ -179,7 +185,9 @@ def is_payload_present(store: Store, beacon_block_root: Root) -> bool: ```python def is_parent_node_full(store: Store, block: BeaconBlock) -> bool: parent = store.blocks[block.parent_root] - return block.body.signed_execution_payload_header.message.parent_block_hash == parent.body.signed_execution_payload_header.message.block_hash + parent_block_hash = block.body.signed_execution_payload_header.message.parent_block_hash + message_block_hash = parent.body.signed_execution_payload_header.message.block_hash + return parent_block_hash == message_block_hash ``` ### Modified `get_ancestor` @@ -230,7 +238,7 @@ def is_supporting_vote(store: Store, node: ChildNode, message: LatestMessage) -> message_block = store.blocks[message.root] if node.slot >= message_block.slot: return False - ancestor = get_ancestor(store, message.root, node.slot) + ancestor = get_ancestor(store, message.root, node.slot) return (node.root == ancestor.root) and (node.is_payload_present == ancestor.is_payload_present) ``` @@ -250,7 +258,7 @@ def compute_proposer_boost(store: Store, state: BeaconState, node: ChildNode) -> if (node.slot < proposer_boost_slot) and (ancestor.is_payload_present != node.is_payload_present): return Gwei(0) committee_weight = get_total_active_balance(state) // SLOTS_PER_EPOCH - return (committee_weight * PROPOSER_SCORE_BOOST) // 100 + return (committee_weight * PROPOSER_SCORE_BOOST) // 100 ``` ### New `compute_withhold_boost` @@ -269,7 +277,7 @@ def compute_withhold_boost(store: Store, state: BeaconState, node: ChildNode) -> return Gwei(0) committee_weight = get_total_active_balance(state) // SLOTS_PER_EPOCH - return (committee_weight * PAYLOAD_WITHHOLD_BOOST) // 100 + return (committee_weight * PAYLOAD_WITHHOLD_BOOST) // 100 ``` ### New `compute_reveal_boost` @@ -287,7 +295,7 @@ def compute_reveal_boost(store: Store, state: BeaconState, node: ChildNode) -> G if ancestor.is_payload_present != node.is_payload_present: return Gwei(0) committee_weight = get_total_active_balance(state) // SLOTS_PER_EPOCH - return (committee_weight * PAYLOAD_REVEAL_BOOST) // 100 + return (committee_weight * PAYLOAD_REVEAL_BOOST) // 100 ``` ### Modified `get_weight` @@ -497,8 +505,8 @@ def on_tick_per_slot(store: Store, time: uint64) -> None: ### `on_payload_attestation_message` ```python -def on_payload_attestation_message(store: Store, - ptc_message: PayloadAttestationMessage, is_from_block: bool=False) -> None: +def on_payload_attestation_message( + store: Store, ptc_message: PayloadAttestationMessage, is_from_block: bool=False) -> None: """ Run ``on_payload_attestation_message`` upon receiving a new ``ptc_message`` directly on the wire. """ @@ -518,9 +526,14 @@ def on_payload_attestation_message(store: Store, # Check that the attestation is for the current slot assert data.slot == get_current_slot(store) # Verify the signature - assert is_valid_indexed_payload_attestation(state, - IndexedPayloadAttestation(attesting_indices = [ptc_message.validator_index], data = data, - signature = ptc_message.signature)) + assert is_valid_indexed_payload_attestation( + state, + IndexedPayloadAttestation( + attesting_indices=[ptc_message.validator_index], + data=data, + signature=ptc_message.signature + ) + ) # Update the ptc vote for the block ptc_index = ptc.index(ptc_message.validator_index) ptc_vote = store.ptc_vote[data.beacon_block_root] diff --git a/specs/_features/eipxxxx/fork.md b/specs/_features/eipxxxx/fork.md index f08f5da7c0..7f7966b9a6 100644 --- a/specs/_features/eipxxxx/fork.md +++ b/specs/_features/eipxxxx/fork.md @@ -114,16 +114,16 @@ def upgrade_to_epbs(pre: deneb.BeaconState) -> BeaconState: current_sync_committee=pre.current_sync_committee, next_sync_committee=pre.next_sync_committee, # Execution-layer - latest_execution_payload_header=ExecutionPayloadHeader(), # [Modified in EIP-XXXX] + latest_execution_payload_header=ExecutionPayloadHeader(), # [Modified in EIP-XXXX] # Withdrawals next_withdrawal_index=pre.next_withdrawal_index, next_withdrawal_validator_index=pre.next_withdrawal_validator_index, # Deep history valid from Capella onwards historical_summaries=pre.historical_summaries, # ePBS - latest_block_hash=pre.latest_execution_payload_header.block_hash, # [New in EIP-XXXX] - latest_full_slot=pre.slot, # [New in EIP-XXXX] - last_withdrawals_root=Root(), # [New in EIP-XXXX] + latest_block_hash=pre.latest_execution_payload_header.block_hash, # [New in EIP-XXXX] + latest_full_slot=pre.slot, # [New in EIP-XXXX] + last_withdrawals_root=Root(), # [New in EIP-XXXX] ) return post diff --git a/specs/_features/eipxxxx/validator.md b/specs/_features/eipxxxx/validator.md index 78c9f178b2..8982067729 100644 --- a/specs/_features/eipxxxx/validator.md +++ b/specs/_features/eipxxxx/validator.md @@ -27,10 +27,10 @@ A validator may be a member of the new Payload Timeliness Committee (PTC) for a PTC committee selection is only stable within the context of the current and next epoch. ```python -def get_ptc_assignment(state: BeaconState, - epoch: Epoch, - validator_index: ValidatorIndex - ) -> Optional[Slot]: +def get_ptc_assignment( + state: BeaconState, + epoch: Epoch, + validator_index: ValidatorIndex) -> Optional[Slot]: """ Returns the slot during the requested epoch in which the validator with index `validator_index` is a member of the PTC. Returns None if no assignment is found. @@ -40,8 +40,8 @@ def get_ptc_assignment(state: BeaconState, start_slot = compute_start_slot_at_epoch(epoch) for slot in range(start_slot, start_slot + SLOTS_PER_EPOCH): - if validator_index in get_ptc(state, Slot(slot)): - return Slot(slot) + if validator_index in get_ptc(state, Slot(slot)): + return Slot(slot) return None ``` @@ -122,7 +122,8 @@ The validator creates `payload_attestation_message` as follows: Notice that the attester only signs the `PayloadAttestationData` and not the `validator_index` field in the message. Proposers need to aggregate these attestations as described above. ```python -def get_payload_attestation_message_signature(state: BeaconState, attestation: PayloadAttestationMessage, privkey: int) -> BLSSignature: +def get_payload_attestation_message_signature( + state: BeaconState, attestation: PayloadAttestationMessage, privkey: int) -> BLSSignature: domain = get_domain(state, DOMAIN_PTC_ATTESTER, compute_epoch_at_slot(attestation.data.slot)) signing_root = compute_signing_root(attestation.data, domain) return bls.Sign(privkey, signing_root) From 5013e5724e52801f0f18df92e5239434563ce839 Mon Sep 17 00:00:00 2001 From: terence Date: Wed, 3 Jul 2024 03:53:04 -0700 Subject: [PATCH 065/130] Fix epbs pre fork (#13) --- .gitignore | 1 + pysetup/spec_builders/eipxxxx.py | 2 +- specs/_features/eipxxxx/fork.md | 13 +++++++++++-- 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/.gitignore b/.gitignore index 3586b356ca..e6fad84fe9 100644 --- a/.gitignore +++ b/.gitignore @@ -25,6 +25,7 @@ tests/core/pyspec/eth2spec/electra/ tests/core/pyspec/eth2spec/whisk/ tests/core/pyspec/eth2spec/eip7594/ tests/core/pyspec/eth2spec/eip6800/ +tests/core/pyspec/eth2spec/eipxxxx/ # coverage reports .htmlcov diff --git a/pysetup/spec_builders/eipxxxx.py b/pysetup/spec_builders/eipxxxx.py index dcb267d736..594d3e43a9 100644 --- a/pysetup/spec_builders/eipxxxx.py +++ b/pysetup/spec_builders/eipxxxx.py @@ -10,7 +10,7 @@ class EIPXXXXSpecBuilder(BaseSpecBuilder): @classmethod def imports(cls, preset_name: str): return f''' -from eth2spec.eipxxxx import {preset_name} as eipxxxx +from eth2spec.electra import {preset_name} as electra ''' @classmethod diff --git a/specs/_features/eipxxxx/fork.md b/specs/_features/eipxxxx/fork.md index 7f7966b9a6..fc719b3eae 100644 --- a/specs/_features/eipxxxx/fork.md +++ b/specs/_features/eipxxxx/fork.md @@ -71,8 +71,8 @@ If `state.slot % SLOTS_PER_EPOCH == 0` and `compute_epoch_at_slot(state.slot) == an irregular state change is made to upgrade to ePBS. ```python -def upgrade_to_epbs(pre: deneb.BeaconState) -> BeaconState: - epoch = deneb.get_current_epoch(pre) +def upgrade_to_epbs(pre: electra.BeaconState) -> BeaconState: + epoch = electra.get_current_epoch(pre) post = BeaconState( # Versioning @@ -120,6 +120,15 @@ def upgrade_to_epbs(pre: deneb.BeaconState) -> BeaconState: next_withdrawal_validator_index=pre.next_withdrawal_validator_index, # Deep history valid from Capella onwards historical_summaries=pre.historical_summaries, + deposit_requests_start_index=pre.deposit_requests_start_index, + deposit_balance_to_consumer=pre.deposit_balance_to_consumer, + exit_balance_to_consumer=pre.exit_balance_to_consumer, + earliest_exit_epoch=pre.earliest_exit_epoch, + consolidation_balance_to_consumer=pre.consolidation_balance_to_consumer, + earliest_consolidation_epoch=pre.earliest_consolidation_epoch, + pending_balance_deposits=pre.pending_balance_deposits, + pending_partial_withdrawals=pre.pending_partial_withdrawals, + pending_consolidations=pre.pending_consolidations, # ePBS latest_block_hash=pre.latest_execution_payload_header.block_hash, # [New in EIP-XXXX] latest_full_slot=pre.slot, # [New in EIP-XXXX] From 181ce5e9dcee6396cdb0837c6f5cc54bb08f1d30 Mon Sep 17 00:00:00 2001 From: Potuz Date: Wed, 3 Jul 2024 08:03:24 -0300 Subject: [PATCH 066/130] add type annotation --- specs/_features/eipxxxx/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/_features/eipxxxx/beacon-chain.md b/specs/_features/eipxxxx/beacon-chain.md index 88075df5a9..efcf59b914 100644 --- a/specs/_features/eipxxxx/beacon-chain.md +++ b/specs/_features/eipxxxx/beacon-chain.md @@ -583,7 +583,7 @@ def verify_execution_payload_envelope_signature( ```python def process_execution_payload(state: BeaconState, signed_envelope: SignedExecutionPayloadEnvelope, - execution_engine: ExecutionEngine, verify=True) -> None: + execution_engine: ExecutionEngine, verify: bool = True) -> None: # Verify signature if verify: assert verify_execution_payload_envelope_signature(state, signed_envelope) From a11ffb5278a5d0bfed341187b897ddfd0c0cc2c6 Mon Sep 17 00:00:00 2001 From: Potuz Date: Wed, 3 Jul 2024 08:27:20 -0300 Subject: [PATCH 067/130] use definitive EIP number --- Makefile | 2 +- .../mainnet/{eip-xxxx.yaml => eip-7732.yaml} | 4 +- .../minimal/{eip-xxxx.yaml => eip-7732.yaml} | 2 +- pysetup/constants.py | 2 +- pysetup/md_doc_paths.py | 4 +- pysetup/spec_builders/__init__.py | 4 +- .../spec_builders/{eipxxxx.py => eip7732.py} | 8 ++-- .../{eipxxxx => eip7732}/beacon-chain.md | 40 +++++++++---------- .../_features/{eipxxxx => eip7732}/builder.md | 2 +- .../{eipxxxx => eip7732}/fork-choice.md | 20 +++++----- specs/_features/{eipxxxx => eip7732}/fork.md | 14 +++---- .../{eipxxxx => eip7732}/p2p-interface.md | 22 +++++----- .../{eipxxxx => eip7732}/validator.md | 2 +- .../eth2spec/{eipxxxx => eip7732}/__init__.py | 0 .../pyspec/eth2spec/test/helpers/constants.py | 10 ++--- 15 files changed, 68 insertions(+), 68 deletions(-) rename presets/mainnet/{eip-xxxx.yaml => eip-7732.yaml} (68%) rename presets/minimal/{eip-xxxx.yaml => eip-7732.yaml} (84%) rename pysetup/spec_builders/{eipxxxx.py => eip7732.py} (81%) rename specs/_features/{eipxxxx => eip7732}/beacon-chain.md (96%) rename specs/_features/{eipxxxx => eip7732}/builder.md (99%) rename specs/_features/{eipxxxx => eip7732}/fork-choice.md (98%) rename specs/_features/{eipxxxx => eip7732}/fork.md (94%) rename specs/_features/{eipxxxx => eip7732}/p2p-interface.md (96%) rename specs/_features/{eipxxxx => eip7732}/validator.md (99%) rename tests/core/pyspec/eth2spec/{eipxxxx => eip7732}/__init__.py (100%) diff --git a/Makefile b/Makefile index 07dc301735..b0274f1f68 100644 --- a/Makefile +++ b/Makefile @@ -35,7 +35,7 @@ MARKDOWN_FILES = $(wildcard $(SPEC_DIR)/*/*.md) \ $(wildcard $(SPEC_DIR)/_features/*/*/*.md) \ $(wildcard $(SSZ_DIR)/*.md) -ALL_EXECUTABLE_SPEC_NAMES = phase0 altair bellatrix capella deneb electra whisk eip6800 eipxxxx +ALL_EXECUTABLE_SPEC_NAMES = phase0 altair bellatrix capella deneb electra whisk eip6800 eip7732 # The parameters for commands. Use `foreach` to avoid listing specs again. COVERAGE_SCOPE := $(foreach S,$(ALL_EXECUTABLE_SPEC_NAMES), --cov=eth2spec.$S.$(TEST_PRESET_TYPE)) PYLINT_SCOPE := $(foreach S,$(ALL_EXECUTABLE_SPEC_NAMES), ./eth2spec/$S) diff --git a/presets/mainnet/eip-xxxx.yaml b/presets/mainnet/eip-7732.yaml similarity index 68% rename from presets/mainnet/eip-xxxx.yaml rename to presets/mainnet/eip-7732.yaml index 6b081bbf34..f0a513cd21 100644 --- a/presets/mainnet/eip-xxxx.yaml +++ b/presets/mainnet/eip-7732.yaml @@ -1,8 +1,8 @@ -# Mainnet preset - EIPXXXX +# Mainnet preset - EIP7732 # Execution # --------------------------------------------------------------- # 2**9 (= 512) PTC_SIZE: 512 # 2**2 (= 4) -MAX_PAYLOAD_ATTESTATIONS: 4 \ No newline at end of file +MAX_PAYLOAD_ATTESTATIONS: 4 diff --git a/presets/minimal/eip-xxxx.yaml b/presets/minimal/eip-7732.yaml similarity index 84% rename from presets/minimal/eip-xxxx.yaml rename to presets/minimal/eip-7732.yaml index d596ed8971..c00e04f4bf 100644 --- a/presets/minimal/eip-xxxx.yaml +++ b/presets/minimal/eip-7732.yaml @@ -1,4 +1,4 @@ -# Minimal preset - EIPXXXX +# Minimal preset - EIP7732 # Execution # --------------------------------------------------------------- diff --git a/pysetup/constants.py b/pysetup/constants.py index 0f91c981d9..6bf22865b2 100644 --- a/pysetup/constants.py +++ b/pysetup/constants.py @@ -8,7 +8,7 @@ EIP7594 = 'eip7594' EIP6800 = 'eip6800' WHISK = 'whisk' -EIPXXXX = 'eipxxxx' +EIP7732 = 'eip7732' # The helper functions that are used when defining constants diff --git a/pysetup/md_doc_paths.py b/pysetup/md_doc_paths.py index 96d27a74e5..d99fc122ac 100644 --- a/pysetup/md_doc_paths.py +++ b/pysetup/md_doc_paths.py @@ -10,7 +10,7 @@ WHISK, EIP7594, EIP6800, - EIPXXXX, + EIP7732, ) @@ -24,7 +24,7 @@ WHISK: CAPELLA, EIP7594: DENEB, EIP6800: DENEB, - EIPXXXX: ELECTRA, + EIP7732: ELECTRA, } ALL_FORKS = list(PREVIOUS_FORK_OF.keys()) diff --git a/pysetup/spec_builders/__init__.py b/pysetup/spec_builders/__init__.py index 3c8be80b16..c5bbcf39eb 100644 --- a/pysetup/spec_builders/__init__.py +++ b/pysetup/spec_builders/__init__.py @@ -7,13 +7,13 @@ from .whisk import WhiskSpecBuilder from .eip7594 import EIP7594SpecBuilder from .eip6800 import EIP6800SpecBuilder -from .eipxxxx import EIPXXXXSpecBuilder +from .eip7732 import EIP7732SpecBuilder spec_builders = { builder.fork: builder for builder in ( Phase0SpecBuilder, AltairSpecBuilder, BellatrixSpecBuilder, CapellaSpecBuilder, DenebSpecBuilder, - ElectraSpecBuilder, WhiskSpecBuilder, EIP7594SpecBuilder, EIP6800SpecBuilder, EIPXXXXSpecBuilder, + ElectraSpecBuilder, WhiskSpecBuilder, EIP7594SpecBuilder, EIP6800SpecBuilder, EIP7732SpecBuilder, ) } diff --git a/pysetup/spec_builders/eipxxxx.py b/pysetup/spec_builders/eip7732.py similarity index 81% rename from pysetup/spec_builders/eipxxxx.py rename to pysetup/spec_builders/eip7732.py index 594d3e43a9..2e595a6649 100644 --- a/pysetup/spec_builders/eipxxxx.py +++ b/pysetup/spec_builders/eip7732.py @@ -1,11 +1,11 @@ from typing import Dict from .base import BaseSpecBuilder -from ..constants import EIPXXXX +from ..constants import EIP7732 -class EIPXXXXSpecBuilder(BaseSpecBuilder): - fork: str = EIPXXXX +class EIP7732SpecBuilder(BaseSpecBuilder): + fork: str = EIP7732 @classmethod def imports(cls, preset_name: str): @@ -18,4 +18,4 @@ def hardcoded_custom_type_dep_constants(cls, spec_object) -> Dict[str, str]: return { 'PTC_SIZE': spec_object.preset_vars['PTC_SIZE'].value, 'MAX_PAYLOAD_ATTESTATIONS': spec_object.preset_vars['MAX_PAYLOAD_ATTESTATIONS'].value, - } \ No newline at end of file + } diff --git a/specs/_features/eipxxxx/beacon-chain.md b/specs/_features/eip7732/beacon-chain.md similarity index 96% rename from specs/_features/eipxxxx/beacon-chain.md rename to specs/_features/eip7732/beacon-chain.md index efcf59b914..7094423b82 100644 --- a/specs/_features/eipxxxx/beacon-chain.md +++ b/specs/_features/eip7732/beacon-chain.md @@ -1,4 +1,4 @@ -# EIP-XXXX -- The Beacon Chain +# EIP-7732 -- The Beacon Chain ## Table of contents @@ -80,20 +80,20 @@ At any given slot, the status of the blockchain's head may be either | Name | Value | | - | - | -| `PTC_SIZE` | `uint64(2**9)` (=512) # (New in EIP-XXXX) | +| `PTC_SIZE` | `uint64(2**9)` (=512) # (New in EIP-7732) | ### Domain types | Name | Value | | - | - | -| `DOMAIN_BEACON_BUILDER` | `DomainType('0x1B000000')` # (New in EIP-XXXX)| -| `DOMAIN_PTC_ATTESTER` | `DomainType('0x0C000000')` # (New in EIP-XXXX)| +| `DOMAIN_BEACON_BUILDER` | `DomainType('0x1B000000')` # (New in EIP-7732)| +| `DOMAIN_PTC_ATTESTER` | `DomainType('0x0C000000')` # (New in EIP-7732)| ### Max operations per block | Name | Value | | - | - | -| `MAX_PAYLOAD_ATTESTATIONS` | `2**2` (= 4) # (New in EIP-XXXX) | +| `MAX_PAYLOAD_ATTESTATIONS` | `2**2` (= 4) # (New in EIP-7732) | ## Containers @@ -182,12 +182,12 @@ class BeaconBlockBody(Container): voluntary_exits: List[SignedVoluntaryExit, MAX_VOLUNTARY_EXITS] sync_aggregate: SyncAggregate # Execution - # Removed execution_payload [Removed in EIP-XXXX] - # Removed blob_kzg_commitments [Removed in EIP-XXXX] + # Removed execution_payload [Removed in EIP-7732] + # Removed blob_kzg_commitments [Removed in EIP-7732] bls_to_execution_changes: List[SignedBLSToExecutionChange, MAX_BLS_TO_EXECUTION_CHANGES] # PBS - signed_execution_payload_header: SignedExecutionPayloadHeader # [New in EIP-XXXX] - payload_attestations: List[PayloadAttestation, MAX_PAYLOAD_ATTESTATIONS] # [New in EIP-XXXX] + signed_execution_payload_header: SignedExecutionPayloadHeader # [New in EIP-7732] + payload_attestations: List[PayloadAttestation, MAX_PAYLOAD_ATTESTATIONS] # [New in EIP-7732] ``` #### `ExecutionPayloadHeader` @@ -262,9 +262,9 @@ class BeaconState(Container): pending_partial_withdrawals: List[PendingPartialWithdrawal, PENDING_PARTIAL_WITHDRAWALS_LIMIT] pending_consolidations: List[PendingConsolidation, PENDING_CONSOLIDATIONS_LIMIT] # PBS - latest_block_hash: Hash32 # [New in EIP-XXXX] - latest_full_slot: Slot # [New in EIP-XXXX] - latest_withdrawals_root: Root # [New in EIP-XXXX] + latest_block_hash: Hash32 # [New in EIP-7732] + latest_full_slot: Slot # [New in EIP-7732] + latest_withdrawals_root: Root # [New in EIP-7732] ``` ## Helper functions @@ -394,7 +394,7 @@ def get_indexed_payload_attestation(state: BeaconState, slot: Slot, ## Beacon chain state transition function -*Note*: state transition is fundamentally modified in EIP-XXXX. The full state transition is broken in two parts, first importing a signed block and then importing an execution payload. +*Note*: state transition is fundamentally modified in EIP-7732. The full state transition is broken in two parts, first importing a signed block and then importing an execution payload. The post-state corresponding to a pre-state `state` and a signed beacon block `signed_block` is defined as `state_transition(state, signed_block)`. State transitions that trigger an unhandled exception (e.g. a failed `assert` or an out-of-range list access) are considered invalid. State transitions that cause a `uint64` overflow or underflow are also considered invalid. @@ -405,11 +405,11 @@ The post-state corresponding to a pre-state `state` and a signed execution paylo ```python def process_block(state: BeaconState, block: BeaconBlock) -> None: process_block_header(state, block) - process_withdrawals(state) # [Modified in EIP-XXXX] - process_execution_payload_header(state, block) # [Modified in EIP-XXXX, removed process_execution_payload] + process_withdrawals(state) # [Modified in EIP-7732] + process_execution_payload_header(state, block) # [Modified in EIP-7732, removed process_execution_payload] process_randao(state, block.body) process_eth1_data(state, block.body) - process_operations(state, block.body) # [Modified in EIP-XXXX] + process_operations(state, block.body) # [Modified in EIP-7732] process_sync_aggregate(state, block.body.sync_aggregate) ``` @@ -505,10 +505,10 @@ def process_operations(state: BeaconState, body: BeaconBlockBody) -> None: for_ops(body.deposits, process_deposit) for_ops(body.voluntary_exits, process_voluntary_exit) for_ops(body.bls_to_execution_changes, process_bls_to_execution_change) - # Removed `process_deposit_request` in EIP-XXXX - # Removed `process_withdrawal_request` in EIP-XXXX - # Removed `process_consolidation_request` in EIP-XXXX - for_ops(body.payload_attestations, process_payload_attestation) # [New in EIP-XXXX] + # Removed `process_deposit_request` in EIP-7732 + # Removed `process_withdrawal_request` in EIP-7732 + # Removed `process_consolidation_request` in EIP-7732 + for_ops(body.payload_attestations, process_payload_attestation) # [New in EIP-7732] ``` ##### Payload Attestations diff --git a/specs/_features/eipxxxx/builder.md b/specs/_features/eip7732/builder.md similarity index 99% rename from specs/_features/eipxxxx/builder.md rename to specs/_features/eip7732/builder.md index 35cf296039..1a95baac08 100644 --- a/specs/_features/eipxxxx/builder.md +++ b/specs/_features/eip7732/builder.md @@ -52,7 +52,7 @@ The builder assembles then `signed_execution_payload_header = SignedExecutionPay ### Constructing the `BlobSidecar`s -[Modified in EIP-XXXX] +[Modified in EIP-7732] The `BlobSidecar` container is modified indirectly because the constant `KZG_COMMITMENT_INCLUSION_PROOF_DEPTH` is modified. Each sidecar is obtained from the modified diff --git a/specs/_features/eipxxxx/fork-choice.md b/specs/_features/eip7732/fork-choice.md similarity index 98% rename from specs/_features/eipxxxx/fork-choice.md rename to specs/_features/eip7732/fork-choice.md index 4ee870ffef..ca7bdf051f 100644 --- a/specs/_features/eipxxxx/fork-choice.md +++ b/specs/_features/eip7732/fork-choice.md @@ -45,8 +45,8 @@ This is the modification of the fork choice accompanying the ePBS upgrade. | Name | Value | | -------------------- | ----------- | | `PAYLOAD_TIMELY_THRESHOLD` | `PTC_SIZE / 2` (=`uint64(256)`) | -| `INTERVALS_PER_SLOT` | `4` # [modified in EIP-XXXX] | -| `PROPOSER_SCORE_BOOST` | `20` # [modified in EIP-XXXX] | +| `INTERVALS_PER_SLOT` | `4` # [modified in EIP-7732] | +| `PROPOSER_SCORE_BOOST` | `20` # [modified in EIP-7732] | | `PAYLOAD_WITHHOLD_BOOST` | `40` | | `PAYLOAD_REVEAL_BOOST` | `40` | @@ -100,9 +100,9 @@ class Store(object): unrealized_justified_checkpoint: Checkpoint unrealized_finalized_checkpoint: Checkpoint proposer_boost_root: Root - payload_withhold_boost_root: Root # [New in EIP-XXXX] - payload_withhold_boost_full: boolean # [New in EIP-XXXX] - payload_reveal_boost_root: Root # [New in EIP-XXXX] + payload_withhold_boost_root: Root # [New in EIP-7732] + payload_withhold_boost_full: boolean # [New in EIP-7732] + payload_reveal_boost_root: Root # [New in EIP-7732] equivocating_indices: Set[ValidatorIndex] blocks: Dict[Root, BeaconBlock] = field(default_factory=dict) block_states: Dict[Root, BeaconState] = field(default_factory=dict) @@ -110,8 +110,8 @@ class Store(object): checkpoint_states: Dict[Checkpoint, BeaconState] = field(default_factory=dict) latest_messages: Dict[ValidatorIndex, LatestMessage] = field(default_factory=dict) unrealized_justifications: Dict[Root, Checkpoint] = field(default_factory=dict) - execution_payload_states: Dict[Root, BeaconState] = field(default_factory=dict) # [New in EIP-XXXX] - ptc_vote: Dict[Root, Vector[uint8, PTC_SIZE]] = field(default_factory=dict) # [New in EIP-XXXX] + execution_payload_states: Dict[Root, BeaconState] = field(default_factory=dict) # [New in EIP-7732] + ptc_vote: Dict[Root, Vector[uint8, PTC_SIZE]] = field(default_factory=dict) # [New in EIP-7732] ``` ### Modified `get_forkchoice_store` @@ -132,9 +132,9 @@ def get_forkchoice_store(anchor_state: BeaconState, anchor_block: BeaconBlock) - unrealized_justified_checkpoint=justified_checkpoint, unrealized_finalized_checkpoint=finalized_checkpoint, proposer_boost_root=proposer_boost_root, - payload_withhold_boost_root=proposer_boost_root, # [New in EIP-XXXX] - payload_withhold_boost_full=True, # [New in EIP-XXXX] - payload_reveal_boost_root=proposer_boost_root, # [New in EIP-XXXX] + payload_withhold_boost_root=proposer_boost_root, # [New in EIP-7732] + payload_withhold_boost_full=True, # [New in EIP-7732] + payload_reveal_boost_root=proposer_boost_root, # [New in EIP-7732] equivocating_indices=set(), blocks={anchor_root: copy(anchor_block)}, block_states={anchor_root: copy(anchor_state)}, diff --git a/specs/_features/eipxxxx/fork.md b/specs/_features/eip7732/fork.md similarity index 94% rename from specs/_features/eipxxxx/fork.md rename to specs/_features/eip7732/fork.md index fc719b3eae..a41bd32d21 100644 --- a/specs/_features/eipxxxx/fork.md +++ b/specs/_features/eip7732/fork.md @@ -12,7 +12,7 @@ - [Helper functions](#helper-functions) - [Misc](#misc) - [Modified `compute_fork_version`](#modified-compute_fork_version) -- [Fork to EIP-XXXX](#fork-to-eip-xxxx) +- [Fork to EIP-7732](#fork-to-eip-7732) - [Fork trigger](#fork-trigger) - [Upgrading the state](#upgrading-the-state) @@ -20,7 +20,7 @@ ## Introduction -This document describes the process of the EIP-XXXX upgrade. +This document describes the process of the EIP-7732 upgrade. ## Configuration @@ -57,7 +57,7 @@ def compute_fork_version(epoch: Epoch) -> Version: return GENESIS_FORK_VERSION ``` -## Fork to EIP-XXXX +## Fork to EIP-7732 ### Fork trigger @@ -114,7 +114,7 @@ def upgrade_to_epbs(pre: electra.BeaconState) -> BeaconState: current_sync_committee=pre.current_sync_committee, next_sync_committee=pre.next_sync_committee, # Execution-layer - latest_execution_payload_header=ExecutionPayloadHeader(), # [Modified in EIP-XXXX] + latest_execution_payload_header=ExecutionPayloadHeader(), # [Modified in EIP-7732] # Withdrawals next_withdrawal_index=pre.next_withdrawal_index, next_withdrawal_validator_index=pre.next_withdrawal_validator_index, @@ -130,9 +130,9 @@ def upgrade_to_epbs(pre: electra.BeaconState) -> BeaconState: pending_partial_withdrawals=pre.pending_partial_withdrawals, pending_consolidations=pre.pending_consolidations, # ePBS - latest_block_hash=pre.latest_execution_payload_header.block_hash, # [New in EIP-XXXX] - latest_full_slot=pre.slot, # [New in EIP-XXXX] - last_withdrawals_root=Root(), # [New in EIP-XXXX] + latest_block_hash=pre.latest_execution_payload_header.block_hash, # [New in EIP-7732] + latest_full_slot=pre.slot, # [New in EIP-7732] + last_withdrawals_root=Root(), # [New in EIP-7732] ) return post diff --git a/specs/_features/eipxxxx/p2p-interface.md b/specs/_features/eip7732/p2p-interface.md similarity index 96% rename from specs/_features/eipxxxx/p2p-interface.md rename to specs/_features/eip7732/p2p-interface.md index eae9552fdc..099b40a203 100644 --- a/specs/_features/eipxxxx/p2p-interface.md +++ b/specs/_features/eip7732/p2p-interface.md @@ -5,7 +5,7 @@ This document contains the consensus-layer networking specification for ePBS. -- [Modification in EIP-XXXX](#modification-in-eip-xxxx) +- [Modification in EIP-7732](#modification-in-eip-7732) - [Preset](#preset) - [Containers](#containers) - [`BlobSidecar`](#blobsidecar) @@ -27,11 +27,11 @@ This document contains the consensus-layer networking specification for ePBS. -## Modification in EIP-XXXX +## Modification in EIP-7732 ### Preset -*[Modified in EIP-XXXX]* +*[Modified in EIP-7732]* | Name | Value | Description | |------------------------------------------|-----------------------------------|---------------------------------------------------------------------| @@ -58,7 +58,7 @@ class BlobSidecar(Container): ##### Modified `verify_blob_sidecar_inclusion_proof` -`verify_blob_sidecar_inclusion_proof` is modified in EIP-XXXX to account for the fact that the KZG commitments are included in the `ExecutionPayloadEnvelope` and no longer in the beacon block body. +`verify_blob_sidecar_inclusion_proof` is modified in EIP-7732 to account for the fact that the KZG commitments are included in the `ExecutionPayloadEnvelope` and no longer in the beacon block body. ```python def verify_blob_sidecar_inclusion_proof(blob_sidecar: BlobSidecar) -> bool: @@ -76,7 +76,7 @@ def verify_blob_sidecar_inclusion_proof(blob_sidecar: BlobSidecar) -> bool: ### The gossip domain: gossipsub -Some gossip meshes are upgraded in the fork of EIP-XXXX to support upgraded types. +Some gossip meshes are upgraded in the fork of EIP-7732 to support upgraded types. #### Topics and messages @@ -85,23 +85,23 @@ Topics follow the same specification as in prior upgrades. The `beacon_block` topic is updated to support the modified type | Name | Message Type | | --- | --- | -| `beacon_block` | `SignedBeaconBlock` [modified in EIP-XXXX] | +| `beacon_block` | `SignedBeaconBlock` [modified in EIP-7732] | The new topics along with the type of the `data` field of a gossipsub message are given in this table: | Name | Message Type | |-------------------------------|------------------------------------------------------| -| `execution_payload_header` | `SignedExecutionPayloadHeader` [New in EIP-XXXX] | -| `execution_payload` | `SignedExecutionPayloadEnvelope` [New in EIP-XXXX] | -| `payload_attestation_message` | `PayloadAttestationMessage` [New in EIP-XXXX] | +| `execution_payload_header` | `SignedExecutionPayloadHeader` [New in EIP-7732] | +| `execution_payload` | `SignedExecutionPayloadEnvelope` [New in EIP-7732] | +| `payload_attestation_message` | `PayloadAttestationMessage` [New in EIP-7732] | ##### Global topics -EIP-XXXX introduces new global topics for execution header, execution payload and payload attestation. +EIP-7732 introduces new global topics for execution header, execution payload and payload attestation. ###### `beacon_block` -[Modified in EIP-XXXX] +[Modified in EIP-7732] The *type* of the payload of this topic changes to the (modified) `SignedBeaconBlock` found in [the Beacon Chain changes](./beacon-chain.md). diff --git a/specs/_features/eipxxxx/validator.md b/specs/_features/eip7732/validator.md similarity index 99% rename from specs/_features/eipxxxx/validator.md rename to specs/_features/eip7732/validator.md index 8982067729..e9d30d2726 100644 --- a/specs/_features/eipxxxx/validator.md +++ b/specs/_features/eip7732/validator.md @@ -47,7 +47,7 @@ def get_ptc_assignment( ### Lookahead -[New in EIP-XXXX] +[New in EIP-7732] `get_ptc_assignment` should be called at the start of each epoch to get the assignment for the next epoch (`current_epoch + 1`). A validator should plan for future assignments by noting their assigned PTC slot. diff --git a/tests/core/pyspec/eth2spec/eipxxxx/__init__.py b/tests/core/pyspec/eth2spec/eip7732/__init__.py similarity index 100% rename from tests/core/pyspec/eth2spec/eipxxxx/__init__.py rename to tests/core/pyspec/eth2spec/eip7732/__init__.py diff --git a/tests/core/pyspec/eth2spec/test/helpers/constants.py b/tests/core/pyspec/eth2spec/test/helpers/constants.py index 2fbbfb90d6..4267ad21c8 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/constants.py +++ b/tests/core/pyspec/eth2spec/test/helpers/constants.py @@ -20,7 +20,7 @@ ELECTRA = SpecForkName('electra') WHISK = SpecForkName('whisk') EIP7594 = SpecForkName('eip7594') -EIPXXXX = SpecForkName('eipxxxx') +EIP7732 = SpecForkName('eip7732') # # SpecFork settings @@ -38,12 +38,12 @@ ELECTRA, # Experimental patches EIP7594, - EIPXXXX, + EIP7732, ) # The forks that have light client specs LIGHT_CLIENT_TESTING_FORKS = (*[item for item in MAINNET_FORKS if item != PHASE0],) # The forks that output to the test vectors. -TESTGEN_FORKS = (*MAINNET_FORKS, ELECTRA, EIP7594, WHISK, EIPXXXX) +TESTGEN_FORKS = (*MAINNET_FORKS, ELECTRA, EIP7594, WHISK, EIP7732) # Forks allowed in the test runner `--fork` flag, to fail fast in case of typos ALLOWED_TEST_RUNNER_FORKS = (*ALL_PHASES, WHISK) @@ -59,7 +59,7 @@ # Experimental patches WHISK: CAPELLA, EIP7594: DENEB, - EIPXXXX: ELECTRA, + EIP7732: ELECTRA, } # For fork transition tests @@ -70,7 +70,7 @@ BELLATRIX: CAPELLA, CAPELLA: DENEB, DENEB: ELECTRA, - ELECTRA: EIPXXXX, + ELECTRA: EIP7732, } ALL_PRE_POST_FORKS = POST_FORK_OF.items() From b230a42584aee0d1d0f41a6c2cfc40a9fdd9c2e3 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 3 Jul 2024 20:48:56 +0800 Subject: [PATCH 068/130] Add deprecation helpers in spec builder --- pysetup/helpers.py | 13 +++++++++++-- pysetup/spec_builders/base.py | 10 +++++++++- pysetup/spec_builders/eip7732.py | 14 +++++++++++++- 3 files changed, 33 insertions(+), 4 deletions(-) diff --git a/pysetup/helpers.py b/pysetup/helpers.py index 589ae6ab58..212eb98c10 100644 --- a/pysetup/helpers.py +++ b/pysetup/helpers.py @@ -124,13 +124,22 @@ def format_constant(name: str, vardef: VariableDefinition) -> str: # Keep engine from the most recent fork execution_engine_cls = reduce(lambda txt, builder: builder.execution_engine_cls() or txt, builders, "") + # Remove deprecated constants + deprecate_constants = reduce(lambda obj, builder: obj.union(builder.deprecate_constants()), builders, set()) + # constant_vars = {k: v for k, v in spec_object.constant_vars.items() if k not in deprecate_constants} + filtered_ssz_dep_constants = {k: v for k, v in hardcoded_ssz_dep_constants.items() if k not in deprecate_constants} + # Remove deprecated presets + deprecate_presets = reduce(lambda obj, builder: obj.union(builder.deprecate_presets()), builders, set()) + # preset_vars = {k: v for k, v in spec_object.constant_vars.items() if k not in deprecate_constants} + filtered_hardcoded_func_dep_presets = {k: v for k, v in hardcoded_func_dep_presets.items() if k not in deprecate_presets} + constant_vars_spec = '# Constant vars\n' + '\n'.join(format_constant(k, v) for k, v in spec_object.constant_vars.items()) preset_vars_spec = '# Preset vars\n' + '\n'.join(format_constant(k, v) for k, v in spec_object.preset_vars.items()) ordered_class_objects_spec = '\n\n\n'.join(ordered_class_objects.values()) ssz_dep_constants = '\n'.join(map(lambda x: '%s = %s' % (x, hardcoded_ssz_dep_constants[x]), hardcoded_ssz_dep_constants)) - ssz_dep_constants_verification = '\n'.join(map(lambda x: 'assert %s == %s' % (x, spec_object.ssz_dep_constants[x]), hardcoded_ssz_dep_constants)) + ssz_dep_constants_verification = '\n'.join(map(lambda x: 'assert %s == %s' % (x, spec_object.ssz_dep_constants[x]), filtered_ssz_dep_constants)) custom_type_dep_constants = '\n'.join(map(lambda x: '%s = %s' % (x, hardcoded_custom_type_dep_constants[x]), hardcoded_custom_type_dep_constants)) - func_dep_presets_verification = '\n'.join(map(lambda x: 'assert %s == %s # noqa: E501' % (x, spec_object.func_dep_presets[x]), hardcoded_func_dep_presets)) + func_dep_presets_verification = '\n'.join(map(lambda x: 'assert %s == %s # noqa: E501' % (x, spec_object.func_dep_presets[x]), filtered_hardcoded_func_dep_presets)) spec_strs = [ imports, preparations, diff --git a/pysetup/spec_builders/base.py b/pysetup/spec_builders/base.py index ad9a2cb4c4..a8c648a0f0 100644 --- a/pysetup/spec_builders/base.py +++ b/pysetup/spec_builders/base.py @@ -1,5 +1,5 @@ from abc import ABC, abstractmethod -from typing import Sequence, Dict +from typing import Sequence, Dict, Set from pathlib import Path class BaseSpecBuilder(ABC): @@ -54,3 +54,11 @@ def hardcoded_func_dep_presets(cls, spec_object) -> Dict[str, str]: @classmethod def implement_optimizations(cls, functions: Dict[str, str]) -> Dict[str, str]: return functions + + @classmethod + def deprecate_constants(cls) -> Set[str]: + return set() + + @classmethod + def deprecate_presets(cls) -> Set[str]: + return set() diff --git a/pysetup/spec_builders/eip7732.py b/pysetup/spec_builders/eip7732.py index 2e595a6649..49c1c880b7 100644 --- a/pysetup/spec_builders/eip7732.py +++ b/pysetup/spec_builders/eip7732.py @@ -1,4 +1,4 @@ -from typing import Dict +from typing import Dict, Set from .base import BaseSpecBuilder from ..constants import EIP7732 @@ -19,3 +19,15 @@ def hardcoded_custom_type_dep_constants(cls, spec_object) -> Dict[str, str]: 'PTC_SIZE': spec_object.preset_vars['PTC_SIZE'].value, 'MAX_PAYLOAD_ATTESTATIONS': spec_object.preset_vars['MAX_PAYLOAD_ATTESTATIONS'].value, } + + @classmethod + def deprecate_constants(cls) -> Set[str]: + return set([ + 'EXECUTION_PAYLOAD_GINDEX', + ]) + + @classmethod + def deprecate_presets(cls) -> Set[str]: + return set([ + 'KZG_COMMITMENT_INCLUSION_PROOF_DEPTH', + ]) From b2d6e68e62fe783787d546b91010edc5bd7fd91f Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 3 Jul 2024 20:49:56 +0800 Subject: [PATCH 069/130] Fix and add `EIP7732_FORK_EPOCH` and `EIP7594_FORK_VERSION` --- configs/mainnet.yaml | 3 +++ configs/minimal.yaml | 3 +++ specs/_features/eip7732/fork.md | 14 +++++++------- specs/_features/eip7732/p2p-interface.md | 8 ++++---- 4 files changed, 17 insertions(+), 11 deletions(-) diff --git a/configs/mainnet.yaml b/configs/mainnet.yaml index 7051873ce9..cf761c60c2 100644 --- a/configs/mainnet.yaml +++ b/configs/mainnet.yaml @@ -59,6 +59,9 @@ EIP7594_FORK_EPOCH: 18446744073709551615 # WHISK WHISK_FORK_VERSION: 0x08000000 # temporary stub WHISK_FORK_EPOCH: 18446744073709551615 +# EIP7732 +EIP7732_FORK_VERSION: 0x09000000 # temporary stub +EIP7732_FORK_EPOCH: 18446744073709551615 # Time parameters # --------------------------------------------------------------- diff --git a/configs/minimal.yaml b/configs/minimal.yaml index 8e2a222d47..dbc46afc1e 100644 --- a/configs/minimal.yaml +++ b/configs/minimal.yaml @@ -58,6 +58,9 @@ EIP7594_FORK_EPOCH: 18446744073709551615 # WHISK WHISK_FORK_VERSION: 0x08000001 WHISK_FORK_EPOCH: 18446744073709551615 +# EIP7732 +EIP7732_FORK_VERSION: 0x09000001 +EIP7732_FORK_EPOCH: 18446744073709551615 # Time parameters # --------------------------------------------------------------- diff --git a/specs/_features/eip7732/fork.md b/specs/_features/eip7732/fork.md index a41bd32d21..0212f8fce9 100644 --- a/specs/_features/eip7732/fork.md +++ b/specs/_features/eip7732/fork.md @@ -28,8 +28,8 @@ Warning: this configuration is not definitive. | Name | Value | |---------------------| - | -| `EPBS_FORK_VERSION` | `Version('0x06000000')` | -| `EPBS_FORK_EPOCH` | `Epoch(18446744073709551615)` **TBD** | +| `EIP7732_FORK_VERSION` | `Version('0x09000000')` | +| `EIP7732_FORK_EPOCH` | `Epoch(18446744073709551615)` **TBD** | ## Helper functions @@ -42,8 +42,8 @@ def compute_fork_version(epoch: Epoch) -> Version: """ Return the fork version at the given ``epoch``. """ - if epoch >= EPBS_FORK_EPOCH: - return EPBS_FORK_VERSION + if epoch >= EIP7732_FORK_EPOCH: + return EIP7732_FORK_VERSION if epoch >= ELECTRA_FORK_EPOCH: return ELECTRA_FORK_VERSION if epoch >= DENEB_FORK_EPOCH: @@ -63,11 +63,11 @@ def compute_fork_version(epoch: Epoch) -> Version: TBD. This fork is defined for testing purposes, the EIP may be combined with other consensus-layer upgrade. -For now, we assume the condition will be triggered at epoch `EPBS_FORK_EPOCH`. +For now, we assume the condition will be triggered at epoch `EIP7732_FORK_EPOCH`. ### Upgrading the state -If `state.slot % SLOTS_PER_EPOCH == 0` and `compute_epoch_at_slot(state.slot) == EPBS_FORK_EPOCH`, +If `state.slot % SLOTS_PER_EPOCH == 0` and `compute_epoch_at_slot(state.slot) == EIP7732_FORK_EPOCH`, an irregular state change is made to upgrade to ePBS. ```python @@ -81,7 +81,7 @@ def upgrade_to_epbs(pre: electra.BeaconState) -> BeaconState: slot=pre.slot, fork=Fork( previous_version=pre.fork.current_version, - current_version=EPBS_FORK_EPOCH, # [Modified in ePBS] + current_version=EIP7732_FORK_EPOCH, # [Modified in ePBS] epoch=epoch, ), # History diff --git a/specs/_features/eip7732/p2p-interface.md b/specs/_features/eip7732/p2p-interface.md index 099b40a203..1bb2d19e53 100644 --- a/specs/_features/eip7732/p2p-interface.md +++ b/specs/_features/eip7732/p2p-interface.md @@ -184,7 +184,7 @@ The following validations MUST pass before forwarding the `signed_execution_payl | `BELLATRIX_FORK_VERSION` | `bellatrix.SignedBeaconBlock` | | `CAPELLA_FORK_VERSION` | `capella.SignedBeaconBlock` | | `DENEB_FORK_VERSION` | `deneb.SignedBeaconBlock` | -| `EPBS_FORK_VERSION` | `epbs.SignedBeaconBlock` | +| `EIP7732_FORK_VERSION` | `epbs.SignedBeaconBlock` | ##### BeaconBlocksByRoot v3 @@ -201,7 +201,7 @@ Per `context = compute_fork_digest(fork_version, genesis_validators_root)`: | `BELLATRIX_FORK_VERSION` | `bellatrix.SignedBeaconBlock` | | `CAPELLA_FORK_VERSION` | `capella.SignedBeaconBlock` | | `DENEB_FORK_VERSION` | `deneb.SignedBeaconBlock` | -| `EPBS_FORK_VERSION` | `epbs.SignedBeaconBlock` | +| `EIP7732_FORK_VERSION` | `epbs.SignedBeaconBlock` | ##### BlobSidecarsByRoot v2 @@ -213,7 +213,7 @@ Per `context = compute_fork_digest(fork_version, genesis_validators_root)`: | `fork_version` | Chunk SSZ type | |--------------------------|-------------------------------| | `DENEB_FORK_VERSION` | `deneb.BlobSidecar` | -| `EPBS_FORK_VERSION` | `epbs.BlobSidecar` | +| `EIP7732_FORK_VERSION` | `epbs.BlobSidecar` | ##### ExecutionPayloadEnvelopeByRoot v1 @@ -226,7 +226,7 @@ The `` field is calculated as `context = compute_fork_digest(fork | `fork_version` | Chunk SSZ type | |---------------------|---------------------------------------| -| `EPBS_FORK_VERSION` | `epbs.SignedExecutionPayloadEnvelope` | +| `EIP7732_FORK_VERSION` | `epbs.SignedExecutionPayloadEnvelope` | Request Content: From 11276403f8e6f9948bfacb60c82470c43334b766 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Wed, 3 Jul 2024 20:51:44 +0800 Subject: [PATCH 070/130] Add `eip7732` to .gitignore --- .gitignore | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index e6fad84fe9..ad62aa177e 100644 --- a/.gitignore +++ b/.gitignore @@ -25,7 +25,7 @@ tests/core/pyspec/eth2spec/electra/ tests/core/pyspec/eth2spec/whisk/ tests/core/pyspec/eth2spec/eip7594/ tests/core/pyspec/eth2spec/eip6800/ -tests/core/pyspec/eth2spec/eipxxxx/ +tests/core/pyspec/eth2spec/eip7732/ # coverage reports .htmlcov From 603cf53fab8d3929b286e480a994f5796b4639be Mon Sep 17 00:00:00 2001 From: Potuz Date: Wed, 3 Jul 2024 11:02:12 -0300 Subject: [PATCH 071/130] add fork fixes --- specs/_features/eip7732/fork.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/specs/_features/eip7732/fork.md b/specs/_features/eip7732/fork.md index 0212f8fce9..765b032d0d 100644 --- a/specs/_features/eip7732/fork.md +++ b/specs/_features/eip7732/fork.md @@ -71,7 +71,7 @@ If `state.slot % SLOTS_PER_EPOCH == 0` and `compute_epoch_at_slot(state.slot) == an irregular state change is made to upgrade to ePBS. ```python -def upgrade_to_epbs(pre: electra.BeaconState) -> BeaconState: +def upgrade_to_eip7732(pre: electra.BeaconState) -> BeaconState: epoch = electra.get_current_epoch(pre) post = BeaconState( @@ -81,7 +81,7 @@ def upgrade_to_epbs(pre: electra.BeaconState) -> BeaconState: slot=pre.slot, fork=Fork( previous_version=pre.fork.current_version, - current_version=EIP7732_FORK_EPOCH, # [Modified in ePBS] + current_version=EIP7732_FORK_VERSION, # [Modified in EIP-7732] epoch=epoch, ), # History @@ -121,10 +121,10 @@ def upgrade_to_epbs(pre: electra.BeaconState) -> BeaconState: # Deep history valid from Capella onwards historical_summaries=pre.historical_summaries, deposit_requests_start_index=pre.deposit_requests_start_index, - deposit_balance_to_consumer=pre.deposit_balance_to_consumer, - exit_balance_to_consumer=pre.exit_balance_to_consumer, + deposit_balance_to_consume=pre.deposit_balance_to_consume, + exit_balance_to_consume=pre.exit_balance_to_consume, earliest_exit_epoch=pre.earliest_exit_epoch, - consolidation_balance_to_consumer=pre.consolidation_balance_to_consumer, + consolidation_balance_to_consume=pre.consolidation_balance_to_consume, earliest_consolidation_epoch=pre.earliest_consolidation_epoch, pending_balance_deposits=pre.pending_balance_deposits, pending_partial_withdrawals=pre.pending_partial_withdrawals, @@ -132,7 +132,7 @@ def upgrade_to_epbs(pre: electra.BeaconState) -> BeaconState: # ePBS latest_block_hash=pre.latest_execution_payload_header.block_hash, # [New in EIP-7732] latest_full_slot=pre.slot, # [New in EIP-7732] - last_withdrawals_root=Root(), # [New in EIP-7732] + latest_withdrawals_root=Root(), # [New in EIP-7732] ) return post From e9a8ddad03fc4a28ce2c942f4feba6b2712be7b1 Mon Sep 17 00:00:00 2001 From: Potuz Date: Wed, 3 Jul 2024 12:49:32 -0300 Subject: [PATCH 072/130] Fix python execution 4 --- .../pyspec/eth2spec/test/helpers/block.py | 9 ++- .../test/helpers/execution_payload.py | 55 +++++++++++++++++-- .../pyspec/eth2spec/test/helpers/forks.py | 6 +- 3 files changed, 63 insertions(+), 7 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/helpers/block.py b/tests/core/pyspec/eth2spec/test/helpers/block.py index 84770623e3..a724b968bc 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/block.py +++ b/tests/core/pyspec/eth2spec/test/helpers/block.py @@ -1,5 +1,5 @@ -from eth2spec.test.helpers.execution_payload import build_empty_execution_payload -from eth2spec.test.helpers.forks import is_post_whisk, is_post_altair, is_post_bellatrix +from eth2spec.test.helpers.execution_payload import build_empty_execution_payload, build_empty_signed_execution_payload_header +from eth2spec.test.helpers.forks import is_post_whisk, is_post_altair, is_post_bellatrix, is_post_eip7732 from eth2spec.test.helpers.keys import privkeys, whisk_ks_initial, whisk_ks_final from eth2spec.utils import bls from eth2spec.utils.bls import only_with_bls @@ -117,6 +117,11 @@ def build_empty_block(spec, state, slot=None, proposer_index=None): if is_post_altair(spec): empty_block.body.sync_aggregate.sync_committee_signature = spec.G2_POINT_AT_INFINITY + if is_post_eip7732(spec): + signed_header = build_empty_signed_execution_payload_header(spec, state) + empty_block.body.signed_execution_payload_header = signed_header + return empty_block + if is_post_bellatrix(spec): empty_block.body.execution_payload = build_empty_execution_payload(spec, state) diff --git a/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py index d8be4a92ae..e1bd890dd5 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py @@ -3,16 +3,31 @@ from rlp import encode from rlp.sedes import big_endian_int, Binary, List +from eth2spec.test.helpers.keys import privkeys +from eth2spec.utils.ssz.ssz_impl import hash_tree_root from eth2spec.debug.random_value import get_random_bytes_list from eth2spec.test.helpers.withdrawals import get_expected_withdrawals from eth2spec.test.helpers.forks import ( is_post_capella, is_post_deneb, is_post_electra, + is_post_eip7732, ) def get_execution_payload_header(spec, execution_payload): + if is_post_eip7732(spec): + return spec.ExecutionPayloadHeader( + parent_block_hash=execution_payload.parent_hash, + parent_block_root=spec.Root(), # TODO: Fix this + block_hash=execution_payload.block_hash, + gas_limit=execution_payload.gas_limit, + builder_index=spec.ValidatorIndex(0), # TODO: Fix this + slot=spec.Slot(0), # TODO: Fix this + value=spec.Gwei(0), # TODO: Fix this + blob_kzg_commitments_root=spec.Root() # TODO: Fix this + ) + payload_header = spec.ExecutionPayloadHeader( parent_hash=execution_payload.parent_hash, fee_recipient=execution_payload.fee_recipient, @@ -64,6 +79,9 @@ def compute_el_header_block_hash(spec, """ Computes the RLP execution block hash described by an `ExecutionPayloadHeader`. """ + if is_post_eip7732(spec): + return spec.Hash32() + execution_payload_header_rlp = [ # parent_hash (Binary(32, 32), payload_header.parent_hash), @@ -218,6 +236,34 @@ def compute_el_block_hash(spec, payload, pre_state): ) +def build_empty_post_eip7732_execution_payload_header(spec, state): + if not is_post_eip7732(spec): + return + parent_block_root = hash_tree_root(state.latest_block_header) + return spec.ExecutionPayloadHeader( + parent_block_hash=state.latest_block_hash, + parent_block_root=parent_block_root, + block_hash=spec.Hash32(), + gas_limit=spec.uint64(0), + builder_index=spec.ValidatorIndex(0), + slot=state.slot, + value=spec.Gwei(0), + blob_kzg_commitments_root=spec.Root() + ) + + +def build_empty_signed_execution_payload_header(spec, state): + if not is_post_eip7732(spec): + return + message = build_empty_post_eip7732_execution_payload_header(spec, state) + privkey = privkeys[0] + signature = spec.get_execution_payload_header_signature(state, message, privkey) + return spec.SignedExecutionPayloadHeader( + message=message, + signature=signature, + ) + + def build_empty_execution_payload(spec, state, randao_mix=None): """ Assuming a pre-state of the same slot, build a valid ExecutionPayload without any transactions. @@ -232,18 +278,19 @@ def build_empty_execution_payload(spec, state, randao_mix=None): payload = spec.ExecutionPayload( parent_hash=latest.block_hash, fee_recipient=spec.ExecutionAddress(), - state_root=latest.state_root, # no changes to the state receipts_root=spec.Bytes32(bytes.fromhex("1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347")), logs_bloom=spec.ByteVector[spec.BYTES_PER_LOGS_BLOOM](), # TODO: zeroed logs bloom for empty logs ok? - block_number=latest.block_number + 1, prev_randao=randao_mix, - gas_limit=latest.gas_limit, # retain same limit gas_used=0, # empty block, 0 gas timestamp=timestamp, extra_data=spec.ByteList[spec.MAX_EXTRA_DATA_BYTES](), - base_fee_per_gas=latest.base_fee_per_gas, # retain same base_fee transactions=empty_txs, ) + if not is_post_eip7732(spec): + payload.state_root = latest.state_root, # no changes to the state + payload.block_number = latest.block_number + 1, + payload.gas_limit = latest.gas_limit, # retain same limit + payload.base_fee_per_gas = latest.base_fee_per_gas, # retain same base_fee if is_post_capella(spec): payload.withdrawals = get_expected_withdrawals(spec, state) if is_post_deneb(spec): diff --git a/tests/core/pyspec/eth2spec/test/helpers/forks.py b/tests/core/pyspec/eth2spec/test/helpers/forks.py index 5ea03c31b3..288ad0d9e9 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/forks.py +++ b/tests/core/pyspec/eth2spec/test/helpers/forks.py @@ -1,6 +1,6 @@ from .constants import ( PHASE0, ALTAIR, BELLATRIX, CAPELLA, DENEB, - ELECTRA, WHISK, + ELECTRA, WHISK, EIP7732, PREVIOUS_FORK_OF, ) @@ -45,6 +45,10 @@ def is_post_whisk(spec): return is_post_fork(spec.fork, WHISK) +def is_post_eip7732(spec): + return is_post_fork(spec.fork, EIP7732) + + def get_spec_for_fork_version(spec, fork_version, phases): if phases is None: return spec From e3cd216d3a2ed5c883a72e1e78f250af7dd9c904 Mon Sep 17 00:00:00 2001 From: Potuz Date: Wed, 3 Jul 2024 13:05:04 -0300 Subject: [PATCH 073/130] Fix python execution 5 --- tests/core/pyspec/eth2spec/test/helpers/block.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/core/pyspec/eth2spec/test/helpers/block.py b/tests/core/pyspec/eth2spec/test/helpers/block.py index a724b968bc..96a0155732 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/block.py +++ b/tests/core/pyspec/eth2spec/test/helpers/block.py @@ -1,4 +1,5 @@ -from eth2spec.test.helpers.execution_payload import build_empty_execution_payload, build_empty_signed_execution_payload_header +from eth2spec.test.helpers.execution_payload import build_empty_execution_payload +from eth2spec.test.helpers.execution_payload import build_empty_signed_execution_payload_header from eth2spec.test.helpers.forks import is_post_whisk, is_post_altair, is_post_bellatrix, is_post_eip7732 from eth2spec.test.helpers.keys import privkeys, whisk_ks_initial, whisk_ks_final from eth2spec.utils import bls From dba3cda3aa18ef9310ebbb5d8c435a1125316973 Mon Sep 17 00:00:00 2001 From: Potuz Date: Wed, 3 Jul 2024 13:20:58 -0300 Subject: [PATCH 074/130] Fix python execution 6 --- .../pyspec/eth2spec/test/helpers/execution_payload.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py index e1bd890dd5..1fbb12d7ba 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py +++ b/tests/core/pyspec/eth2spec/test/helpers/execution_payload.py @@ -287,10 +287,10 @@ def build_empty_execution_payload(spec, state, randao_mix=None): transactions=empty_txs, ) if not is_post_eip7732(spec): - payload.state_root = latest.state_root, # no changes to the state - payload.block_number = latest.block_number + 1, - payload.gas_limit = latest.gas_limit, # retain same limit - payload.base_fee_per_gas = latest.base_fee_per_gas, # retain same base_fee + payload.state_root = latest.state_root # no changes to the state + payload.block_number = latest.block_number + 1 + payload.gas_limit = latest.gas_limit # retain same limit + payload.base_fee_per_gas = latest.base_fee_per_gas # retain same base_fee if is_post_capella(spec): payload.withdrawals = get_expected_withdrawals(spec, state) if is_post_deneb(spec): From 53aaf26ba22de8fac8f5deb3cc39f6140f2c0b7f Mon Sep 17 00:00:00 2001 From: Potuz Date: Wed, 3 Jul 2024 13:40:16 -0300 Subject: [PATCH 075/130] Fix python execution 7 --- specs/_features/eip7732/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/_features/eip7732/beacon-chain.md b/specs/_features/eip7732/beacon-chain.md index 7094423b82..85b90695e8 100644 --- a/specs/_features/eip7732/beacon-chain.md +++ b/specs/_features/eip7732/beacon-chain.md @@ -359,7 +359,7 @@ def get_attesting_indices(state: BeaconState, attestation: Attestation) -> Set[V committee_offset += len(committee) ptc = get_ptc(state, attestation.data.slot) - return Set([i for i in output if i not in ptc]) + return set([i for i in output if i not in ptc]) ``` #### `get_payload_attesting_indices` From 3d48e72570338d065de0fc623717df8824285d5f Mon Sep 17 00:00:00 2001 From: Potuz Date: Wed, 3 Jul 2024 14:22:50 -0300 Subject: [PATCH 076/130] Deal with attestations from previous forks --- specs/_features/eip7732/beacon-chain.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/specs/_features/eip7732/beacon-chain.md b/specs/_features/eip7732/beacon-chain.md index 85b90695e8..1779d26b20 100644 --- a/specs/_features/eip7732/beacon-chain.md +++ b/specs/_features/eip7732/beacon-chain.md @@ -358,8 +358,10 @@ def get_attesting_indices(state: BeaconState, attestation: Attestation) -> Set[V committee_offset += len(committee) + if compute_epoch_at_slot(attestation.data.slot) < EIP7732_FORK_EPOCH: + return output ptc = get_ptc(state, attestation.data.slot) - return set([i for i in output if i not in ptc]) + return set(i for i in output if i not in ptc) ``` #### `get_payload_attesting_indices` From a5d216189dbb02541066fd648e151ce7d6566da8 Mon Sep 17 00:00:00 2001 From: Potuz Date: Wed, 3 Jul 2024 14:57:04 -0300 Subject: [PATCH 077/130] add CI workflow --- .circleci/config.yml | 13 +++++++++++++ .github/workflows/run-tests.yml | 2 +- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index bf8b7ada8a..7a95cc201b 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -194,6 +194,19 @@ jobs: command: make citest fork=eip7594 - store_test_results: path: tests/core/pyspec/test-reports + test-eip7732: + docker: + - image: circleci/python:3.9 + working_directory: ~/specs-repo + steps: + - restore_cache: + key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} + - restore_pyspec_cached_venv + - run: + name: Run py-tests + command: make citest fork=eip7732 + - store_test_results: + path: tests/core/pyspec/test-reports table_of_contents: docker: - image: circleci/node:10.16.3 diff --git a/.github/workflows/run-tests.yml b/.github/workflows/run-tests.yml index a56270030f..b581338c16 100644 --- a/.github/workflows/run-tests.yml +++ b/.github/workflows/run-tests.yml @@ -72,7 +72,7 @@ jobs: needs: [lint,codespell,table_of_contents] strategy: matrix: - version: ["phase0", "altair", "bellatrix", "capella", "deneb", "electra", "whisk", "eip7594"] + version: ["phase0", "altair", "bellatrix", "capella", "deneb", "electra", "whisk", "eip7594", "eip7732"] steps: - name: Checkout repository uses: actions/checkout@v4 From 614284c32a134ae6c3f744f17e0edcabef3d06e7 Mon Sep 17 00:00:00 2001 From: terence Date: Wed, 3 Jul 2024 11:11:38 -0700 Subject: [PATCH 078/130] Fix p2p to use eip7732 (#14) --- specs/_features/eip7732/builder.md | 4 ++-- specs/_features/eip7732/fork-choice.md | 4 ++-- specs/_features/eip7732/fork.md | 4 ++-- specs/_features/eip7732/p2p-interface.md | 12 ++++++------ specs/_features/eip7732/validator.md | 4 ++-- 5 files changed, 14 insertions(+), 14 deletions(-) diff --git a/specs/_features/eip7732/builder.md b/specs/_features/eip7732/builder.md index 1a95baac08..82fa0df088 100644 --- a/specs/_features/eip7732/builder.md +++ b/specs/_features/eip7732/builder.md @@ -1,4 +1,4 @@ -# ePBS -- Honest Builder +# EIP-7732 -- Honest Builder This is an accompanying document which describes the expected actions of a "builder" participating in the Ethereum proof-of-stake protocol. @@ -16,7 +16,7 @@ This is an accompanying document which describes the expected actions of a "buil ## Introduction -With the ePBS Fork, the protocol includes new staked participants of the protocol called *Builders*. While Builders are a subset of the validator set, they have extra attributions that are optional. Validators may opt to not be builders and as such we collect the set of guidelines for those validators that want to act as builders in this document. +With the EIP-7732 Fork, the protocol includes new staked participants of the protocol called *Builders*. While Builders are a subset of the validator set, they have extra attributions that are optional. Validators may opt to not be builders and as such we collect the set of guidelines for those validators that want to act as builders in this document. ## Builders attributions diff --git a/specs/_features/eip7732/fork-choice.md b/specs/_features/eip7732/fork-choice.md index ca7bdf051f..4bf68cfbc0 100644 --- a/specs/_features/eip7732/fork-choice.md +++ b/specs/_features/eip7732/fork-choice.md @@ -1,4 +1,4 @@ -# ePBS -- Fork Choice +# EIP-7732 -- Fork Choice ## Table of contents @@ -38,7 +38,7 @@ ## Introduction -This is the modification of the fork choice accompanying the ePBS upgrade. +This is the modification of the fork choice accompanying the EIP-7732 upgrade. ## Constants diff --git a/specs/_features/eip7732/fork.md b/specs/_features/eip7732/fork.md index 765b032d0d..942b9c30cc 100644 --- a/specs/_features/eip7732/fork.md +++ b/specs/_features/eip7732/fork.md @@ -1,4 +1,4 @@ -# ePBS -- Fork Logic +# EIP-7732 -- Fork Logic **Notice**: This document is a work-in-progress for researchers and implementers. @@ -68,7 +68,7 @@ For now, we assume the condition will be triggered at epoch `EIP7732_FORK_EPOCH` ### Upgrading the state If `state.slot % SLOTS_PER_EPOCH == 0` and `compute_epoch_at_slot(state.slot) == EIP7732_FORK_EPOCH`, -an irregular state change is made to upgrade to ePBS. +an irregular state change is made to upgrade to EIP-7732. ```python def upgrade_to_eip7732(pre: electra.BeaconState) -> BeaconState: diff --git a/specs/_features/eip7732/p2p-interface.md b/specs/_features/eip7732/p2p-interface.md index 1bb2d19e53..83a6579357 100644 --- a/specs/_features/eip7732/p2p-interface.md +++ b/specs/_features/eip7732/p2p-interface.md @@ -1,6 +1,6 @@ -# ePBS -- Networking +# EIP-7732 -- Networking -This document contains the consensus-layer networking specification for ePBS. +This document contains the consensus-layer networking specification for EIP7732. @@ -184,7 +184,7 @@ The following validations MUST pass before forwarding the `signed_execution_payl | `BELLATRIX_FORK_VERSION` | `bellatrix.SignedBeaconBlock` | | `CAPELLA_FORK_VERSION` | `capella.SignedBeaconBlock` | | `DENEB_FORK_VERSION` | `deneb.SignedBeaconBlock` | -| `EIP7732_FORK_VERSION` | `epbs.SignedBeaconBlock` | +| `EIP7732_FORK_VERSION` | `eip7732.SignedBeaconBlock` | ##### BeaconBlocksByRoot v3 @@ -201,7 +201,7 @@ Per `context = compute_fork_digest(fork_version, genesis_validators_root)`: | `BELLATRIX_FORK_VERSION` | `bellatrix.SignedBeaconBlock` | | `CAPELLA_FORK_VERSION` | `capella.SignedBeaconBlock` | | `DENEB_FORK_VERSION` | `deneb.SignedBeaconBlock` | -| `EIP7732_FORK_VERSION` | `epbs.SignedBeaconBlock` | +| `EIP7732_FORK_VERSION` | `eip7732.SignedBeaconBlock` | ##### BlobSidecarsByRoot v2 @@ -213,7 +213,7 @@ Per `context = compute_fork_digest(fork_version, genesis_validators_root)`: | `fork_version` | Chunk SSZ type | |--------------------------|-------------------------------| | `DENEB_FORK_VERSION` | `deneb.BlobSidecar` | -| `EIP7732_FORK_VERSION` | `epbs.BlobSidecar` | +| `EIP7732_FORK_VERSION` | `eip7732.BlobSidecar` | ##### ExecutionPayloadEnvelopeByRoot v1 @@ -226,7 +226,7 @@ The `` field is calculated as `context = compute_fork_digest(fork | `fork_version` | Chunk SSZ type | |---------------------|---------------------------------------| -| `EIP7732_FORK_VERSION` | `epbs.SignedExecutionPayloadEnvelope` | +| `EIP7732_FORK_VERSION` | `eip7732.SignedExecutionPayloadEnvelope` | Request Content: diff --git a/specs/_features/eip7732/validator.md b/specs/_features/eip7732/validator.md index e9d30d2726..4a520ba985 100644 --- a/specs/_features/eip7732/validator.md +++ b/specs/_features/eip7732/validator.md @@ -1,6 +1,6 @@ -# ePBS -- Honest Validator +# EIP-7732 -- Honest Validator -This document represents the changes and additions to the Honest validator guide included in the ePBS fork. +This document represents the changes and additions to the Honest validator guide included in the EIP-7732 fork. From eb3382faa59685e20fb20d3671ed1aa1fbff0394 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 4 Jul 2024 23:43:22 +0800 Subject: [PATCH 079/130] Disable eip7732 tests like whisk-style --- .circleci/config.yml | 13 ------------- .github/workflows/run-tests.yml | 2 +- .../core/pyspec/eth2spec/test/helpers/constants.py | 6 ++---- 3 files changed, 3 insertions(+), 18 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 7a95cc201b..bf8b7ada8a 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -194,19 +194,6 @@ jobs: command: make citest fork=eip7594 - store_test_results: path: tests/core/pyspec/test-reports - test-eip7732: - docker: - - image: circleci/python:3.9 - working_directory: ~/specs-repo - steps: - - restore_cache: - key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} - - restore_pyspec_cached_venv - - run: - name: Run py-tests - command: make citest fork=eip7732 - - store_test_results: - path: tests/core/pyspec/test-reports table_of_contents: docker: - image: circleci/node:10.16.3 diff --git a/.github/workflows/run-tests.yml b/.github/workflows/run-tests.yml index b581338c16..a56270030f 100644 --- a/.github/workflows/run-tests.yml +++ b/.github/workflows/run-tests.yml @@ -72,7 +72,7 @@ jobs: needs: [lint,codespell,table_of_contents] strategy: matrix: - version: ["phase0", "altair", "bellatrix", "capella", "deneb", "electra", "whisk", "eip7594", "eip7732"] + version: ["phase0", "altair", "bellatrix", "capella", "deneb", "electra", "whisk", "eip7594"] steps: - name: Checkout repository uses: actions/checkout@v4 diff --git a/tests/core/pyspec/eth2spec/test/helpers/constants.py b/tests/core/pyspec/eth2spec/test/helpers/constants.py index 4267ad21c8..3389311c3c 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/constants.py +++ b/tests/core/pyspec/eth2spec/test/helpers/constants.py @@ -38,14 +38,13 @@ ELECTRA, # Experimental patches EIP7594, - EIP7732, ) # The forks that have light client specs LIGHT_CLIENT_TESTING_FORKS = (*[item for item in MAINNET_FORKS if item != PHASE0],) # The forks that output to the test vectors. -TESTGEN_FORKS = (*MAINNET_FORKS, ELECTRA, EIP7594, WHISK, EIP7732) +TESTGEN_FORKS = (*MAINNET_FORKS, ELECTRA, EIP7594, WHISK) # Forks allowed in the test runner `--fork` flag, to fail fast in case of typos -ALLOWED_TEST_RUNNER_FORKS = (*ALL_PHASES, WHISK) +ALLOWED_TEST_RUNNER_FORKS = (*ALL_PHASES, WHISK, EIP7732) # NOTE: the same definition as in `pysetup/md_doc_paths.py` PREVIOUS_FORK_OF = { @@ -70,7 +69,6 @@ BELLATRIX: CAPELLA, CAPELLA: DENEB, DENEB: ELECTRA, - ELECTRA: EIP7732, } ALL_PRE_POST_FORKS = POST_FORK_OF.items() From 2af6e7d22c972088244f0b888dada86b6ba7a893 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Thu, 4 Jul 2024 23:43:42 +0800 Subject: [PATCH 080/130] Minor refactoring --- specs/_features/eip7732/beacon-chain.md | 65 ++++++++++++++++-------- specs/_features/eip7732/p2p-interface.md | 10 ++-- 2 files changed, 48 insertions(+), 27 deletions(-) diff --git a/specs/_features/eip7732/beacon-chain.md b/specs/_features/eip7732/beacon-chain.md index 1779d26b20..0b1d330f6d 100644 --- a/specs/_features/eip7732/beacon-chain.md +++ b/specs/_features/eip7732/beacon-chain.md @@ -29,6 +29,8 @@ - [Helper functions](#helper-functions) - [Math](#math) - [`bit_floor`](#bit_floor) + - [Misc](#misc-1) + - [`remove_flag`](#remove_flag) - [Predicates](#predicates) - [`is_valid_indexed_payload_attestation`](#is_valid_indexed_payload_attestation) - [`is_parent_block_full`](#is_parent_block_full) @@ -39,13 +41,17 @@ - [`get_indexed_payload_attestation`](#get_indexed_payload_attestation) - [Beacon chain state transition function](#beacon-chain-state-transition-function) - [Block processing](#block-processing) - - [Modified `process_withdrawals`](#modified-process_withdrawals) - - [New `verify_execution_payload_header_signature`](#new-verify_execution_payload_header_signature) - - [New `process_execution_payload_header`](#new-process_execution_payload_header) - - [Modified `process_operations`](#modified-process_operations) + - [Withdrawals](#withdrawals) + - [Modified `process_withdrawals`](#modified-process_withdrawals) + - [Execution payload header](#execution-payload-header) + - [New `verify_execution_payload_header_signature`](#new-verify_execution_payload_header_signature) + - [New `process_execution_payload_header`](#new-process_execution_payload_header) + - [Operations](#operations) + - [Modified `process_operations`](#modified-process_operations) - [Payload Attestations](#payload-attestations) - - [New `verify_execution_payload_envelope_signature`](#new-verify_execution_payload_envelope_signature) + - [`process_payload_attestation`](#process_payload_attestation) - [Modified `process_execution_payload`](#modified-process_execution_payload) + - [New `verify_execution_payload_envelope_signature`](#new-verify_execution_payload_envelope_signature) @@ -165,8 +171,8 @@ class SignedExecutionPayloadEnvelope(Container): ### Modified containers +#### `BeaconBlockBody` -#### `BeaconBlockBody` **Note:** The Beacon Block body is modified to contain a `Signed ExecutionPayloadHeader`. The containers `BeaconBlock` and `SignedBeaconBlock` are modified indirectly. ```python @@ -207,6 +213,7 @@ class ExecutionPayloadHeader(Container): ``` #### `BeaconState` + *Note*: The `BeaconState` is modified to track the last withdrawals honored in the CL. The `latest_execution_payload_header` is modified semantically to refer not to a past committed `ExecutionPayload` but instead it corresponds to the state's slot builder's bid. Another addition is to track the last committed block hash and the last slot that was full, that is in which there were both consensus and execution blocks included. ```python @@ -282,7 +289,17 @@ def bit_floor(n: uint64) -> uint64: return 0 return uint64(1) << (n.bit_length() - 1) ``` - + +### Misc + +#### `remove_flag` + +```python +def remove_flag(flags: ParticipationFlags, flag_index: int) -> ParticipationFlags: + flag = ParticipationFlags(2**flag_index) + return flags & ~flag +``` + ### Predicates #### `is_valid_indexed_payload_attestation` @@ -303,6 +320,7 @@ def is_valid_indexed_payload_attestation( indices = indexed_payload_attestation.attesting_indices if len(indices) == 0 or not indices == sorted(set(indices)): return False + # Verify aggregate signature pubkeys = [state.validators[i].pubkey for i in indices] domain = get_domain(state, DOMAIN_PTC_ATTESTER, None) @@ -340,6 +358,7 @@ def get_ptc(state: BeaconState, slot: Slot) -> Vector[ValidatorIndex, PTC_SIZE]: ``` #### Modified `get_attesting_indices` + `get_attesting_indices` is modified to ignore PTC votes ```python @@ -355,7 +374,6 @@ def get_attesting_indices(state: BeaconState, attestation: Attestation) -> Set[V committee_attesters = set( index for i, index in enumerate(committee) if attestation.aggregation_bits[committee_offset + i]) output = output.union(committee_attesters) - committee_offset += len(committee) if compute_epoch_at_slot(attestation.data.slot) < EIP7732_FORK_EPOCH: @@ -376,7 +394,6 @@ def get_payload_attesting_indices(state: BeaconState, slot: Slot, return set(index for i, index in enumerate(ptc) if payload_attestation.aggregation_bits[i]) ``` - #### `get_indexed_payload_attestation` ```python @@ -398,9 +415,9 @@ def get_indexed_payload_attestation(state: BeaconState, slot: Slot, *Note*: state transition is fundamentally modified in EIP-7732. The full state transition is broken in two parts, first importing a signed block and then importing an execution payload. -The post-state corresponding to a pre-state `state` and a signed beacon block `signed_block` is defined as `state_transition(state, signed_block)`. State transitions that trigger an unhandled exception (e.g. a failed `assert` or an out-of-range list access) are considered invalid. State transitions that cause a `uint64` overflow or underflow are also considered invalid. +The post-state corresponding to a pre-state `state` and a signed beacon block `signed_block` is defined as `state_transition(state, signed_block)`. State transitions that trigger an unhandled exception (e.g. a failed `assert` or an out-of-range list access) are considered invalid. State transitions that cause a `uint64` overflow or underflow are also considered invalid. -The post-state corresponding to a pre-state `state` and a signed execution payload envelope `signed_envelope` is defined as `process_execution_payload(state, signed_envelope)`. State transitions that trigger an unhandled exception (e.g. a failed `assert` or an out-of-range list access) are considered invalid. State transitions that cause a `uint64` overflow or underflow are also considered invalid. +The post-state corresponding to a pre-state `state` and a signed execution payload envelope `signed_envelope` is defined as `process_execution_payload(state, signed_envelope)`. State transitions that trigger an unhandled exception (e.g. a failed `assert` or an out-of-range list access) are considered invalid. State transitions that cause an `uint64` overflow or underflow are also considered invalid. ### Block processing @@ -415,7 +432,10 @@ def process_block(state: BeaconState, block: BeaconBlock) -> None: process_sync_aggregate(state, block.body.sync_aggregate) ``` -#### Modified `process_withdrawals` +#### Withdrawals + +##### Modified `process_withdrawals` + **Note:** This is modified to take only the `state` as parameter. Withdrawals are deterministic given the beacon state, any execution payload that has the corresponding block as parent beacon block is required to honor these withdrawals in the execution layer. This function must be called before `process_execution_payload_header` as this latter function affects validator balances. ```python @@ -449,7 +469,9 @@ def process_withdrawals(state: BeaconState) -> None: state.next_withdrawal_validator_index = next_validator_index ``` -#### New `verify_execution_payload_header_signature` +#### Execution payload header + +##### New `verify_execution_payload_header_signature` ```python def verify_execution_payload_header_signature(state: BeaconState, @@ -460,7 +482,7 @@ def verify_execution_payload_header_signature(state: BeaconState, return bls.Verify(builder.pubkey, signing_root, signed_header.signature) ``` -#### New `process_execution_payload_header` +##### New `process_execution_payload_header` ```python def process_execution_payload_header(state: BeaconState, block: BeaconBlock) -> None: @@ -488,7 +510,9 @@ def process_execution_payload_header(state: BeaconState, block: BeaconBlock) -> state.latest_execution_payload_header = header ``` -#### Modified `process_operations` +#### Operations + +##### Modified `process_operations` **Note:** `process_operations` is modified to process PTC attestations @@ -515,11 +539,7 @@ def process_operations(state: BeaconState, body: BeaconBlockBody) -> None: ##### Payload Attestations -```python -def remove_flag(flags: ParticipationFlags, flag_index: int) -> ParticipationFlags: - flag = ParticipationFlags(2**flag_index) - return flags & ~flag -``` +###### `process_payload_attestation` ```python def process_payload_attestation(state: BeaconState, payload_attestation: PayloadAttestation) -> None: @@ -569,7 +589,9 @@ def process_payload_attestation(state: BeaconState, payload_attestation: Payload increase_balance(state, proposer_index, proposer_reward) ``` -#### New `verify_execution_payload_envelope_signature` +#### Modified `process_execution_payload` + +##### New `verify_execution_payload_envelope_signature` ```python def verify_execution_payload_envelope_signature( @@ -579,7 +601,6 @@ def verify_execution_payload_envelope_signature( return bls.Verify(builder.pubkey, signing_root, signed_envelope.signature) ``` -#### Modified `process_execution_payload` *Note*: `process_execution_payload` is now an independent check in state transition. It is called when importing a signed execution payload proposed by the builder of the current slot. ```python diff --git a/specs/_features/eip7732/p2p-interface.md b/specs/_features/eip7732/p2p-interface.md index 83a6579357..5979e3975c 100644 --- a/specs/_features/eip7732/p2p-interface.md +++ b/specs/_features/eip7732/p2p-interface.md @@ -184,7 +184,7 @@ The following validations MUST pass before forwarding the `signed_execution_payl | `BELLATRIX_FORK_VERSION` | `bellatrix.SignedBeaconBlock` | | `CAPELLA_FORK_VERSION` | `capella.SignedBeaconBlock` | | `DENEB_FORK_VERSION` | `deneb.SignedBeaconBlock` | -| `EIP7732_FORK_VERSION` | `eip7732.SignedBeaconBlock` | +| `EIP7732_FORK_VERSION` | `eip7732.SignedBeaconBlock` | ##### BeaconBlocksByRoot v3 @@ -201,7 +201,7 @@ Per `context = compute_fork_digest(fork_version, genesis_validators_root)`: | `BELLATRIX_FORK_VERSION` | `bellatrix.SignedBeaconBlock` | | `CAPELLA_FORK_VERSION` | `capella.SignedBeaconBlock` | | `DENEB_FORK_VERSION` | `deneb.SignedBeaconBlock` | -| `EIP7732_FORK_VERSION` | `eip7732.SignedBeaconBlock` | +| `EIP7732_FORK_VERSION` | `eip7732.SignedBeaconBlock` | ##### BlobSidecarsByRoot v2 @@ -213,7 +213,7 @@ Per `context = compute_fork_digest(fork_version, genesis_validators_root)`: | `fork_version` | Chunk SSZ type | |--------------------------|-------------------------------| | `DENEB_FORK_VERSION` | `deneb.BlobSidecar` | -| `EIP7732_FORK_VERSION` | `eip7732.BlobSidecar` | +| `EIP7732_FORK_VERSION` | `eip7732.BlobSidecar` | ##### ExecutionPayloadEnvelopeByRoot v1 @@ -224,8 +224,8 @@ The `` field is calculated as `context = compute_fork_digest(fork [1]: # (eth2spec: skip) -| `fork_version` | Chunk SSZ type | -|---------------------|---------------------------------------| +| `fork_version` | Chunk SSZ type | +|------------------------|------------------------------------------| | `EIP7732_FORK_VERSION` | `eip7732.SignedExecutionPayloadEnvelope` | Request Content: From bb7ce4fa6efe73d6268ce3efa16e7b6dae746eb6 Mon Sep 17 00:00:00 2001 From: Potuz Date: Fri, 5 Jul 2024 09:12:04 -0300 Subject: [PATCH 081/130] minor fixes --- specs/_features/eip7732/beacon-chain.md | 2 +- specs/_features/eip7732/fork-choice.md | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/specs/_features/eip7732/beacon-chain.md b/specs/_features/eip7732/beacon-chain.md index 0b1d330f6d..4ff040c7e7 100644 --- a/specs/_features/eip7732/beacon-chain.md +++ b/specs/_features/eip7732/beacon-chain.md @@ -616,7 +616,7 @@ def process_execution_payload(state: BeaconState, assert envelope.beacon_block_root == hash_tree_root(state.latest_block_header) # Verify consistency with the committed header - committed_header = state.execution_payload_header + committed_header = state.latest_execution_payload_header assert envelope.builder_index == committed_header.builder_index assert committed_header.blob_kzg_commitments_root == hash_tree_root(envelope.blob_kzg_commitments) diff --git a/specs/_features/eip7732/fork-choice.md b/specs/_features/eip7732/fork-choice.md index 4bf68cfbc0..32b67c8712 100644 --- a/specs/_features/eip7732/fork-choice.md +++ b/specs/_features/eip7732/fork-choice.md @@ -140,6 +140,8 @@ def get_forkchoice_store(anchor_state: BeaconState, anchor_block: BeaconBlock) - block_states={anchor_root: copy(anchor_state)}, checkpoint_states={justified_checkpoint: copy(anchor_state)}, unrealized_justifications={anchor_root: justified_checkpoint}, + execution_payload_states={anchor_root: copy(anchor_state)}, # [New in EIP-7732] + ptc_vote={anchor_root: Vector[uint8, PTC_SIZE]()}, ) ``` From e93f923bb5aab79bdbc546392ac92c45267a18a8 Mon Sep 17 00:00:00 2001 From: Potuz Date: Fri, 5 Jul 2024 09:28:58 -0300 Subject: [PATCH 082/130] fix typo last_withdrawals_root --- specs/_features/eip7732/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/_features/eip7732/beacon-chain.md b/specs/_features/eip7732/beacon-chain.md index 4ff040c7e7..db81f9cd12 100644 --- a/specs/_features/eip7732/beacon-chain.md +++ b/specs/_features/eip7732/beacon-chain.md @@ -622,7 +622,7 @@ def process_execution_payload(state: BeaconState, if not envelope.payload_withheld: # Verify the withdrawals root - assert hash_tree_root(payload.withdrawals) == state.last_withdrawals_root + assert hash_tree_root(payload.withdrawals) == state.latest_withdrawals_root # Verify the gas_limit assert committed_header.gas_limit == payload.gas_limit From 9181e2937f0b0d6bc279e71a1fc1ea53679ef8b9 Mon Sep 17 00:00:00 2001 From: Potuz Date: Fri, 5 Jul 2024 12:24:17 -0300 Subject: [PATCH 083/130] modify get_blob_sidecars to take the commitments --- specs/_features/eip7732/builder.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/specs/_features/eip7732/builder.md b/specs/_features/eip7732/builder.md index 82fa0df088..f3fbb9e906 100644 --- a/specs/_features/eip7732/builder.md +++ b/specs/_features/eip7732/builder.md @@ -54,11 +54,12 @@ The builder assembles then `signed_execution_payload_header = SignedExecutionPay [Modified in EIP-7732] -The `BlobSidecar` container is modified indirectly because the constant `KZG_COMMITMENT_INCLUSION_PROOF_DEPTH` is modified. Each sidecar is obtained from the modified +The `BlobSidecar` container is modified indirectly because the constant `KZG_COMMITMENT_INCLUSION_PROOF_DEPTH` is modified. The function `get_blob_sidecars` is modified because the KZG commitments are no longer included in the beacon block but rather in the `ExecutionPayloadEnvelope`, the builder has to send the commitments as parameters to this function. ```python def get_blob_sidecars(signed_block: SignedBeaconBlock, blobs: Sequence[Blob], + blob_kzg_commitments: Sequence[KZGCommitment], blob_kzg_proofs: Sequence[KZGProof]) -> Sequence[BlobSidecar]: block = signed_block.message block_header = BeaconBlockHeader( @@ -73,7 +74,7 @@ def get_blob_sidecars(signed_block: SignedBeaconBlock, BlobSidecar( index=index, blob=blob, - kzg_commitment=block.body.blob_kzg_commitments[index], + kzg_commitment=blob_kzg_commitments[index], kzg_proof=blob_kzg_proofs[index], signed_block_header=signed_block_header, kzg_commitment_inclusion_proof=compute_merkle_proof( From 17b7905e5c01988aa39369a1431837c1e3b13672 Mon Sep 17 00:00:00 2001 From: Potuz Date: Mon, 8 Jul 2024 09:42:01 -0300 Subject: [PATCH 084/130] modify get_blob_sidecars to use nested proof --- specs/_features/eip7732/builder.md | 38 ++++++++++++++++-------- specs/_features/eip7732/p2p-interface.md | 5 ++-- 2 files changed, 28 insertions(+), 15 deletions(-) diff --git a/specs/_features/eip7732/builder.md b/specs/_features/eip7732/builder.md index f3fbb9e906..d0a7d73383 100644 --- a/specs/_features/eip7732/builder.md +++ b/specs/_features/eip7732/builder.md @@ -59,7 +59,7 @@ The `BlobSidecar` container is modified indirectly because the constant `KZG_COM ```python def get_blob_sidecars(signed_block: SignedBeaconBlock, blobs: Sequence[Blob], - blob_kzg_commitments: Sequence[KZGCommitment], + blob_kzg_commitments: List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK], blob_kzg_proofs: Sequence[KZGProof]) -> Sequence[BlobSidecar]: block = signed_block.message block_header = BeaconBlockHeader( @@ -70,20 +70,32 @@ def get_blob_sidecars(signed_block: SignedBeaconBlock, body_root=hash_tree_root(block.body), ) signed_block_header = SignedBeaconBlockHeader(message=block_header, signature=signed_block.signature) - return [ - BlobSidecar( - index=index, - blob=blob, - kzg_commitment=blob_kzg_commitments[index], - kzg_proof=blob_kzg_proofs[index], - signed_block_header=signed_block_header, - kzg_commitment_inclusion_proof=compute_merkle_proof( - block.body, - GeneralizedIndex(KZG_GENERALIZED_INDEX_PREFIX + index), + sidecars: List[BlobSidecar] = [] + for index, blob in enumerate(blobs): + inner_proof = compute_merkle_proof( + blob_kzg_commitments, + get_generalized_index(List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK], index), + ) + outer_proof = compute_merkle_proof( + block.body, + get_generalized_index( + BeaconBlockBody, + "signed_execution_payload_header", + "message", + "blob_kzg_commitments_root", ), ) - for index, blob in enumerate(blobs) - ] + sidecars.append( + BlobSidecar( + index=index, + blob=blob, + kzg_commitment=blob_kzg_commitments[index], + kzg_proof=blob_kzg_proofs[index], + signed_block_header=signed_block_header, + kzg_commitment_inclusion_proof=outer_proof + inner_proof, + ) + ) + return sidecars ``` ### Constructing the execution payload envelope diff --git a/specs/_features/eip7732/p2p-interface.md b/specs/_features/eip7732/p2p-interface.md index 5979e3975c..429f0a17d4 100644 --- a/specs/_features/eip7732/p2p-interface.md +++ b/specs/_features/eip7732/p2p-interface.md @@ -35,8 +35,9 @@ This document contains the consensus-layer networking specification for EIP7732. | Name | Value | Description | |------------------------------------------|-----------------------------------|---------------------------------------------------------------------| -| `KZG_COMMITMENT_INCLUSION_PROOF_DEPTH` | `0` # TODO: Compute it when the spec stabilizes | Merkle proof depth for the `blob_kzg_commitments` list item | -| `KZG_GENERALIZED_INDEX_PREFIX` | `0` # TODO: Compute it when the spec stabilizes | Generalized index for the first item in the `blob_kzg_commitments` list | +| `KZG_COMMITMENT_INCLUSION_PROOF_DEPTH` | `13` # TODO: Compute it when the spec stabilizes | Merkle proof depth for the `blob_kzg_commitments` list item | +| `KZG_GENERALIZED_INDEX_PREFIX` | `13` # TODO: Compute it when the spec stabilizes | Generalized index for the first item in the `blob_kzg_commitments` list | + ### Containers From 19c6f8456225a161afcb8819cfefb7829a3bd792 Mon Sep 17 00:00:00 2001 From: Potuz Date: Mon, 8 Jul 2024 10:21:56 -0300 Subject: [PATCH 085/130] linter --- specs/_features/eip7732/builder.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/specs/_features/eip7732/builder.md b/specs/_features/eip7732/builder.md index d0a7d73383..e51503bf34 100644 --- a/specs/_features/eip7732/builder.md +++ b/specs/_features/eip7732/builder.md @@ -85,6 +85,7 @@ def get_blob_sidecars(signed_block: SignedBeaconBlock, "blob_kzg_commitments_root", ), ) + proof = Vector[Bytes32, KZG_COMMITMENT_INCLUSION_PROOF_DEPTH](outer_proof + inner_proof) sidecars.append( BlobSidecar( index=index, @@ -92,7 +93,7 @@ def get_blob_sidecars(signed_block: SignedBeaconBlock, kzg_commitment=blob_kzg_commitments[index], kzg_proof=blob_kzg_proofs[index], signed_block_header=signed_block_header, - kzg_commitment_inclusion_proof=outer_proof + inner_proof, + kzg_commitment_inclusion_proof=proof, ) ) return sidecars From b556420e5ab1bf5b70e2c0210216a748fc9d0b15 Mon Sep 17 00:00:00 2001 From: Potuz Date: Mon, 8 Jul 2024 10:27:45 -0300 Subject: [PATCH 086/130] linter 2 --- specs/_features/eip7732/builder.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/specs/_features/eip7732/builder.md b/specs/_features/eip7732/builder.md index e51503bf34..9d28e2e6cc 100644 --- a/specs/_features/eip7732/builder.md +++ b/specs/_features/eip7732/builder.md @@ -72,11 +72,7 @@ def get_blob_sidecars(signed_block: SignedBeaconBlock, signed_block_header = SignedBeaconBlockHeader(message=block_header, signature=signed_block.signature) sidecars: List[BlobSidecar] = [] for index, blob in enumerate(blobs): - inner_proof = compute_merkle_proof( - blob_kzg_commitments, - get_generalized_index(List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK], index), - ) - outer_proof = compute_merkle_proof( + proof = compute_merkle_proof( block.body, get_generalized_index( BeaconBlockBody, @@ -85,7 +81,10 @@ def get_blob_sidecars(signed_block: SignedBeaconBlock, "blob_kzg_commitments_root", ), ) - proof = Vector[Bytes32, KZG_COMMITMENT_INCLUSION_PROOF_DEPTH](outer_proof + inner_proof) + proof += compute_merkle_proof( + blob_kzg_commitments, + get_generalized_index(List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK], index), + ) sidecars.append( BlobSidecar( index=index, @@ -93,7 +92,7 @@ def get_blob_sidecars(signed_block: SignedBeaconBlock, kzg_commitment=blob_kzg_commitments[index], kzg_proof=blob_kzg_proofs[index], signed_block_header=signed_block_header, - kzg_commitment_inclusion_proof=proof, + kzg_commitment_inclusion_proof=proof ) ) return sidecars From 0e94fc956b7e056a528753c25ec0934b4e382d27 Mon Sep 17 00:00:00 2001 From: Potuz Date: Mon, 8 Jul 2024 10:48:51 -0300 Subject: [PATCH 087/130] Rename changed preset --- presets/mainnet/eip-7732.yaml | 2 ++ presets/minimal/eip-7732.yaml | 2 ++ pysetup/spec_builders/eip7732.py | 4 ++++ specs/_features/eip7732/p2p-interface.md | 8 ++++---- 4 files changed, 12 insertions(+), 4 deletions(-) diff --git a/presets/mainnet/eip-7732.yaml b/presets/mainnet/eip-7732.yaml index f0a513cd21..eca2dd7056 100644 --- a/presets/mainnet/eip-7732.yaml +++ b/presets/mainnet/eip-7732.yaml @@ -6,3 +6,5 @@ PTC_SIZE: 512 # 2**2 (= 4) MAX_PAYLOAD_ATTESTATIONS: 4 +KZG_COMMITMENT_INCLUSION_PROOF_DEPTH_EIP7732: 13 +KZG_GENERALIZED_INDEX_PREFIX: 486 diff --git a/presets/minimal/eip-7732.yaml b/presets/minimal/eip-7732.yaml index c00e04f4bf..efbfdfad90 100644 --- a/presets/minimal/eip-7732.yaml +++ b/presets/minimal/eip-7732.yaml @@ -6,3 +6,5 @@ PTC_SIZE: 2 # 2**2 (= 4) MAX_PAYLOAD_ATTESTATIONS: 4 +KZG_COMMITMENT_INCLUSION_PROOF_DEPTH_EIP7732: 13 +KZG_GENERALIZED_INDEX_PREFIX: 486 diff --git a/pysetup/spec_builders/eip7732.py b/pysetup/spec_builders/eip7732.py index 49c1c880b7..2117f96332 100644 --- a/pysetup/spec_builders/eip7732.py +++ b/pysetup/spec_builders/eip7732.py @@ -18,6 +18,10 @@ def hardcoded_custom_type_dep_constants(cls, spec_object) -> Dict[str, str]: return { 'PTC_SIZE': spec_object.preset_vars['PTC_SIZE'].value, 'MAX_PAYLOAD_ATTESTATIONS': spec_object.preset_vars['MAX_PAYLOAD_ATTESTATIONS'].value, + 'KZG_COMMITMENT_INCLUSION_PROOF_DEPTH_EIP7732': + spec_object.preset_vars['KZG_COMMITMENT_INCLUSION_PROOF_DEPTH_EIP7732'].value, + 'KZG_GENERALIZED_INDEX_PREFIX': + spec_object.preset_vars['KZG_GENERALIZED_INDEX_PREFIX'].value, } @classmethod diff --git a/specs/_features/eip7732/p2p-interface.md b/specs/_features/eip7732/p2p-interface.md index 429f0a17d4..dd4e7a8271 100644 --- a/specs/_features/eip7732/p2p-interface.md +++ b/specs/_features/eip7732/p2p-interface.md @@ -35,8 +35,8 @@ This document contains the consensus-layer networking specification for EIP7732. | Name | Value | Description | |------------------------------------------|-----------------------------------|---------------------------------------------------------------------| -| `KZG_COMMITMENT_INCLUSION_PROOF_DEPTH` | `13` # TODO: Compute it when the spec stabilizes | Merkle proof depth for the `blob_kzg_commitments` list item | -| `KZG_GENERALIZED_INDEX_PREFIX` | `13` # TODO: Compute it when the spec stabilizes | Generalized index for the first item in the `blob_kzg_commitments` list | +| `KZG_COMMITMENT_INCLUSION_PROOF_DEPTH_EIP7732` | `13` # TODO: Compute it when the spec stabilizes | Merkle proof depth for the `blob_kzg_commitments` list item | +| `KZG_GENERALIZED_INDEX_PREFIX` | `486` # TODO: Compute it when the spec stabilizes | Generalized index for the first item in the `blob_kzg_commitments` list | ### Containers @@ -52,7 +52,7 @@ class BlobSidecar(Container): kzg_commitment: KZGCommitment kzg_proof: KZGProof # Allows for quick verification of kzg_commitment signed_block_header: SignedBeaconBlockHeader - kzg_commitment_inclusion_proof: Vector[Bytes32, KZG_COMMITMENT_INCLUSION_PROOF_DEPTH] + kzg_commitment_inclusion_proof: Vector[Bytes32, KZG_COMMITMENT_INCLUSION_PROOF_DEPTH_EIP7732] ``` #### Helpers @@ -69,7 +69,7 @@ def verify_blob_sidecar_inclusion_proof(blob_sidecar: BlobSidecar) -> bool: return is_valid_merkle_branch( leaf=blob_sidecar.kzg_commitment.hash_tree_root(), branch=blob_sidecar.kzg_commitment_inclusion_proof, - depth=KZG_COMMITMENT_INCLUSION_PROOF_DEPTH, + depth=KZG_COMMITMENT_INCLUSION_PROOF_DEPTH_EIP7732, index=gindex, root=blob_sidecar.signed_block_header.message.body_root, ) From 93aa7d6beed4b2ced3befd7fed33f08447ae62e3 Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Mon, 8 Jul 2024 21:52:52 +0800 Subject: [PATCH 088/130] Fix linter error --- pysetup/spec_builders/altair.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pysetup/spec_builders/altair.py b/pysetup/spec_builders/altair.py index aed5a8a1e9..830f396a9d 100644 --- a/pysetup/spec_builders/altair.py +++ b/pysetup/spec_builders/altair.py @@ -35,7 +35,7 @@ def get_generalized_index(ssz_class: Any, *path: PyUnion[int, SSZVariableName]) def compute_merkle_proof(object: SSZObject, - index: GeneralizedIndex) -> Sequence[Bytes32]: + index: GeneralizedIndex) -> list[Bytes32]: return build_proof(object.get_backing(), index)''' From 9274a661993d315a40f0e5ec3b3cac3a80e78825 Mon Sep 17 00:00:00 2001 From: Potuz Date: Mon, 8 Jul 2024 13:32:33 -0300 Subject: [PATCH 089/130] fix Merkle proof generator --- presets/mainnet/eip-7732.yaml | 1 - presets/minimal/eip-7732.yaml | 1 - pysetup/spec_builders/eip7732.py | 12 ++++++++++-- specs/_features/eip7732/builder.md | 8 ++++---- specs/_features/eip7732/p2p-interface.md | 14 +++++++++++--- 5 files changed, 25 insertions(+), 11 deletions(-) diff --git a/presets/mainnet/eip-7732.yaml b/presets/mainnet/eip-7732.yaml index eca2dd7056..eb43c981a9 100644 --- a/presets/mainnet/eip-7732.yaml +++ b/presets/mainnet/eip-7732.yaml @@ -7,4 +7,3 @@ PTC_SIZE: 512 # 2**2 (= 4) MAX_PAYLOAD_ATTESTATIONS: 4 KZG_COMMITMENT_INCLUSION_PROOF_DEPTH_EIP7732: 13 -KZG_GENERALIZED_INDEX_PREFIX: 486 diff --git a/presets/minimal/eip-7732.yaml b/presets/minimal/eip-7732.yaml index efbfdfad90..751c1325a5 100644 --- a/presets/minimal/eip-7732.yaml +++ b/presets/minimal/eip-7732.yaml @@ -7,4 +7,3 @@ PTC_SIZE: 2 # 2**2 (= 4) MAX_PAYLOAD_ATTESTATIONS: 4 KZG_COMMITMENT_INCLUSION_PROOF_DEPTH_EIP7732: 13 -KZG_GENERALIZED_INDEX_PREFIX: 486 diff --git a/pysetup/spec_builders/eip7732.py b/pysetup/spec_builders/eip7732.py index 2117f96332..0c335fe1d9 100644 --- a/pysetup/spec_builders/eip7732.py +++ b/pysetup/spec_builders/eip7732.py @@ -13,6 +13,16 @@ def imports(cls, preset_name: str): from eth2spec.electra import {preset_name} as electra ''' + @classmethod + def sundry_functions(cls) -> str: + return ''' +def concat_generalized_indices(*indices: GeneralizedIndex) -> GeneralizedIndex: + o = GeneralizedIndex(1) + for i in indices: + o = GeneralizedIndex(o * bit_floor(i) + (i - bit_floor(i))) + return o''' + + @classmethod def hardcoded_custom_type_dep_constants(cls, spec_object) -> Dict[str, str]: return { @@ -20,8 +30,6 @@ def hardcoded_custom_type_dep_constants(cls, spec_object) -> Dict[str, str]: 'MAX_PAYLOAD_ATTESTATIONS': spec_object.preset_vars['MAX_PAYLOAD_ATTESTATIONS'].value, 'KZG_COMMITMENT_INCLUSION_PROOF_DEPTH_EIP7732': spec_object.preset_vars['KZG_COMMITMENT_INCLUSION_PROOF_DEPTH_EIP7732'].value, - 'KZG_GENERALIZED_INDEX_PREFIX': - spec_object.preset_vars['KZG_GENERALIZED_INDEX_PREFIX'].value, } @classmethod diff --git a/specs/_features/eip7732/builder.md b/specs/_features/eip7732/builder.md index 9d28e2e6cc..491f625d89 100644 --- a/specs/_features/eip7732/builder.md +++ b/specs/_features/eip7732/builder.md @@ -73,6 +73,10 @@ def get_blob_sidecars(signed_block: SignedBeaconBlock, sidecars: List[BlobSidecar] = [] for index, blob in enumerate(blobs): proof = compute_merkle_proof( + blob_kzg_commitments, + get_generalized_index(List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK], index), + ) + proof += compute_merkle_proof( block.body, get_generalized_index( BeaconBlockBody, @@ -81,10 +85,6 @@ def get_blob_sidecars(signed_block: SignedBeaconBlock, "blob_kzg_commitments_root", ), ) - proof += compute_merkle_proof( - blob_kzg_commitments, - get_generalized_index(List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK], index), - ) sidecars.append( BlobSidecar( index=index, diff --git a/specs/_features/eip7732/p2p-interface.md b/specs/_features/eip7732/p2p-interface.md index dd4e7a8271..52546070f3 100644 --- a/specs/_features/eip7732/p2p-interface.md +++ b/specs/_features/eip7732/p2p-interface.md @@ -36,7 +36,6 @@ This document contains the consensus-layer networking specification for EIP7732. | Name | Value | Description | |------------------------------------------|-----------------------------------|---------------------------------------------------------------------| | `KZG_COMMITMENT_INCLUSION_PROOF_DEPTH_EIP7732` | `13` # TODO: Compute it when the spec stabilizes | Merkle proof depth for the `blob_kzg_commitments` list item | -| `KZG_GENERALIZED_INDEX_PREFIX` | `486` # TODO: Compute it when the spec stabilizes | Generalized index for the first item in the `blob_kzg_commitments` list | ### Containers @@ -63,8 +62,17 @@ class BlobSidecar(Container): ```python def verify_blob_sidecar_inclusion_proof(blob_sidecar: BlobSidecar) -> bool: - # hardcoded here because the block does not include the commitments but only their root. - gindex = GeneralizedIndex(KZG_GENERALIZED_INDEX_PREFIX + blob_sidecar.index) + inner_gindex = get_generalized_index( + List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK], + blob_sidecar.index + ) + outer_gindex = get_generalized_index( + BeaconBlockBody, + "signed_execution_payload_header", + "message", + "blob_kzg_commitments_root", + ) + gindex = get_subtree_index(concat_generalized_indices(outer_gindex, inner_gindex)) return is_valid_merkle_branch( leaf=blob_sidecar.kzg_commitment.hash_tree_root(), From f97f0b4ef235de858c0ceebb392feddfc203e798 Mon Sep 17 00:00:00 2001 From: Potuz Date: Mon, 8 Jul 2024 15:12:56 -0300 Subject: [PATCH 090/130] use List with limit to hash withdrawals --- specs/_features/eip7732/beacon-chain.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/specs/_features/eip7732/beacon-chain.md b/specs/_features/eip7732/beacon-chain.md index db81f9cd12..d4a37be461 100644 --- a/specs/_features/eip7732/beacon-chain.md +++ b/specs/_features/eip7732/beacon-chain.md @@ -445,7 +445,8 @@ def process_withdrawals(state: BeaconState) -> None: return withdrawals, partial_withdrawals_count = get_expected_withdrawals(state) - state.latest_withdrawals_root = hash_tree_root(withdrawals) + withdrawals_list = List[Withdrawal, MAX_WITHDRAWALS_PER_PAYLOAD](withdrawals) + state.latest_withdrawals_root = hash_tree_root(withdrawals_list) for withdrawal in withdrawals: decrease_balance(state, withdrawal.validator_index, withdrawal.amount) From 1dbb68d3d72cdfa40427582440229de07a2710cc Mon Sep 17 00:00:00 2001 From: Justin Traglia Date: Mon, 8 Jul 2024 14:07:53 -0500 Subject: [PATCH 091/130] Update get_data_column_sidecars to take cells/proofs --- specs/_features/eip7594/das-core.md | 34 ++++++++++++------- .../merkle_proof/test_single_merkle_proof.py | 3 +- 2 files changed, 24 insertions(+), 13 deletions(-) diff --git a/specs/_features/eip7594/das-core.md b/specs/_features/eip7594/das-core.md index 0d6530226d..30cb411bf9 100644 --- a/specs/_features/eip7594/das-core.md +++ b/specs/_features/eip7594/das-core.md @@ -195,27 +195,37 @@ def recover_matrix(partial_matrix: Sequence[MatrixEntry], ```python def get_data_column_sidecars(signed_block: SignedBeaconBlock, - blobs: Sequence[Blob]) -> Sequence[DataColumnSidecar]: + cells_and_kzg_proofs: Sequence[ + Tuple[ + Vector[Cell, CELLS_PER_EXT_BLOB], + Vector[KZGProof, CELLS_PER_EXT_BLOB]] + ]) -> Sequence[DataColumnSidecar]: + """ + Given a signed block and the cells/proofs associated with each blob in the + blob, assemble the sidecars which can be distributed to peers. + + Since there is no method which converts cells back to a blob, this method + takes cells/proofs instead of blobs so that it can be re-create sidecars + after recovery. + """ + blob_kzg_commitments = signed_block.message.body.blob_kzg_commitments + assert len(cells_and_kzg_proofs) == len(blob_kzg_commitments) signed_block_header = compute_signed_block_header(signed_block) - block = signed_block.message kzg_commitments_inclusion_proof = compute_merkle_proof( - block.body, + signed_block.message.body, get_generalized_index(BeaconBlockBody, 'blob_kzg_commitments'), ) - cells_and_proofs = [compute_cells_and_kzg_proofs(blob) for blob in blobs] - blob_count = len(blobs) - cells = [cells_and_proofs[i][0] for i in range(blob_count)] - proofs = [cells_and_proofs[i][1] for i in range(blob_count)] + sidecars = [] for column_index in range(NUMBER_OF_COLUMNS): - column_cells = [cells[row_index][column_index] - for row_index in range(blob_count)] - column_proofs = [proofs[row_index][column_index] - for row_index in range(blob_count)] + column_cells, column_proofs = [], [] + for cells, proofs in cells_and_kzg_proofs: + column_cells.append(cells[column_index]) + column_proofs.append(proofs[column_index]) sidecars.append(DataColumnSidecar( index=column_index, column=column_cells, - kzg_commitments=block.body.blob_kzg_commitments, + kzg_commitments=blob_kzg_commitments, kzg_proofs=column_proofs, signed_block_header=signed_block_header, kzg_commitments_inclusion_proof=kzg_commitments_inclusion_proof, diff --git a/tests/core/pyspec/eth2spec/test/eip7594/merkle_proof/test_single_merkle_proof.py b/tests/core/pyspec/eth2spec/test/eip7594/merkle_proof/test_single_merkle_proof.py index 3cdf3685e9..98c751508d 100644 --- a/tests/core/pyspec/eth2spec/test/eip7594/merkle_proof/test_single_merkle_proof.py +++ b/tests/core/pyspec/eth2spec/test/eip7594/merkle_proof/test_single_merkle_proof.py @@ -38,7 +38,8 @@ def _run_blob_kzg_commitments_merkle_proof_test(spec, state, rng=None): block.body.execution_payload.transactions = [opaque_tx] block.body.execution_payload.block_hash = compute_el_block_hash(spec, block.body.execution_payload, state) signed_block = sign_block(spec, state, block, proposer_index=0) - column_sidcars = spec.get_data_column_sidecars(signed_block, blobs) + cells_and_kzg_proofs = [spec.compute_cells_and_kzg_proofs(blob) for blob in blobs] + column_sidcars = spec.get_data_column_sidecars(signed_block, cells_and_kzg_proofs) column_sidcar = column_sidcars[0] yield "object", block.body From 8da90b27020813d0a90abc156d68446984fe37a1 Mon Sep 17 00:00:00 2001 From: Justin Traglia Date: Mon, 8 Jul 2024 14:20:45 -0500 Subject: [PATCH 092/130] Fix linter issues --- specs/_features/eip7594/das-core.md | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/specs/_features/eip7594/das-core.md b/specs/_features/eip7594/das-core.md index 30cb411bf9..c3f19b4cb0 100644 --- a/specs/_features/eip7594/das-core.md +++ b/specs/_features/eip7594/das-core.md @@ -195,18 +195,15 @@ def recover_matrix(partial_matrix: Sequence[MatrixEntry], ```python def get_data_column_sidecars(signed_block: SignedBeaconBlock, - cells_and_kzg_proofs: Sequence[ - Tuple[ - Vector[Cell, CELLS_PER_EXT_BLOB], - Vector[KZGProof, CELLS_PER_EXT_BLOB]] - ]) -> Sequence[DataColumnSidecar]: + cells_and_kzg_proofs: Sequence[Tuple[ + Vector[Cell, CELLS_PER_EXT_BLOB], + Vector[KZGProof, CELLS_PER_EXT_BLOB]]]) -> Sequence[DataColumnSidecar]: """ Given a signed block and the cells/proofs associated with each blob in the - blob, assemble the sidecars which can be distributed to peers. + block, assemble the sidecars which can be distributed to peers. - Since there is no method which converts cells back to a blob, this method - takes cells/proofs instead of blobs so that it can be re-create sidecars - after recovery. + Since there is no method which converts cells to a blob, this method takes + cells/proofs instead of blobs so it can re-create sidecars after recovery. """ blob_kzg_commitments = signed_block.message.body.blob_kzg_commitments assert len(cells_and_kzg_proofs) == len(blob_kzg_commitments) From b7149ef5faa4463aff7acf56d7dcb626863c030f Mon Sep 17 00:00:00 2001 From: Alex Stokes Date: Mon, 8 Jul 2024 17:52:06 -0600 Subject: [PATCH 093/130] Update beacon-chain.md Typo fix in deneb comment --- specs/deneb/beacon-chain.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/deneb/beacon-chain.md b/specs/deneb/beacon-chain.md index 4f036d3c93..cf5d1acd55 100644 --- a/specs/deneb/beacon-chain.md +++ b/specs/deneb/beacon-chain.md @@ -260,7 +260,7 @@ def is_valid_block_hash(self: ExecutionEngine, def is_valid_versioned_hashes(self: ExecutionEngine, new_payload_request: NewPayloadRequest) -> bool: """ Return ``True`` if and only if the version hashes computed by the blob transactions of - ``new_payload_request.execution_payload`` matches ``new_payload_request.version_hashes``. + ``new_payload_request.execution_payload`` matches ``new_payload_request.versioned_hashes``. """ ... ``` From 096e5649c9bb500c0d6f5b06a7519731b3e35904 Mon Sep 17 00:00:00 2001 From: Potuz Date: Tue, 9 Jul 2024 13:58:10 -0300 Subject: [PATCH 094/130] Verify kzg-commitments length with the payload --- specs/_features/eip7732/beacon-chain.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/specs/_features/eip7732/beacon-chain.md b/specs/_features/eip7732/beacon-chain.md index d4a37be461..d9f67b4712 100644 --- a/specs/_features/eip7732/beacon-chain.md +++ b/specs/_features/eip7732/beacon-chain.md @@ -635,6 +635,8 @@ def process_execution_payload(state: BeaconState, assert payload.prev_randao == get_randao_mix(state, get_current_epoch(state)) # Verify timestamp assert payload.timestamp == compute_timestamp_at_slot(state, state.slot) + # Verify commitments are under limit + assert len(envelope.blob_kzg_commitments) <= MAX_BLOBS_PER_BLOCK # Verify the execution payload is valid versioned_hashes = [kzg_commitment_to_versioned_hash(commitment) for commitment in envelope.blob_kzg_commitments] From 63433ec84795584b4c62523fc81f7104436b04e3 Mon Sep 17 00:00:00 2001 From: Etan Kissling Date: Tue, 9 Jul 2024 20:15:31 +0200 Subject: [PATCH 095/130] Use separate constant names for post-Electra gindices --- pysetup/spec_builders/electra.py | 6 +-- specs/altair/light-client/full-node.md | 6 +-- specs/electra/light-client/fork.md | 8 ++-- specs/electra/light-client/sync-protocol.md | 26 +++++++---- .../light_client/test_single_merkle_proof.py | 44 +++++++++++-------- .../test/altair/light_client/test_sync.py | 3 +- .../light_client/test_single_merkle_proof.py | 13 +++--- .../eth2spec/test/helpers/light_client.py | 30 ++++++++++--- 8 files changed, 85 insertions(+), 51 deletions(-) diff --git a/pysetup/spec_builders/electra.py b/pysetup/spec_builders/electra.py index 2b8febd347..ca02ee927c 100644 --- a/pysetup/spec_builders/electra.py +++ b/pysetup/spec_builders/electra.py @@ -15,7 +15,7 @@ def imports(cls, preset_name: str): @classmethod def hardcoded_ssz_dep_constants(cls) -> Dict[str, str]: return { - 'FINALIZED_ROOT_GINDEX': 'GeneralizedIndex(169)', - 'CURRENT_SYNC_COMMITTEE_GINDEX': 'GeneralizedIndex(86)', - 'NEXT_SYNC_COMMITTEE_GINDEX': 'GeneralizedIndex(87)', + 'FINALIZED_ROOT_GINDEX_ELECTRA': 'GeneralizedIndex(169)', + 'CURRENT_SYNC_COMMITTEE_GINDEX_ELECTRA': 'GeneralizedIndex(86)', + 'NEXT_SYNC_COMMITTEE_GINDEX_ELECTRA': 'GeneralizedIndex(87)', } diff --git a/specs/altair/light-client/full-node.md b/specs/altair/light-client/full-node.md index cea38613db..d887691c67 100644 --- a/specs/altair/light-client/full-node.md +++ b/specs/altair/light-client/full-node.md @@ -76,7 +76,7 @@ def create_light_client_bootstrap(state: BeaconState, header=block_to_light_client_header(block), current_sync_committee=state.current_sync_committee, current_sync_committee_branch=CurrentSyncCommitteeBranch( - compute_merkle_proof(state, CURRENT_SYNC_COMMITTEE_GINDEX)), + compute_merkle_proof(state, current_sync_committee_gindex_at_slot(state.slot))), ) ``` @@ -124,7 +124,7 @@ def create_light_client_update(state: BeaconState, if update_attested_period == update_signature_period: update.next_sync_committee = attested_state.next_sync_committee update.next_sync_committee_branch = NextSyncCommitteeBranch( - compute_merkle_proof(attested_state, NEXT_SYNC_COMMITTEE_GINDEX)) + compute_merkle_proof(attested_state, next_sync_committee_gindex_at_slot(attested_state.slot))) # Indicate finality whenever possible if finalized_block is not None: @@ -134,7 +134,7 @@ def create_light_client_update(state: BeaconState, else: assert attested_state.finalized_checkpoint.root == Bytes32() update.finality_branch = FinalityBranch( - compute_merkle_proof(attested_state, FINALIZED_ROOT_GINDEX)) + compute_merkle_proof(attested_state, finalized_root_gindex_at_slot(attested_state.slot))) update.sync_aggregate = block.message.body.sync_aggregate update.signature_slot = block.message.slot diff --git a/specs/electra/light-client/fork.md b/specs/electra/light-client/fork.md index 7bae9c3f4f..d613df56a9 100644 --- a/specs/electra/light-client/fork.md +++ b/specs/electra/light-client/fork.md @@ -71,7 +71,7 @@ def upgrade_lc_bootstrap_to_electra(pre: deneb.LightClientBootstrap) -> LightCli header=upgrade_lc_header_to_electra(pre.header), current_sync_committee=pre.current_sync_committee, current_sync_committee_branch=normalize_merkle_branch( - pre.current_sync_committee_branch, CURRENT_SYNC_COMMITTEE_GINDEX), + pre.current_sync_committee_branch, CURRENT_SYNC_COMMITTEE_GINDEX_ELECTRA), ) ``` @@ -81,10 +81,10 @@ def upgrade_lc_update_to_electra(pre: deneb.LightClientUpdate) -> LightClientUpd attested_header=upgrade_lc_header_to_electra(pre.attested_header), next_sync_committee=pre.next_sync_committee, next_sync_committee_branch=normalize_merkle_branch( - pre.next_sync_committee_branch, NEXT_SYNC_COMMITTEE_GINDEX), + pre.next_sync_committee_branch, NEXT_SYNC_COMMITTEE_GINDEX_ELECTRA), finalized_header=upgrade_lc_header_to_electra(pre.finalized_header), finality_branch=normalize_merkle_branch( - pre.finality_branch, FINALIZED_ROOT_GINDEX), + pre.finality_branch, FINALIZED_ROOT_GINDEX_ELECTRA), sync_aggregate=pre.sync_aggregate, signature_slot=pre.signature_slot, ) @@ -96,7 +96,7 @@ def upgrade_lc_finality_update_to_electra(pre: deneb.LightClientFinalityUpdate) attested_header=upgrade_lc_header_to_electra(pre.attested_header), finalized_header=upgrade_lc_header_to_electra(pre.finalized_header), finality_branch=normalize_merkle_branch( - pre.finality_branch, FINALIZED_ROOT_GINDEX), + pre.finality_branch, FINALIZED_ROOT_GINDEX_ELECTRA), sync_aggregate=pre.sync_aggregate, signature_slot=pre.signature_slot, ) diff --git a/specs/electra/light-client/sync-protocol.md b/specs/electra/light-client/sync-protocol.md index 53a3bcb78b..5c6436dd5c 100644 --- a/specs/electra/light-client/sync-protocol.md +++ b/specs/electra/light-client/sync-protocol.md @@ -28,13 +28,21 @@ Additional documents describes the impact of the upgrade on certain roles: - [Full node](./full-node.md) - [Networking](./p2p-interface.md) +## Custom types + +| Name | SSZ equivalent | Description | +| - | - | - | +| `FinalityBranch` | `Vector[Bytes32, floorlog2(FINALIZED_ROOT_GINDEX_ELECTRA)]` | Merkle branch of `finalized_checkpoint.root` within `BeaconState` | +| `CurrentSyncCommitteeBranch` | `Vector[Bytes32, floorlog2(CURRENT_SYNC_COMMITTEE_GINDEX_ELECTRA)]` | Merkle branch of `current_sync_committee` within `BeaconState` | +| `NextSyncCommitteeBranch` | `Vector[Bytes32, floorlog2(NEXT_SYNC_COMMITTEE_GINDEX_ELECTRA)]` | Merkle branch of `next_sync_committee` within `BeaconState` | + ## Constants | Name | Value | | - | - | -| `FINALIZED_ROOT_GINDEX` | `get_generalized_index(BeaconState, 'finalized_checkpoint', 'root')` (= 169) | -| `CURRENT_SYNC_COMMITTEE_GINDEX` | `get_generalized_index(BeaconState, 'current_sync_committee')` (= 86) | -| `NEXT_SYNC_COMMITTEE_GINDEX` | `get_generalized_index(BeaconState, 'next_sync_committee')` (= 87) | +| `FINALIZED_ROOT_GINDEX_ELECTRA` | `get_generalized_index(BeaconState, 'finalized_checkpoint', 'root')` (= 169) | +| `CURRENT_SYNC_COMMITTEE_GINDEX_ELECTRA` | `get_generalized_index(BeaconState, 'current_sync_committee')` (= 86) | +| `NEXT_SYNC_COMMITTEE_GINDEX_ELECTRA` | `get_generalized_index(BeaconState, 'next_sync_committee')` (= 87) | ## Helper functions @@ -46,8 +54,8 @@ def finalized_root_gindex_at_slot(slot: Slot) -> GeneralizedIndex: # [Modified in Electra] if epoch >= ELECTRA_FORK_EPOCH: - return FINALIZED_ROOT_GINDEX - return GeneralizedIndex(altair.FINALIZED_ROOT_GINDEX) + return FINALIZED_ROOT_GINDEX_ELECTRA + return GeneralizedIndex(FINALIZED_ROOT_GINDEX) ``` ### Modified `current_sync_committee_gindex_at_slot` @@ -58,8 +66,8 @@ def current_sync_committee_gindex_at_slot(slot: Slot) -> GeneralizedIndex: # [Modified in Electra] if epoch >= ELECTRA_FORK_EPOCH: - return CURRENT_SYNC_COMMITTEE_GINDEX - return GeneralizedIndex(altair.CURRENT_SYNC_COMMITTEE_GINDEX) + return CURRENT_SYNC_COMMITTEE_GINDEX_ELECTRA + return GeneralizedIndex(CURRENT_SYNC_COMMITTEE_GINDEX) ``` ### Modified `next_sync_committee_gindex_at_slot` @@ -70,8 +78,8 @@ def next_sync_committee_gindex_at_slot(slot: Slot) -> GeneralizedIndex: # [Modified in Electra] if epoch >= ELECTRA_FORK_EPOCH: - return NEXT_SYNC_COMMITTEE_GINDEX - return GeneralizedIndex(altair.NEXT_SYNC_COMMITTEE_GINDEX) + return NEXT_SYNC_COMMITTEE_GINDEX_ELECTRA + return GeneralizedIndex(NEXT_SYNC_COMMITTEE_GINDEX) ``` ### Modified `get_lc_execution_root` diff --git a/tests/core/pyspec/eth2spec/test/altair/light_client/test_single_merkle_proof.py b/tests/core/pyspec/eth2spec/test/altair/light_client/test_single_merkle_proof.py index 6418caafd9..dd2f4a7164 100644 --- a/tests/core/pyspec/eth2spec/test/altair/light_client/test_single_merkle_proof.py +++ b/tests/core/pyspec/eth2spec/test/altair/light_client/test_single_merkle_proof.py @@ -3,6 +3,11 @@ with_light_client, with_test_suite_name, ) +from eth2spec.test.helpers.light_client import ( + latest_current_sync_committee_gindex, + latest_finalized_root_gindex, + latest_next_sync_committee_gindex, +) @with_test_suite_name("BeaconState") @@ -10,17 +15,18 @@ @spec_state_test def test_current_sync_committee_merkle_proof(spec, state): yield "object", state - current_sync_committee_branch = spec.compute_merkle_proof(state, spec.CURRENT_SYNC_COMMITTEE_GINDEX) + gindex = latest_current_sync_committee_gindex(spec) + branch = spec.compute_merkle_proof(state, gindex) yield "proof", { "leaf": "0x" + state.current_sync_committee.hash_tree_root().hex(), - "leaf_index": spec.CURRENT_SYNC_COMMITTEE_GINDEX, - "branch": ['0x' + root.hex() for root in current_sync_committee_branch] + "leaf_index": gindex, + "branch": ['0x' + root.hex() for root in branch] } assert spec.is_valid_merkle_branch( leaf=state.current_sync_committee.hash_tree_root(), - branch=current_sync_committee_branch, - depth=spec.floorlog2(spec.CURRENT_SYNC_COMMITTEE_GINDEX), - index=spec.get_subtree_index(spec.CURRENT_SYNC_COMMITTEE_GINDEX), + branch=branch, + depth=spec.floorlog2(gindex), + index=spec.get_subtree_index(gindex), root=state.hash_tree_root(), ) @@ -30,17 +36,18 @@ def test_current_sync_committee_merkle_proof(spec, state): @spec_state_test def test_next_sync_committee_merkle_proof(spec, state): yield "object", state - next_sync_committee_branch = spec.compute_merkle_proof(state, spec.NEXT_SYNC_COMMITTEE_GINDEX) + gindex = latest_next_sync_committee_gindex(spec) + branch = spec.compute_merkle_proof(state, gindex) yield "proof", { "leaf": "0x" + state.next_sync_committee.hash_tree_root().hex(), - "leaf_index": spec.NEXT_SYNC_COMMITTEE_GINDEX, - "branch": ['0x' + root.hex() for root in next_sync_committee_branch] + "leaf_index": gindex, + "branch": ['0x' + root.hex() for root in branch] } assert spec.is_valid_merkle_branch( leaf=state.next_sync_committee.hash_tree_root(), - branch=next_sync_committee_branch, - depth=spec.floorlog2(spec.NEXT_SYNC_COMMITTEE_GINDEX), - index=spec.get_subtree_index(spec.NEXT_SYNC_COMMITTEE_GINDEX), + branch=branch, + depth=spec.floorlog2(gindex), + index=spec.get_subtree_index(gindex), root=state.hash_tree_root(), ) @@ -50,17 +57,18 @@ def test_next_sync_committee_merkle_proof(spec, state): @spec_state_test def test_finality_root_merkle_proof(spec, state): yield "object", state - finality_branch = spec.compute_merkle_proof(state, spec.FINALIZED_ROOT_GINDEX) + gindex = latest_finalized_root_gindex(spec) + branch = spec.compute_merkle_proof(state, gindex) yield "proof", { "leaf": "0x" + state.finalized_checkpoint.root.hex(), - "leaf_index": spec.FINALIZED_ROOT_GINDEX, - "branch": ['0x' + root.hex() for root in finality_branch] + "leaf_index": gindex, + "branch": ['0x' + root.hex() for root in branch] } assert spec.is_valid_merkle_branch( leaf=state.finalized_checkpoint.root, - branch=finality_branch, - depth=spec.floorlog2(spec.FINALIZED_ROOT_GINDEX), - index=spec.get_subtree_index(spec.FINALIZED_ROOT_GINDEX), + branch=branch, + depth=spec.floorlog2(gindex), + index=spec.get_subtree_index(gindex), root=state.hash_tree_root(), ) diff --git a/tests/core/pyspec/eth2spec/test/altair/light_client/test_sync.py b/tests/core/pyspec/eth2spec/test/altair/light_client/test_sync.py index acd64cf3bd..45c7d77887 100644 --- a/tests/core/pyspec/eth2spec/test/altair/light_client/test_sync.py +++ b/tests/core/pyspec/eth2spec/test/altair/light_client/test_sync.py @@ -157,8 +157,7 @@ def emit_update(test, spec, state, block, attested_state, attested_block, finali data = d_spec.create_light_client_update(state, block, attested_state, attested_block, finalized_block) if not with_next: data.next_sync_committee = spec.SyncCommittee() - data.next_sync_committee_branch = \ - [spec.Bytes32() for _ in range(spec.floorlog2(spec.NEXT_SYNC_COMMITTEE_GINDEX))] + data.next_sync_committee_branch = spec.NextSyncCommitteeBranch() current_slot = state.slot upgraded = upgrade_lc_update_to_new_spec(d_spec, test.s_spec, data, phases) diff --git a/tests/core/pyspec/eth2spec/test/capella/light_client/test_single_merkle_proof.py b/tests/core/pyspec/eth2spec/test/capella/light_client/test_single_merkle_proof.py index 41bb3f307e..7f414ab285 100644 --- a/tests/core/pyspec/eth2spec/test/capella/light_client/test_single_merkle_proof.py +++ b/tests/core/pyspec/eth2spec/test/capella/light_client/test_single_merkle_proof.py @@ -15,16 +15,17 @@ def test_execution_merkle_proof(spec, state): block = state_transition_with_full_block(spec, state, True, False) yield "object", block.message.body - execution_branch = spec.compute_merkle_proof(block.message.body, spec.EXECUTION_PAYLOAD_GINDEX) + gindex = spec.EXECUTION_PAYLOAD_GINDEX + branch = spec.compute_merkle_proof(block.message.body, gindex) yield "proof", { "leaf": "0x" + block.message.body.execution_payload.hash_tree_root().hex(), - "leaf_index": spec.EXECUTION_PAYLOAD_GINDEX, - "branch": ['0x' + root.hex() for root in execution_branch] + "leaf_index": gindex, + "branch": ['0x' + root.hex() for root in branch] } assert spec.is_valid_merkle_branch( leaf=block.message.body.execution_payload.hash_tree_root(), - branch=execution_branch, - depth=spec.floorlog2(spec.EXECUTION_PAYLOAD_GINDEX), - index=spec.get_subtree_index(spec.EXECUTION_PAYLOAD_GINDEX), + branch=branch, + depth=spec.floorlog2(gindex), + index=spec.get_subtree_index(gindex), root=block.message.body.hash_tree_root(), ) diff --git a/tests/core/pyspec/eth2spec/test/helpers/light_client.py b/tests/core/pyspec/eth2spec/test/helpers/light_client.py index 53544c5555..a0acf14151 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/light_client.py +++ b/tests/core/pyspec/eth2spec/test/helpers/light_client.py @@ -14,6 +14,24 @@ from math import floor +def latest_finalized_root_gindex(spec): + if hasattr(spec, 'FINALIZED_ROOT_GINDEX_ELECTRA'): + return spec.FINALIZED_ROOT_GINDEX_ELECTRA + return spec.FINALIZED_ROOT_GINDEX + + +def latest_current_sync_committee_gindex(spec): + if hasattr(spec, 'CURRENT_SYNC_COMMITTEE_GINDEX_ELECTRA'): + return spec.CURRENT_SYNC_COMMITTEE_GINDEX_ELECTRA + return spec.CURRENT_SYNC_COMMITTEE_GINDEX + + +def latest_next_sync_committee_gindex(spec): + if hasattr(spec, 'NEXT_SYNC_COMMITTEE_GINDEX_ELECTRA'): + return spec.NEXT_SYNC_COMMITTEE_GINDEX_ELECTRA + return spec.NEXT_SYNC_COMMITTEE_GINDEX + + def compute_start_slot_at_sync_committee_period(spec, sync_committee_period): return spec.compute_start_slot_at_epoch(sync_committee_period * spec.EPOCHS_PER_SYNC_COMMITTEE_PERIOD) @@ -71,11 +89,11 @@ def create_update(spec, if with_next: update.next_sync_committee = attested_state.next_sync_committee - update.next_sync_committee_branch = spec.compute_merkle_proof(attested_state, spec.NEXT_SYNC_COMMITTEE_GINDEX) + update.next_sync_committee_branch = spec.compute_merkle_proof(attested_state, latest_next_sync_committee_gindex(spec)) if with_finality: update.finalized_header = spec.block_to_light_client_header(finalized_block) - update.finality_branch = spec.compute_merkle_proof(attested_state, spec.FINALIZED_ROOT_GINDEX) + update.finality_branch = spec.compute_merkle_proof(attested_state, latest_finalized_root_gindex(spec)) update.sync_aggregate, update.signature_slot = get_sync_aggregate( spec, attested_state, num_participants) @@ -141,7 +159,7 @@ def check_lc_bootstrap_equal(spec, new_spec, data, upgraded): new_spec, data.current_sync_committee_branch, upgraded.current_sync_committee_branch, - new_spec.CURRENT_SYNC_COMMITTEE_GINDEX, + latest_current_sync_committee_gindex(new_spec), ) @@ -171,7 +189,7 @@ def check_lc_update_equal(spec, new_spec, data, upgraded): new_spec, data.next_sync_committee_branch, upgraded.next_sync_committee_branch, - new_spec.NEXT_SYNC_COMMITTEE_GINDEX, + latest_next_sync_committee_gindex(new_spec), ) check_lc_header_equal(spec, new_spec, data.finalized_header, upgraded.finalized_header) check_merkle_branch_equal( @@ -179,7 +197,7 @@ def check_lc_update_equal(spec, new_spec, data, upgraded): new_spec, data.finality_branch, upgraded.finality_branch, - new_spec.FINALIZED_ROOT_GINDEX, + latest_finalized_root_gindex(new_spec), ) assert upgraded.sync_aggregate == data.sync_aggregate assert upgraded.signature_slot == data.signature_slot @@ -211,7 +229,7 @@ def check_lc_finality_update_equal(spec, new_spec, data, upgraded): new_spec, data.finality_branch, upgraded.finality_branch, - new_spec.FINALIZED_ROOT_GINDEX, + latest_finalized_root_gindex(new_spec), ) assert upgraded.sync_aggregate == data.sync_aggregate assert upgraded.signature_slot == data.signature_slot From a86eeb3ee094be35e3faee7dfc884e54a2ec79dc Mon Sep 17 00:00:00 2001 From: Etan Kissling Date: Tue, 9 Jul 2024 20:23:27 +0200 Subject: [PATCH 096/130] Freeze Altair constants at Deneb --- specs/electra/light-client/sync-protocol.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/specs/electra/light-client/sync-protocol.md b/specs/electra/light-client/sync-protocol.md index 5c6436dd5c..724d3c8497 100644 --- a/specs/electra/light-client/sync-protocol.md +++ b/specs/electra/light-client/sync-protocol.md @@ -9,7 +9,10 @@ - [Introduction](#introduction) +- [Custom types](#custom-types) - [Constants](#constants) + - [Frozen constants](#frozen-constants) + - [New constants](#new-constants) - [Helper functions](#helper-functions) - [Modified `finalized_root_gindex_at_slot`](#modified-finalized_root_gindex_at_slot) - [Modified `current_sync_committee_gindex_at_slot`](#modified-current_sync_committee_gindex_at_slot) @@ -38,6 +41,18 @@ Additional documents describes the impact of the upgrade on certain roles: ## Constants +### Frozen constants + +Existing `GeneralizedIndex` constants are frozen at their [Altair](../../altair/light-client/sync-protocol#constants) values. + +| Name | Value | +| - | - | +| `FINALIZED_ROOT_GINDEX` | `get_generalized_index(altair.BeaconState, 'finalized_checkpoint', 'root')` (= 105) | +| `CURRENT_SYNC_COMMITTEE_GINDEX` | `get_generalized_index(altair.BeaconState, 'current_sync_committee')` (= 54) | +| `NEXT_SYNC_COMMITTEE_GINDEX` | `get_generalized_index(altair.BeaconState, 'next_sync_committee')` (= 55) | + +### New constants + | Name | Value | | - | - | | `FINALIZED_ROOT_GINDEX_ELECTRA` | `get_generalized_index(BeaconState, 'finalized_checkpoint', 'root')` (= 169) | From 82fe862a47b4e783144716a9f867dfcb9049d6f0 Mon Sep 17 00:00:00 2001 From: Etan Kissling Date: Tue, 9 Jul 2024 21:22:30 +0200 Subject: [PATCH 097/130] Fix lint --- tests/core/pyspec/eth2spec/test/helpers/light_client.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/helpers/light_client.py b/tests/core/pyspec/eth2spec/test/helpers/light_client.py index a0acf14151..4638c988b5 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/light_client.py +++ b/tests/core/pyspec/eth2spec/test/helpers/light_client.py @@ -89,11 +89,13 @@ def create_update(spec, if with_next: update.next_sync_committee = attested_state.next_sync_committee - update.next_sync_committee_branch = spec.compute_merkle_proof(attested_state, latest_next_sync_committee_gindex(spec)) + update.next_sync_committee_branch = spec.compute_merkle_proof( + attested_state, latest_next_sync_committee_gindex(spec)) if with_finality: update.finalized_header = spec.block_to_light_client_header(finalized_block) - update.finality_branch = spec.compute_merkle_proof(attested_state, latest_finalized_root_gindex(spec)) + update.finality_branch = spec.compute_merkle_proof( + attested_state, latest_finalized_root_gindex(spec)) update.sync_aggregate, update.signature_slot = get_sync_aggregate( spec, attested_state, num_participants) From 088f77161b8882ba93ee6461fb9c4395fd4eaeba Mon Sep 17 00:00:00 2001 From: Matt Kocubinski Date: Tue, 9 Jul 2024 21:26:12 +0200 Subject: [PATCH 098/130] fix spec error in get_generalized_index function --- ssz/merkle-proofs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ssz/merkle-proofs.md b/ssz/merkle-proofs.md index 919ff07f59..9b9127d63a 100644 --- a/ssz/merkle-proofs.md +++ b/ssz/merkle-proofs.md @@ -176,8 +176,8 @@ def get_generalized_index(typ: SSZType, *path: PyUnion[int, SSZVariableName]) -> for p in path: assert not issubclass(typ, BasicValue) # If we descend to a basic type, the path cannot continue further if p == '__len__': - typ = uint64 assert issubclass(typ, (List, ByteList)) + typ = uint64 root = GeneralizedIndex(root * 2 + 1) else: pos, _, _ = get_item_position(typ, p) From 9a9fa965b0b24544ad6eefe5afa0901c096987cd Mon Sep 17 00:00:00 2001 From: Justin Traglia <95511699+jtraglia@users.noreply.github.com> Date: Wed, 10 Jul 2024 10:41:37 -0500 Subject: [PATCH 099/130] In batch cell verification, take commitment for each cell (#3834) --- specs/_features/eip7594/p2p-interface.md | 8 +- .../polynomial-commitments-sampling.md | 39 +-- .../test_polynomial_commitments.py | 17 +- .../kzg_7594/verify_cell_kzg_proof_batch.md | 8 +- tests/generators/kzg_7594/main.py | 228 ++++++------------ 5 files changed, 106 insertions(+), 194 deletions(-) diff --git a/specs/_features/eip7594/p2p-interface.md b/specs/_features/eip7594/p2p-interface.md index 582ee65140..0c9a7cd178 100644 --- a/specs/_features/eip7594/p2p-interface.md +++ b/specs/_features/eip7594/p2p-interface.md @@ -68,19 +68,17 @@ class DataColumnIdentifier(Container): ```python def verify_data_column_sidecar_kzg_proofs(sidecar: DataColumnSidecar) -> bool: """ - Verify if the proofs are correct + Verify if the proofs are correct. """ assert sidecar.index < NUMBER_OF_COLUMNS assert len(sidecar.column) == len(sidecar.kzg_commitments) == len(sidecar.kzg_proofs) - row_indices = [RowIndex(i) for i in range(len(sidecar.column))] column_indices = [sidecar.index] * len(sidecar.column) # KZG batch verifies that the cells match the corresponding commitments and proofs return verify_cell_kzg_proof_batch( - row_commitments_bytes=sidecar.kzg_commitments, - row_indices=row_indices, # all rows - column_indices=column_indices, # specific column + commitments_bytes=sidecar.kzg_commitments, + column_indices=column_indices, cells=sidecar.column, proofs_bytes=sidecar.kzg_proofs, ) diff --git a/specs/_features/eip7594/polynomial-commitments-sampling.md b/specs/_features/eip7594/polynomial-commitments-sampling.md index 999e5ef343..2d343dc6ea 100644 --- a/specs/_features/eip7594/polynomial-commitments-sampling.md +++ b/specs/_features/eip7594/polynomial-commitments-sampling.md @@ -481,12 +481,18 @@ def verify_cell_kzg_proof_batch_impl(row_commitments: Sequence[KZGCommitment], cosets_evals: Sequence[CosetEvals], proofs: Sequence[KZGProof]) -> bool: """ - Verify a set of cells, given their corresponding proofs and their coordinates (row_index, column_index) in the blob - matrix. The i-th cell is in row row_indices[i] and in column column_indices[i]. - The list of all commitments is provided in row_commitments_bytes. + Helper: Verify that a set of cells belong to their corresponding commitment. - This function is the internal implementation of verify_cell_kzg_proof_batch. + Given a list of ``row_commitments`` and four lists representing tuples of (``row_index``, + ``column_index``, ``evals``, ``proof``), the function verifies ``proof`` which shows that + ``evals`` are the evaluations of the polynomial associated with ``row_commitments[row_index]``, + evaluated over the domain specified by ``column_index``. + + This function is the internal implementation of ``verify_cell_kzg_proof_batch``. """ + assert len(row_indices) == len(column_indices) == len(cosets_evals) == len(proofs) + for row_index in row_indices: + assert row_index < len(row_commitments) # The verification equation that we will check is pairing (LL, LR) = pairing (RL, [1]), where # LL = sum_k r^k proofs[k], @@ -670,15 +676,16 @@ def verify_cell_kzg_proof(commitment_bytes: Bytes48, #### `verify_cell_kzg_proof_batch` ```python -def verify_cell_kzg_proof_batch(row_commitments_bytes: Sequence[Bytes48], - row_indices: Sequence[RowIndex], +def verify_cell_kzg_proof_batch(commitments_bytes: Sequence[Bytes48], column_indices: Sequence[ColumnIndex], cells: Sequence[Cell], proofs_bytes: Sequence[Bytes48]) -> bool: """ - Verify a set of cells, given their corresponding proofs and their coordinates (row_index, column_index) in the blob - matrix. The i-th cell is in row = row_indices[i] and in column = column_indices[i]. - The list of all commitments is provided in row_commitments_bytes. + Verify that a set of cells belong to their corresponding commitments. + + Given four lists representing tuples of (``commitment``, ``column_index``, ``cell``, ``proof``), + the function verifies ``proof`` which shows that ``cell`` are the evaluations of the polynomial + associated with ``commitment``, evaluated over the domain specified by ``column_index``. This function implements the universal verification equation that has been introduced here: https://ethresear.ch/t/a-universal-verification-equation-for-data-availability-sampling/13240 @@ -686,11 +693,9 @@ def verify_cell_kzg_proof_batch(row_commitments_bytes: Sequence[Bytes48], Public method. """ - assert len(cells) == len(proofs_bytes) == len(row_indices) == len(column_indices) - for commitment_bytes in row_commitments_bytes: + assert len(commitments_bytes) == len(cells) == len(proofs_bytes) == len(column_indices) + for commitment_bytes in commitments_bytes: assert len(commitment_bytes) == BYTES_PER_COMMITMENT - for row_index in row_indices: - assert row_index < len(row_commitments_bytes) for column_index in column_indices: assert column_index < CELLS_PER_EXT_BLOB for cell in cells: @@ -698,8 +703,12 @@ def verify_cell_kzg_proof_batch(row_commitments_bytes: Sequence[Bytes48], for proof_bytes in proofs_bytes: assert len(proof_bytes) == BYTES_PER_PROOF - # Get objects from bytes - row_commitments = [bytes_to_kzg_commitment(commitment_bytes) for commitment_bytes in row_commitments_bytes] + # Create the list of unique commitments we are dealing with + deduplicated_commitments = list(set(commitments_bytes)) + row_commitments = [bytes_to_kzg_commitment(commitment_bytes) for commitment_bytes in deduplicated_commitments] + # Create indices list mapping initial commitments (that potentially contains duplicates) to the unique commitments. + row_indices = [deduplicated_commitments.index(commitment_bytes) for commitment_bytes in commitments_bytes] + cosets_evals = [cell_to_coset_evals(cell) for cell in cells] proofs = [bytes_to_kzg_proof(proof_bytes) for proof_bytes in proofs_bytes] diff --git a/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py b/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py index dcb43f4fa0..c94b085c5a 100644 --- a/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py +++ b/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py @@ -128,8 +128,7 @@ def test_verify_cell_kzg_proof(spec): def test_verify_cell_kzg_proof_batch_zero_cells(spec): # Verify with zero cells should return true assert spec.verify_cell_kzg_proof_batch( - row_commitments_bytes=[], - row_indices=[], + commitments_bytes=[], column_indices=[], cells=[], proofs_bytes=[], @@ -149,8 +148,7 @@ def test_verify_cell_kzg_proof_batch(spec): assert len(cells) == len(proofs) assert spec.verify_cell_kzg_proof_batch( - row_commitments_bytes=[commitment], - row_indices=[0, 0], + commitments_bytes=[commitment, commitment], column_indices=[0, 4], cells=[cells[0], cells[4]], proofs_bytes=[proofs[0], proofs[4]], @@ -178,11 +176,11 @@ def test_verify_cell_kzg_proof_batch(spec): column_indices = [0, 4, 0, 1, 2] cells = [all_cells[i][j] for (i, j) in zip(row_indices, column_indices)] proofs = [all_proofs[i][j] for (i, j) in zip(row_indices, column_indices)] + commitments = [all_commitments[i] for i in row_indices] # do the check assert spec.verify_cell_kzg_proof_batch( - row_commitments_bytes=all_commitments, - row_indices=row_indices, + commitments_bytes=commitments, column_indices=column_indices, cells=cells, proofs_bytes=proofs, @@ -202,8 +200,7 @@ def test_verify_cell_kzg_proof_batch_invalid(spec): assert len(cells) == len(proofs) assert not spec.verify_cell_kzg_proof_batch( - row_commitments_bytes=[commitment], - row_indices=[0, 0], + commitments_bytes=[commitment, commitment], column_indices=[0, 4], cells=[cells[0], cells[5]], # Note: this is where it should go wrong proofs_bytes=[proofs[0], proofs[4]], @@ -231,14 +228,14 @@ def test_verify_cell_kzg_proof_batch_invalid(spec): column_indices = [0, 4, 0, 1, 2] cells = [all_cells[i][j] for (i, j) in zip(row_indices, column_indices)] proofs = [all_proofs[i][j] for (i, j) in zip(row_indices, column_indices)] + commitments = [all_commitments[i] for i in row_indices] # let's change one of the cells. Then it should not verify cells[1] = all_cells[1][3] # do the check assert not spec.verify_cell_kzg_proof_batch( - row_commitments_bytes=all_commitments, - row_indices=row_indices, + commitments_bytes=commitments, column_indices=column_indices, cells=cells, proofs_bytes=proofs, diff --git a/tests/formats/kzg_7594/verify_cell_kzg_proof_batch.md b/tests/formats/kzg_7594/verify_cell_kzg_proof_batch.md index 439d1e25ae..270ad3e893 100644 --- a/tests/formats/kzg_7594/verify_cell_kzg_proof_batch.md +++ b/tests/formats/kzg_7594/verify_cell_kzg_proof_batch.md @@ -1,6 +1,6 @@ # Test format: Verify cell KZG proof batch -Use the cell KZG `proofs` to verify that the KZG `row_commitments` for the given `cells` are correct. +Use the cell KZG `proofs` to verify that the KZG `commitments` for the given `cells` are correct. ## Test case format @@ -8,8 +8,7 @@ The test data is declared in a `data.yaml` file: ```yaml input: - row_commitments: List[Bytes48] -- the KZG commitments - row_indices: List[RowIndex] -- the commitment index for each cell + commitments: List[Bytes48] -- the KZG commitments for each cell column_indices: List[ColumnIndex] -- the column index for each cell cells: List[Cell] -- the cells proofs: List[Bytes48] -- the KZG proof for each cell @@ -17,7 +16,6 @@ output: bool -- true (all proofs are correct) or false (some proofs incorrect) ``` - `Bytes48` is a 48-byte hexadecimal string, prefixed with `0x`. -- `RowIndex` is an unsigned 64-bit integer. - `ColumnIndex` is an unsigned 64-bit integer. - `Cell` is a 2048-byte hexadecimal string, prefixed with `0x`. @@ -25,4 +23,4 @@ All byte(s) fields are encoded as strings, hexadecimal encoding, prefixed with ` ## Condition -The `verify_cell_kzg_proof_batch` handler should verify that `row_commitments` are correct KZG commitments to `cells` by using the cell KZG proofs `proofs`, and the result should match the expected `output`. If any of the commitments or proofs are invalid (e.g. not on the curve or not in the G1 subgroup of the BLS curve), any cell is invalid (e.g. incorrect length or one of the 32-byte blocks does not represent a BLS field element), or any `cell_index` is invalid (e.g. greater than the number of cells for an extended blob), it should error, i.e. the output should be `null`. +The `verify_cell_kzg_proof_batch` handler should verify that `commitments` are correct KZG commitments to `cells` by using the cell KZG proofs `proofs`, and the result should match the expected `output`. If any of the commitments or proofs are invalid (e.g. not on the curve or not in the G1 subgroup of the BLS curve), any cell is invalid (e.g. incorrect length or one of the 32-byte blocks does not represent a BLS field element), or any `cell_index` is invalid (e.g. greater than the number of cells for an extended blob), it should error, i.e. the output should be `null`. diff --git a/tests/generators/kzg_7594/main.py b/tests/generators/kzg_7594/main.py index 428baa566c..4d537c2941 100644 --- a/tests/generators/kzg_7594/main.py +++ b/tests/generators/kzg_7594/main.py @@ -220,15 +220,13 @@ def case_verify_cell_kzg_proof_batch(): # Valid cases for i in range(len(VALID_BLOBS)): cells, proofs = VALID_CELLS_AND_PROOFS[i] - row_commitments = [VALID_COMMITMENTS[i]] - row_indices = [0] * spec.CELLS_PER_EXT_BLOB + commitments = [VALID_COMMITMENTS[i] for _ in cells] column_indices = list(range(spec.CELLS_PER_EXT_BLOB)) - assert spec.verify_cell_kzg_proof_batch(row_commitments, row_indices, column_indices, cells, proofs) - identifier = make_id(row_commitments, row_indices, column_indices, cells, proofs) + assert spec.verify_cell_kzg_proof_batch(commitments, column_indices, cells, proofs) + identifier = make_id(commitments, column_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_valid_{identifier}', { 'input': { - 'row_commitments': encode_hex_list(row_commitments), - 'row_indices': row_indices, + 'commitments': encode_hex_list(commitments), 'column_indices': column_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), @@ -237,13 +235,12 @@ def case_verify_cell_kzg_proof_batch(): } # Valid: zero cells - cells, row_commitments, row_indices, column_indices, proofs = [], [], [], [], [] - assert spec.verify_cell_kzg_proof_batch(row_commitments, row_indices, column_indices, cells, proofs) - identifier = make_id(row_commitments, row_indices, column_indices, cells, proofs) + cells, commitments, column_indices, proofs = [], [], [], [] + assert spec.verify_cell_kzg_proof_batch(commitments, column_indices, cells, proofs) + identifier = make_id(commitments, column_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_valid_zero_cells_{identifier}', { 'input': { - 'row_commitments': encode_hex_list(row_commitments), - 'row_indices': row_indices, + 'commitments': encode_hex_list(commitments), 'column_indices': column_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), @@ -254,37 +251,15 @@ def case_verify_cell_kzg_proof_batch(): # Valid: Verify cells from multiple blobs cells0, proofs0 = VALID_CELLS_AND_PROOFS[0] cells1, proofs1 = VALID_CELLS_AND_PROOFS[1] - row_commitments = VALID_COMMITMENTS[:2] - row_indices = [0, 1] + commitments = [VALID_COMMITMENTS[0], VALID_COMMITMENTS[1]] column_indices = [0, 0] cells = [cells0[0], cells1[0]] proofs = [proofs0[0], proofs1[0]] - assert spec.verify_cell_kzg_proof_batch(row_commitments, row_indices, column_indices, cells, proofs) - identifier = make_id(row_commitments, row_indices, column_indices, cells, proofs) + assert spec.verify_cell_kzg_proof_batch(commitments, column_indices, cells, proofs) + identifier = make_id(commitments, column_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_valid_multiple_blobs_{identifier}', { 'input': { - 'row_commitments': encode_hex_list(row_commitments), - 'row_indices': row_indices, - 'column_indices': column_indices, - 'cells': encode_hex_list(cells), - 'proofs': encode_hex_list(proofs), - }, - 'output': True - } - - # Valid: Unused row commitments - cells, proofs = VALID_CELLS_AND_PROOFS[2] - cells, proofs = cells[:3], proofs[:3] - # Provide list of all commitments - row_commitments = VALID_COMMITMENTS - row_indices = [2] * len(cells) - column_indices = list(range(len(cells))) - assert spec.verify_cell_kzg_proof_batch(row_commitments, row_indices, column_indices, cells, proofs) - identifier = make_id(row_commitments, row_indices, column_indices, cells, proofs) - yield f'verify_cell_kzg_proof_batch_case_valid_unused_row_commitments_{identifier}', { - 'input': { - 'row_commitments': encode_hex_list(row_commitments), - 'row_indices': row_indices, + 'commitments': encode_hex_list(commitments), 'column_indices': column_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), @@ -293,18 +268,16 @@ def case_verify_cell_kzg_proof_batch(): } # Valid: Same cell multiple times - row_commitments = [VALID_COMMITMENTS[3]] num_duplicates = 3 - row_indices = [0] * num_duplicates + commitments = [VALID_COMMITMENTS[3]] * num_duplicates column_indices = [0] * num_duplicates cells = [VALID_CELLS_AND_PROOFS[3][0][0]] * num_duplicates proofs = [VALID_CELLS_AND_PROOFS[3][1][0]] * num_duplicates - assert spec.verify_cell_kzg_proof_batch(row_commitments, row_indices, column_indices, cells, proofs) - identifier = make_id(row_commitments, row_indices, column_indices, cells, proofs) + assert spec.verify_cell_kzg_proof_batch(commitments, column_indices, cells, proofs) + identifier = make_id(commitments, column_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_valid_same_cell_multiple_times_{identifier}', { 'input': { - 'row_commitments': encode_hex_list(row_commitments), - 'row_indices': row_indices, + 'commitments': encode_hex_list(commitments), 'column_indices': column_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), @@ -312,19 +285,17 @@ def case_verify_cell_kzg_proof_batch(): 'output': True } - # Incorrect row commitment + # Incorrect commitment cells, proofs = VALID_CELLS_AND_PROOFS[5] cells, proofs = cells[:1], proofs[:1] - # Change commitment so it's wrong - row_commitments = [bls_add_one(VALID_COMMITMENTS[5])] - row_indices = [0] * len(cells) + # Use the wrong commitment + commitments = [bls_add_one(VALID_COMMITMENTS[5])] column_indices = list(range(len(cells))) - assert not spec.verify_cell_kzg_proof_batch(row_commitments, row_indices, column_indices, cells, proofs) - identifier = make_id(row_commitments, row_indices, column_indices, cells, proofs) - yield f'verify_cell_kzg_proof_batch_case_incorrect_row_commitment_{identifier}', { + assert not spec.verify_cell_kzg_proof_batch(commitments, column_indices, cells, proofs) + identifier = make_id(commitments, column_indices, cells, proofs) + yield f'verify_cell_kzg_proof_batch_case_incorrect_commitment_{identifier}', { 'input': { - 'row_commitments': encode_hex_list(row_commitments), - 'row_indices': row_indices, + 'commitments': encode_hex_list(commitments), 'column_indices': column_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), @@ -335,17 +306,15 @@ def case_verify_cell_kzg_proof_batch(): # Incorrect cell cells, proofs = VALID_CELLS_AND_PROOFS[6] cells, proofs = cells[:1], proofs[:1] - row_commitments = [VALID_COMMITMENTS[6]] - row_indices = [0] * len(cells) + commitments = [VALID_COMMITMENTS[6]] column_indices = list(range(len(cells))) # Change last cell so it's wrong cells[-1] = CELL_RANDOM_VALID2 - assert not spec.verify_cell_kzg_proof_batch(row_commitments, row_indices, column_indices, cells, proofs) - identifier = make_id(row_commitments, row_indices, column_indices, cells, proofs) + assert not spec.verify_cell_kzg_proof_batch(commitments, column_indices, cells, proofs) + identifier = make_id(commitments, column_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_incorrect_cell_{identifier}', { 'input': { - 'row_commitments': encode_hex_list(row_commitments), - 'row_indices': row_indices, + 'commitments': encode_hex_list(commitments), 'column_indices': column_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), @@ -356,17 +325,15 @@ def case_verify_cell_kzg_proof_batch(): # Incorrect proof cells, proofs = VALID_CELLS_AND_PROOFS[0] cells, proofs = cells[:1], proofs[:1] - row_commitments = [VALID_COMMITMENTS[0]] - row_indices = [0] * len(cells) + commitments = [VALID_COMMITMENTS[0]] column_indices = list(range(len(cells))) # Change last proof so it's wrong proofs[-1] = bls_add_one(proofs[-1]) - assert not spec.verify_cell_kzg_proof_batch(row_commitments, row_indices, column_indices, cells, proofs) - identifier = make_id(row_commitments, row_indices, column_indices, cells, proofs) + assert not spec.verify_cell_kzg_proof_batch(commitments, column_indices, cells, proofs) + identifier = make_id(commitments, column_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_incorrect_proof_{identifier}', { 'input': { - 'row_commitments': encode_hex_list(row_commitments), - 'row_indices': row_indices, + 'commitments': encode_hex_list(commitments), 'column_indices': column_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), @@ -374,20 +341,18 @@ def case_verify_cell_kzg_proof_batch(): 'output': False } - # Edge case: Invalid row commitment + # Edge case: Invalid commitment for i, commitment in enumerate(INVALID_G1_POINTS): cells, proofs = VALID_CELLS_AND_PROOFS[i % len(INVALID_G1_POINTS)] cells, proofs = cells[:1], proofs[:1] - # Set row_commitments to the invalid commitment - row_commitments = [commitment] - row_indices = [0] * len(cells) + # Set commitments to the invalid commitment + commitments = [commitment] column_indices = list(range(len(cells))) - expect_exception(spec.verify_cell_kzg_proof_batch, row_commitments, row_indices, column_indices, cells, proofs) - identifier = make_id(row_commitments, row_indices, column_indices, cells, proofs) - yield f'verify_cell_kzg_proof_batch_case_invalid_row_commitment_{identifier}', { + expect_exception(spec.verify_cell_kzg_proof_batch, commitments, column_indices, cells, proofs) + identifier = make_id(commitments, column_indices, cells, proofs) + yield f'verify_cell_kzg_proof_batch_case_invalid_commitment_{identifier}', { 'input': { - 'row_commitments': encode_hex_list(row_commitments), - 'row_indices': row_indices, + 'commitments': encode_hex_list(commitments), 'column_indices': column_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), @@ -395,41 +360,18 @@ def case_verify_cell_kzg_proof_batch(): 'output': None } - # Edge case: Invalid row_index - cells, proofs = VALID_CELLS_AND_PROOFS[0] - cells, proofs = cells[:1], proofs[:1] - row_commitments = [VALID_COMMITMENTS[0]] - row_indices = [0] * len(cells) - # Set first row index to an invalid value - row_indices[0] = 1 - column_indices = list(range(len(cells))) - expect_exception(spec.verify_cell_kzg_proof_batch, row_commitments, row_indices, column_indices, cells, proofs) - identifier = make_id(row_commitments, row_indices, column_indices, cells, proofs) - yield f'verify_cell_kzg_proof_batch_case_invalid_row_index_{identifier}', { - 'input': { - 'row_commitments': encode_hex_list(row_commitments), - 'row_indices': row_indices, - 'column_indices': column_indices, - 'cells': encode_hex_list(cells), - 'proofs': encode_hex_list(proofs), - }, - 'output': None - } - # Edge case: Invalid column_index cells, proofs = VALID_CELLS_AND_PROOFS[1] cells, proofs = cells[:1], proofs[:1] - row_commitments = [VALID_COMMITMENTS[1]] - row_indices = [0] * len(cells) + commitments = [VALID_COMMITMENTS[1]] column_indices = list(range(len(cells))) # Set first column index to an invalid value column_indices[0] = spec.CELLS_PER_EXT_BLOB - expect_exception(spec.verify_cell_kzg_proof_batch, row_commitments, row_indices, column_indices, cells, proofs) - identifier = make_id(row_commitments, row_indices, column_indices, cells, proofs) + expect_exception(spec.verify_cell_kzg_proof_batch, commitments, column_indices, cells, proofs) + identifier = make_id(commitments, column_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_invalid_column_index_{identifier}', { 'input': { - 'row_commitments': encode_hex_list(row_commitments), - 'row_indices': row_indices, + 'commitments': encode_hex_list(commitments), 'column_indices': column_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), @@ -441,17 +383,15 @@ def case_verify_cell_kzg_proof_batch(): for i, cell in enumerate(INVALID_INDIVIDUAL_CELL_BYTES): cells, proofs = VALID_CELLS_AND_PROOFS[i % len(INVALID_INDIVIDUAL_CELL_BYTES)] cells, proofs = cells[:1], proofs[:1] - row_commitments = [VALID_COMMITMENTS[i % len(INVALID_INDIVIDUAL_CELL_BYTES)]] - row_indices = [0] * len(cells) + commitments = [VALID_COMMITMENTS[i % len(INVALID_INDIVIDUAL_CELL_BYTES)]] column_indices = list(range(len(cells))) # Set first cell to the invalid cell cells[0] = cell - expect_exception(spec.verify_cell_kzg_proof_batch, row_commitments, row_indices, column_indices, cells, proofs) - identifier = make_id(row_commitments, row_indices, column_indices, cells, proofs) + expect_exception(spec.verify_cell_kzg_proof_batch, commitments, column_indices, cells, proofs) + identifier = make_id(commitments, column_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_invalid_cell_{identifier}', { 'input': { - 'row_commitments': encode_hex_list(row_commitments), - 'row_indices': row_indices, + 'commitments': encode_hex_list(commitments), 'column_indices': column_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), @@ -463,17 +403,15 @@ def case_verify_cell_kzg_proof_batch(): for i, proof in enumerate(INVALID_G1_POINTS): cells, proofs = VALID_CELLS_AND_PROOFS[i % len(INVALID_G1_POINTS)] cells, proofs = cells[:1], proofs[:1] - row_commitments = [VALID_COMMITMENTS[i % len(INVALID_G1_POINTS)]] - row_indices = [0] * len(cells) + commitments = [VALID_COMMITMENTS[i % len(INVALID_G1_POINTS)]] column_indices = list(range(len(cells))) # Set first proof to the invalid proof proofs[0] = proof - expect_exception(spec.verify_cell_kzg_proof_batch, row_commitments, row_indices, column_indices, cells, proofs) - identifier = make_id(row_commitments, row_indices, column_indices, cells, proofs) + expect_exception(spec.verify_cell_kzg_proof_batch, commitments, column_indices, cells, proofs) + identifier = make_id(commitments, column_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_invalid_proof_{identifier}', { 'input': { - 'row_commitments': encode_hex_list(row_commitments), - 'row_indices': row_indices, + 'commitments': encode_hex_list(commitments), 'column_indices': column_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), @@ -481,39 +419,17 @@ def case_verify_cell_kzg_proof_batch(): 'output': None } - # Edge case: Missing a row commitment + # Edge case: Missing a commitment cells, proofs = VALID_CELLS_AND_PROOFS[0] - cells, proofs = cells[:1], proofs[:1] - # Do not include the row commitment - row_commitments = [] - row_indices = [0] * len(cells) - column_indices = list(range(len(cells))) - expect_exception(spec.verify_cell_kzg_proof_batch, row_commitments, row_indices, column_indices, cells, proofs) - identifier = make_id(row_commitments, row_indices, column_indices, cells, proofs) - yield f'verify_cell_kzg_proof_batch_case_invalid_missing_row_commitment_{identifier}', { - 'input': { - 'row_commitments': encode_hex_list(row_commitments), - 'row_indices': row_indices, - 'column_indices': column_indices, - 'cells': encode_hex_list(cells), - 'proofs': encode_hex_list(proofs), - }, - 'output': None - } - - # Edge case: Missing a row index - cells, proofs = VALID_CELLS_AND_PROOFS[1] cells, proofs = cells[:2], proofs[:2] - row_commitments = [VALID_COMMITMENTS[1]] - # Leave off one of the row indices - row_indices = [0] * (len(cells) - 1) + # Do not include the second commitment + commitments = [VALID_COMMITMENTS[0]] column_indices = list(range(len(cells))) - expect_exception(spec.verify_cell_kzg_proof_batch, row_commitments, row_indices, column_indices, cells, proofs) - identifier = make_id(row_commitments, row_indices, column_indices, cells, proofs) - yield f'verify_cell_kzg_proof_batch_case_invalid_missing_row_index_{identifier}', { + expect_exception(spec.verify_cell_kzg_proof_batch, commitments, column_indices, cells, proofs) + identifier = make_id(commitments, column_indices, cells, proofs) + yield f'verify_cell_kzg_proof_batch_case_invalid_missing_commitment_{identifier}', { 'input': { - 'row_commitments': encode_hex_list(row_commitments), - 'row_indices': row_indices, + 'commitments': encode_hex_list(commitments), 'column_indices': column_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), @@ -524,16 +440,14 @@ def case_verify_cell_kzg_proof_batch(): # Edge case: Missing a column index cells, proofs = VALID_CELLS_AND_PROOFS[2] cells, proofs = cells[:2], proofs[:2] - row_commitments = [VALID_COMMITMENTS[2]] - row_indices = [0] * len(cells) + commitments = [VALID_COMMITMENTS[2], VALID_COMMITMENTS[2]] # Leave off one of the column indices column_indices = list(range(len(cells) - 1)) - expect_exception(spec.verify_cell_kzg_proof_batch, row_commitments, row_indices, column_indices, cells, proofs) - identifier = make_id(row_commitments, row_indices, column_indices, cells, proofs) + expect_exception(spec.verify_cell_kzg_proof_batch, commitments, column_indices, cells, proofs) + identifier = make_id(commitments, column_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_invalid_missing_column_index_{identifier}', { 'input': { - 'row_commitments': encode_hex_list(row_commitments), - 'row_indices': row_indices, + 'commitments': encode_hex_list(commitments), 'column_indices': column_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), @@ -544,17 +458,15 @@ def case_verify_cell_kzg_proof_batch(): # Edge case: Missing a cell cells, proofs = VALID_CELLS_AND_PROOFS[3] cells, proofs = cells[:2], proofs[:2] - row_commitments = [VALID_COMMITMENTS[3]] - row_indices = [0] * len(cells) + commitments = [VALID_COMMITMENTS[3], VALID_COMMITMENTS[3]] column_indices = list(range(len(cells))) # Remove the last proof cells = cells[:-1] - expect_exception(spec.verify_cell_kzg_proof_batch, row_commitments, row_indices, column_indices, cells, proofs) - identifier = make_id(row_commitments, row_indices, column_indices, cells, proofs) + expect_exception(spec.verify_cell_kzg_proof_batch, commitments, column_indices, cells, proofs) + identifier = make_id(commitments, column_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_invalid_missing_cell_{identifier}', { 'input': { - 'row_commitments': encode_hex_list(row_commitments), - 'row_indices': row_indices, + 'commitments': encode_hex_list(commitments), 'column_indices': column_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), @@ -565,17 +477,15 @@ def case_verify_cell_kzg_proof_batch(): # Edge case: Missing a proof cells, proofs = VALID_CELLS_AND_PROOFS[4] cells, proofs = cells[:2], proofs[:2] - row_commitments = [VALID_COMMITMENTS[4]] - row_indices = [0] * len(cells) + commitments = [VALID_COMMITMENTS[4], VALID_COMMITMENTS[4]] column_indices = list(range(len(cells))) # Remove the last proof proofs = proofs[:-1] - expect_exception(spec.verify_cell_kzg_proof_batch, row_commitments, row_indices, column_indices, cells, proofs) - identifier = make_id(row_commitments, row_indices, column_indices, cells, proofs) + expect_exception(spec.verify_cell_kzg_proof_batch, commitments, column_indices, cells, proofs) + identifier = make_id(commitments, column_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_invalid_missing_proof_{identifier}', { 'input': { - 'row_commitments': encode_hex_list(row_commitments), - 'row_indices': row_indices, + 'commitments': encode_hex_list(commitments), 'column_indices': column_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), From db717702a82f552611db4be34eff58dec61b943f Mon Sep 17 00:00:00 2001 From: dapplion <35266934+dapplion@users.noreply.github.com> Date: Thu, 11 Jul 2024 14:07:16 +0200 Subject: [PATCH 100/130] Move to peerdas eip --- specs/_features/eip7594/p2p-interface.md | 36 ++++++++++++++++++++ specs/electra/p2p-interface.md | 42 ------------------------ 2 files changed, 36 insertions(+), 42 deletions(-) diff --git a/specs/_features/eip7594/p2p-interface.md b/specs/_features/eip7594/p2p-interface.md index 582ee65140..4dbd3238b1 100644 --- a/specs/_features/eip7594/p2p-interface.md +++ b/specs/_features/eip7594/p2p-interface.md @@ -17,6 +17,7 @@ - [`verify_data_column_sidecar_kzg_proofs`](#verify_data_column_sidecar_kzg_proofs) - [`verify_data_column_sidecar_inclusion_proof`](#verify_data_column_sidecar_inclusion_proof) - [`compute_subnet_for_data_column_sidecar`](#compute_subnet_for_data_column_sidecar) + - [MetaData](#metadata) - [The gossip domain: gossipsub](#the-gossip-domain-gossipsub) - [Topics and messages](#topics-and-messages) - [Blob subnets](#blob-subnets) @@ -26,6 +27,7 @@ - [Messages](#messages) - [DataColumnSidecarsByRoot v1](#datacolumnsidecarsbyroot-v1) - [DataColumnSidecarsByRange v1](#datacolumnsidecarsbyrange-v1) + - [GetMetaData v3](#getmetadata-v3) - [The discovery domain: discv5](#the-discovery-domain-discv5) - [ENR structure](#enr-structure) - [Custody subnet count](#custody-subnet-count) @@ -110,6 +112,24 @@ def compute_subnet_for_data_column_sidecar(column_index: ColumnIndex) -> SubnetI return SubnetID(column_index % DATA_COLUMN_SIDECAR_SUBNET_COUNT) ``` +### MetaData + +The `MetaData` stored locally by clients is updated with an additional field to communicate the custody subnet count. + +``` +( + seq_number: uint64 + attnets: Bitvector[ATTESTATION_SUBNET_COUNT] + syncnets: Bitvector[SYNC_COMMITTEE_SUBNET_COUNT] + custody_subnet_count: uint64 +) +``` + +Where + +- `seq_number`, `attnets`, and `syncnets` have the same meaning defined in the Altair document. +- `custody_subnet_count` represents the node's custody subnet count. Clients MAY reject ENRs with a value less than `CUSTODY_REQUIREMENT`. + ### The gossip domain: gossipsub Some gossip meshes are upgraded in the EIP-7594 fork to support upgraded types. @@ -280,6 +300,22 @@ Clients MUST respond with data column sidecars that are consistent from a single After the initial data column sidecar, clients MAY stop in the process of responding if their fork choice changes the view of the chain in the context of the request. +##### GetMetaData v3 + +**Protocol ID:** `/eth2/beacon_chain/req/metadata/3/` + +No Request Content. + +Response Content: + +``` +( + MetaData +) +``` + +Requests the MetaData of a peer, using the new `MetaData` definition given above that is extended from Altair. Other conditions for the `GetMetaData` protocol are unchanged from the Altair p2p networking document. + ### The discovery domain: discv5 #### ENR structure diff --git a/specs/electra/p2p-interface.md b/specs/electra/p2p-interface.md index 33d3b6592c..ebdcaaa831 100644 --- a/specs/electra/p2p-interface.md +++ b/specs/electra/p2p-interface.md @@ -11,39 +11,17 @@ The specification of these changes continues in the same format as the network s - [Modifications in Electra](#modifications-in-electra) - - [MetaData](#metadata) - [The gossip domain: gossipsub](#the-gossip-domain-gossipsub) - [Topics and messages](#topics-and-messages) - [Global topics](#global-topics) - [`beacon_aggregate_and_proof`](#beacon_aggregate_and_proof) - [`beacon_attestation_{subnet_id}`](#beacon_attestation_subnet_id) - - [The Req/Resp domain](#the-reqresp-domain) - - [Messages](#messages) - - [GetMetaData v3](#getmetadata-v3) ## Modifications in Electra -### MetaData - -The `MetaData` stored locally by clients is updated with an additional field to communicate the custody subnet count. - -``` -( - seq_number: uint64 - attnets: Bitvector[ATTESTATION_SUBNET_COUNT] - syncnets: Bitvector[SYNC_COMMITTEE_SUBNET_COUNT] - custody_subnet_count: uint64 -) -``` - -Where - -- `seq_number`, `attnets`, and `syncnets` have the same meaning defined in the Altair document. -- `custody_subnet_count` represents the node's custody subnet count. Clients MAY reject ENRs with a value less than `CUSTODY_REQUIREMENT`. - ### The gossip domain: gossipsub Some gossip meshes are upgraded in the fork of Electra to support upgraded types. @@ -79,23 +57,3 @@ The following convenience variables are re-defined The following validations are added: * [REJECT] `len(committee_indices) == 1`, where `committee_indices = get_committee_indices(attestation)`. * [REJECT] `attestation.data.index == 0` - -### The Req/Resp domain - -#### Messages - -##### GetMetaData v3 - -**Protocol ID:** `/eth2/beacon_chain/req/metadata/3/` - -No Request Content. - -Response Content: - -``` -( - MetaData -) -``` - -Requests the MetaData of a peer, using the new `MetaData` definition given above that is extended from Altair. Other conditions for the `GetMetaData` protocol are unchanged from the Altair p2p networking document. From e2c06c32b1e40fe40e908c229eef8d2c38da1ffc Mon Sep 17 00:00:00 2001 From: Etan Kissling Date: Mon, 15 Jul 2024 13:56:54 +0200 Subject: [PATCH 101/130] Fix link --- specs/electra/light-client/sync-protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/electra/light-client/sync-protocol.md b/specs/electra/light-client/sync-protocol.md index 724d3c8497..54dc076dbb 100644 --- a/specs/electra/light-client/sync-protocol.md +++ b/specs/electra/light-client/sync-protocol.md @@ -43,7 +43,7 @@ Additional documents describes the impact of the upgrade on certain roles: ### Frozen constants -Existing `GeneralizedIndex` constants are frozen at their [Altair](../../altair/light-client/sync-protocol#constants) values. +Existing `GeneralizedIndex` constants are frozen at their [Altair](../../altair/light-client/sync-protocol.md#constants) values. | Name | Value | | - | - | From 9db6d3f2ef7d3d6e64b9b033d6185da1034f10ed Mon Sep 17 00:00:00 2001 From: Etan Kissling Date: Mon, 15 Jul 2024 14:02:07 +0200 Subject: [PATCH 102/130] Remove unnecessary cast --- specs/electra/light-client/sync-protocol.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specs/electra/light-client/sync-protocol.md b/specs/electra/light-client/sync-protocol.md index 54dc076dbb..ef9dcd5987 100644 --- a/specs/electra/light-client/sync-protocol.md +++ b/specs/electra/light-client/sync-protocol.md @@ -70,7 +70,7 @@ def finalized_root_gindex_at_slot(slot: Slot) -> GeneralizedIndex: # [Modified in Electra] if epoch >= ELECTRA_FORK_EPOCH: return FINALIZED_ROOT_GINDEX_ELECTRA - return GeneralizedIndex(FINALIZED_ROOT_GINDEX) + return FINALIZED_ROOT_GINDEX ``` ### Modified `current_sync_committee_gindex_at_slot` @@ -82,7 +82,7 @@ def current_sync_committee_gindex_at_slot(slot: Slot) -> GeneralizedIndex: # [Modified in Electra] if epoch >= ELECTRA_FORK_EPOCH: return CURRENT_SYNC_COMMITTEE_GINDEX_ELECTRA - return GeneralizedIndex(CURRENT_SYNC_COMMITTEE_GINDEX) + return CURRENT_SYNC_COMMITTEE_GINDEX ``` ### Modified `next_sync_committee_gindex_at_slot` @@ -94,7 +94,7 @@ def next_sync_committee_gindex_at_slot(slot: Slot) -> GeneralizedIndex: # [Modified in Electra] if epoch >= ELECTRA_FORK_EPOCH: return NEXT_SYNC_COMMITTEE_GINDEX_ELECTRA - return GeneralizedIndex(NEXT_SYNC_COMMITTEE_GINDEX) + return NEXT_SYNC_COMMITTEE_GINDEX ``` ### Modified `get_lc_execution_root` From 3529b46149767e4e653699774b933642a0288788 Mon Sep 17 00:00:00 2001 From: Potuz Date: Mon, 15 Jul 2024 11:31:39 -0300 Subject: [PATCH 103/130] Fix block_header.state_root and forkchoice filtering --- specs/_features/eip7732/beacon-chain.md | 47 +++++++++++++++++++++++++ specs/_features/eip7732/fork-choice.md | 3 +- 2 files changed, 48 insertions(+), 2 deletions(-) diff --git a/specs/_features/eip7732/beacon-chain.md b/specs/_features/eip7732/beacon-chain.md index d9f67b4712..939adf08da 100644 --- a/specs/_features/eip7732/beacon-chain.md +++ b/specs/_features/eip7732/beacon-chain.md @@ -613,6 +613,11 @@ def process_execution_payload(state: BeaconState, assert verify_execution_payload_envelope_signature(state, signed_envelope) envelope = signed_envelope.message payload = envelope.payload + # Cache latest block header state root + previous_state_root = hash_tree_root(state) + if state.latest_block_header.state_root == Root(): + state.latest_block_header.state_root = previous_state_root + # Verify consistency with the beacon block assert envelope.beacon_block_root == hash_tree_root(state.latest_block_header) @@ -665,3 +670,45 @@ def process_execution_payload(state: BeaconState, if verify: assert envelope.state_root == hash_tree_root(state) ``` + +#### Modified `is_merge_transition_complete` + +`is_merge_transition_complete` is modified only for testing purposes to add the blob kzg commitments root for an empty list + +```python +def is_merge_transition_complete(state: BeaconState) -> bool: + header = ExecutionPayloadHeader() + kzgs = List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK]() + header.blob_kzg_commitments_root = kzgs.hash_tree_root() + + return state.latest_execution_payload_header != header +``` + +#### Modified `validate_merge_block` +`validate_merge_block` is modified to use the new `signed_execution_payload_header` message in the Beacon Block Body + +```python +def validate_merge_block(block: BeaconBlock) -> None: + """ + Check the parent PoW block of execution payload is a valid terminal PoW block. + + Note: Unavailable PoW block(s) may later become available, + and a client software MAY delay a call to ``validate_merge_block`` + until the PoW block(s) become available. + """ + if config.TERMINAL_BLOCK_HASH != Hash32(): + # If `config.TERMINAL_BLOCK_HASH` is used as an override, the activation epoch must be reached. + assert compute_epoch_at_slot(block.slot) >= config.TERMINAL_BLOCK_HASH_ACTIVATION_EPOCH + assert block.body.signed_execution_payload_header.message.parent_block_hash == config.TERMINAL_BLOCK_HASH + return + + # Modified in EIP-7732 + pow_block = get_pow_block(block.body.signed_execution_payload_header.message.parent_block_hash) + # Check if `pow_block` is available + assert pow_block is not None + pow_parent = get_pow_block(pow_block.parent_hash) + # Check if `pow_parent` is available + assert pow_parent is not None + # Check if `pow_block` is a valid terminal PoW block + assert is_valid_terminal_pow_block(pow_block, pow_parent) +``` diff --git a/specs/_features/eip7732/fork-choice.md b/specs/_features/eip7732/fork-choice.md index 32b67c8712..98ff0e233e 100644 --- a/specs/_features/eip7732/fork-choice.md +++ b/specs/_features/eip7732/fork-choice.md @@ -344,8 +344,7 @@ def get_head(store: Store) -> ChildNode: children = [ ChildNode(root=root, slot=block.slot, is_payload_present=present) for (root, block) in blocks.items() if block.parent_root == best_child.root and - is_parent_node_full(store, block) == best_child.is_payload_present if - root != store.justified_checkpoint.root + (best_child.root == justified_root or is_parent_node_full(store, block) == best_child.is_payload_present) for present in (True, False) if root in store.execution_payload_states or not present ] if len(children) == 0: From 252b852a61289efeb961a497f56302a70aeb7c77 Mon Sep 17 00:00:00 2001 From: Justin Traglia <95511699+jtraglia@users.noreply.github.com> Date: Mon, 15 Jul 2024 11:41:33 -0500 Subject: [PATCH 104/130] Rename column_index to cell_index in KZG spec (#3841) * Rename column_index to cell_index in KZG spec * Fix table of contents * Replace list(set()) with set() * Change coset_evals to cell_indices in some places * Remove punctuation in table description * Remove "row" from comments & improve docstring * Replace unique with deduplicated --- specs/_features/eip7594/das-core.md | 4 +- specs/_features/eip7594/p2p-interface.md | 7 +- .../polynomial-commitments-sampling.md | 132 ++++++++--------- .../test_polynomial_commitments.py | 30 ++-- .../kzg_7594/verify_cell_kzg_proof_batch.md | 4 +- tests/generators/kzg_7594/main.py | 134 +++++++++--------- 6 files changed, 149 insertions(+), 162 deletions(-) diff --git a/specs/_features/eip7594/das-core.md b/specs/_features/eip7594/das-core.md index 0d6530226d..ee5dc557f5 100644 --- a/specs/_features/eip7594/das-core.md +++ b/specs/_features/eip7594/das-core.md @@ -68,8 +68,8 @@ The following values are (non-configurable) constants used throughout the specif | Name | Value | Description | | - | - | - | -| `NUMBER_OF_COLUMNS` | `uint64(CELLS_PER_EXT_BLOB)` (= 128) | Number of columns in the extended data matrix. | -| `MAX_CELLS_IN_EXTENDED_MATRIX` | `uint64(MAX_BLOBS_PER_BLOCK * NUMBER_OF_COLUMNS)` (= 768) | The data size of `ExtendedMatrix`. | +| `NUMBER_OF_COLUMNS` | `uint64(CELLS_PER_EXT_BLOB)` (= 128) | Number of columns in the extended data matrix | +| `MAX_CELLS_IN_EXTENDED_MATRIX` | `uint64(MAX_BLOBS_PER_BLOCK * NUMBER_OF_COLUMNS)` (= 768) | The data size of `ExtendedMatrix` | ### Networking diff --git a/specs/_features/eip7594/p2p-interface.md b/specs/_features/eip7594/p2p-interface.md index 0c9a7cd178..4584a6c8e7 100644 --- a/specs/_features/eip7594/p2p-interface.md +++ b/specs/_features/eip7594/p2p-interface.md @@ -73,12 +73,13 @@ def verify_data_column_sidecar_kzg_proofs(sidecar: DataColumnSidecar) -> bool: assert sidecar.index < NUMBER_OF_COLUMNS assert len(sidecar.column) == len(sidecar.kzg_commitments) == len(sidecar.kzg_proofs) - column_indices = [sidecar.index] * len(sidecar.column) + # The column index also represents the cell index + cell_indices = [CellIndex(sidecar.index)] * len(sidecar.column) - # KZG batch verifies that the cells match the corresponding commitments and proofs + # Batch verify that the cells match the corresponding commitments and proofs return verify_cell_kzg_proof_batch( commitments_bytes=sidecar.kzg_commitments, - column_indices=column_indices, + cell_indices=cell_indices, cells=sidecar.column, proofs_bytes=sidecar.kzg_proofs, ) diff --git a/specs/_features/eip7594/polynomial-commitments-sampling.md b/specs/_features/eip7594/polynomial-commitments-sampling.md index 253644fe7a..2993d50bf9 100644 --- a/specs/_features/eip7594/polynomial-commitments-sampling.md +++ b/specs/_features/eip7594/polynomial-commitments-sampling.md @@ -9,7 +9,6 @@ - [Introduction](#introduction) - [Public Methods](#public-methods) - [Custom types](#custom-types) -- [Constants](#constants) - [Preset](#preset) - [Cells](#cells) - [Helper functions](#helper-functions) @@ -77,11 +76,7 @@ The following is a list of the public methods: | `CosetEvals` | `Vector[BLSFieldElement, FIELD_ELEMENTS_PER_CELL]` | The internal representation of a cell (the evaluations over its Coset) | | `Cell` | `ByteVector[BYTES_PER_FIELD_ELEMENT * FIELD_ELEMENTS_PER_CELL]` | The unit of blob data that can come with its own KZG proof | | `CellIndex` | `uint64` | Validation: `x < CELLS_PER_EXT_BLOB` | - -## Constants - -| Name | Value | Notes | -| - | - | - | +| `CommitmentIndex` | `uint64` | The type which represents the index of an element in the list of commitments | ## Preset @@ -226,46 +221,30 @@ def coset_fft_field(vals: Sequence[BLSFieldElement], #### `compute_verify_cell_kzg_proof_batch_challenge` ```python -def compute_verify_cell_kzg_proof_batch_challenge(row_commitments: Sequence[KZGCommitment], - row_indices: Sequence[RowIndex], - column_indices: Sequence[ColumnIndex], +def compute_verify_cell_kzg_proof_batch_challenge(commitments: Sequence[KZGCommitment], + commitment_indices: Sequence[CommitmentIndex], + cell_indices: Sequence[CellIndex], cosets_evals: Sequence[CosetEvals], proofs: Sequence[KZGProof]) -> BLSFieldElement: """ - Compute a random challenge r used in the universal verification equation. - This is used in verify_cell_kzg_proof_batch_impl. - - To compute the challenge, `RANDOM_CHALLENGE_KZG_CELL_BATCH_DOMAIN` is used as a hash prefix. + Compute a random challenge ``r`` used in the universal verification equation. To compute the + challenge, ``RANDOM_CHALLENGE_KZG_CELL_BATCH_DOMAIN`` and all data that can influence the + verification is hashed together to deterministically generate a "random" field element via + the Fiat-Shamir heuristic. """ - # input the domain separator hashinput = RANDOM_CHALLENGE_KZG_CELL_BATCH_DOMAIN - - # input the degree bound of the polynomial hashinput += int.to_bytes(FIELD_ELEMENTS_PER_BLOB, 8, KZG_ENDIANNESS) - - # input the field elements per cell hashinput += int.to_bytes(FIELD_ELEMENTS_PER_CELL, 8, KZG_ENDIANNESS) - - # input the number of commitments - num_commitments = len(row_commitments) - hashinput += int.to_bytes(num_commitments, 8, KZG_ENDIANNESS) - - # input the number of cells - num_cells = len(row_indices) - hashinput += int.to_bytes(num_cells, 8, KZG_ENDIANNESS) - - # input all commitments - for commitment in row_commitments: + hashinput += int.to_bytes(len(commitments), 8, KZG_ENDIANNESS) + hashinput += int.to_bytes(len(cell_indices), 8, KZG_ENDIANNESS) + for commitment in commitments: hashinput += commitment - - # input each cell with its indices and proof - for k in range(num_cells): - hashinput += int.to_bytes(row_indices[k], 8, KZG_ENDIANNESS) - hashinput += int.to_bytes(column_indices[k], 8, KZG_ENDIANNESS) - for eval in cosets_evals[k]: - hashinput += bls_field_to_bytes(eval) + for k, coset_evals in enumerate(cosets_evals): + hashinput += int.to_bytes(commitment_indices[k], 8, KZG_ENDIANNESS) + hashinput += int.to_bytes(cell_indices[k], 8, KZG_ENDIANNESS) + for coset_eval in coset_evals: + hashinput += bls_field_to_bytes(coset_eval) hashinput += proofs[k] - return hash_to_bls_field(hashinput) ``` @@ -434,24 +413,25 @@ def compute_kzg_proof_multi_impl( #### `verify_cell_kzg_proof_batch_impl` ```python -def verify_cell_kzg_proof_batch_impl(row_commitments: Sequence[KZGCommitment], - row_indices: Sequence[RowIndex], - column_indices: Sequence[ColumnIndex], +def verify_cell_kzg_proof_batch_impl(commitments: Sequence[KZGCommitment], + commitment_indices: Sequence[CommitmentIndex], + cell_indices: Sequence[CellIndex], cosets_evals: Sequence[CosetEvals], proofs: Sequence[KZGProof]) -> bool: """ Helper: Verify that a set of cells belong to their corresponding commitment. - Given a list of ``row_commitments`` and four lists representing tuples of (``row_index``, - ``column_index``, ``evals``, ``proof``), the function verifies ``proof`` which shows that - ``evals`` are the evaluations of the polynomial associated with ``row_commitments[row_index]``, - evaluated over the domain specified by ``column_index``. + Given a list of ``commitments`` (which contains no duplicates) and four lists representing + tuples of (``commitment_index``, ``cell_index``, ``evals``, ``proof``), the function + verifies ``proof`` which shows that ``evals`` are the evaluations of the polynomial associated + with ``commitments[commitment_index]``, evaluated over the domain specified by ``cell_index``. This function is the internal implementation of ``verify_cell_kzg_proof_batch``. """ - assert len(row_indices) == len(column_indices) == len(cosets_evals) == len(proofs) - for row_index in row_indices: - assert row_index < len(row_commitments) + assert len(commitment_indices) == len(cell_indices) == len(cosets_evals) == len(proofs) + assert len(commitments) == len(set(commitments)) + for commitment_index in commitment_indices: + assert commitment_index < len(commitments) # The verification equation that we will check is pairing (LL, LR) = pairing (RL, [1]), where # LL = sum_k r^k proofs[k], @@ -462,25 +442,26 @@ def verify_cell_kzg_proof_batch_impl(row_commitments: Sequence[KZGCommitment], # RLP = sum_k (r^k * h_k^n) proofs[k] # # Here, the variables have the following meaning: - # - k < len(row_indices) is an index iterating over all cells in the input + # - k < len(cell_indices) is an index iterating over all cells in the input # - r is a random coefficient, derived from hashing all data provided by the prover # - s is the secret embedded in the KZG setup # - n = FIELD_ELEMENTS_PER_CELL is the size of the evaluation domain - # - i ranges over all rows that are touched - # - weights[i] is a weight computed for row i. It depends on r and on which cells are in row i + # - i ranges over all provided commitments + # - weights[i] is a weight computed for commitment i + # - It depends on r and on which cells are associated with commitment i # - interpolation_poly_k is the interpolation polynomial for the kth cell # - h_k is the coset shift specifying the evaluation domain of the kth cell # Preparation - num_cells = len(row_indices) + num_cells = len(cell_indices) n = FIELD_ELEMENTS_PER_CELL - num_rows = len(row_commitments) + num_commitments = len(commitments) # Step 1: Compute a challenge r and its powers r^0, ..., r^{num_cells-1} r = compute_verify_cell_kzg_proof_batch_challenge( - row_commitments, - row_indices, - column_indices, + commitments, + commitment_indices, + cell_indices, cosets_evals, proofs ) @@ -494,20 +475,20 @@ def verify_cell_kzg_proof_batch_impl(row_commitments: Sequence[KZGCommitment], # Step 4: Compute RL = RLC - RLI + RLP # Step 4.1: Compute RLC = sum_i weights[i] commitments[i] - # Step 4.1a: Compute weights[i]: the sum of all r^k for which cell k is in row i. + # Step 4.1a: Compute weights[i]: the sum of all r^k for which cell k is associated with commitment i. # Note: we do that by iterating over all k and updating the correct weights[i] accordingly - weights = [0] * num_rows + weights = [0] * num_commitments for k in range(num_cells): - i = row_indices[k] + i = commitment_indices[k] weights[i] = (weights[i] + int(r_powers[k])) % BLS_MODULUS # Step 4.1b: Linearly combine the weights with the commitments to get RLC - rlc = bls.bytes48_to_G1(g1_lincomb(row_commitments, weights)) + rlc = bls.bytes48_to_G1(g1_lincomb(commitments, weights)) # Step 4.2: Compute RLI = [sum_k r^k interpolation_poly_k(s)] # Note: an efficient implementation would use the IDFT based method explained in the blog post sum_interp_polys_coeff = [0] * n for k in range(num_cells): - interp_poly_coeff = interpolate_polynomialcoeff(coset_for_cell(column_indices[k]), cosets_evals[k]) + interp_poly_coeff = interpolate_polynomialcoeff(coset_for_cell(cell_indices[k]), cosets_evals[k]) interp_poly_scaled_coeff = multiply_polynomialcoeff([r_powers[k]], interp_poly_coeff) sum_interp_polys_coeff = add_polynomialcoeff(sum_interp_polys_coeff, interp_poly_scaled_coeff) rli = bls.bytes48_to_G1(g1_lincomb(KZG_SETUP_G1_MONOMIAL[:n], sum_interp_polys_coeff)) @@ -515,7 +496,7 @@ def verify_cell_kzg_proof_batch_impl(row_commitments: Sequence[KZGCommitment], # Step 4.3: Compute RLP = sum_k (r^k * h_k^n) proofs[k] weighted_r_powers = [] for k in range(num_cells): - h_k = int(coset_shift_for_cell(column_indices[k])) + h_k = int(coset_shift_for_cell(cell_indices[k])) h_k_pow = pow(h_k, n, BLS_MODULUS) wrp = (int(r_powers[k]) * h_k_pow) % BLS_MODULUS weighted_r_powers.append(wrp) @@ -610,15 +591,15 @@ def compute_cells_and_kzg_proofs(blob: Blob) -> Tuple[ ```python def verify_cell_kzg_proof_batch(commitments_bytes: Sequence[Bytes48], - column_indices: Sequence[ColumnIndex], + cell_indices: Sequence[CellIndex], cells: Sequence[Cell], proofs_bytes: Sequence[Bytes48]) -> bool: """ Verify that a set of cells belong to their corresponding commitments. - Given four lists representing tuples of (``commitment``, ``column_index``, ``cell``, ``proof``), + Given four lists representing tuples of (``commitment``, ``cell_index``, ``cell``, ``proof``), the function verifies ``proof`` which shows that ``cell`` are the evaluations of the polynomial - associated with ``commitment``, evaluated over the domain specified by ``column_index``. + associated with ``commitment``, evaluated over the domain specified by ``cell_index``. This function implements the universal verification equation that has been introduced here: https://ethresear.ch/t/a-universal-verification-equation-for-data-availability-sampling/13240 @@ -626,27 +607,32 @@ def verify_cell_kzg_proof_batch(commitments_bytes: Sequence[Bytes48], Public method. """ - assert len(commitments_bytes) == len(cells) == len(proofs_bytes) == len(column_indices) + assert len(commitments_bytes) == len(cells) == len(proofs_bytes) == len(cell_indices) for commitment_bytes in commitments_bytes: assert len(commitment_bytes) == BYTES_PER_COMMITMENT - for column_index in column_indices: - assert column_index < CELLS_PER_EXT_BLOB + for cell_index in cell_indices: + assert cell_index < CELLS_PER_EXT_BLOB for cell in cells: assert len(cell) == BYTES_PER_CELL for proof_bytes in proofs_bytes: assert len(proof_bytes) == BYTES_PER_PROOF - # Create the list of unique commitments we are dealing with - deduplicated_commitments = list(set(commitments_bytes)) - row_commitments = [bytes_to_kzg_commitment(commitment_bytes) for commitment_bytes in deduplicated_commitments] - # Create indices list mapping initial commitments (that potentially contains duplicates) to the unique commitments. - row_indices = [deduplicated_commitments.index(commitment_bytes) for commitment_bytes in commitments_bytes] + # Create the list of deduplicated commitments we are dealing with + deduplicated_commitments = [bytes_to_kzg_commitment(commitment_bytes) + for commitment_bytes in set(commitments_bytes)] + # Create indices list mapping initial commitments (that may contain duplicates) to the deduplicated commitments + commitment_indices = [deduplicated_commitments.index(commitment_bytes) for commitment_bytes in commitments_bytes] cosets_evals = [cell_to_coset_evals(cell) for cell in cells] proofs = [bytes_to_kzg_proof(proof_bytes) for proof_bytes in proofs_bytes] # Do the actual verification - return verify_cell_kzg_proof_batch_impl(row_commitments, row_indices, column_indices, cosets_evals, proofs) + return verify_cell_kzg_proof_batch_impl( + deduplicated_commitments, + commitment_indices, + cell_indices, + cosets_evals, + proofs) ``` ## Reconstruction diff --git a/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py b/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py index 1e8d86f6ef..c68c606765 100644 --- a/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py +++ b/tests/core/pyspec/eth2spec/test/eip7594/unittests/polynomial_commitments/test_polynomial_commitments.py @@ -115,7 +115,7 @@ def test_verify_cell_kzg_proof_batch_zero_cells(spec): # Verify with zero cells should return true assert spec.verify_cell_kzg_proof_batch( commitments_bytes=[], - column_indices=[], + cell_indices=[], cells=[], proofs_bytes=[], ) @@ -135,7 +135,7 @@ def test_verify_cell_kzg_proof_batch(spec): assert spec.verify_cell_kzg_proof_batch( commitments_bytes=[commitment, commitment], - column_indices=[0, 4], + cell_indices=[0, 4], cells=[cells[0], cells[4]], proofs_bytes=[proofs[0], proofs[4]], ) @@ -158,16 +158,16 @@ def test_verify_cell_kzg_proof_batch(spec): all_proofs.append(proofs) # the cells of interest - row_indices = [0, 0, 1, 2, 1] - column_indices = [0, 4, 0, 1, 2] - cells = [all_cells[i][j] for (i, j) in zip(row_indices, column_indices)] - proofs = [all_proofs[i][j] for (i, j) in zip(row_indices, column_indices)] - commitments = [all_commitments[i] for i in row_indices] + commitment_indices = [0, 0, 1, 2, 1] + cell_indices = [0, 4, 0, 1, 2] + cells = [all_cells[i][j] for (i, j) in zip(commitment_indices, cell_indices)] + proofs = [all_proofs[i][j] for (i, j) in zip(commitment_indices, cell_indices)] + commitments = [all_commitments[i] for i in commitment_indices] # do the check assert spec.verify_cell_kzg_proof_batch( commitments_bytes=commitments, - column_indices=column_indices, + cell_indices=cell_indices, cells=cells, proofs_bytes=proofs, ) @@ -187,7 +187,7 @@ def test_verify_cell_kzg_proof_batch_invalid(spec): assert not spec.verify_cell_kzg_proof_batch( commitments_bytes=[commitment, commitment], - column_indices=[0, 4], + cell_indices=[0, 4], cells=[cells[0], cells[5]], # Note: this is where it should go wrong proofs_bytes=[proofs[0], proofs[4]], ) @@ -210,11 +210,11 @@ def test_verify_cell_kzg_proof_batch_invalid(spec): all_proofs.append(proofs) # the cells of interest - row_indices = [0, 0, 1, 2, 1] - column_indices = [0, 4, 0, 1, 2] - cells = [all_cells[i][j] for (i, j) in zip(row_indices, column_indices)] - proofs = [all_proofs[i][j] for (i, j) in zip(row_indices, column_indices)] - commitments = [all_commitments[i] for i in row_indices] + commitment_indices = [0, 0, 1, 2, 1] + cell_indices = [0, 4, 0, 1, 2] + cells = [all_cells[i][j] for (i, j) in zip(commitment_indices, cell_indices)] + proofs = [all_proofs[i][j] for (i, j) in zip(commitment_indices, cell_indices)] + commitments = [all_commitments[i] for i in commitment_indices] # let's change one of the cells. Then it should not verify cells[1] = all_cells[1][3] @@ -222,7 +222,7 @@ def test_verify_cell_kzg_proof_batch_invalid(spec): # do the check assert not spec.verify_cell_kzg_proof_batch( commitments_bytes=commitments, - column_indices=column_indices, + cell_indices=cell_indices, cells=cells, proofs_bytes=proofs, ) diff --git a/tests/formats/kzg_7594/verify_cell_kzg_proof_batch.md b/tests/formats/kzg_7594/verify_cell_kzg_proof_batch.md index 270ad3e893..84bfaa1c2e 100644 --- a/tests/formats/kzg_7594/verify_cell_kzg_proof_batch.md +++ b/tests/formats/kzg_7594/verify_cell_kzg_proof_batch.md @@ -9,14 +9,14 @@ The test data is declared in a `data.yaml` file: ```yaml input: commitments: List[Bytes48] -- the KZG commitments for each cell - column_indices: List[ColumnIndex] -- the column index for each cell + cell_indices: List[CellIndex] -- the cell index for each cell cells: List[Cell] -- the cells proofs: List[Bytes48] -- the KZG proof for each cell output: bool -- true (all proofs are correct) or false (some proofs incorrect) ``` - `Bytes48` is a 48-byte hexadecimal string, prefixed with `0x`. -- `ColumnIndex` is an unsigned 64-bit integer. +- `CellIndex` is an unsigned 64-bit integer. - `Cell` is a 2048-byte hexadecimal string, prefixed with `0x`. All byte(s) fields are encoded as strings, hexadecimal encoding, prefixed with `0x`. diff --git a/tests/generators/kzg_7594/main.py b/tests/generators/kzg_7594/main.py index f21fa41258..d96c677440 100644 --- a/tests/generators/kzg_7594/main.py +++ b/tests/generators/kzg_7594/main.py @@ -66,13 +66,13 @@ def case_verify_cell_kzg_proof_batch(): for i in range(len(VALID_BLOBS)): cells, proofs = VALID_CELLS_AND_PROOFS[i] commitments = [VALID_COMMITMENTS[i] for _ in cells] - column_indices = list(range(spec.CELLS_PER_EXT_BLOB)) - assert spec.verify_cell_kzg_proof_batch(commitments, column_indices, cells, proofs) - identifier = make_id(commitments, column_indices, cells, proofs) + cell_indices = list(range(spec.CELLS_PER_EXT_BLOB)) + assert spec.verify_cell_kzg_proof_batch(commitments, cell_indices, cells, proofs) + identifier = make_id(commitments, cell_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_valid_{identifier}', { 'input': { 'commitments': encode_hex_list(commitments), - 'column_indices': column_indices, + 'cell_indices': cell_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), }, @@ -80,13 +80,13 @@ def case_verify_cell_kzg_proof_batch(): } # Valid: zero cells - cells, commitments, column_indices, proofs = [], [], [], [] - assert spec.verify_cell_kzg_proof_batch(commitments, column_indices, cells, proofs) - identifier = make_id(commitments, column_indices, cells, proofs) + cells, commitments, cell_indices, proofs = [], [], [], [] + assert spec.verify_cell_kzg_proof_batch(commitments, cell_indices, cells, proofs) + identifier = make_id(commitments, cell_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_valid_zero_cells_{identifier}', { 'input': { 'commitments': encode_hex_list(commitments), - 'column_indices': column_indices, + 'cell_indices': cell_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), }, @@ -97,15 +97,15 @@ def case_verify_cell_kzg_proof_batch(): cells0, proofs0 = VALID_CELLS_AND_PROOFS[0] cells1, proofs1 = VALID_CELLS_AND_PROOFS[1] commitments = [VALID_COMMITMENTS[0], VALID_COMMITMENTS[1]] - column_indices = [0, 0] + cell_indices = [0, 0] cells = [cells0[0], cells1[0]] proofs = [proofs0[0], proofs1[0]] - assert spec.verify_cell_kzg_proof_batch(commitments, column_indices, cells, proofs) - identifier = make_id(commitments, column_indices, cells, proofs) + assert spec.verify_cell_kzg_proof_batch(commitments, cell_indices, cells, proofs) + identifier = make_id(commitments, cell_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_valid_multiple_blobs_{identifier}', { 'input': { 'commitments': encode_hex_list(commitments), - 'column_indices': column_indices, + 'cell_indices': cell_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), }, @@ -115,15 +115,15 @@ def case_verify_cell_kzg_proof_batch(): # Valid: Same cell multiple times num_duplicates = 3 commitments = [VALID_COMMITMENTS[3]] * num_duplicates - column_indices = [0] * num_duplicates + cell_indices = [0] * num_duplicates cells = [VALID_CELLS_AND_PROOFS[3][0][0]] * num_duplicates proofs = [VALID_CELLS_AND_PROOFS[3][1][0]] * num_duplicates - assert spec.verify_cell_kzg_proof_batch(commitments, column_indices, cells, proofs) - identifier = make_id(commitments, column_indices, cells, proofs) + assert spec.verify_cell_kzg_proof_batch(commitments, cell_indices, cells, proofs) + identifier = make_id(commitments, cell_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_valid_same_cell_multiple_times_{identifier}', { 'input': { 'commitments': encode_hex_list(commitments), - 'column_indices': column_indices, + 'cell_indices': cell_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), }, @@ -135,13 +135,13 @@ def case_verify_cell_kzg_proof_batch(): cells, proofs = cells[:1], proofs[:1] # Use the wrong commitment commitments = [bls_add_one(VALID_COMMITMENTS[5])] - column_indices = list(range(len(cells))) - assert not spec.verify_cell_kzg_proof_batch(commitments, column_indices, cells, proofs) - identifier = make_id(commitments, column_indices, cells, proofs) + cell_indices = list(range(len(cells))) + assert not spec.verify_cell_kzg_proof_batch(commitments, cell_indices, cells, proofs) + identifier = make_id(commitments, cell_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_incorrect_commitment_{identifier}', { 'input': { 'commitments': encode_hex_list(commitments), - 'column_indices': column_indices, + 'cell_indices': cell_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), }, @@ -152,15 +152,15 @@ def case_verify_cell_kzg_proof_batch(): cells, proofs = VALID_CELLS_AND_PROOFS[6] cells, proofs = cells[:1], proofs[:1] commitments = [VALID_COMMITMENTS[6]] - column_indices = list(range(len(cells))) + cell_indices = list(range(len(cells))) # Change last cell so it's wrong cells[-1] = CELL_RANDOM_VALID2 - assert not spec.verify_cell_kzg_proof_batch(commitments, column_indices, cells, proofs) - identifier = make_id(commitments, column_indices, cells, proofs) + assert not spec.verify_cell_kzg_proof_batch(commitments, cell_indices, cells, proofs) + identifier = make_id(commitments, cell_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_incorrect_cell_{identifier}', { 'input': { 'commitments': encode_hex_list(commitments), - 'column_indices': column_indices, + 'cell_indices': cell_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), }, @@ -171,15 +171,15 @@ def case_verify_cell_kzg_proof_batch(): cells, proofs = VALID_CELLS_AND_PROOFS[0] cells, proofs = cells[:1], proofs[:1] commitments = [VALID_COMMITMENTS[0]] - column_indices = list(range(len(cells))) + cell_indices = list(range(len(cells))) # Change last proof so it's wrong proofs[-1] = bls_add_one(proofs[-1]) - assert not spec.verify_cell_kzg_proof_batch(commitments, column_indices, cells, proofs) - identifier = make_id(commitments, column_indices, cells, proofs) + assert not spec.verify_cell_kzg_proof_batch(commitments, cell_indices, cells, proofs) + identifier = make_id(commitments, cell_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_incorrect_proof_{identifier}', { 'input': { 'commitments': encode_hex_list(commitments), - 'column_indices': column_indices, + 'cell_indices': cell_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), }, @@ -192,32 +192,32 @@ def case_verify_cell_kzg_proof_batch(): cells, proofs = cells[:1], proofs[:1] # Set commitments to the invalid commitment commitments = [commitment] - column_indices = list(range(len(cells))) - expect_exception(spec.verify_cell_kzg_proof_batch, commitments, column_indices, cells, proofs) - identifier = make_id(commitments, column_indices, cells, proofs) + cell_indices = list(range(len(cells))) + expect_exception(spec.verify_cell_kzg_proof_batch, commitments, cell_indices, cells, proofs) + identifier = make_id(commitments, cell_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_invalid_commitment_{identifier}', { 'input': { 'commitments': encode_hex_list(commitments), - 'column_indices': column_indices, + 'cell_indices': cell_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), }, 'output': None } - # Edge case: Invalid column_index + # Edge case: Invalid cell_index cells, proofs = VALID_CELLS_AND_PROOFS[1] cells, proofs = cells[:1], proofs[:1] commitments = [VALID_COMMITMENTS[1]] - column_indices = list(range(len(cells))) - # Set first column index to an invalid value - column_indices[0] = spec.CELLS_PER_EXT_BLOB - expect_exception(spec.verify_cell_kzg_proof_batch, commitments, column_indices, cells, proofs) - identifier = make_id(commitments, column_indices, cells, proofs) - yield f'verify_cell_kzg_proof_batch_case_invalid_column_index_{identifier}', { + cell_indices = list(range(len(cells))) + # Set first cell index to an invalid value + cell_indices[0] = spec.CELLS_PER_EXT_BLOB + expect_exception(spec.verify_cell_kzg_proof_batch, commitments, cell_indices, cells, proofs) + identifier = make_id(commitments, cell_indices, cells, proofs) + yield f'verify_cell_kzg_proof_batch_case_invalid_cell_index_{identifier}', { 'input': { 'commitments': encode_hex_list(commitments), - 'column_indices': column_indices, + 'cell_indices': cell_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), }, @@ -229,15 +229,15 @@ def case_verify_cell_kzg_proof_batch(): cells, proofs = VALID_CELLS_AND_PROOFS[i % len(INVALID_INDIVIDUAL_CELL_BYTES)] cells, proofs = cells[:1], proofs[:1] commitments = [VALID_COMMITMENTS[i % len(INVALID_INDIVIDUAL_CELL_BYTES)]] - column_indices = list(range(len(cells))) + cell_indices = list(range(len(cells))) # Set first cell to the invalid cell cells[0] = cell - expect_exception(spec.verify_cell_kzg_proof_batch, commitments, column_indices, cells, proofs) - identifier = make_id(commitments, column_indices, cells, proofs) + expect_exception(spec.verify_cell_kzg_proof_batch, commitments, cell_indices, cells, proofs) + identifier = make_id(commitments, cell_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_invalid_cell_{identifier}', { 'input': { 'commitments': encode_hex_list(commitments), - 'column_indices': column_indices, + 'cell_indices': cell_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), }, @@ -249,15 +249,15 @@ def case_verify_cell_kzg_proof_batch(): cells, proofs = VALID_CELLS_AND_PROOFS[i % len(INVALID_G1_POINTS)] cells, proofs = cells[:1], proofs[:1] commitments = [VALID_COMMITMENTS[i % len(INVALID_G1_POINTS)]] - column_indices = list(range(len(cells))) + cell_indices = list(range(len(cells))) # Set first proof to the invalid proof proofs[0] = proof - expect_exception(spec.verify_cell_kzg_proof_batch, commitments, column_indices, cells, proofs) - identifier = make_id(commitments, column_indices, cells, proofs) + expect_exception(spec.verify_cell_kzg_proof_batch, commitments, cell_indices, cells, proofs) + identifier = make_id(commitments, cell_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_invalid_proof_{identifier}', { 'input': { 'commitments': encode_hex_list(commitments), - 'column_indices': column_indices, + 'cell_indices': cell_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), }, @@ -269,31 +269,31 @@ def case_verify_cell_kzg_proof_batch(): cells, proofs = cells[:2], proofs[:2] # Do not include the second commitment commitments = [VALID_COMMITMENTS[0]] - column_indices = list(range(len(cells))) - expect_exception(spec.verify_cell_kzg_proof_batch, commitments, column_indices, cells, proofs) - identifier = make_id(commitments, column_indices, cells, proofs) + cell_indices = list(range(len(cells))) + expect_exception(spec.verify_cell_kzg_proof_batch, commitments, cell_indices, cells, proofs) + identifier = make_id(commitments, cell_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_invalid_missing_commitment_{identifier}', { 'input': { 'commitments': encode_hex_list(commitments), - 'column_indices': column_indices, + 'cell_indices': cell_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), }, 'output': None } - # Edge case: Missing a column index + # Edge case: Missing a cell index cells, proofs = VALID_CELLS_AND_PROOFS[2] cells, proofs = cells[:2], proofs[:2] commitments = [VALID_COMMITMENTS[2], VALID_COMMITMENTS[2]] - # Leave off one of the column indices - column_indices = list(range(len(cells) - 1)) - expect_exception(spec.verify_cell_kzg_proof_batch, commitments, column_indices, cells, proofs) - identifier = make_id(commitments, column_indices, cells, proofs) - yield f'verify_cell_kzg_proof_batch_case_invalid_missing_column_index_{identifier}', { + # Leave off one of the cell indices + cell_indices = list(range(len(cells) - 1)) + expect_exception(spec.verify_cell_kzg_proof_batch, commitments, cell_indices, cells, proofs) + identifier = make_id(commitments, cell_indices, cells, proofs) + yield f'verify_cell_kzg_proof_batch_case_invalid_missing_cell_index_{identifier}', { 'input': { 'commitments': encode_hex_list(commitments), - 'column_indices': column_indices, + 'cell_indices': cell_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), }, @@ -304,15 +304,15 @@ def case_verify_cell_kzg_proof_batch(): cells, proofs = VALID_CELLS_AND_PROOFS[3] cells, proofs = cells[:2], proofs[:2] commitments = [VALID_COMMITMENTS[3], VALID_COMMITMENTS[3]] - column_indices = list(range(len(cells))) + cell_indices = list(range(len(cells))) # Remove the last proof cells = cells[:-1] - expect_exception(spec.verify_cell_kzg_proof_batch, commitments, column_indices, cells, proofs) - identifier = make_id(commitments, column_indices, cells, proofs) + expect_exception(spec.verify_cell_kzg_proof_batch, commitments, cell_indices, cells, proofs) + identifier = make_id(commitments, cell_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_invalid_missing_cell_{identifier}', { 'input': { 'commitments': encode_hex_list(commitments), - 'column_indices': column_indices, + 'cell_indices': cell_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), }, @@ -323,15 +323,15 @@ def case_verify_cell_kzg_proof_batch(): cells, proofs = VALID_CELLS_AND_PROOFS[4] cells, proofs = cells[:2], proofs[:2] commitments = [VALID_COMMITMENTS[4], VALID_COMMITMENTS[4]] - column_indices = list(range(len(cells))) + cell_indices = list(range(len(cells))) # Remove the last proof proofs = proofs[:-1] - expect_exception(spec.verify_cell_kzg_proof_batch, commitments, column_indices, cells, proofs) - identifier = make_id(commitments, column_indices, cells, proofs) + expect_exception(spec.verify_cell_kzg_proof_batch, commitments, cell_indices, cells, proofs) + identifier = make_id(commitments, cell_indices, cells, proofs) yield f'verify_cell_kzg_proof_batch_case_invalid_missing_proof_{identifier}', { 'input': { 'commitments': encode_hex_list(commitments), - 'column_indices': column_indices, + 'cell_indices': cell_indices, 'cells': encode_hex_list(cells), 'proofs': encode_hex_list(proofs), }, From 4b43b96f831f4269c596199592c21b394f13c263 Mon Sep 17 00:00:00 2001 From: Potuz Date: Mon, 15 Jul 2024 15:17:20 -0300 Subject: [PATCH 105/130] doctoc --- specs/_features/eip7732/beacon-chain.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/specs/_features/eip7732/beacon-chain.md b/specs/_features/eip7732/beacon-chain.md index 939adf08da..441ea9c0f1 100644 --- a/specs/_features/eip7732/beacon-chain.md +++ b/specs/_features/eip7732/beacon-chain.md @@ -52,6 +52,8 @@ - [`process_payload_attestation`](#process_payload_attestation) - [Modified `process_execution_payload`](#modified-process_execution_payload) - [New `verify_execution_payload_envelope_signature`](#new-verify_execution_payload_envelope_signature) + - [Modified `is_merge_transition_complete`](#modified-is_merge_transition_complete) + - [Modified `validate_merge_block`](#modified-validate_merge_block) From 5961e269316c8dda36b481390623eee9b789fe2f Mon Sep 17 00:00:00 2001 From: Justin Traglia <95511699+jtraglia@users.noreply.github.com> Date: Mon, 15 Jul 2024 15:09:01 -0500 Subject: [PATCH 106/130] Remove unnecessary comment --- specs/_features/eip7594/das-core.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/specs/_features/eip7594/das-core.md b/specs/_features/eip7594/das-core.md index c3f19b4cb0..f25e29ce3a 100644 --- a/specs/_features/eip7594/das-core.md +++ b/specs/_features/eip7594/das-core.md @@ -201,9 +201,6 @@ def get_data_column_sidecars(signed_block: SignedBeaconBlock, """ Given a signed block and the cells/proofs associated with each blob in the block, assemble the sidecars which can be distributed to peers. - - Since there is no method which converts cells to a blob, this method takes - cells/proofs instead of blobs so it can re-create sidecars after recovery. """ blob_kzg_commitments = signed_block.message.body.blob_kzg_commitments assert len(cells_and_kzg_proofs) == len(blob_kzg_commitments) From 5bb25c8088612ecf08646d5e27f0e2fc682f54a3 Mon Sep 17 00:00:00 2001 From: Potuz Date: Mon, 15 Jul 2024 17:11:26 -0300 Subject: [PATCH 107/130] remove spurious config prefix --- specs/_features/eip7732/beacon-chain.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/specs/_features/eip7732/beacon-chain.md b/specs/_features/eip7732/beacon-chain.md index 441ea9c0f1..fdcb2b3008 100644 --- a/specs/_features/eip7732/beacon-chain.md +++ b/specs/_features/eip7732/beacon-chain.md @@ -698,10 +698,10 @@ def validate_merge_block(block: BeaconBlock) -> None: and a client software MAY delay a call to ``validate_merge_block`` until the PoW block(s) become available. """ - if config.TERMINAL_BLOCK_HASH != Hash32(): - # If `config.TERMINAL_BLOCK_HASH` is used as an override, the activation epoch must be reached. - assert compute_epoch_at_slot(block.slot) >= config.TERMINAL_BLOCK_HASH_ACTIVATION_EPOCH - assert block.body.signed_execution_payload_header.message.parent_block_hash == config.TERMINAL_BLOCK_HASH + if TERMINAL_BLOCK_HASH != Hash32(): + # If `TERMINAL_BLOCK_HASH` is used as an override, the activation epoch must be reached. + assert compute_epoch_at_slot(block.slot) >= TERMINAL_BLOCK_HASH_ACTIVATION_EPOCH + assert block.body.signed_execution_payload_header.message.parent_block_hash == TERMINAL_BLOCK_HASH return # Modified in EIP-7732 From 6d146e2bc115c3172341d03e3aef1626952e6ca1 Mon Sep 17 00:00:00 2001 From: Justin Traglia Date: Tue, 16 Jul 2024 14:16:18 -0500 Subject: [PATCH 108/130] For pytest, replace NUMBER_OF_CORES with auto --- Makefile | 7 +++---- docker/README.md | 8 ++++---- scripts/build_run_docker_tests.sh | 10 +++------- 3 files changed, 10 insertions(+), 15 deletions(-) diff --git a/Makefile b/Makefile index bdf4bdde7a..04b25267e7 100644 --- a/Makefile +++ b/Makefile @@ -14,7 +14,6 @@ SOLIDITY_FILE_NAME = deposit_contract.json DEPOSIT_CONTRACT_TESTER_DIR = ${SOLIDITY_DEPOSIT_CONTRACT_DIR}/web3_tester CONFIGS_DIR = ./configs TEST_PRESET_TYPE ?= minimal -NUMBER_OF_CORES=16 # Collect a list of generator names GENERATORS = $(sort $(dir $(wildcard $(GENERATOR_DIR)/*/.))) # Map this list of generator paths to "gen_{generator name}" entries @@ -118,7 +117,7 @@ install_test: preinstallation # Testing against `minimal` or `mainnet` config by default test: pyspec . venv/bin/activate; cd $(PY_SPEC_DIR); \ - python3 -m pytest -n 4 --disable-bls $(COVERAGE_SCOPE) --cov-report="html:$(COV_HTML_OUT)" --cov-branch eth2spec + python3 -m pytest -n auto --disable-bls $(COVERAGE_SCOPE) --cov-report="html:$(COV_HTML_OUT)" --cov-branch eth2spec # Testing against `minimal` or `mainnet` config by default find_test: pyspec @@ -129,10 +128,10 @@ citest: pyspec mkdir -p $(TEST_REPORT_DIR); ifdef fork . venv/bin/activate; cd $(PY_SPEC_DIR); \ - python3 -m pytest -n $(NUMBER_OF_CORES) --bls-type=fastest --preset=$(TEST_PRESET_TYPE) --fork=$(fork) --junitxml=test-reports/test_results.xml eth2spec + python3 -m pytest -n auto --bls-type=fastest --preset=$(TEST_PRESET_TYPE) --fork=$(fork) --junitxml=test-reports/test_results.xml eth2spec else . venv/bin/activate; cd $(PY_SPEC_DIR); \ - python3 -m pytest -n $(NUMBER_OF_CORES) --bls-type=fastest --preset=$(TEST_PRESET_TYPE) --junitxml=test-reports/test_results.xml eth2spec + python3 -m pytest -n auto --bls-type=fastest --preset=$(TEST_PRESET_TYPE) --junitxml=test-reports/test_results.xml eth2spec endif diff --git a/docker/README.md b/docker/README.md index 6d5b21e59d..4f0c1ecca9 100644 --- a/docker/README.md +++ b/docker/README.md @@ -6,15 +6,15 @@ This dockerfile sets up the dependencies required to run consensus-spec tests. T Handy commands: - `docker run -it $IMAGE_NAME /bin/sh` will give you a shell inside the docker container to manually run any tests -- `docker run $IMAGE_NAME make citest` will run the make citest command inside the docker container +- `docker run $IMAGE_NAME make citest` will run the make citest command inside the docker container Ideally manual running of docker containers is for advanced users, we recommend the script based approach described below for most users. -The `scripts/build_run_docker_tests.sh` script will cover most usecases. The script allows the user to configure the fork(altair/bellatrix/capella..), `$IMAGE_NAME` (specifies the container to use), number of cores, preset type (mainnet/minimal), and test all forks flags. Ideally, this is the main way that users interact with the spec tests instead of running it locally with varying versions of dependencies. +The `scripts/build_run_docker_tests.sh` script will cover most usecases. The script allows the user to configure the fork(altair/bellatrix/capella..), `$IMAGE_NAME` (specifies the container to use), preset type (mainnet/minimal), and test all forks flags. Ideally, this is the main way that users interact with the spec tests instead of running it locally with varying versions of dependencies. E.g: -- `./build_run_test.sh --p mainnet --n 16` will run the mainnet preset tests with 16 threads +- `./build_run_test.sh --p mainnet` will run the mainnet preset tests - `./build_run_test.sh --a` will run all the tests across all the forks -- `./build_run_test.sh --f deneb --n 16` will only run deneb tests on 16 threads +- `./build_run_test.sh --f deneb` will only run deneb tests Results are always placed in a folder called `./testResults`. The results are `.xml` files and contain the fork they represent and the date/time they were run at. \ No newline at end of file diff --git a/scripts/build_run_docker_tests.sh b/scripts/build_run_docker_tests.sh index 8b20cfae62..9d2740ca27 100755 --- a/scripts/build_run_docker_tests.sh +++ b/scripts/build_run_docker_tests.sh @@ -13,7 +13,6 @@ ALL_EXECUTABLE_SPECS=("phase0" "altair" "bellatrix" "capella" "deneb" "electra" "whisk") TEST_PRESET_TYPE=minimal FORK_TO_TEST=phase0 -NUMBER_OF_CORES=4 WORKDIR="//consensus-specs//tests//core//pyspec" ETH2SPEC_FOLDER_NAME="eth2spec" CONTAINER_NAME="consensus-specs-tests" @@ -21,17 +20,15 @@ DATE=$(date +"%Y%m%d-%H-%M") # Default flag values version=$(git log --pretty=format:'%h' -n 1) IMAGE_NAME="consensus-specs:$version" -number_of_core=4 # displays the available options display_help() { echo "Run 'consensus-specs' tests from a container instance." echo "Be sure to launch Docker before running this script." echo - echo "Syntax: build_run_test.sh [--v TAG | --n NUMBER_OF_CORE | --f FORK_TO_TEST | --p PRESET_TYPE | --a | --h HELP]" + echo "Syntax: build_run_test.sh [--v TAG | --f FORK_TO_TEST | --p PRESET_TYPE | --a | --h HELP]" echo " --f Specify the fork to test" echo " --i Specify the docker image to use" - echo " --n Specify the number of cores" echo " --p Specify the test preset type" echo " --a Test all forks" echo " --h Display this help and exit" @@ -63,7 +60,6 @@ while [[ "$#" -gt 0 ]]; do case $1 in --f) FORK_TO_TEST="$2"; shift ;; --v) IMAGE_NAME="$2"; shift ;; - --n) NUMBER_OF_CORES="$2"; shift ;; --p) TEST_PRESET_TYPE="$2"; shift ;; --a) FORK_TO_TEST="all" ;; --h) display_help; exit 0 ;; @@ -90,12 +86,12 @@ fi if [ "$FORK_TO_TEST" == "all" ]; then for fork in "${ALL_EXECUTABLE_SPECS[@]}"; do docker run --name $CONTAINER_NAME $IMAGE_NAME \ - make citest fork=$fork TEST_PRESET_TYPE=$TEST_PRESET_TYPE NUMBER_OF_CORES=$NUMBER_OF_CORES + make citest fork=$fork TEST_PRESET_TYPE=$TEST_PRESET_TYPE copy_test_results $fork done else docker run --name $CONTAINER_NAME $IMAGE_NAME \ - make citest fork=$FORK_TO_TEST TEST_PRESET_TYPE=$TEST_PRESET_TYPE NUMBER_OF_CORES=$NUMBER_OF_CORES + make citest fork=$FORK_TO_TEST TEST_PRESET_TYPE=$TEST_PRESET_TYPE copy_test_results $FORK_TO_TEST fi From 233129122bd289316f9aa5afbeefa25392618aa5 Mon Sep 17 00:00:00 2001 From: kevaundray Date: Wed, 17 Jul 2024 14:23:59 +0100 Subject: [PATCH 109/130] Replace `recover_data` with `recover_polynomialcoeff` (#3820) * chore: remove recover_data * make it look closer to final code * Improve comments * Fix lint issue * Fix tests & clean things up a bit * Replace a couple uses of "monomial" with "coefficient" * Revert "Replace a couple uses of "monomial" with "coefficient"" This reverts commit c9a1a757d1a09190eee78767b3d36b2a84066e42. * Only replace "monomial" with "coefficient" --------- Co-authored-by: Justin Traglia <95511699+jtraglia@users.noreply.github.com> Co-authored-by: Justin Traglia --- .../polynomial-commitments-sampling.md | 97 ++++++++----------- 1 file changed, 41 insertions(+), 56 deletions(-) diff --git a/specs/_features/eip7594/polynomial-commitments-sampling.md b/specs/_features/eip7594/polynomial-commitments-sampling.md index 2993d50bf9..7fec2a52d7 100644 --- a/specs/_features/eip7594/polynomial-commitments-sampling.md +++ b/specs/_features/eip7594/polynomial-commitments-sampling.md @@ -39,12 +39,13 @@ - [`coset_for_cell`](#coset_for_cell) - [Cells](#cells-1) - [Cell computation](#cell-computation) + - [`compute_cells_and_kzg_proofs_polynomialcoeff`](#compute_cells_and_kzg_proofs_polynomialcoeff) - [`compute_cells_and_kzg_proofs`](#compute_cells_and_kzg_proofs) - [Cell verification](#cell-verification) - [`verify_cell_kzg_proof_batch`](#verify_cell_kzg_proof_batch) - [Reconstruction](#reconstruction) - [`construct_vanishing_polynomial`](#construct_vanishing_polynomial) - - [`recover_data`](#recover_data) + - [`recover_polynomialcoeff`](#recover_polynomialcoeff) - [`recover_cells_and_kzg_proofs`](#recover_cells_and_kzg_proofs) @@ -555,6 +556,24 @@ def coset_for_cell(cell_index: CellIndex) -> Coset: ### Cell computation +#### `compute_cells_and_kzg_proofs_polynomialcoeff` + +```python +def compute_cells_and_kzg_proofs_polynomialcoeff(polynomial_coeff: PolynomialCoeff) -> Tuple[ + Vector[Cell, CELLS_PER_EXT_BLOB], + Vector[KZGProof, CELLS_PER_EXT_BLOB]]: + """ + Helper function which computes cells/proofs for a polynomial in coefficient form. + """ + cells, proofs = [], [] + for i in range(CELLS_PER_EXT_BLOB): + coset = coset_for_cell(CellIndex(i)) + proof, ys = compute_kzg_proof_multi_impl(polynomial_coeff, coset) + cells.append(coset_evals_to_cell(ys)) + proofs.append(proof) + return cells, proofs +``` + #### `compute_cells_and_kzg_proofs` ```python @@ -572,17 +591,7 @@ def compute_cells_and_kzg_proofs(blob: Blob) -> Tuple[ polynomial = blob_to_polynomial(blob) polynomial_coeff = polynomial_eval_to_coeff(polynomial) - - cells = [] - proofs = [] - - for i in range(CELLS_PER_EXT_BLOB): - coset = coset_for_cell(CellIndex(i)) - proof, ys = compute_kzg_proof_multi_impl(polynomial_coeff, coset) - cells.append(coset_evals_to_cell(ys)) - proofs.append(proof) - - return cells, proofs + return compute_cells_and_kzg_proofs_polynomialcoeff(polynomial_coeff) ``` ### Cell verification @@ -668,21 +677,19 @@ def construct_vanishing_polynomial(missing_cell_indices: Sequence[CellIndex]) -> return zero_poly_coeff ``` -### `recover_data` +### `recover_polynomialcoeff` ```python -def recover_data(cell_indices: Sequence[CellIndex], - cells: Sequence[Cell], - ) -> Sequence[BLSFieldElement]: +def recover_polynomialcoeff(cell_indices: Sequence[CellIndex], + cells: Sequence[Cell]) -> Sequence[BLSFieldElement]: """ - Recover the missing evaluations for the extended blob, given at least half of the evaluations. + Recover the polynomial in coefficient form that when evaluated at the roots of unity will give the extended blob. """ - - # Get the extended domain. This will be referred to as the FFT domain. + # Get the extended domain. This will be referred to as the FFT domain roots_of_unity_extended = compute_roots_of_unity(FIELD_ELEMENTS_PER_EXT_BLOB) - # Flatten the cells into evaluations. - # If a cell is missing, then its evaluation is zero. + # Flatten the cells into evaluations + # If a cell is missing, then its evaluation is zero extended_evaluation_rbo = [0] * FIELD_ELEMENTS_PER_EXT_BLOB for cell_index, cell in zip(cell_indices, cells): start = cell_index * FIELD_ELEMENTS_PER_CELL @@ -690,7 +697,7 @@ def recover_data(cell_indices: Sequence[CellIndex], extended_evaluation_rbo[start:end] = cell extended_evaluation = bit_reversal_permutation(extended_evaluation_rbo) - # Compute Z(x) in monomial form + # Compute Z(x) in coefficient form # Z(x) is the polynomial which vanishes on all of the evaluations which are missing missing_cell_indices = [CellIndex(cell_index) for cell_index in range(CELLS_PER_EXT_BLOB) if cell_index not in cell_indices] @@ -703,7 +710,7 @@ def recover_data(cell_indices: Sequence[CellIndex], extended_evaluation_times_zero = [BLSFieldElement(int(a) * int(b) % BLS_MODULUS) for a, b in zip(zero_poly_eval, extended_evaluation)] - # Convert (E*Z)(x) to monomial form + # Convert (E*Z)(x) to coefficient form extended_evaluation_times_zero_coeffs = fft_field(extended_evaluation_times_zero, roots_of_unity_extended, inv=True) # Convert (E*Z)(x) to evaluation form over a coset of the FFT domain @@ -713,18 +720,12 @@ def recover_data(cell_indices: Sequence[CellIndex], zero_poly_over_coset = coset_fft_field(zero_poly_coeff, roots_of_unity_extended) # Compute Q_3(x) = (E*Z)(x) / Z(x) in evaluation form over a coset of the FFT domain - reconstructed_poly_over_coset = [ - div(a, b) - for a, b in zip(extended_evaluations_over_coset, zero_poly_over_coset) - ] + reconstructed_poly_over_coset = [div(a, b) for a, b in zip(extended_evaluations_over_coset, zero_poly_over_coset)] - # Convert Q_3(x) to monomial form + # Convert Q_3(x) to coefficient form reconstructed_poly_coeff = coset_fft_field(reconstructed_poly_over_coset, roots_of_unity_extended, inv=True) - # Convert Q_3(x) to evaluation form over the FFT domain and bit reverse the result - reconstructed_data = bit_reversal_permutation(fft_field(reconstructed_poly_coeff, roots_of_unity_extended)) - - return reconstructed_data + return reconstructed_poly_coeff[:FIELD_ELEMENTS_PER_BLOB] ``` ### `recover_cells_and_kzg_proofs` @@ -735,7 +736,7 @@ def recover_cells_and_kzg_proofs(cell_indices: Sequence[CellIndex], Vector[Cell, CELLS_PER_EXT_BLOB], Vector[KZGProof, CELLS_PER_EXT_BLOB]]: """ - Given at least 50% of cells/proofs for a blob, recover all the cells/proofs. + Given at least 50% of cells for a blob, recover all the cells/proofs. This algorithm uses FFTs to recover cells faster than using Lagrange implementation, as can be seen here: https://ethresear.ch/t/reed-solomon-erasure-code-recovery-in-n-log-2-n-time-with-ffts/3039 @@ -745,6 +746,7 @@ def recover_cells_and_kzg_proofs(cell_indices: Sequence[CellIndex], Public method. """ + # Check we have the same number of cells and indices assert len(cell_indices) == len(cells) # Check we have enough cells to be able to perform the reconstruction assert CELLS_PER_EXT_BLOB / 2 <= len(cell_indices) <= CELLS_PER_EXT_BLOB @@ -757,29 +759,12 @@ def recover_cells_and_kzg_proofs(cell_indices: Sequence[CellIndex], for cell in cells: assert len(cell) == BYTES_PER_CELL - # Convert cells to coset evals + # Convert cells to coset evaluations cosets_evals = [cell_to_coset_evals(cell) for cell in cells] - reconstructed_data = recover_data(cell_indices, cosets_evals) - - for cell_index, coset_evals in zip(cell_indices, cosets_evals): - start = cell_index * FIELD_ELEMENTS_PER_CELL - end = (cell_index + 1) * FIELD_ELEMENTS_PER_CELL - assert reconstructed_data[start:end] == coset_evals - - recovered_cells = [ - coset_evals_to_cell(reconstructed_data[i * FIELD_ELEMENTS_PER_CELL:(i + 1) * FIELD_ELEMENTS_PER_CELL]) - for i in range(CELLS_PER_EXT_BLOB)] - - polynomial_eval = reconstructed_data[:FIELD_ELEMENTS_PER_BLOB] - polynomial_coeff = polynomial_eval_to_coeff(polynomial_eval) - recovered_proofs = [None] * CELLS_PER_EXT_BLOB - - for i in range(CELLS_PER_EXT_BLOB): - coset = coset_for_cell(CellIndex(i)) - proof, ys = compute_kzg_proof_multi_impl(polynomial_coeff, coset) - assert coset_evals_to_cell(ys) == recovered_cells[i] - recovered_proofs[i] = proof - - return recovered_cells, recovered_proofs + # Given the coset evaluations, recover the polynomial in coefficient form + polynomial_coeff = recover_polynomialcoeff(cell_indices, cosets_evals) + + # Recompute all cells/proofs + return compute_cells_and_kzg_proofs_polynomialcoeff(polynomial_coeff) ``` From f3dcc597e4fed4053f3a4d869c1b7357761c47fd Mon Sep 17 00:00:00 2001 From: Justin Traglia <95511699+jtraglia@users.noreply.github.com> Date: Thu, 18 Jul 2024 16:18:30 -0500 Subject: [PATCH 110/130] Update description for test format --- tests/formats/kzg_7594/recover_cells_and_kzg_proofs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/formats/kzg_7594/recover_cells_and_kzg_proofs.md b/tests/formats/kzg_7594/recover_cells_and_kzg_proofs.md index 37c5950c55..b12f49c265 100644 --- a/tests/formats/kzg_7594/recover_cells_and_kzg_proofs.md +++ b/tests/formats/kzg_7594/recover_cells_and_kzg_proofs.md @@ -1,6 +1,6 @@ # Test format: Recover cells and KZG proofs -Recover all cells/proofs given at least 50% of the original `cells` and `proofs`. +Recover all cells/proofs given at least 50% of the original `cells`. ## Test case format From 80092b1b14827dc29247132576bb98a0a8093013 Mon Sep 17 00:00:00 2001 From: 0xTylerHolmes Date: Fri, 19 Jul 2024 12:52:59 -0400 Subject: [PATCH 111/130] remove beacon_attestation_subnet from global topics --- specs/electra/p2p-interface.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/specs/electra/p2p-interface.md b/specs/electra/p2p-interface.md index ebdcaaa831..88d14813b3 100644 --- a/specs/electra/p2p-interface.md +++ b/specs/electra/p2p-interface.md @@ -15,6 +15,7 @@ The specification of these changes continues in the same format as the network s - [Topics and messages](#topics-and-messages) - [Global topics](#global-topics) - [`beacon_aggregate_and_proof`](#beacon_aggregate_and_proof) + - [Attestation subnets](#attestation-subnets) - [`beacon_attestation_{subnet_id}`](#beacon_attestation_subnet_id) @@ -49,6 +50,8 @@ The following validations are added: * [REJECT] `len(committee_indices) == 1`, where `committee_indices = get_committee_indices(aggregate)`. * [REJECT] `aggregate.data.index == 0` +#### Attestation subnets + ##### `beacon_attestation_{subnet_id}` The following convenience variables are re-defined From 8c9514ecc1ce04ad9cbbde74a318430367f838eb Mon Sep 17 00:00:00 2001 From: Potuz Date: Mon, 22 Jul 2024 16:29:55 -0300 Subject: [PATCH 112/130] Restore proposer_boost timeliness calculation --- specs/_features/eip7732/fork-choice.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/specs/_features/eip7732/fork-choice.md b/specs/_features/eip7732/fork-choice.md index 98ff0e233e..4bab92e8f9 100644 --- a/specs/_features/eip7732/fork-choice.md +++ b/specs/_features/eip7732/fork-choice.md @@ -429,7 +429,12 @@ def on_block(store: Store, signed_block: SignedBeaconBlock) -> None: # Add proposer score boost if the block is timely time_into_slot = (store.time - store.genesis_time) % SECONDS_PER_SLOT is_before_attesting_interval = time_into_slot < SECONDS_PER_SLOT // INTERVALS_PER_SLOT - if get_current_slot(store) == block.slot and is_before_attesting_interval: + is_timely = get_current_slot(store) == block.slot and is_before_attesting_interval + store.block_timeliness[hash_tree_root(block)] = is_timely + + # Add proposer score boost if the block is timely and not conflicting with an existing block + is_first_block = store.proposer_boost_root == Root() + if is_timely and is_first_block: store.proposer_boost_root = hash_tree_root(block) # Update checkpoints in store if necessary From b7461ac174b56244a34e654f2083f35da3b4e29d Mon Sep 17 00:00:00 2001 From: Potuz Date: Tue, 23 Jul 2024 07:53:50 -0300 Subject: [PATCH 113/130] Remove block_timeliness from the store --- specs/_features/eip7732/fork-choice.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/specs/_features/eip7732/fork-choice.md b/specs/_features/eip7732/fork-choice.md index 4bab92e8f9..2fd33d052e 100644 --- a/specs/_features/eip7732/fork-choice.md +++ b/specs/_features/eip7732/fork-choice.md @@ -106,7 +106,7 @@ class Store(object): equivocating_indices: Set[ValidatorIndex] blocks: Dict[Root, BeaconBlock] = field(default_factory=dict) block_states: Dict[Root, BeaconState] = field(default_factory=dict) - block_timeliness: Dict[Root, boolean] = field(default_factory=dict) + # block_timeliness removed in EIP-7732 checkpoint_states: Dict[Checkpoint, BeaconState] = field(default_factory=dict) latest_messages: Dict[ValidatorIndex, LatestMessage] = field(default_factory=dict) unrealized_justifications: Dict[Root, Checkpoint] = field(default_factory=dict) @@ -430,7 +430,6 @@ def on_block(store: Store, signed_block: SignedBeaconBlock) -> None: time_into_slot = (store.time - store.genesis_time) % SECONDS_PER_SLOT is_before_attesting_interval = time_into_slot < SECONDS_PER_SLOT // INTERVALS_PER_SLOT is_timely = get_current_slot(store) == block.slot and is_before_attesting_interval - store.block_timeliness[hash_tree_root(block)] = is_timely # Add proposer score boost if the block is timely and not conflicting with an existing block is_first_block = store.proposer_boost_root == Root() From d0fad1f3809e82e430b07ad5a1ec632f845bda84 Mon Sep 17 00:00:00 2001 From: Potuz Date: Tue, 23 Jul 2024 13:06:54 -0300 Subject: [PATCH 114/130] Advance best child with empty until it is higher than every children --- specs/_features/eip7732/fork-choice.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/specs/_features/eip7732/fork-choice.md b/specs/_features/eip7732/fork-choice.md index 2fd33d052e..1cb3e4d82f 100644 --- a/specs/_features/eip7732/fork-choice.md +++ b/specs/_features/eip7732/fork-choice.md @@ -343,13 +343,14 @@ def get_head(store: Store) -> ChildNode: while True: children = [ ChildNode(root=root, slot=block.slot, is_payload_present=present) for (root, block) in blocks.items() - if block.parent_root == best_child.root and + if block.parent_root == best_child.root and block.slot > best_child.slot and (best_child.root == justified_root or is_parent_node_full(store, block) == best_child.is_payload_present) for present in (True, False) if root in store.execution_payload_states or not present ] if len(children) == 0: return best_child # if we have children we consider the current head advanced as a possible head + highest_child_slot = max(child.slot for child in children) children += [ ChildNode(root=best_child.root, slot=best_child.slot + 1, is_payload_present=best_child.is_payload_present) ] @@ -366,7 +367,7 @@ def get_head(store: Store) -> ChildNode: child.root ) ) - if new_best_child.root == best_child.root: + if new_best_child.root == best_child.root and new_best_child.slot >= highest_child_slot: return new_best_child best_child = new_best_child ``` From 0c9ca6de24e27ee717ef8bcfec299be5b9ee77da Mon Sep 17 00:00:00 2001 From: Potuz Date: Tue, 23 Jul 2024 13:31:21 -0300 Subject: [PATCH 115/130] solve ties by block slot first --- specs/_features/eip7732/fork-choice.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/specs/_features/eip7732/fork-choice.md b/specs/_features/eip7732/fork-choice.md index 1cb3e4d82f..19c9b88dd6 100644 --- a/specs/_features/eip7732/fork-choice.md +++ b/specs/_features/eip7732/fork-choice.md @@ -354,16 +354,16 @@ def get_head(store: Store) -> ChildNode: children += [ ChildNode(root=best_child.root, slot=best_child.slot + 1, is_payload_present=best_child.is_payload_present) ] - # Sort by latest attesting balance with ties broken lexicographically - # Ties broken by favoring full blocks according to the PTC vote + # Sort by latest attesting balance with + # Ties broken by the block's slot + # Ties are broken by the PTC vote # Ties are then broken by favoring full blocks - # Ties broken then by favoring higher slot numbers # Ties then broken by favoring block with lexicographically higher root new_best_child = max(children, key=lambda child: ( get_weight(store, child), + blocks[child.root].slot, is_payload_present(store, child.root), child.is_payload_present, - child.slot, child.root ) ) From b082140312aad8248ffb7dafe1bf9c5d0c66c221 Mon Sep 17 00:00:00 2001 From: Potuz Date: Wed, 24 Jul 2024 16:02:53 -0300 Subject: [PATCH 116/130] Revert "Remove block_timeliness from the store" The damned linter complains and there's no problem having it This reverts commit b7461ac174b56244a34e654f2083f35da3b4e29d. --- specs/_features/eip7732/fork-choice.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/specs/_features/eip7732/fork-choice.md b/specs/_features/eip7732/fork-choice.md index 19c9b88dd6..0eb49ddfc1 100644 --- a/specs/_features/eip7732/fork-choice.md +++ b/specs/_features/eip7732/fork-choice.md @@ -106,7 +106,7 @@ class Store(object): equivocating_indices: Set[ValidatorIndex] blocks: Dict[Root, BeaconBlock] = field(default_factory=dict) block_states: Dict[Root, BeaconState] = field(default_factory=dict) - # block_timeliness removed in EIP-7732 + block_timeliness: Dict[Root, boolean] = field(default_factory=dict) checkpoint_states: Dict[Checkpoint, BeaconState] = field(default_factory=dict) latest_messages: Dict[ValidatorIndex, LatestMessage] = field(default_factory=dict) unrealized_justifications: Dict[Root, Checkpoint] = field(default_factory=dict) @@ -431,6 +431,7 @@ def on_block(store: Store, signed_block: SignedBeaconBlock) -> None: time_into_slot = (store.time - store.genesis_time) % SECONDS_PER_SLOT is_before_attesting_interval = time_into_slot < SECONDS_PER_SLOT // INTERVALS_PER_SLOT is_timely = get_current_slot(store) == block.slot and is_before_attesting_interval + store.block_timeliness[hash_tree_root(block)] = is_timely # Add proposer score boost if the block is timely and not conflicting with an existing block is_first_block = store.proposer_boost_root == Root() From 2d59c55524bf852785632075e266d0bc92f025cb Mon Sep 17 00:00:00 2001 From: Justin Traglia Date: Thu, 25 Jul 2024 17:07:53 +0900 Subject: [PATCH 117/130] Fix indentation for kzg_7594 tests Somehow (probably because of a merge conflict) the last three verify_cell_kzg_proof_batch test cases were indented too far and as a result were part of the loop above, and as a result executed multiple times. This doesn't change the generated tests because it would try to do the same test with the same inputs. --- tests/generators/kzg_7594/main.py | 140 +++++++++++++++--------------- 1 file changed, 70 insertions(+), 70 deletions(-) diff --git a/tests/generators/kzg_7594/main.py b/tests/generators/kzg_7594/main.py index d96c677440..814a840537 100644 --- a/tests/generators/kzg_7594/main.py +++ b/tests/generators/kzg_7594/main.py @@ -264,79 +264,79 @@ def case_verify_cell_kzg_proof_batch(): 'output': None } - # Edge case: Missing a commitment - cells, proofs = VALID_CELLS_AND_PROOFS[0] - cells, proofs = cells[:2], proofs[:2] - # Do not include the second commitment - commitments = [VALID_COMMITMENTS[0]] - cell_indices = list(range(len(cells))) - expect_exception(spec.verify_cell_kzg_proof_batch, commitments, cell_indices, cells, proofs) - identifier = make_id(commitments, cell_indices, cells, proofs) - yield f'verify_cell_kzg_proof_batch_case_invalid_missing_commitment_{identifier}', { - 'input': { - 'commitments': encode_hex_list(commitments), - 'cell_indices': cell_indices, - 'cells': encode_hex_list(cells), - 'proofs': encode_hex_list(proofs), - }, - 'output': None - } + # Edge case: Missing a commitment + cells, proofs = VALID_CELLS_AND_PROOFS[0] + cells, proofs = cells[:2], proofs[:2] + # Do not include the second commitment + commitments = [VALID_COMMITMENTS[0]] + cell_indices = list(range(len(cells))) + expect_exception(spec.verify_cell_kzg_proof_batch, commitments, cell_indices, cells, proofs) + identifier = make_id(commitments, cell_indices, cells, proofs) + yield f'verify_cell_kzg_proof_batch_case_invalid_missing_commitment_{identifier}', { + 'input': { + 'commitments': encode_hex_list(commitments), + 'cell_indices': cell_indices, + 'cells': encode_hex_list(cells), + 'proofs': encode_hex_list(proofs), + }, + 'output': None + } - # Edge case: Missing a cell index - cells, proofs = VALID_CELLS_AND_PROOFS[2] - cells, proofs = cells[:2], proofs[:2] - commitments = [VALID_COMMITMENTS[2], VALID_COMMITMENTS[2]] - # Leave off one of the cell indices - cell_indices = list(range(len(cells) - 1)) - expect_exception(spec.verify_cell_kzg_proof_batch, commitments, cell_indices, cells, proofs) - identifier = make_id(commitments, cell_indices, cells, proofs) - yield f'verify_cell_kzg_proof_batch_case_invalid_missing_cell_index_{identifier}', { - 'input': { - 'commitments': encode_hex_list(commitments), - 'cell_indices': cell_indices, - 'cells': encode_hex_list(cells), - 'proofs': encode_hex_list(proofs), - }, - 'output': None - } + # Edge case: Missing a cell index + cells, proofs = VALID_CELLS_AND_PROOFS[2] + cells, proofs = cells[:2], proofs[:2] + commitments = [VALID_COMMITMENTS[2], VALID_COMMITMENTS[2]] + # Leave off one of the cell indices + cell_indices = list(range(len(cells) - 1)) + expect_exception(spec.verify_cell_kzg_proof_batch, commitments, cell_indices, cells, proofs) + identifier = make_id(commitments, cell_indices, cells, proofs) + yield f'verify_cell_kzg_proof_batch_case_invalid_missing_cell_index_{identifier}', { + 'input': { + 'commitments': encode_hex_list(commitments), + 'cell_indices': cell_indices, + 'cells': encode_hex_list(cells), + 'proofs': encode_hex_list(proofs), + }, + 'output': None + } - # Edge case: Missing a cell - cells, proofs = VALID_CELLS_AND_PROOFS[3] - cells, proofs = cells[:2], proofs[:2] - commitments = [VALID_COMMITMENTS[3], VALID_COMMITMENTS[3]] - cell_indices = list(range(len(cells))) - # Remove the last proof - cells = cells[:-1] - expect_exception(spec.verify_cell_kzg_proof_batch, commitments, cell_indices, cells, proofs) - identifier = make_id(commitments, cell_indices, cells, proofs) - yield f'verify_cell_kzg_proof_batch_case_invalid_missing_cell_{identifier}', { - 'input': { - 'commitments': encode_hex_list(commitments), - 'cell_indices': cell_indices, - 'cells': encode_hex_list(cells), - 'proofs': encode_hex_list(proofs), - }, - 'output': None - } + # Edge case: Missing a cell + cells, proofs = VALID_CELLS_AND_PROOFS[3] + cells, proofs = cells[:2], proofs[:2] + commitments = [VALID_COMMITMENTS[3], VALID_COMMITMENTS[3]] + cell_indices = list(range(len(cells))) + # Remove the last proof + cells = cells[:-1] + expect_exception(spec.verify_cell_kzg_proof_batch, commitments, cell_indices, cells, proofs) + identifier = make_id(commitments, cell_indices, cells, proofs) + yield f'verify_cell_kzg_proof_batch_case_invalid_missing_cell_{identifier}', { + 'input': { + 'commitments': encode_hex_list(commitments), + 'cell_indices': cell_indices, + 'cells': encode_hex_list(cells), + 'proofs': encode_hex_list(proofs), + }, + 'output': None + } - # Edge case: Missing a proof - cells, proofs = VALID_CELLS_AND_PROOFS[4] - cells, proofs = cells[:2], proofs[:2] - commitments = [VALID_COMMITMENTS[4], VALID_COMMITMENTS[4]] - cell_indices = list(range(len(cells))) - # Remove the last proof - proofs = proofs[:-1] - expect_exception(spec.verify_cell_kzg_proof_batch, commitments, cell_indices, cells, proofs) - identifier = make_id(commitments, cell_indices, cells, proofs) - yield f'verify_cell_kzg_proof_batch_case_invalid_missing_proof_{identifier}', { - 'input': { - 'commitments': encode_hex_list(commitments), - 'cell_indices': cell_indices, - 'cells': encode_hex_list(cells), - 'proofs': encode_hex_list(proofs), - }, - 'output': None - } + # Edge case: Missing a proof + cells, proofs = VALID_CELLS_AND_PROOFS[4] + cells, proofs = cells[:2], proofs[:2] + commitments = [VALID_COMMITMENTS[4], VALID_COMMITMENTS[4]] + cell_indices = list(range(len(cells))) + # Remove the last proof + proofs = proofs[:-1] + expect_exception(spec.verify_cell_kzg_proof_batch, commitments, cell_indices, cells, proofs) + identifier = make_id(commitments, cell_indices, cells, proofs) + yield f'verify_cell_kzg_proof_batch_case_invalid_missing_proof_{identifier}', { + 'input': { + 'commitments': encode_hex_list(commitments), + 'cell_indices': cell_indices, + 'cells': encode_hex_list(cells), + 'proofs': encode_hex_list(proofs), + }, + 'output': None + } ############################################################################### From 85a42cd7bf40800a32cb8b2633b68be02d951e94 Mon Sep 17 00:00:00 2001 From: b-wagn Date: Wed, 31 Jul 2024 11:56:39 +0200 Subject: [PATCH 118/130] improve comments in recovery --- .../polynomial-commitments-sampling.md | 24 +++++++++++++------ 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/specs/_features/eip7594/polynomial-commitments-sampling.md b/specs/_features/eip7594/polynomial-commitments-sampling.md index 7fec2a52d7..ec711f346b 100644 --- a/specs/_features/eip7594/polynomial-commitments-sampling.md +++ b/specs/_features/eip7594/polynomial-commitments-sampling.md @@ -690,6 +690,8 @@ def recover_polynomialcoeff(cell_indices: Sequence[CellIndex], # Flatten the cells into evaluations # If a cell is missing, then its evaluation is zero + # We let E(x) be a polynomial of degree FIELD_ELEMENTS_PER_EXT_BLOB - 1 + # that interpolates the evaluations including the zeros for missing ones extended_evaluation_rbo = [0] * FIELD_ELEMENTS_PER_EXT_BLOB for cell_index, cell in zip(cell_indices, cells): start = cell_index * FIELD_ELEMENTS_PER_CELL @@ -697,7 +699,7 @@ def recover_polynomialcoeff(cell_indices: Sequence[CellIndex], extended_evaluation_rbo[start:end] = cell extended_evaluation = bit_reversal_permutation(extended_evaluation_rbo) - # Compute Z(x) in coefficient form + # Compute the vanishing polynomial Z(x) in coefficient form # Z(x) is the polynomial which vanishes on all of the evaluations which are missing missing_cell_indices = [CellIndex(cell_index) for cell_index in range(CELLS_PER_EXT_BLOB) if cell_index not in cell_indices] @@ -707,22 +709,30 @@ def recover_polynomialcoeff(cell_indices: Sequence[CellIndex], zero_poly_eval = fft_field(zero_poly_coeff, roots_of_unity_extended) # Compute (E*Z)(x) = E(x) * Z(x) in evaluation form over the FFT domain + # Note: over the FFT domain, the polynomials (E*Z)(x) and (P*Z)(x) agree, where + # P(x) is the polynomial we want to reconstruct (degree FIELD_ELEMENTS_PER_BLOB - 1) extended_evaluation_times_zero = [BLSFieldElement(int(a) * int(b) % BLS_MODULUS) for a, b in zip(zero_poly_eval, extended_evaluation)] - # Convert (E*Z)(x) to coefficient form + # We know that (E*Z)(x) and (P*Z)(x) agree over the FFT domain + # and we know that (P*Z)(x) has degree at most FIELD_ELEMENTS_PER_EXT_BLOB - 1 + # Thus, an inverse FFT of the evaluations of (E*Z)(x) (= evaluations of (P*Z)(x)) + # yields the coefficient form of (P*Z)(x). extended_evaluation_times_zero_coeffs = fft_field(extended_evaluation_times_zero, roots_of_unity_extended, inv=True) - # Convert (E*Z)(x) to evaluation form over a coset of the FFT domain + # Next step is to divide the polynomial (P*Z)(x) by polynomial Z(x) to get P(x) + # Do this in evaluation form over a coset of the FFT domain to avoid division by 0 + + # Convert (P*Z)(x) to evaluation form over a coset of the FFT domain extended_evaluations_over_coset = coset_fft_field(extended_evaluation_times_zero_coeffs, roots_of_unity_extended) # Convert Z(x) to evaluation form over a coset of the FFT domain zero_poly_over_coset = coset_fft_field(zero_poly_coeff, roots_of_unity_extended) - # Compute Q_3(x) = (E*Z)(x) / Z(x) in evaluation form over a coset of the FFT domain + # Compute P(x) = (P*Z)(x) / Z(x) in evaluation form over a coset of the FFT domain reconstructed_poly_over_coset = [div(a, b) for a, b in zip(extended_evaluations_over_coset, zero_poly_over_coset)] - # Convert Q_3(x) to coefficient form + # Convert P(x) to coefficient form reconstructed_poly_coeff = coset_fft_field(reconstructed_poly_over_coset, roots_of_unity_extended, inv=True) return reconstructed_poly_coeff[:FIELD_ELEMENTS_PER_BLOB] @@ -762,9 +772,9 @@ def recover_cells_and_kzg_proofs(cell_indices: Sequence[CellIndex], # Convert cells to coset evaluations cosets_evals = [cell_to_coset_evals(cell) for cell in cells] - # Given the coset evaluations, recover the polynomial in coefficient form + # Given the coset evaluations, recover the polynomial in coefficient form polynomial_coeff = recover_polynomialcoeff(cell_indices, cosets_evals) - + # Recompute all cells/proofs return compute_cells_and_kzg_proofs_polynomialcoeff(polynomial_coeff) ``` From 225c4861838cf64e6386a6109b51aaae4d31e922 Mon Sep 17 00:00:00 2001 From: b-wagn Date: Wed, 31 Jul 2024 17:30:36 +0200 Subject: [PATCH 119/130] some punctuation --- .../polynomial-commitments-sampling.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/specs/_features/eip7594/polynomial-commitments-sampling.md b/specs/_features/eip7594/polynomial-commitments-sampling.md index ec711f346b..da26c1cfa3 100644 --- a/specs/_features/eip7594/polynomial-commitments-sampling.md +++ b/specs/_features/eip7594/polynomial-commitments-sampling.md @@ -685,13 +685,13 @@ def recover_polynomialcoeff(cell_indices: Sequence[CellIndex], """ Recover the polynomial in coefficient form that when evaluated at the roots of unity will give the extended blob. """ - # Get the extended domain. This will be referred to as the FFT domain + # Get the extended domain. This will be referred to as the FFT domain. roots_of_unity_extended = compute_roots_of_unity(FIELD_ELEMENTS_PER_EXT_BLOB) # Flatten the cells into evaluations - # If a cell is missing, then its evaluation is zero + # If a cell is missing, then its evaluation is zero. # We let E(x) be a polynomial of degree FIELD_ELEMENTS_PER_EXT_BLOB - 1 - # that interpolates the evaluations including the zeros for missing ones + # that interpolates the evaluations including the zeros for missing ones. extended_evaluation_rbo = [0] * FIELD_ELEMENTS_PER_EXT_BLOB for cell_index, cell in zip(cell_indices, cells): start = cell_index * FIELD_ELEMENTS_PER_CELL @@ -699,8 +699,8 @@ def recover_polynomialcoeff(cell_indices: Sequence[CellIndex], extended_evaluation_rbo[start:end] = cell extended_evaluation = bit_reversal_permutation(extended_evaluation_rbo) - # Compute the vanishing polynomial Z(x) in coefficient form - # Z(x) is the polynomial which vanishes on all of the evaluations which are missing + # Compute the vanishing polynomial Z(x) in coefficient form. + # Z(x) is the polynomial which vanishes on all of the evaluations which are missing. missing_cell_indices = [CellIndex(cell_index) for cell_index in range(CELLS_PER_EXT_BLOB) if cell_index not in cell_indices] zero_poly_coeff = construct_vanishing_polynomial(missing_cell_indices) @@ -710,18 +710,18 @@ def recover_polynomialcoeff(cell_indices: Sequence[CellIndex], # Compute (E*Z)(x) = E(x) * Z(x) in evaluation form over the FFT domain # Note: over the FFT domain, the polynomials (E*Z)(x) and (P*Z)(x) agree, where - # P(x) is the polynomial we want to reconstruct (degree FIELD_ELEMENTS_PER_BLOB - 1) + # P(x) is the polynomial we want to reconstruct (degree FIELD_ELEMENTS_PER_BLOB - 1). extended_evaluation_times_zero = [BLSFieldElement(int(a) * int(b) % BLS_MODULUS) for a, b in zip(zero_poly_eval, extended_evaluation)] - # We know that (E*Z)(x) and (P*Z)(x) agree over the FFT domain - # and we know that (P*Z)(x) has degree at most FIELD_ELEMENTS_PER_EXT_BLOB - 1 + # We know that (E*Z)(x) and (P*Z)(x) agree over the FFT domain, + # and we know that (P*Z)(x) has degree at most FIELD_ELEMENTS_PER_EXT_BLOB - 1. # Thus, an inverse FFT of the evaluations of (E*Z)(x) (= evaluations of (P*Z)(x)) # yields the coefficient form of (P*Z)(x). extended_evaluation_times_zero_coeffs = fft_field(extended_evaluation_times_zero, roots_of_unity_extended, inv=True) - # Next step is to divide the polynomial (P*Z)(x) by polynomial Z(x) to get P(x) - # Do this in evaluation form over a coset of the FFT domain to avoid division by 0 + # Next step is to divide the polynomial (P*Z)(x) by polynomial Z(x) to get P(x). + # We do this in evaluation form over a coset of the FFT domain to avoid division by 0. # Convert (P*Z)(x) to evaluation form over a coset of the FFT domain extended_evaluations_over_coset = coset_fft_field(extended_evaluation_times_zero_coeffs, roots_of_unity_extended) From 31c23f8dbfbcf01e43f01e56692a5a23c617c5ce Mon Sep 17 00:00:00 2001 From: Agnish Ghosh <80243668+agnxsh@users.noreply.github.com> Date: Wed, 31 Jul 2024 22:39:16 +0530 Subject: [PATCH 120/130] Remove compute_cells test format --- tests/formats/kzg_7594/compute_cells.md | 22 ---------------------- 1 file changed, 22 deletions(-) delete mode 100644 tests/formats/kzg_7594/compute_cells.md diff --git a/tests/formats/kzg_7594/compute_cells.md b/tests/formats/kzg_7594/compute_cells.md deleted file mode 100644 index aec118c20b..0000000000 --- a/tests/formats/kzg_7594/compute_cells.md +++ /dev/null @@ -1,22 +0,0 @@ -# Test format: Compute cells - -Compute the cells for a given `blob`. - -## Test case format - -The test data is declared in a `data.yaml` file: - -```yaml -input: - blob: Blob -- the data blob -output: List[Cell] -- the cells -``` - -- `Blob` is a 131072-byte hexadecimal string, prefixed with `0x`. -- `Cell` is a 2048-byte hexadecimal string, prefixed with `0x`. - -All byte(s) fields are encoded as strings, hexadecimal encoding, prefixed with `0x`. - -## Condition - -The `compute_cells` handler should compute the cells (chunks of an extended blob) for `blob`, and the result should match the expected `output`. If the blob is invalid (e.g. incorrect length or one of the 32-byte blocks does not represent a BLS field element) it should error, i.e. the output should be `null`. From ef3c9dc84dd5fb8b5940a2d953c70326452ada09 Mon Sep 17 00:00:00 2001 From: Benedikt Wagner <113296072+b-wagn@users.noreply.github.com> Date: Fri, 2 Aug 2024 08:35:02 +0200 Subject: [PATCH 121/130] Remove dead link to no longer existing public method. --- specs/_features/eip7594/polynomial-commitments-sampling.md | 1 - 1 file changed, 1 deletion(-) diff --git a/specs/_features/eip7594/polynomial-commitments-sampling.md b/specs/_features/eip7594/polynomial-commitments-sampling.md index da26c1cfa3..869f471844 100644 --- a/specs/_features/eip7594/polynomial-commitments-sampling.md +++ b/specs/_features/eip7594/polynomial-commitments-sampling.md @@ -64,7 +64,6 @@ Public functions MUST accept raw bytes as input and perform the required cryptog The following is a list of the public methods: - [`compute_cells_and_kzg_proofs`](#compute_cells_and_kzg_proofs) -- [`verify_cell_kzg_proof`](#verify_cell_kzg_proof) - [`verify_cell_kzg_proof_batch`](#verify_cell_kzg_proof_batch) - [`recover_cells_and_kzg_proofs`](#recover_cells_and_kzg_proofs) From 7ab2bbe89b26799fcc1f0ecacc8ea041119852f4 Mon Sep 17 00:00:00 2001 From: vbuterin Date: Fri, 2 Aug 2024 22:35:25 +0900 Subject: [PATCH 122/130] Make process_withdrawals slightly cleaner (equivalent functionality) (#3761) --- specs/capella/beacon-chain.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/specs/capella/beacon-chain.md b/specs/capella/beacon-chain.md index b6d0f28f81..103530bf8b 100644 --- a/specs/capella/beacon-chain.md +++ b/specs/capella/beacon-chain.md @@ -379,10 +379,9 @@ def get_expected_withdrawals(state: BeaconState) -> Sequence[Withdrawal]: ```python def process_withdrawals(state: BeaconState, payload: ExecutionPayload) -> None: expected_withdrawals = get_expected_withdrawals(state) - assert len(payload.withdrawals) == len(expected_withdrawals) + assert payload.withdrawals == expected_withdrawals - for expected_withdrawal, withdrawal in zip(expected_withdrawals, payload.withdrawals): - assert withdrawal == expected_withdrawal + for withdrawal in expected_withdrawals: decrease_balance(state, withdrawal.validator_index, withdrawal.amount) # Update the next withdrawal index if this block contained withdrawals From 74124f521b4c0d1bd68ac92595177ec8fcc0e931 Mon Sep 17 00:00:00 2001 From: Potuz Date: Fri, 2 Aug 2024 12:50:25 -0300 Subject: [PATCH 123/130] Typo in validator.md --- specs/_features/eip7732/validator.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/_features/eip7732/validator.md b/specs/_features/eip7732/validator.md index 4a520ba985..6a6cbeecef 100644 --- a/specs/_features/eip7732/validator.md +++ b/specs/_features/eip7732/validator.md @@ -12,7 +12,7 @@ This document represents the changes and additions to the Honest validator guide - [Attestation](#attestation) - [Sync Committee participations](#sync-committee-participations) - [Block proposal](#block-proposal) - - [Constructing the new `signed_execution_payload_header_envelope` field in `BeaconBlockBody`](#constructing-the-new-signed_execution_payload_header_envelope-field-in--beaconblockbody) + - [Constructing the new `signed_execution_payload_header` field in `BeaconBlockBody`](#constructing-the-new-signed_execution_payload_header-field-in--beaconblockbody) - [Constructing the new `payload_attestations` field in `BeaconBlockBody`](#constructing-the-new-payload_attestations-field-in--beaconblockbody) - [Blob sidecars](#blob-sidecars) - [Payload timeliness attestation](#payload-timeliness-attestation) @@ -71,7 +71,7 @@ Sync committee duties are not changed for validators, however the submission dea Validators are still expected to propose `SignedBeaconBlock` at the beginning of any slot during which `is_proposer(state, validator_index)` returns `true`. The mechanism to prepare this beacon block and related sidecars differs from previous forks as follows -#### Constructing the new `signed_execution_payload_header_envelope` field in `BeaconBlockBody` +#### Constructing the new `signed_execution_payload_header` field in `BeaconBlockBody` To obtain `signed_execution_payload_header`, a block proposer building a block on top of a `state` must take the following actions: * Listen to the `execution_payload_header` gossip global topic and save an accepted `signed_execution_payload_header` from a builder. Proposer MAY obtain these signed messages by other off-protocol means. From b5cea48d4540f22059ee10c287c302186666037d Mon Sep 17 00:00:00 2001 From: Hsiao-Wei Wang Date: Sat, 3 Aug 2024 19:14:40 +0900 Subject: [PATCH 124/130] bump to 1.5.0-alpha.4 --- tests/core/pyspec/eth2spec/VERSION.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/core/pyspec/eth2spec/VERSION.txt b/tests/core/pyspec/eth2spec/VERSION.txt index 86f9d092d6..2ac49a507e 100644 --- a/tests/core/pyspec/eth2spec/VERSION.txt +++ b/tests/core/pyspec/eth2spec/VERSION.txt @@ -1 +1 @@ -1.5.0-alpha.3 +1.5.0-alpha.4 From fcca2b5c7153cf7833c8a195e2d330b9d945f9cb Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Tue, 6 Aug 2024 17:17:26 +0600 Subject: [PATCH 125/130] Fix off-by-one in process_pending_consolidations --- specs/electra/beacon-chain.md | 3 +- .../test_process_pending_consolidations.py | 106 +++++++++++++++++- .../eth2spec/test/helpers/epoch_processing.py | 8 ++ .../pyspec/eth2spec/test/helpers/state.py | 8 ++ 4 files changed, 123 insertions(+), 2 deletions(-) diff --git a/specs/electra/beacon-chain.md b/specs/electra/beacon-chain.md index d9185b2030..f7019f8d30 100644 --- a/specs/electra/beacon-chain.md +++ b/specs/electra/beacon-chain.md @@ -894,13 +894,14 @@ def process_pending_balance_deposits(state: BeaconState) -> None: ```python def process_pending_consolidations(state: BeaconState) -> None: + next_epoch = Epoch(get_current_epoch(state) + 1) next_pending_consolidation = 0 for pending_consolidation in state.pending_consolidations: source_validator = state.validators[pending_consolidation.source_index] if source_validator.slashed: next_pending_consolidation += 1 continue - if source_validator.withdrawable_epoch > get_current_epoch(state): + if source_validator.withdrawable_epoch > next_epoch: break # Churn any target excess active balance of target and raise its max diff --git a/tests/core/pyspec/eth2spec/test/electra/epoch_processing/test_process_pending_consolidations.py b/tests/core/pyspec/eth2spec/test/electra/epoch_processing/test_process_pending_consolidations.py index 6c21a722ff..9cb3886dac 100644 --- a/tests/core/pyspec/eth2spec/test/electra/epoch_processing/test_process_pending_consolidations.py +++ b/tests/core/pyspec/eth2spec/test/electra/epoch_processing/test_process_pending_consolidations.py @@ -1,8 +1,14 @@ -from eth2spec.test.helpers.epoch_processing import run_epoch_processing_with +from eth2spec.test.helpers.epoch_processing import ( + run_epoch_processing_with, + compute_state_by_epoch_processing_to, +) from eth2spec.test.context import ( spec_state_test, with_electra_and_later, ) +from eth2spec.test.helpers.state import ( + next_epoch_with_full_participation, +) # *********************** # * CONSOLIDATION TESTS * @@ -185,3 +191,101 @@ def test_all_consolidation_cases_together(spec, state): assert state.balances[target_index[i]] == pre_balances[target_index[i]] # First consolidation is processed, second is skipped, last two are left in the queue state.pending_consolidations = pre_pending_consolidations[2:] + + +@with_electra_and_later +@spec_state_test +def test_pending_consolidation_future_epoch(spec, state): + current_epoch = spec.get_current_epoch(state) + source_index = spec.get_active_validator_indices(state, current_epoch)[0] + target_index = spec.get_active_validator_indices(state, current_epoch)[1] + # initiate source exit + spec.initiate_validator_exit(state, source_index) + # set withdrawable_epoch to exit_epoch + 1 + state.validators[source_index].withdrawable_epoch = state.validators[source_index].exit_epoch + spec.Epoch(1) + # append pending consolidation + state.pending_consolidations.append( + spec.PendingConsolidation(source_index=source_index, target_index=target_index) + ) + # Set the target withdrawal credential to eth1 + eth1_withdrawal_credential = ( + spec.ETH1_ADDRESS_WITHDRAWAL_PREFIX + b"\x00" * 11 + b"\x11" * 20 + ) + state.validators[target_index].withdrawal_credentials = eth1_withdrawal_credential + + # Advance to withdrawable_epoch - 1 with full participation + target_epoch = state.validators[source_index].withdrawable_epoch - spec.Epoch(1) + while spec.get_current_epoch(state) < target_epoch: + next_epoch_with_full_participation(spec, state) + + # Obtain state before the call to process_pending_consolidations + state_before_consolidation = compute_state_by_epoch_processing_to(spec, state, "process_pending_consolidations") + + yield from run_epoch_processing_with(spec, state, "process_pending_consolidations") + + # Pending consolidation was successfully processed + expected_source_balance = state_before_consolidation.balances[source_index] - spec.MIN_ACTIVATION_BALANCE + assert ( + state.validators[target_index].withdrawal_credentials[:1] + == spec.COMPOUNDING_WITHDRAWAL_PREFIX + ) + assert state.balances[target_index] == 2 * spec.MIN_ACTIVATION_BALANCE + assert state.balances[source_index] == expected_source_balance + assert state.pending_consolidations == [] + + # Pending balance deposit to the target is created + expected_pending_balance = state_before_consolidation.balances[target_index] - spec.MIN_ACTIVATION_BALANCE + assert len(state.pending_balance_deposits) > 0 + pending_balance_deposit = state.pending_balance_deposits[len(state.pending_balance_deposits) - 1] + assert pending_balance_deposit.index == target_index + assert pending_balance_deposit.amount == expected_pending_balance + + +@with_electra_and_later +@spec_state_test +def test_pending_consolidation_compounding_creds(spec, state): + current_epoch = spec.get_current_epoch(state) + source_index = spec.get_active_validator_indices(state, current_epoch)[0] + target_index = spec.get_active_validator_indices(state, current_epoch)[1] + # initiate source exit + spec.initiate_validator_exit(state, source_index) + # set withdrawable_epoch to exit_epoch + 1 + state.validators[source_index].withdrawable_epoch = state.validators[source_index].exit_epoch + spec.Epoch(1) + # append pending consolidation + state.pending_consolidations.append( + spec.PendingConsolidation(source_index=source_index, target_index=target_index) + ) + # Set the source and the target withdrawal credential to compounding + compounding_withdrawal_credential_1 = ( + spec.COMPOUNDING_WITHDRAWAL_PREFIX + b"\x00" * 11 + b"\x11" * 20 + ) + compounding_withdrawal_credential_2 = ( + spec.COMPOUNDING_WITHDRAWAL_PREFIX + b"\x00" * 11 + b"\x12" * 20 + ) + state.validators[source_index].withdrawal_credentials = compounding_withdrawal_credential_1 + state.validators[target_index].withdrawal_credentials = compounding_withdrawal_credential_2 + + # Advance to withdrawable_epoch - 1 with full participation + target_epoch = state.validators[source_index].withdrawable_epoch - spec.Epoch(1) + while spec.get_current_epoch(state) < target_epoch: + next_epoch_with_full_participation(spec, state) + + # Obtain state before the call to process_pending_consolidations + state_before_consolidation = compute_state_by_epoch_processing_to(spec, state, "process_pending_consolidations") + + yield from run_epoch_processing_with(spec, state, "process_pending_consolidations") + + # Pending consolidation was successfully processed + expected_target_balance = ( + state_before_consolidation.balances[source_index] + state_before_consolidation.balances[target_index] + ) + assert ( + state.validators[target_index].withdrawal_credentials[:1] + == spec.COMPOUNDING_WITHDRAWAL_PREFIX + ) + assert state.balances[target_index] == expected_target_balance + assert state.balances[source_index] == 0 + assert state.pending_consolidations == [] + + # Pending balance deposit to the target is not created + assert len(state.pending_balance_deposits) == 0 diff --git a/tests/core/pyspec/eth2spec/test/helpers/epoch_processing.py b/tests/core/pyspec/eth2spec/test/helpers/epoch_processing.py index 44b42aff91..80302e111d 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/epoch_processing.py +++ b/tests/core/pyspec/eth2spec/test/helpers/epoch_processing.py @@ -22,6 +22,8 @@ def get_process_calls(spec): 'charge_confirmed_header_fees', # sharding 'reset_pending_headers', # sharding 'process_eth1_data_reset', + 'process_pending_balance_deposits', # electra + 'process_pending_consolidations', # electra 'process_effective_balance_updates', 'process_slashings_reset', 'process_randao_mixes_reset', @@ -72,3 +74,9 @@ def run_epoch_processing_with(spec, state, process_name: str): yield 'pre', state getattr(spec, process_name)(state) yield 'post', state + + +def compute_state_by_epoch_processing_to(spec, state, process_name: str): + state_copy = state.copy() + run_epoch_processing_to(spec, state_copy, process_name) + return state_copy diff --git a/tests/core/pyspec/eth2spec/test/helpers/state.py b/tests/core/pyspec/eth2spec/test/helpers/state.py index 1e64bd4db2..07e7bfb478 100644 --- a/tests/core/pyspec/eth2spec/test/helpers/state.py +++ b/tests/core/pyspec/eth2spec/test/helpers/state.py @@ -60,6 +60,14 @@ def next_epoch(spec, state): spec.process_slots(state, slot) +def next_epoch_with_full_participation(spec, state): + """ + Transition to the start slot of the next epoch with full participation + """ + set_full_participation(spec, state) + next_epoch(spec, state) + + def next_epoch_via_block(spec, state, insert_state_root=False): """ Transition to the start slot of the next epoch via a full block transition From b6656983506dafc6715470d91e8d9beb3309a64f Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Tue, 6 Aug 2024 17:29:13 +0600 Subject: [PATCH 126/130] Fix off-by-one in process_pending_balance_deposits --- specs/electra/beacon-chain.md | 3 +- .../test_process_pending_consolidations.py | 60 +++++++++++++++++-- 2 files changed, 58 insertions(+), 5 deletions(-) diff --git a/specs/electra/beacon-chain.md b/specs/electra/beacon-chain.md index f7019f8d30..cb5ee6a7a9 100644 --- a/specs/electra/beacon-chain.md +++ b/specs/electra/beacon-chain.md @@ -854,6 +854,7 @@ def process_registry_updates(state: BeaconState) -> None: ```python def process_pending_balance_deposits(state: BeaconState) -> None: + next_epoch = Epoch(get_current_epoch(state) + 1) available_for_processing = state.deposit_balance_to_consume + get_activation_exit_churn_limit(state) processed_amount = 0 next_deposit_index = 0 @@ -863,7 +864,7 @@ def process_pending_balance_deposits(state: BeaconState) -> None: validator = state.validators[deposit.index] # Validator is exiting, postpone the deposit until after withdrawable epoch if validator.exit_epoch < FAR_FUTURE_EPOCH: - if get_current_epoch(state) <= validator.withdrawable_epoch: + if next_epoch <= validator.withdrawable_epoch: deposits_to_postpone.append(deposit) # Deposited balance will never become active. Increase balance but do not consume churn else: diff --git a/tests/core/pyspec/eth2spec/test/electra/epoch_processing/test_process_pending_consolidations.py b/tests/core/pyspec/eth2spec/test/electra/epoch_processing/test_process_pending_consolidations.py index 9cb3886dac..2682a535b5 100644 --- a/tests/core/pyspec/eth2spec/test/electra/epoch_processing/test_process_pending_consolidations.py +++ b/tests/core/pyspec/eth2spec/test/electra/epoch_processing/test_process_pending_consolidations.py @@ -256,14 +256,12 @@ def test_pending_consolidation_compounding_creds(spec, state): spec.PendingConsolidation(source_index=source_index, target_index=target_index) ) # Set the source and the target withdrawal credential to compounding - compounding_withdrawal_credential_1 = ( + state.validators[source_index].withdrawal_credentials = ( spec.COMPOUNDING_WITHDRAWAL_PREFIX + b"\x00" * 11 + b"\x11" * 20 ) - compounding_withdrawal_credential_2 = ( + state.validators[target_index].withdrawal_credentials = ( spec.COMPOUNDING_WITHDRAWAL_PREFIX + b"\x00" * 11 + b"\x12" * 20 ) - state.validators[source_index].withdrawal_credentials = compounding_withdrawal_credential_1 - state.validators[target_index].withdrawal_credentials = compounding_withdrawal_credential_2 # Advance to withdrawable_epoch - 1 with full participation target_epoch = state.validators[source_index].withdrawable_epoch - spec.Epoch(1) @@ -289,3 +287,57 @@ def test_pending_consolidation_compounding_creds(spec, state): # Pending balance deposit to the target is not created assert len(state.pending_balance_deposits) == 0 + + +@with_electra_and_later +@spec_state_test +def test_pending_consolidation_with_pending_deposit(spec, state): + current_epoch = spec.get_current_epoch(state) + source_index = spec.get_active_validator_indices(state, current_epoch)[0] + target_index = spec.get_active_validator_indices(state, current_epoch)[1] + # initiate source exit + spec.initiate_validator_exit(state, source_index) + # set withdrawable_epoch to exit_epoch + 1 + state.validators[source_index].withdrawable_epoch = state.validators[source_index].exit_epoch + spec.Epoch(1) + # append pending consolidation + state.pending_consolidations.append( + spec.PendingConsolidation(source_index=source_index, target_index=target_index) + ) + # append pending deposit + state.pending_balance_deposits.append( + spec.PendingBalanceDeposit(index=source_index, amount=spec.MIN_ACTIVATION_BALANCE) + ) + # Set the source and the target withdrawal credential to compounding + state.validators[source_index].withdrawal_credentials = ( + spec.COMPOUNDING_WITHDRAWAL_PREFIX + b"\x00" * 11 + b"\x11" * 20 + ) + state.validators[target_index].withdrawal_credentials = ( + spec.COMPOUNDING_WITHDRAWAL_PREFIX + b"\x00" * 11 + b"\x12" * 20 + ) + + # Advance to withdrawable_epoch - 1 with full participation + target_epoch = state.validators[source_index].withdrawable_epoch - spec.Epoch(1) + while spec.get_current_epoch(state) < target_epoch: + next_epoch_with_full_participation(spec, state) + + # Obtain state before the call to process_pending_balance_deposits + state_before_consolidation = compute_state_by_epoch_processing_to(spec, state, "process_pending_balance_deposits") + + yield from run_epoch_processing_with(spec, state, "process_pending_consolidations") + + # Pending consolidation was successfully processed + expected_target_balance = ( + state_before_consolidation.balances[source_index] + state_before_consolidation.balances[target_index] + ) + assert ( + state.validators[target_index].withdrawal_credentials[:1] + == spec.COMPOUNDING_WITHDRAWAL_PREFIX + ) + assert state.balances[target_index] == expected_target_balance + assert state.balances[source_index] == 0 + assert state.pending_consolidations == [] + + # Pending balance deposit to the source was not processed + assert len(state.pending_balance_deposits) == 1 + assert state.pending_balance_deposits[0] == spec.PendingBalanceDeposit( + index=source_index, amount=spec.MIN_ACTIVATION_BALANCE) From 7c1092216c8914f1f718914daae1dbf911690b05 Mon Sep 17 00:00:00 2001 From: terence tsao Date: Sat, 3 Aug 2024 17:33:17 -0400 Subject: [PATCH 127/130] p2p: clean up payload_attestation_message conditions --- specs/_features/eip7732/p2p-interface.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/specs/_features/eip7732/p2p-interface.md b/specs/_features/eip7732/p2p-interface.md index 52546070f3..86ab89568c 100644 --- a/specs/_features/eip7732/p2p-interface.md +++ b/specs/_features/eip7732/p2p-interface.md @@ -154,13 +154,13 @@ This topic is used to propagate signed payload attestation message. The following validations MUST pass before forwarding the `payload_attestation_message` on the network, assuming the alias `data = payload_attestation_message.data`: -- _[IGNORE]_ `data.slot` is the current slot. -- _[REJECT]_ `data.payload_status < PAYLOAD_INVALID_STATUS` -- _[IGNORE]_ the `payload_attestation_message` is the first valid payload attestation message received from the validator index. -- _[IGNORE]_ The attestation's `data.beacon_block_root` has been seen (via both gossip and non-gossip sources) (a client MAY queue attestation for processing once the block is retrieved. Note a client might want to request payload after). -_ _[REJECT]_ The beacon block with root `data.beacon_block_root` passes validation. -- _[REJECT]_ The validator index is within the payload committee in `get_ptc(state, data.slot)`. For the current's slot head state. -- _[REJECT]_ The signature of `payload_attestation_message.signature` is valid with respect to the validator index. +- _[IGNORE]_ The message's slot is for the current slot (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance), i.e. `data.slot == current_slot`. +- _[REJECT]_ The message's payload status is a valid status, i.e. `data.payload_status < PAYLOAD_INVALID_STATUS`. +- _[IGNORE]_ The `payload_attestation_message` is the first valid message received from the validator with index `payload_attestation_message.validate_index`. +- _[IGNORE]_ The message's block `data.beacon_block_root` has been seen (via both gossip and non-gossip sources) (a client MAY queue attestation for processing once the block is retrieved. Note a client might want to request payload after). +- _[REJECT]_ The message's block `data.beacon_block_root` passes validation. +- _[REJECT]_ The message's validator index is within the payload committee in `get_ptc(state, data.slot)`. The `state` is the head state corresponding to processing the block up to the current slot as determined by the fork choice. +- _[REJECT]_ The message's signature of `payload_attestation_message.signature` is valid with respect to the validator index. ###### `execution_payload_header` From 5e06787db8cce44d615f79eccba04809bcb1d715 Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Wed, 7 Aug 2024 15:54:50 +0600 Subject: [PATCH 128/130] Apply suggestions by @fradamt Co-authored-by: fradamt <104826920+fradamt@users.noreply.github.com> --- .../test_process_pending_consolidations.py | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/electra/epoch_processing/test_process_pending_consolidations.py b/tests/core/pyspec/eth2spec/test/electra/epoch_processing/test_process_pending_consolidations.py index 2682a535b5..aa0d8a3b97 100644 --- a/tests/core/pyspec/eth2spec/test/electra/epoch_processing/test_process_pending_consolidations.py +++ b/tests/core/pyspec/eth2spec/test/electra/epoch_processing/test_process_pending_consolidations.py @@ -233,7 +233,8 @@ def test_pending_consolidation_future_epoch(spec, state): assert state.balances[source_index] == expected_source_balance assert state.pending_consolidations == [] - # Pending balance deposit to the target is created + # Pending balance deposit to the target is created as part of `switch_to_compounding_validator`. + # The excess balance to queue are the rewards accumulated over the previous epoch transitions. expected_pending_balance = state_before_consolidation.balances[target_index] - spec.MIN_ACTIVATION_BALANCE assert len(state.pending_balance_deposits) > 0 pending_balance_deposit = state.pending_balance_deposits[len(state.pending_balance_deposits) - 1] @@ -282,10 +283,13 @@ def test_pending_consolidation_compounding_creds(spec, state): == spec.COMPOUNDING_WITHDRAWAL_PREFIX ) assert state.balances[target_index] == expected_target_balance + # All source balance is active and moved to the target, + # because the source validator has compounding credentials assert state.balances[source_index] == 0 assert state.pending_consolidations == [] - # Pending balance deposit to the target is not created + # Pending balance deposit to the target is not created, + # because the target already has compounding credentials assert len(state.pending_balance_deposits) == 0 @@ -337,7 +341,8 @@ def test_pending_consolidation_with_pending_deposit(spec, state): assert state.balances[source_index] == 0 assert state.pending_consolidations == [] - # Pending balance deposit to the source was not processed + # Pending balance deposit to the source was not processed. + # It should only be processed in the next epoch transition assert len(state.pending_balance_deposits) == 1 assert state.pending_balance_deposits[0] == spec.PendingBalanceDeposit( index=source_index, amount=spec.MIN_ACTIVATION_BALANCE) From 024ee04373a27d989d59cb49f7cd1b67636365f7 Mon Sep 17 00:00:00 2001 From: Mikhail Kalinin Date: Wed, 7 Aug 2024 16:03:39 +0600 Subject: [PATCH 129/130] Fix lint --- .../epoch_processing/test_process_pending_consolidations.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/core/pyspec/eth2spec/test/electra/epoch_processing/test_process_pending_consolidations.py b/tests/core/pyspec/eth2spec/test/electra/epoch_processing/test_process_pending_consolidations.py index aa0d8a3b97..d750149839 100644 --- a/tests/core/pyspec/eth2spec/test/electra/epoch_processing/test_process_pending_consolidations.py +++ b/tests/core/pyspec/eth2spec/test/electra/epoch_processing/test_process_pending_consolidations.py @@ -283,12 +283,12 @@ def test_pending_consolidation_compounding_creds(spec, state): == spec.COMPOUNDING_WITHDRAWAL_PREFIX ) assert state.balances[target_index] == expected_target_balance - # All source balance is active and moved to the target, + # All source balance is active and moved to the target, # because the source validator has compounding credentials assert state.balances[source_index] == 0 assert state.pending_consolidations == [] - # Pending balance deposit to the target is not created, + # Pending balance deposit to the target is not created, # because the target already has compounding credentials assert len(state.pending_balance_deposits) == 0 From 54093964c95fbd2e48be5de672e3baae8531a964 Mon Sep 17 00:00:00 2001 From: Justin Traglia Date: Wed, 7 Aug 2024 21:22:00 -0500 Subject: [PATCH 130/130] Fix nits in eip7594 p2p-interface spec --- specs/_features/eip7594/p2p-interface.md | 31 ++++++++++++------------ 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/specs/_features/eip7594/p2p-interface.md b/specs/_features/eip7594/p2p-interface.md index 203f524ae8..1b25c5fc5a 100644 --- a/specs/_features/eip7594/p2p-interface.md +++ b/specs/_features/eip7594/p2p-interface.md @@ -39,19 +39,18 @@ ### Preset -| Name | Value | Description | -| - | - | - | +| Name | Value | Description | +|-----------------------------------------|-------------------------------------------------------------------------------------------|-------------------------------------------------------------------| | `KZG_COMMITMENTS_INCLUSION_PROOF_DEPTH` | `uint64(floorlog2(get_generalized_index(BeaconBlockBody, 'blob_kzg_commitments')))` (= 4) | Merkle proof index for `blob_kzg_commitments` | - ### Configuration -*[New in Deneb:EIP4844]* +*[New in EIP7594]* -| Name | Value | Description | -|------------------------------------------|-----------------------------------|---------------------------------------------------------------------| -| `MAX_REQUEST_DATA_COLUMN_SIDECARS` | `MAX_REQUEST_BLOCKS_DENEB * NUMBER_OF_COLUMNS` | Maximum number of data column sidecars in a single request | -| `MIN_EPOCHS_FOR_DATA_COLUMN_SIDECARS_REQUESTS` | `2**12` (= 4096 epochs, ~18 days) | The minimum epoch range over which a node must serve data column sidecars | +| Name | Value | Description | +|------------------------------------------------|------------------------------------------------|---------------------------------------------------------------------------| +| `MAX_REQUEST_DATA_COLUMN_SIDECARS` | `MAX_REQUEST_BLOCKS_DENEB * NUMBER_OF_COLUMNS` | Maximum number of data column sidecars in a single request | +| `MIN_EPOCHS_FOR_DATA_COLUMN_SIDECARS_REQUESTS` | `2**12` (= 4096 epochs, ~18 days) | The minimum epoch range over which a node must serve data column sidecars | ### Containers @@ -180,8 +179,8 @@ The `` field is calculated as `context = compute_fork_digest(fork [1]: # (eth2spec: skip) -| `fork_version` | Chunk SSZ type | -| - | - | +| `fork_version` | Chunk SSZ type | +|------------------------|-----------------------------| | `EIP7594_FORK_VERSION` | `eip7594.DataColumnSidecar` | Request Content: @@ -228,9 +227,9 @@ The `` field is calculated as `context = compute_fork_digest(fork [1]: # (eth2spec: skip) -| `fork_version` | Chunk SSZ type | -|--------------------------|-------------------------------| -| `EIP7594_FORK_VERSION` | `eip7594.DataColumnSidecar` | +| `fork_version` | Chunk SSZ type | +|------------------------|-----------------------------| +| `EIP7594_FORK_VERSION` | `eip7594.DataColumnSidecar` | Request Content: ``` @@ -323,6 +322,6 @@ Requests the MetaData of a peer, using the new `MetaData` definition given above A new field is added to the ENR under the key `csc` to facilitate custody data column discovery. -| Key | Value | -|:------|:-----------------------------------------| -| `csc` | Custody subnet count, big endian integer | +| Key | Value | +|--------|------------------------------------------| +| `csc` | Custody subnet count, big endian integer |