[WIP] New shard proposal #1427

vbuterin · 2019-10-12T03:06:04Z

Still very incomplete and being worked on.

Known TODOs:

Flesh out fraud proofs into a complete specification

…alik71

djrtwo

Great start! We'll need a little bit of back and forth on cleaning up and figuring out how to best integrate this as an executable extension to Phase 0.

Take a look at the more question focused items at least.
From there I can do a round of cleanup

specs/core/1_new_shards.md

hwwhww

👍👍
First look before going to bed. 👀

specs/core/1_new_shards.md

Co-Authored-By: Hsiao-Wei Wang <hwwang156@gmail.com>

specs/core/1_new_shards.md

Co-Authored-By: Hsiao-Wei Wang <hwwang156@gmail.com>

specs/core/1_new_shards.md

hwwhww · 2019-11-06T10:19:23Z

specs/core/1_new_shards.md

+```python
+class ShardState(Container):
+    slot: Slot
+    gasprice: Gwei


Need to set initial gasprice value.

I think no need. It's zero by default, but then after one slot get_updated_gasprice pushes it up to MIN_GASPRICE.

specs/core/1_new_shards.md

hwwhww · 2019-11-06T12:17:57Z

specs/core/1_new_shards.md

+    # Type 2: delayed attestations
+    else:
+        assert state.slot - slot_to_epoch(data.slot) < EPOCH_LENGTH
+        assert data.shard_transition_root == Hash()


Is it a kind of reduced form of attestation data?

If so, does that mean the late attesters have to rebuild AttestationData of the old slot with empty custody_bits and shard_transition_root, and broadcast it?

It seems the proposer can help transform ReducedAttestation in the previous version?

Yes, it's a reduced form, and late attesters do have to rebroadcast.

djrtwo

I think the approach here is reasonable. I'm on board from a high level perspective. We'll need many cleanups and testing to get where we're ultimately going.

I can take a pass on getting the linter happy and a couple of sanity tests running after you address @hwwhww's comments

specs/core/1_new_shards.md

djrtwo · 2019-11-06T19:57:54Z

specs/core/1_new_shards.md

+### Epoch transition
+
+```python
+def phase_1_epoch_transition(state: BeaconState) -> None:


These are additional functions to the existing epoch transition?

We really need to figure out the build process.
This spec currently has 3 different approaches to modifying phase 0

function/container replacement and modifications

field addition to containers

additional functions for epoch/block processing

Co-Authored-By: Hsiao-Wei Wang <hwwang156@gmail.com>

Co-Authored-By: Danny Ryan <dannyjryan@gmail.com>

specs/core/1_new_shards.md

terencechain

Nice work! Implemented all the helpers in Prysm and here's feedback on those

terencechain · 2019-11-15T17:42:04Z

specs/core/1_new_shards.md

+```python
+def get_online_indices(state: BeaconState) -> Set[ValidatorIndex]:
+    active_validators = get_active_validator_indices(state, get_current_epoch(state))
+    return set([i for i in active_validators if state.online_countdown[i] != 0])


Is set really necessary? Trying to optimize from client implementation's perspective

It's not, but when things are unordered and unique we declare it as Set in the spec. Here it would be a free conversion for a client to a set object anyway (or any validator indices container).

terencechain · 2019-11-15T17:42:10Z

specs/core/1_new_shards.md

+| `SHARD_BLOCK_OFFSETS` | `[1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233]` | |
+| `MAX_SHARD_BLOCKS_PER_ATTESTATION` | `len(SHARD_BLOCK_OFFSETS)` | |
+| `EMPTY_CHUNK_ROOT` | `hash_tree_root(BytesN[SHARD_BLOCK_CHUNK_SIZE]())` | |
+| `MAX_GASPRICE` | `2**14` (= 16,384) | Gwei | |


gasprice or gas_price? not sure which is more semantic 🤔

terencechain · 2019-11-15T17:45:12Z

specs/core/1_new_shards.md

+    """
+    Given a state and a list of validator indices, outputs the CompactCommittee representing them.
+    """
+    validators = [state.validators[i] for i in committee]


Can simplify to one loop without sacrificing much readability. Example:

compactValidators := make([]uint64, len(committee)) pubKeys := make([][]byte, len(committee)) for i := 0; i < len(committee); i++ { v := state.Validators[committee[i]] compactValidators[i] = PackCompactValidator(committee[i], v.Slashed, v.EffectiveBalance/params.BeaconConfig().EffectiveBalanceIncrement) pubKeys[i] = v.PublicKey }

terencechain · 2019-11-15T17:46:29Z

specs/core/1_new_shards.md

+```python
+def get_shard_committee(beacon_state: BeaconState, epoch: Epoch, shard: Shard) -> Sequence[ValidatorIndex]:
+    source_epoch = epoch - epoch % SHARD_COMMITTEE_PERIOD 
+    if source_epoch > 0:


This feels safer

Suggested change

if source_epoch > 0:

if source_epoch >= SHARD_COMMITTEE_PERIOD:

terencechain · 2019-11-15T17:47:16Z

specs/core/1_new_shards.md

+### `get_shard_committee`
+
+```python
+def get_shard_committee(beacon_state: BeaconState, epoch: Epoch, shard: Shard) -> Sequence[ValidatorIndex]:


shard: Shard is not used

terencechain · 2019-11-15T17:47:46Z

specs/core/1_new_shards.md

+
+```python
+def get_shard_proposer_index(beacon_state: BeaconState, slot: Slot, shard: Shard) -> ValidatorIndex:
+    committee = get_shard_committee(beacon_state, slot_to_epoch(slot), shard)


Suggested change

committee = get_shard_committee(beacon_state, slot_to_epoch(slot), shard)

committee = get_shard_committee(beacon_state, compute_epoch_at_slot(slot), shard)

terencechain · 2019-11-15T17:48:36Z

specs/core/1_new_shards.md

+
+```python
+def get_shard(state: BeaconState, attestation: Attestation) -> Shard:
+    return Shard((attestation.data.index + get_start_shard(state, data.slot)) % ACTIVE_SHARDS)


get_start_shard is no longer defined in beacon chain spec, I think we have to reintroduce it here

terencechain · 2019-11-15T17:49:52Z

specs/core/1_new_shards.md

+
+```python
+def get_shard(state: BeaconState, attestation: Attestation) -> Shard:
+    return Shard((attestation.data.index + get_start_shard(state, data.slot)) % ACTIVE_SHARDS)


Suggested change

return Shard((attestation.data.index + get_start_shard(state, data.slot)) % ACTIVE_SHARDS)

return Shard((attestation.data.index + get_start_shard(state, attestation.data.slot)) % ACTIVE_SHARDS)

terencechain · 2019-11-15T17:51:18Z

specs/core/1_new_shards.md

+### `AttestationData`
+
+```python
+class AttestationData(Container):


Would it make sense to add shard to the field? can avoid get_shard

Would still need to validate the shard against the committee index for the slot, but I see value in including it here explicitly (similar to including slot even though it used to be able to be calculated from the constituent parts)

protolambda · 2019-11-15T23:15:34Z

Took the latest work and started on the necessary changes to make it run as part of the pyspec, and just organize things (start with custody spec cleanup, integrate with the rest of the spec).
Also rebased on the latest dev, with v0.9.1 changes too. And then some minor misc fixes in this code (blockbody vs block, loop indentation, etc.).

Opened a PR here: #1483
I'll try to merge any continued commits here, but it would be great if we can coordinate to continue phase 1 changes on a fork of this new PR.

djrtwo · 2019-11-20T19:47:33Z

Closing in favor of #1483

Added new shards

0e7c521

vbuterin changed the title ~~Added new shards~~ [WIP] New shard proposal Oct 12, 2019

hwwhww added the phase1 label Oct 12, 2019

vbuterin added 4 commits October 12, 2019 23:59

Added a few things

f371e4b

Added a few more things

c773a82

MAX_CATCHUP -> MAX_CATCHUP_RATIO

dd593f6

Variable max attestation length

391f893

vbuterin mentioned this pull request Oct 13, 2019

New shard proposal issues #1429

Closed

vbuterin added 3 commits October 13, 2019 17:42

formatting

1a863ae

Cleaned up state transition and honest committee member

7fe3547

Made trace a commitment separate from state roots

aa0b3b7

ethereum deleted a comment from Good2no1 Oct 15, 2019

vbuterin and others added 8 commits October 27, 2019 09:01

Some updates

0469d57

Shard slots to 128 bytes

499d323

Fix markdown and ToC

a0dd3d4

Updates to add new proposals

5170df9

Merge branch 'vitalik71' of github.com:ethereum/eth2.0-specs into vit…

c09aa29

…alik71

Changed fraud proof details

c8cee4c

Renaming

430567f

Small edits

54f963c

djrtwo reviewed Oct 31, 2019

View reviewed changes

vbuterin and others added 6 commits November 1, 2019 07:58

Changes to make Danny happy

f6e658b

Reformed attestations

8a2cdbe

Restructured shard blocks

6919571

Fixed pending attestation handling and added empty transition check

760430f

A few cleanups

0732327

Update ToC

8404294

hwwhww reviewed Nov 5, 2019

View reviewed changes

vbuterin and others added 2 commits November 5, 2019 10:31

Update specs/core/1_new_shards.md

7e9f044

Co-Authored-By: Hsiao-Wei Wang <hwwang156@gmail.com>

Update specs/core/1_new_shards.md

e3d61ab

Co-Authored-By: Hsiao-Wei Wang <hwwang156@gmail.com>

vbuterin commented Nov 5, 2019

View reviewed changes

specs/core/1_new_shards.md Outdated Show resolved Hide resolved

vbuterin and others added 8 commits November 5, 2019 12:15

Update specs/core/1_new_shards.md

8e3cf32

Update specs/core/1_new_shards.md

58bb5fc

Co-Authored-By: Hsiao-Wei Wang <hwwang156@gmail.com>

Update specs/core/1_new_shards.md

3dc29ea

Co-Authored-By: Hsiao-Wei Wang <hwwang156@gmail.com>

Update specs/core/1_new_shards.md

14c44c5

Co-Authored-By: Hsiao-Wei Wang <hwwang156@gmail.com>

Update specs/core/1_new_shards.md

dce1117

Co-Authored-By: Hsiao-Wei Wang <hwwang156@gmail.com>

Update specs/core/1_new_shards.md

011df4f

Co-Authored-By: Hsiao-Wei Wang <hwwang156@gmail.com>

Update specs/core/1_new_shards.md

420f56f

Co-Authored-By: Hsiao-Wei Wang <hwwang156@gmail.com>

Update specs/core/1_new_shards.md

563126d

Co-Authored-By: Hsiao-Wei Wang <hwwang156@gmail.com>

hwwhww reviewed Nov 6, 2019

View reviewed changes

djrtwo reviewed Nov 6, 2019

View reviewed changes

vbuterin and others added 6 commits November 6, 2019 15:53

Update specs/core/1_new_shards.md

b9c9fae

Co-Authored-By: Hsiao-Wei Wang <hwwang156@gmail.com>

Update specs/core/1_new_shards.md

45ff8c8

Co-Authored-By: Hsiao-Wei Wang <hwwang156@gmail.com>

Update specs/core/1_new_shards.md

a8a76ff

Co-Authored-By: Hsiao-Wei Wang <hwwang156@gmail.com>

Update specs/core/1_new_shards.md

5100548

Co-Authored-By: Danny Ryan <dannyjryan@gmail.com>

Some initial changes

6daee55

Simplified gasprice update

d98141f

terencechain reviewed Nov 10, 2019

View reviewed changes

specs/core/1_new_shards.md Outdated Show resolved Hide resolved

hwwhww mentioned this pull request Nov 10, 2019

Phase 0 to 1 upgradability: fork/version boundary deserialization issue #1475

Closed

vbuterin added 2 commits November 12, 2019 05:27

Some fixes

736fe98

Added get_shard_proposer_index

7c83172

terencechain mentioned this pull request Nov 13, 2019

Phase 1: Implement custody bits on attestations prysmaticlabs/prysm#3791

Closed

terencechain reviewed Nov 15, 2019

View reviewed changes

protolambda mentioned this pull request Nov 15, 2019

Move new phase 1 changes into the spec #1483

Closed

7 tasks

djrtwo closed this Nov 20, 2019

djrtwo mentioned this pull request Dec 5, 2019

Phase 1 rebase #1504

Merged

7 tasks

djrtwo deleted the vitalik71 branch May 20, 2020 23:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] New shard proposal #1427

[WIP] New shard proposal #1427

vbuterin commented Oct 12, 2019 •

edited

Loading

djrtwo left a comment

hwwhww left a comment

hwwhww Nov 6, 2019

vbuterin Nov 15, 2019

hwwhww Nov 6, 2019 •

edited

Loading

vbuterin Nov 12, 2019

djrtwo left a comment

djrtwo Nov 6, 2019

djrtwo Nov 6, 2019

terencechain left a comment

terencechain Nov 15, 2019

protolambda Nov 15, 2019

terencechain Nov 15, 2019

terencechain Nov 15, 2019

terencechain Nov 15, 2019

terencechain Nov 15, 2019

terencechain Nov 15, 2019

terencechain Nov 15, 2019

terencechain Nov 15, 2019

terencechain Nov 15, 2019

djrtwo Nov 20, 2019

protolambda commented Nov 15, 2019

djrtwo commented Nov 20, 2019

	if source_epoch > 0:
	if source_epoch >= SHARD_COMMITTEE_PERIOD:

	committee = get_shard_committee(beacon_state, slot_to_epoch(slot), shard)
	committee = get_shard_committee(beacon_state, compute_epoch_at_slot(slot), shard)

	return Shard((attestation.data.index + get_start_shard(state, data.slot)) % ACTIVE_SHARDS)
	return Shard((attestation.data.index + get_start_shard(state, attestation.data.slot)) % ACTIVE_SHARDS)

[WIP] New shard proposal #1427

[WIP] New shard proposal #1427

Conversation

vbuterin commented Oct 12, 2019 • edited Loading

djrtwo left a comment

Choose a reason for hiding this comment

hwwhww left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hwwhww Nov 6, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

djrtwo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

terencechain left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

protolambda commented Nov 15, 2019

djrtwo commented Nov 20, 2019

vbuterin commented Oct 12, 2019 •

edited

Loading

hwwhww Nov 6, 2019 •

edited

Loading