feat: unified interface for da worker #1021

drspacemn · 2023-08-18T15:20:35Z

PR is based on unifying the 3 DA Implementations here:
Avail
Celestia
Ethereum

The primary interface draws logic from @Leouarz interface here
(some methods internalized to the config initialization)

#[async_trait]
pub trait DaClient: Send + Sync {
    fn get_mode(&self) -> DaMode;
    async fn last_published_state(&self) -> Result<I256>;
    async fn publish_state_diff(&self, state_diff: Vec<U256>) -> Result<()>;
}

DaClient implementers are meant to be built from config.json files located at <madara_path>/da-config.json, but are also expected to have sensible defaults.

        let da_client: Box<dyn DaClient + Send + Sync> = match da_layer {
            DaLayer::Celestia => {
                let celestia_conf = CelestiaConfig::try_from_file(&da_path)?;
                Box::new(
                    CelestiaClient::try_from_config(celestia_conf).map_err(|e| ServiceError::Other(e.to_string()))?,
                )
            }
            DaLayer::Ethereum => {
                let ethereum_conf = EthereumConfig::try_from_file(&da_path)?;
                Box::new(EthereumClient::try_from_config(ethereum_conf)?)
            }
            DaLayer::Avail => {
                let avail_conf = AvailConfig::try_from_file(&da_path)?;
                Box::new(AvailClient::try_from_config(avail_conf).map_err(|e| ServiceError::Other(e.to_string()))?)
            }
        };

The Enum DaMode defines the potential data availability modes for Madara. As proving functionality is not currently functional, all modes default to Validium:

#[derive(Debug, Copy, Clone, PartialEq, Deserialize, Default)]
pub enum DaMode {
    #[serde(rename = "validity")]
    Validity, 
    #[serde(rename = "volition")]
    Volition,
    #[serde(rename = "validium")]
    #[default]
    Validium,
}

Summary of Changes

Component	Change
CLI	New optional flags in `ExtendedRunCommand`: `da-layer` [ethereum, celestia,avail]
Service	state_diff notifications, push `state_diff` to the selected DA and verify that the `state_diff` output is included in the correct place.

Configuration file

When launching the node da config file is expected at <madara-path>/da-config.json

An example of running a node w/ DA.

RUST_LOG=info cargo run --release -- --da-layer=ethereum --dev

Launch with Avail

da-config.json

{
   "ws_provider": "ws://127.0.0.1:9945",
   "app_id": 0,
   "validate_codegen": true,
   "seed": "//Alice"
}

You can find instruction to launch the Avail node. TLDR:

#  testnet ws wss://kate.avail.tools/ws
git clone https://github.com/availproject/avail
cd avail
cargo run --release -- --dev --tmp --rpc-port 9934 --ws-port 9945 --port 30334

Launch with Celestia

da-config.json

{
   "http_provider": "http://127.0.0.1:26658",
   "ws_provider": "ws://127.0.0.1:26658",
   "nid": "Madara",
   "auth_token": "<see below>"
}

Set up and run a Celestia light node

Requires the installation of the celestia CLI tool (available here). Once you have that set up, run the following:

celestia light start --core.ip consensus-full-arabica-9.celestia-arabica.com --gateway.deprecated-endpoints --gateway --gateway.addr 127.0.0.1 --gateway.port 26659 --p2p.network arabica-9

Next, we need to generate a JWT to access our light node:

celestia light auth admin --p2p.network arabica-9

Launch with Ethereum

da-config.json

{
   "http_provider": "http://127.0.0.1:8545",
   "core_contracts": "0x5FbDB2315678afecb367f032d93F642f64180aa3",
   "sequencer_key": "ac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80",
   "chain_id": 31337
}

Start Anvil and Deploy altered Core Contracts

git clone --recurse-submodules https://github.com/keep-starknet-strange/zaun.git target/zaun 2> /dev/null
./target/zaun/scripts/sn-base-dev.sh target target/zaun 2> /dev/null

Integration Testing to be done via scripts/da_devnet.sh.

codecov · 2023-08-30T06:43:19Z

Codecov Report

Patch coverage has no change and project coverage change: -1.74% ⚠️

Comparison is base (b26042d) 42.45% compared to head (53bfef9) 40.71%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1021      +/-   ##
==========================================
- Coverage   42.45%   40.71%   -1.74%     
==========================================
  Files          86       96      +10     
  Lines       10617    11069     +452     
  Branches    10617    11069     +452     
==========================================
  Hits         4507     4507              
- Misses       5598     6050     +452     
  Partials      512      512

Files Changed	Coverage Δ
...rates/client/data-availability/src/avail/config.rs	`0.00% <0.00%> (ø)`
crates/client/data-availability/src/avail/mod.rs	`0.00% <0.00%> (ø)`
...es/client/data-availability/src/celestia/config.rs	`0.00% <0.00%> (ø)`
...rates/client/data-availability/src/celestia/mod.rs	`0.00% <0.00%> (ø)`
...es/client/data-availability/src/ethereum/config.rs	`0.00% <0.00%> (ø)`
...rates/client/data-availability/src/ethereum/mod.rs	`0.00% <0.00%> (ø)`
crates/client/data-availability/src/lib.rs	`0.00% <0.00%> (ø)`
crates/client/data-availability/src/sharp/mod.rs	`0.00% <0.00%> (ø)`
crates/client/data-availability/src/utils.rs	`0.00% <0.00%> (ø)`
crates/client/db/src/da_db.rs	`0.00% <0.00%> (ø)`
... and 4 more

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

add integration test for Avail DA layer

crates/client/data-availability/Cargo.toml

crates/client/data-availability/src/avail/config.rs

crates/client/data-availability/src/avail/mod.rs

crates/client/data-availability/src/celestia/config.rs

crates/client/data-availability/src/lib.rs

crates/client/data-availability/src/sharp/mod.rs

crates/client/db/src/da_db.rs

crates/node/src/command.rs

EvolveArt

A few questions on top of my mind :

What happens if the madara nodes see that the DA was not updated correctly ?
What does it mean to be in Validity DaMode for a client like CelestiaClient ? Does it just mean we also publish state updates on L1 ?

crates/client/data-availability/src/lib.rs

tdelabro · 2023-09-05T10:11:33Z

@EvolveArt

What does it mean to be in Validity DaMode for a client like CelestiaClient? Does it just mean we also publish state updates on L1 ?

I think it means we publish the state, but also prove the state update computation

…t/da-worker

drspacemn · 2023-09-05T18:01:34Z

A few questions on top of my mind :

What happens if the madara nodes see that the DA was not updated correctly ?

What does it mean to be in Validity DaMode for a client like CelestiaClient ? Does it just mean we also publish state updates on L1 ?

This will be the tradeoffs of the DaModes. If madara node is Validity the Core Contracts (or other solutions see 2) will reject that update. If its Validium, yeah that's the rub, but operators can determine their own consensus around data that is now available on say Celestia
Yet to be fully defined, but there are solutions like:

that can allow us to implement a version of Validity

tdelabro · 2023-09-06T08:18:53Z

@drspacemn just fix the CI and rebase and we can merge

Leouarz · 2023-09-06T08:27:18Z

A few questions on top of my mind :

What happens if the madara nodes see that the DA was not updated correctly ?

What does it mean to be in Validity DaMode for a client like CelestiaClient ? Does it just mean we also publish state updates on L1 ?

This will be the tradeoffs of the DaModes. If madara node is Validity the Core Contracts (or other solutions see 2) will reject that update. If its Validium, yeah that's the rub, but operators can determine their own consensus around data that is now available on say Celestia

Yet to be fully defined, but there are solutions like:

celestia's quantum gravity bridge

avail's attestation bridge

that can allow us to implement a version of Validity

Here's the correct link for Avail btw : https://availproject.github.io/api/use-case-validiums/#verify-data-availability-on-ethereum

EvolveArt · 2023-09-07T12:24:45Z

A few questions on top of my mind :

What happens if the madara nodes see that the DA was not updated correctly ?

What does it mean to be in Validity DaMode for a client like CelestiaClient ? Does it just mean we also publish state updates on L1 ?

This will be the tradeoffs of the DaModes. If madara node is Validity the Core Contracts (or other solutions see 2) will reject that update. If its Validium, yeah that's the rub, but operators can determine their own consensus around data that is now available on say Celestia

Yet to be fully defined, but there are solutions like:

celestia's quantum gravity bridge

avail's attestation bridge

that can allow us to implement a version of Validity

@drspacemn I still find a few things confusing here regarding SHARP/DA coupling

Imo, SHARP and DA (or DP as now people want to say) should be decoupled and not necessarily done as part of same thread
In the current SHARP integration we directly submit the cairo_pie to SHARP, I know it's mocked for now but we need to run the starknet OS program somewhere in the middle right ?
What happens if the app-chain wants to switch of DA Layer, is there anyway to retro-actively publish DA on the new layer ? Would potentially also apply to state migration

drspacemn · 2023-09-07T19:08:56Z

A few questions on top of my mind :

What happens if the madara nodes see that the DA was not updated correctly ?

What does it mean to be in Validity DaMode for a client like CelestiaClient ? Does it just mean we also publish state updates on L1 ?

This will be the tradeoffs of the DaModes. If madara node is Validity the Core Contracts (or other solutions see 2) will reject that update. If its Validium, yeah that's the rub, but operators can determine their own consensus around data that is now available on say Celestia

Yet to be fully defined, but there are solutions like:

celestia's quantum gravity bridge

avail's attestation bridge

that can allow us to implement a version of Validity

@drspacemn I still find a few things confusing here regarding SHARP/DA coupling

Imo, SHARP and DA (or DP as now people want to say) should be decoupled and not necessarily done as part of same thread

In the current SHARP integration we directly submit the cairo_pie to SHARP, I know it's mocked for now but we need to run the starknet OS program somewhere in the middle right ?

What happens if the app-chain wants to switch of DA Layer, is there anyway to retro-actively publish DA on the new layer ? Would potentially also apply to state migration

SHARP and DA are decoupled in this PR. There is a task spawned to handle the state_update and a task spawned to handle the proving. both will handle their business logic via the mode they have been initialized in.
Yes the starknet OS program will need to be run in the prove_current_block task and only if its applicable to the match da_mode { (i.e. not for Valium)
That is a good question probably out of scope for the current work done here.

EvolveArt · 2023-09-08T08:36:10Z

Sounds good, would be great to document all of these questions and features to work on, probably the best place is a GH discussion, wdyt @drspacemn ?

Otherwise let's merge this PR which is a great starting point to on-board new contributors and great work on this again!

drspacemn added 4 commits August 18, 2023 09:16

rebase lock

6d56a14

rebase cargo.lock

bf42d6b

dep lifetime bug

f3f85db

remove chain spec mod

b84f010

drspacemn requested a review from 0xLucqs August 18, 2023 15:23

drspacemn added 8 commits August 18, 2023 13:19

avail config

1f637be

deps

423f2e5

daMode config

77cc31e

update interfaces

255aa72

fmt mods

e575805

devnet

77ae822

celestia da script

0e4def0

wait for height

9aaa599

antiyro mentioned this pull request Aug 23, 2023

net/bitcoin_module KasarLabs/barknet#1

Open

drspacemn added 4 commits August 29, 2023 23:07

update devnet script

b42b81d

attempt save Cargo.lock

82dda64

Merge branch 'main' into feat/da-worker

090ddcb

fmt and clippy

e28a5da

drspacemn mentioned this pull request Aug 30, 2023

Feat/data avail #1020

Closed

1 task

drspacemn marked this pull request as ready for review August 30, 2023 05:53

drspacemn added 2 commits August 30, 2023 00:22

celestia ws conf

ce1f6d3

change log

6e6458e

drspacemn and others added 3 commits August 30, 2023 00:45

changelog prettier

9b6a6f7

add integratiion test for Avail DA layer

0c77456

Merge pull request #1 from Leouarz/test/avail-integration

59def44

add integration test for Avail DA layer

tdelabro suggested changes Aug 31, 2023

View reviewed changes

EvolveArt reviewed Sep 5, 2023

View reviewed changes

tdelabro reviewed Sep 5, 2023

View reviewed changes

crates/client/data-availability/src/lib.rs Outdated Show resolved Hide resolved

drspacemn added 6 commits September 5, 2023 10:01

resolve conflicts

0e0d18a

post merge

68cbd87

Merge branch 'feat/da-worker' of github.com:drspacemn/madara into fea…

dc48f33

…t/da-worker

addressing review

f043c2b

try_froms and match arms

606ebe7

doc for da mode

0d7c721

tdelabro approved these changes Sep 6, 2023

View reviewed changes

drspacemn added 4 commits September 7, 2023 12:37

merge prep

38b8068

Merge remote-tracking branch 'upstream/main' into feat/da-worker

eff2c82

updated cargo lock

3bb1769

fix taplo

c2076c6

Merge branch 'main' into feat/da-worker

53bfef9

tdelabro approved these changes Sep 10, 2023

View reviewed changes

tdelabro merged commit a6f6554 into keep-starknet-strange:main Sep 10, 2023
11 of 12 checks passed

github-actions bot locked and limited conversation to collaborators Sep 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: unified interface for da worker #1021

feat: unified interface for da worker #1021

drspacemn commented Aug 18, 2023 •

edited

Loading

codecov bot commented Aug 30, 2023 •

edited

Loading

EvolveArt left a comment

tdelabro commented Sep 5, 2023

drspacemn commented Sep 5, 2023

tdelabro commented Sep 6, 2023

Leouarz commented Sep 6, 2023

EvolveArt commented Sep 7, 2023

drspacemn commented Sep 7, 2023

EvolveArt commented Sep 8, 2023

feat: unified interface for da worker #1021

feat: unified interface for da worker #1021

Conversation

drspacemn commented Aug 18, 2023 • edited Loading

Summary of Changes

Configuration file

Launch with Avail

Launch with Celestia

Set up and run a Celestia light node

Launch with Ethereum

Start Anvil and Deploy altered Core Contracts

codecov bot commented Aug 30, 2023 • edited Loading

Codecov Report

EvolveArt left a comment

Choose a reason for hiding this comment

tdelabro commented Sep 5, 2023

drspacemn commented Sep 5, 2023

tdelabro commented Sep 6, 2023

Leouarz commented Sep 6, 2023

EvolveArt commented Sep 7, 2023

drspacemn commented Sep 7, 2023

EvolveArt commented Sep 8, 2023

drspacemn commented Aug 18, 2023 •

edited

Loading

codecov bot commented Aug 30, 2023 •

edited

Loading