shuffle API support (#72)

Halo2 support shuffle API at
[here](privacy-scaling-explorations/halo2#185).
Now which is supported through this PR:

|  | Halo2 | halo2_verifier.move |
| --- | --- | --- |
| shuffle API | halo2_backend/src/plonk/shuffle/verifier.rs |
packages/verifier/sources/shuffle.move |
| “struct Argument” implementation | halo2_frontend/src/plonk/shuffle.rs
| packages/verifier/sources/protocol.move |
| verify function | halo2_backend/src/plonk/verifier.rs |
packages/verifier/sources/halo2_verifier.move |
| evaluator | halo2_backend/src/plonk/evaluation.rs | - |
| Halo2 update | - | “Cargo.toml” under crates folder |
| circuit information | - |
crates/verifier-sdk/shape-generator/src/lib.rs::CircuitInfo/struct
Lookup |
| serialized circuit | - |
crates/verifier-sdk/shape-generator/src/serialize.rs::SerializableCircuitInfo
|
| examples |  - | crates/vk-gen-examples/src/examples/shuffle_api.rs |

.gitignore

@@ -8,3 +8,4 @@ build
 vk_deployment/
 
 .aptos/
+.vscode/

README.md

@@ -1,15 +1,15 @@
-## halo2 verifier in Move
+# halo2 verifier in Move
 
-The project is a halo2 verifier written in Move language. 
+The project is a halo2 verifier written in Move language.
 Its objective to enhance the capability of the blockchains in the Move ecosystem by enabling halo2 zero-knowledge proofs to be verified on-chain.
 
-### Why the project
+## Why the project
 
 Halo2 is a widely used plonk implementation. zcash, scroll, axiom, taiko and many other famous projects are developed based on halo2.
-There exists a general verification process for different circuits in rust, 
-however, blockchains cannot utilize the code directly because of the gap in the language or api. 
-several organizations are developing onchain verifiers(like [halo-solidity-verifier](https://github.com/privacy-scaling-explorations/halo2-solidity-verifier)) in solidity for EVM communities, 
-although it’s not a general verifier code, only a general template generator. 
+There exists a general verification process for different circuits in rust,
+however, blockchains cannot utilize the code directly because of the gap in the language or api.
+several organizations are developing onchain verifiers(like [halo-solidity-verifier](https://github.com/privacy-scaling-explorations/halo2-solidity-verifier)) in solidity for EVM communities,
+although it’s not a general verifier code, only a general template generator.
 One still needs to generate a verifier contract for each circuit sharing most of the code.
 
 halo2-verifier.move uses a different approach, it tries to extract the information of a halo2 circuit, and abstract them out into a `Protocol` of the circuit(we call it the shape of a circuit), circuit’s shape includes:
@@ -22,6 +22,6 @@ halo2-verifier.move uses a different approach, it tries to extract the informati
 
 With these information, the general verifier can read commitments and evaluations in proofs of the circuit, and do verification accordingly using a polynomial commitment scheme.
 
-### Give it a try
+## Give it a try
 
 See [TUTORIAL.md](./TUTORIAL.md).

TUTORIAL.md

@@ -1,11 +1,10 @@
-### Tutorial on halo2-verifier
+# Tutorial on halo2-verifier
 
 This doc will guide you through the usage of halo2-verifier.
 
-Cli `aptos` is necessary for the tutorial, make sure you download it from https://github.com/aptos-labs/aptos-core/releases. 
+Cli `aptos` is necessary for the tutorial, make sure you download it from <https://github.com/aptos-labs/aptos-core/releases>.
 Also, the repo is an essence.
 
-
 First, let's create an aptos profile on aptos devnet.
 Execute the command under the project root. It will generate config file of aptos account in `.aptos/config.yaml`.
 
@@ -16,6 +15,7 @@ aptos init --network devnet
 Next, we need to publish the halo2-verifier & halo2-common modules to aptos devnet under the new-generated profile account.
 The following commands will use aptos cli to compile and publish the three packages: `common`, `verifier` and `api`.
 When first executing, it will fetch dependencies, which may need a little time. so be patient here.
+
 ``` shell
 cd packages/common
 aptos move publish --skip-fetch-latest-git-deps --named-addresses halo2_common=default
@@ -31,32 +31,37 @@ But first, we need a params of kzg setup. There exists a setup called [Perpetual
 We're going to use a version of axiom.
 the existing param file `crates/vk-gen-examples/params/challenge_0078-kzg_bn254_16.srs` is downloaded from [axiom page](https://docs.axiom.xyz/transparency-and-security/kzg-trusted-setup).
 
-To view the kzg setup params, run the following cargo commands under directory `crates/vk-gen-examples`. 
+To view the kzg setup params, run the following cargo commands under directory `crates/vk-gen-examples`.
 It will output the g1, g2, and s_g2.
+
 ```shell
 cargo run --release -- --param-path params/challenge_0078-kzg_bn254_16.srs view-param
 ```
 
 We have to send a  create-params transcation to make the params available on aptos.
-Copy paste the `g1`, `g2`, `s_g2` as aptos args, resulting the following aptos command: 
+Copy paste the `g1`, `g2`, `s_g2` as aptos args, resulting the following aptos command:
+
 ```shell
 aptos move run --function-id default::param_store::create --args hex:0x0100000000000000000000000000000000000000000000000000000000000000 hex:0xedf692d95cbdde46ddda5ef7d422436779445c5e66006a42761e1f12efde0018c212f3aeb785e49712e7a9353349aaf1255dfb31b7bf60723a480d9293938e19 hex:e4115200acc86e7670c83ded726335def098657fe8668323e9e41e6781b83b0a9d83b54bbb00215323ce6d7f9d7f331a286d7707d03f7dbdd3125c6163588d13
 ```
 
 Then, we are ready to publish a circuit!
 We're going to use example `vector-mul` in our vk-gen-examples.
 Enter directory `crates/vk-gen-examples`, and run the cargo command, remember **replace the verifier-address with your aptos profile's address!**
+
 ```shell
 cargo run --release -- --param-path params/challenge_0078-kzg_bn254_16.srs --verifier-address c9666cf9a032e81737eb706ce538a423706d86a2a502027fbc909e0817bf313b build-publish-vk-aptos-txn --example vector-mul
 ```
+
 It will output a json file which you can take as input to `aptos move run`.
 
 ```shell
 aptos move run --json-file VectorMul-publish-circuit.json
 ```
 
-Now, the circuit is published. We'll build a verify proof aptos txn and run it on aptos. 
+Now, the circuit is published. We'll build a verify proof aptos txn and run it on aptos.
 Run the command and replace the `verifier-address`/`param-address`/`circuit-address` with your aptos profile's address!
+
 ```shell
 cargo run --release -- --param-path params/challenge_0078-kzg_bn254_16.srs --verifier-address c9666cf9a032e81737eb706ce538a423706d86a2a502027fbc909e0817bf313b build-verify-proof-aptos-txn --example vector-mul --kzg gwc --param-address c9666cf9a032e81737eb706ce538a423706d86a2a502027fbc909e0817bf313b --circuit-address c9666cf9a032e81737eb706ce538a423706d86a2a502027fbc909e0817bf313b
 ```

crates/verifier-sdk/Cargo.toml

@@ -22,11 +22,11 @@ rand_core = { version = "0.6" }
 serde = { version = "1" }
 serde_json = { version = "1" }
 hex = { version = "0.4.3" }
-halo2_proofs = { git = "https://github.com/privacy-scaling-explorations/halo2.git", tag = "v2023_04_20" }
+halo2_proofs = { git = "https://github.com/privacy-scaling-explorations/halo2.git", tag = "v0.3.0" }
 bcs = { version = "0.1.6" }
 blake2b_simd = { version = "1" }
 
 
 
 multipoly = {path = "./multipoly" }
-shape-generator={path="./shape-generator"}
+shape-generator={path="./shape-generator"}

crates/verifier-sdk/aptos-verifier-api/src/proving.rs

@@ -1,4 +1,6 @@
-use halo2_proofs::halo2curves::ff::{FromUniformBytes, PrimeField, WithSmallOrderMulGroup};
+use halo2_proofs::arithmetic::CurveAffine;
+use halo2_proofs::halo2curves::CurveExt;
+use halo2_proofs::halo2curves::ff::{FromUniformBytes, WithSmallOrderMulGroup};
 use halo2_proofs::halo2curves::pairing::{Engine, MultiMillerLoop};
 use halo2_proofs::halo2curves::serde::SerdeObject;
 use halo2_proofs::plonk::{create_proof, verify_proof, Circuit, ProvingKey};
@@ -16,19 +18,18 @@ use rand_core::OsRng;
 
 pub fn prove_with_gwc_and_keccak256<E, ConcreteCircuit>(
     circuit: ConcreteCircuit,
-    instance: &[&[E::Scalar]],
+    instance: &[&[E::Fr]],
     params: &ParamsKZG<E>,
     pk: ProvingKey<E::G1Affine>,
 ) -> Vec<u8>
 where
     E: Engine + Debug + MultiMillerLoop,
-    E::G1Affine: SerdeObject,
-    E::G2Affine: SerdeObject,
-    ConcreteCircuit: Circuit<E::Scalar>,
-    <E as Engine>::Scalar: PrimeField,
-    <E as Engine>::Scalar: Ord,
-    <E as Engine>::Scalar: WithSmallOrderMulGroup<3>,
-    <E as Engine>::Scalar: FromUniformBytes<64>,
+    E::G1Affine:
+        SerdeObject + CurveAffine<ScalarExt = <E as Engine>::Fr, CurveExt = <E as Engine>::G1>,
+    E::G1: CurveExt<AffineExt = E::G1Affine>,
+    E::G2Affine: SerdeObject + CurveAffine,
+    ConcreteCircuit: Circuit<E::Fr>,
+    <E as Engine>::Fr: Ord + WithSmallOrderMulGroup<3> + FromUniformBytes<64>,
 {
     prove_vm_circuit::<
         KZGCommitmentScheme<E>,

crates/verifier-sdk/shape-generator/src/lib.rs

@@ -33,6 +33,7 @@ pub struct CircuitInfo<C: CurveAffine> {
     fixed_queries: Vec<ColumnQuery>,
     permutation_columns: Vec<Column>,
     lookups: Vec<Lookup<C::Scalar>>,
+    shuffles: Vec<Shuffle<C::Scalar>>,
 }
 
 #[derive(Debug)]
@@ -86,6 +87,11 @@ pub struct Lookup<F: Field> {
     pub input_exprs: Vec<MultiVariatePolynomial<F>>,
     pub table_exprs: Vec<MultiVariatePolynomial<F>>,
 }
+#[derive(Debug)]
+pub struct Shuffle<F: Field> {
+    pub input_exprs: Vec<MultiVariatePolynomial<F>>,
+    pub shuffle_exprs: Vec<MultiVariatePolynomial<F>>,
+}
 
 pub type MultiVariatePolynomial<F> = SparsePolynomial<F, SparseTerm>;
 
@@ -194,6 +200,40 @@ where
                     .collect(),
             })
             .collect(),
+        shuffles: cs
+            .shuffles()
+            .iter()
+            .map(|s| Shuffle {
+                input_exprs: s
+                    .input_expressions()
+                    .iter()
+                    .map(|e| {
+                        expression_transform(
+                            cs,
+                            e,
+                            cs.advice_queries().len(),
+                            cs.fixed_queries().len(),
+                            cs.instance_queries().len(),
+                            cs.challenge_phase().len(),
+                        )
+                    })
+                    .collect(),
+                shuffle_exprs: s
+                    .shuffle_expressions()
+                    .iter()
+                    .map(|e| {
+                        expression_transform(
+                            cs,
+                            e,
+                            cs.advice_queries().len(),
+                            cs.fixed_queries().len(),
+                            cs.instance_queries().len(),
+                            cs.challenge_phase().len(),
+                        )
+                    })
+                    .collect(),
+            })
+            .collect(),
         max_num_query_of_advice_column: cs
             .advice_queries()
             .iter()

crates/verifier-sdk/shape-generator/src/serialize.rs

@@ -23,6 +23,8 @@ pub struct SerializableCircuitInfo<C: CurveAffine> {
     gates: Vec<IndexedMultiVariatePolynomial>,
     lookups_input_exprs: Vec<Vec<IndexedMultiVariatePolynomial>>,
     lookups_table_exprs: Vec<Vec<IndexedMultiVariatePolynomial>>,
+    shuffles_input_exprs: Vec<Vec<IndexedMultiVariatePolynomial>>,
+    shuffles_exprs: Vec<Vec<IndexedMultiVariatePolynomial>>,
 }
 
 type IndexedMultiVariatePolynomial = Vec<(u16, SparseTerm)>;
@@ -55,6 +57,7 @@ impl<C: CurveAffine> From<CircuitInfo<C>> for SerializableCircuitInfo<C> {
             fixed_queries,
             permutation_columns,
             lookups,
+            shuffles,
         }: CircuitInfo<C>,
     ) -> Self {
         let mut fields_pool: Vec<C::Scalar> = Vec::new();
@@ -109,6 +112,44 @@ impl<C: CurveAffine> From<CircuitInfo<C>> for SerializableCircuitInfo<C> {
                 (inputs, tables)
             },
         );
+        let shuffles_len = shuffles.len();
+        let (shuffles_input, shuffles_shuffle) = shuffles.into_iter().fold(
+            (
+                Vec::with_capacity(shuffles_len),
+                Vec::with_capacity(shuffles_len),
+            ),
+            |(mut shuffle_inputs, mut shuffles), s| {
+                shuffle_inputs.push(
+                    s.input_exprs
+                        .into_iter()
+                        .map(|expr| {
+                            expr.terms
+                                .iter()
+                                .map(|(coff, t)| {
+                                    let new_coff = index_element(&mut fields_pool, *coff);
+                                    (new_coff as u16, t.clone())
+                                })
+                                .collect::<Vec<_>>()
+                        })
+                        .collect(),
+                );
+                shuffles.push(
+                    s.shuffle_exprs
+                        .into_iter()
+                        .map(|expr| {
+                            expr.terms
+                                .iter()
+                                .map(|(coff, t)| {
+                                    let new_coff = index_element(&mut fields_pool, *coff);
+                                    (new_coff as u16, t.clone())
+                                })
+                                .collect::<Vec<_>>()
+                        })
+                        .collect(),
+                );
+                (shuffle_inputs, shuffles)
+            },
+        );
 
         Self {
             k,
@@ -127,6 +168,8 @@ impl<C: CurveAffine> From<CircuitInfo<C>> for SerializableCircuitInfo<C> {
             gates,
             lookups_input_exprs: inputs,
             lookups_table_exprs: tables,
+            shuffles_input_exprs: shuffles_input,
+            shuffles_exprs: shuffles_shuffle,
             fields_pool,
             vk_transcript_repr,
             fixed_commitments,
@@ -205,6 +248,16 @@ pub fn serialize<C: CurveAffine>(
         .iter()
         .map(serialize_lookup_exprs)
         .collect();
+    let shuffles_input_exprs = circuit_info
+        .shuffles_input_exprs
+        .iter()
+        .map(serialize_lookup_exprs)
+        .collect();
+    let shuffles_shuffle_exprs = circuit_info
+        .shuffles_exprs
+        .iter()
+        .map(serialize_lookup_exprs)
+        .collect();
     Ok(vec![
         general_info,
         advice_queries,
@@ -215,6 +268,8 @@ pub fn serialize<C: CurveAffine>(
         gates,
         lookups_input_exprs,
         lookups_table_exprs,
+        shuffles_input_exprs,
+        shuffles_shuffle_exprs,
     ])
 }