Skip to content

Commit

Permalink
Docs: Integrating with other cddl-codegen gen'd libs (#208)
Browse files Browse the repository at this point in the history 
* Docs: Integrating with other cddl-codegen gen'd libs

New section with tips for generating a library that will depend on
another cddl-codegen'd library e.g. CML.

Also fixes minor isues in other parts of the docs.

* avoid exporting static traits when common dir is overridden

* don't export mod decls with common-import-override

* fix for common export overrides from wasm
  • Loading branch information
@rooooooooob
rooooooooob authored Aug 23, 2023
1 parent 65d8be8 commit 5e96387
Show file tree
Hide file tree
Showing 5 changed files with 122 additions and 76 deletions.
2 changes: 1 addition & 1 deletion docs/docs/command_line_flags.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ cddl-codegen --input=example --output=export --common-import-override=cml_core
<br/><br/>
:::info `--package-json`
:::info `--wasm-cbor-json-api-macro`
If it is passed in, it will call the supplied externally defined macro on each exported type, instead of manually exporting the functions for to/from CBOR bytes + to/from JSON API.
The external macro is assumed to exist at the specified path and will be imported if there are module prefixes.
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/comment_dsl.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ This can also be useful when you have a spec that is either very awkward to use
This can also be useful when you have a spec that is either very awkward to use (so you hand-write or hand-modify after generation) in some type so you don't generate those types and instead manually merge those hand-written/hand-modified structs back in to the code afterwards. This saves you from having to manually remove all code that is generated regarding `Foo` first before merging in your own.


#### _CDDL_CODEGEN_RAW_BYTES_TYPE_
## _CDDL_CODEGEN_RAW_BYTES_TYPE_

Allows encoding as `bytes` but imposing hand-written constraints defined elsewhere.
```cddl
Expand Down
32 changes: 32 additions & 0 deletions docs/docs/integration-other.mdx
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
---
sidebar_position: 7
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';


# Integration with other cddl-codegen libraries

This guide is written in general for integrating with other libraries generated by cddl-codegen, but in particular references CML (cardano-multiplatform-lib) for examples. Most things referencing CML will be relevant to other common cddl-codegen generated libraries used as dependencies.

## Common cddl-codegen traits

When generating a library that has as a dependency another cddl-codegen-generated library you can share the common cddl-codegen types/traits like `Deserialize`, `RawBytesEncoding`, etc. Remember to pass in `--common-import-override` tag. For CML we pass in `--common-import-override=cml_core`. This is where all the common cddl-codegen traits are located so we can avoid having duplicate incompatible traits in other libraries.

## CML macros

In CML we have macros for implementing WASM conversions and JSON/bytes. We pass in `--wasm-cbor-json-api-macro=cml_core_wasm::impl_wasm_cbor_json_api` and `--wasm-conversions-macro=cml_core_wasm::impl_wasm_conversions` which are both located in `cml_core_wasm`. This drastically reduces WASM wrapper boilerplate.

## Externally defined types

### `_CDDL_CODEGEN_EXTERN_TYPE_` vs `_CDDL_CODEGEN_RAW_BYTES_TYPE_`

There are two ways to have explicitly externally-defined types in cddl-codegen: `_CDDL_CODEGEN_EXTERN_TYPE_` and `_CDDL_CODEGEN_RAW_BYTES_TYPE_`. It is important to choose the appropriate one. If the type was defined originally as `_CDDL_CODEGEN_RAW_BYTES_TYPE_` in CML (or whatever library) then it is important to define it using this so it will be encoded correctly. If the type was either defined using `_CDDL_CODEGEN_EXTERN_TYPE_` (hand-written) or was explicitly defined normally in the dependency lib (e.g. CML) then use `_CDDL_CODEGEN_EXTERN_TYPE_`.

### Import pathing

In order to make imports easier it's recommended to make a directory corresponding to the dependency and put the `_CDDL_CODEGEN_RAW_BYTES_TYPE_` and `_CDDL_CODEGEN_EXTERN_TYPE_` external types inside of there and then later delete the output directories containing those modules. For an example see the `cml_chain` directory inside of the [`specs/multiera`](https://github.com/dcSpark/cardano-multiplatform-lib/tree/develop/specs/multiera).

### Non-black-box types

Another important detail, demonstrated in the above `multiera` CDDL spec, is that when using external types that aren't 100% self-contained (i.e. can't be treated as a black box that implements `Serialize` + `Deserialize`, nor as CBOR bytes implementing `RawBytesEncoding`) like `uint` aliases should be explicitly defined and then removed afterwards. Using the above directory/pathing tip makes this trivial to remove after.
24 changes: 17 additions & 7 deletions src/cli.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -67,13 +67,8 @@ pub struct Cli {
/// Location override for default common types (error, serialization, etc)
/// This is useful for integrating into an exisitng project that is based on
/// these types.
#[clap(
long,
value_parser,
value_name = "COMMON_IMPORT_OVERRIDE",
default_value = "crate"
)]
pub common_import_override: String,
#[clap(long, value_parser, value_name = "COMMON_IMPORT_OVERRIDE")]
common_import_override: Option<String>,

/// An external macro to be called instead of manually emitting functions for
/// conversions to/from CBOR bytes or JSON.
Expand All @@ -97,4 +92,19 @@ impl Cli {
pub fn lib_name_code(&self) -> String {
self.lib_name.replace('-', "_")
}

/// If someone override the common imports, we don't want to export them
pub fn export_static_files(&self) -> bool {
self.common_import_override.is_none()
}

pub fn common_import_rust(&self) -> &str {
self.common_import_override.as_deref().unwrap_or("crate")
}

pub fn common_import_wasm(&self) -> String {
self.common_import_override
.clone()
.unwrap_or_else(|| self.lib_name_code())
}
}
138 changes: 71 additions & 67 deletions src/generation.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -747,11 +747,14 @@ impl GenerationScope {
}

// declare modules (root lib specific)
self.rust_lib().raw("pub mod error;");
if cli.export_static_files() {
self.rust_lib().raw("pub mod error;");
if cli.preserve_encodings {
self.rust_lib().raw("pub mod ordered_hash_map;");
}
}
if cli.preserve_encodings {
self.rust_lib()
.raw("pub mod ordered_hash_map;")
.raw("extern crate derivative;");
self.rust_lib().raw("extern crate derivative;");
}
let scope_names = self
.rust_scopes
Expand Down Expand Up @@ -780,17 +783,17 @@ impl GenerationScope {
// needed if there's any params that can fail
content
.push_import("std::convert", "TryFrom", None)
.push_import(format!("{}::error", cli.common_import_override), "*", None);
.push_import(format!("{}::error", cli.common_import_rust()), "*", None);
// in case we store these in enums we're just going to dump them in everywhere
if cli.preserve_encodings {
content
.push_import(
format!("{}::serialization", cli.common_import_override),
format!("{}::serialization", cli.common_import_rust()),
"LenEncoding",
None,
)
.push_import(
format!("{}::serialization", cli.common_import_override),
format!("{}::serialization", cli.common_import_rust()),
"StringEncoding",
None,
);
Expand All @@ -804,12 +807,12 @@ impl GenerationScope {
content
.push_import("std::collections", "BTreeMap", None)
.push_import(
format!("{}::serialization", cli.common_import_override),
format!("{}::serialization", cli.common_import_rust()),
"LenEncoding",
None,
)
.push_import(
format!("{}::serialization", cli.common_import_override),
format!("{}::serialization", cli.common_import_rust()),
"StringEncoding",
None,
);
Expand Down Expand Up @@ -882,15 +885,11 @@ impl GenerationScope {
// Issue (general - not just here): https://github.com/dcSpark/cddl-codegen/issues/139
content.push_import("std::collections", "BTreeMap", None);
if cli.preserve_encodings {
if *scope == *ROOT_SCOPE {
content.push_import("ordered_hash_map", "OrderedHashMap", None);
} else {
content.push_import(
format!("{}::ordered_hash_map", cli.common_import_override),
"OrderedHashMap",
None,
);
}
content.push_import(
format!("{}::ordered_hash_map", cli.common_import_rust()),
"OrderedHashMap",
None,
);
}
}

Expand All @@ -905,7 +904,7 @@ impl GenerationScope {
.push_import("std::io", "Write", None)
.push_import("cbor_event::de", "Deserializer", None)
.push_import("cbor_event::se", "Serializer", None)
.push_import(format!("{}::error", cli.common_import_override), "*", None);
.push_import(format!("{}::error", cli.common_import_rust()), "*", None);
if cli.preserve_encodings {
content.push_import("super::cbor_encodings", "*", None);
}
Expand All @@ -914,7 +913,7 @@ impl GenerationScope {
}
if *scope != *ROOT_SCOPE {
content.push_import(
format!("{}::serialization", cli.common_import_override),
format!("{}::serialization", cli.common_import_rust()),
"*",
None,
);
Expand Down Expand Up @@ -964,7 +963,7 @@ impl GenerationScope {
.push_import("wasm_bindgen::prelude", "JsValue", None);
if cli.preserve_encodings {
content.push_import(
format!("{}::ordered_hash_map", cli.lib_name_code()),
format!("{}::ordered_hash_map", cli.common_import_wasm()),
"OrderedHashMap",
None,
);
Expand Down Expand Up @@ -1071,31 +1070,34 @@ impl GenerationScope {
)?;

// serialiation.rs / {module}/serialization.rs files (if input is a directory)
let mut serialize_paths = vec![cli.static_dir.join("serialization.rs")];
if cli.preserve_encodings {
serialize_paths.push(cli.static_dir.join("serialization_preserve.rs"));
if cli.canonical_form {
serialize_paths.push(
cli.static_dir
.join("serialization_preserve_force_canonical.rs"),
);
let mut merged_rust_serialize_scope = codegen::Scope::new();
if cli.export_static_files() {
let mut serialize_paths = vec![cli.static_dir.join("serialization.rs")];
if cli.preserve_encodings {
serialize_paths.push(cli.static_dir.join("serialization_preserve.rs"));
if cli.canonical_form {
serialize_paths.push(
cli.static_dir
.join("serialization_preserve_force_canonical.rs"),
);
} else {
serialize_paths.push(
cli.static_dir
.join("serialization_preserve_non_force_canonical.rs"),
);
serialize_paths
.push(cli.static_dir.join("serialization_non_force_canonical.rs"));
}
} else {
serialize_paths.push(
cli.static_dir
.join("serialization_preserve_non_force_canonical.rs"),
);
serialize_paths.push(cli.static_dir.join("serialization_non_preserve.rs"));
serialize_paths.push(cli.static_dir.join("serialization_non_force_canonical.rs"));
}
} else {
serialize_paths.push(cli.static_dir.join("serialization_non_preserve.rs"));
serialize_paths.push(cli.static_dir.join("serialization_non_force_canonical.rs"));
}
// raw_bytes_encoding in serialization too
if export_raw_bytes_encoding_trait {
serialize_paths.push(cli.static_dir.join("raw_bytes_encoding.rs"));
// raw_bytes_encoding in serialization too
if export_raw_bytes_encoding_trait {
serialize_paths.push(cli.static_dir.join("raw_bytes_encoding.rs"));
}
merged_rust_serialize_scope.raw(concat_files(&serialize_paths)?);
}
let mut merged_rust_serialize_scope = codegen::Scope::new();
merged_rust_serialize_scope.raw(concat_files(&serialize_paths)?);
merged_rust_serialize_scope.append(&self.rust_serialize_lib_scope);
merge_scopes_and_export(
rust_dir.join("rust/src"),
Expand Down Expand Up @@ -1155,30 +1157,32 @@ impl GenerationScope {
rust_cargo_toml.replace("cddl-lib", &cli.lib_name),
)?;

// error.rs
std::fs::copy(
cli.static_dir.join("error.rs"),
rust_dir.join("rust/src/error.rs"),
)?;
if cli.export_static_files() {
// error.rs
std::fs::copy(
cli.static_dir.join("error.rs"),
rust_dir.join("rust/src/error.rs"),
)?;

// ordered_hash_map.rs
if cli.preserve_encodings {
let mut ordered_hash_map_rs =
std::fs::read_to_string(cli.static_dir.join("ordered_hash_map.rs"))?;
if cli.json_serde_derives {
ordered_hash_map_rs.push_str(&std::fs::read_to_string(
cli.static_dir.join("ordered_hash_map_json.rs"),
)?);
}
if cli.json_schema_export {
ordered_hash_map_rs.push_str(&std::fs::read_to_string(
cli.static_dir.join("ordered_hash_map_schemars.rs"),
)?);
// ordered_hash_map.rs
if cli.preserve_encodings {
let mut ordered_hash_map_rs =
std::fs::read_to_string(cli.static_dir.join("ordered_hash_map.rs"))?;
if cli.json_serde_derives {
ordered_hash_map_rs.push_str(&std::fs::read_to_string(
cli.static_dir.join("ordered_hash_map_json.rs"),
)?);
}
if cli.json_schema_export {
ordered_hash_map_rs.push_str(&std::fs::read_to_string(
cli.static_dir.join("ordered_hash_map_schemars.rs"),
)?);
}
std::fs::write(
rust_dir.join("rust/src/ordered_hash_map.rs"),
rustfmt_generated_string(&ordered_hash_map_rs)?.as_ref(),
)?;
}
std::fs::write(
rust_dir.join("rust/src/ordered_hash_map.rs"),
rustfmt_generated_string(&ordered_hash_map_rs)?.as_ref(),
)?;
}

// wasm crate
Expand Down Expand Up @@ -3415,7 +3419,7 @@ fn create_base_wasm_struct<'a>(
if cli.preserve_encodings && cli.canonical_form {
to_bytes.line(format!(
"{}::serialization::Serialize::to_cbor_bytes(&self.0)",
cli.lib_name_code()
cli.common_import_wasm()
));
let mut to_canonical_bytes =
codegen::Function::new("to_canonical_cbor_bytes");
Expand All @@ -3427,7 +3431,7 @@ fn create_base_wasm_struct<'a>(
} else {
to_bytes.line(format!(
"{}::serialization::ToCBORBytes::to_cbor_bytes(&self.0)",
cli.lib_name_code()
cli.common_import_wasm()
));
}
s_impl.push_fn(to_bytes);
Expand All @@ -3439,7 +3443,7 @@ fn create_base_wasm_struct<'a>(
.vis("pub")
.line(format!(
"{}::serialization::Deserialize::from_cbor_bytes(cbor_bytes).map(Self).map_err(|e| JsValue::from_str(&format!(\"from_bytes: {{}}\", e)))",
cli.lib_name_code()));
cli.common_import_wasm()));
}
}
if cli.json_serde_derives {
Expand Down

0 comments on commit 5e96387

Please sign in to comment.