diff --git a/README.md b/README.md index 57e6026e..8d3ed524 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ CBOR is a [trusted alternative](https://www.rfc-editor.org/rfc/rfc8949.html#name `fxamacker/cbor` is used in projects by Arm Ltd., Cisco, Dapper Labs, EdgeX Foundry, Fraunhofer‑AISEC, Linux Foundation, Microsoft, Mozilla, Oasis Protocol, Tailscale, Teleport, [and others](https://github.com/fxamacker/cbor#who-uses-fxamackercbor). -See [Quick Start](#quick-start) if you have not yet used v2.5.0. +See [Quick Start](#quick-start) and [v2.5.0](https://github.com/fxamacker/cbor/releases/tag/v2.5.0). New funcs `UnmarshalFirst` and `DiagnoseFirst` return remaining bytes. ## fxamacker/cbor @@ -23,24 +23,93 @@ See [Quick Start](#quick-start) if you have not yet used v2.5.0. Features include full support for CBOR tags, [Core Deterministic Encoding](https://www.rfc-editor.org/rfc/rfc8949.html#name-core-deterministic-encoding), duplicate map key detection, etc. -Struct tags (`toarray`, `keyasint`, `omitempty`) reduce encoded size of structs. +Design balances trade-offs between security, speed, concurrency, encoded data size, usability, etc. -![alt text](https://github.com/fxamacker/images/raw/master/cbor/v2.3.0/cbor_struct_tags_api.svg?sanitize=1 "CBOR API and Go Struct Tags") +
Highlights

-API is mostly same as `encoding/json`, plus interfaces that simplify concurrency for CBOR options. +__πŸš€  Speed__ + +Encoding and decoding is fast without using Go's `unsafe` package. Slower settings are opt-in. Default limits allow very fast and memory efficient rejection of malformed CBOR data. + +__πŸ”’  Security__ + +Decoder has configurable limits that defend against malicious inputs. Duplicate map key detection is supported. By contrast, `encoding/gob` is [not designed to be hardened against adversarial inputs](https://pkg.go.dev/encoding/gob#hdr-Security). + +Codec passed multiple confidential security assessments in 2022. No vulnerabilities found in subset of codec in a [nonconfidential security assessment](https://github.com/veraison/go-cose/blob/v1.0.0-rc.1/reports/NCC_Microsoft-go-cose-Report_2022-05-26_v1.0.pdf) prepared by NCC Group for Microsoft Corporation. + +__πŸ—œοΈ  Data Size__ + +Struct tags (`toarray`, `keyasint`, `omitempty`) automatically reduce size of encoded structs. Encoding optionally shrinks float64β†’32β†’16 when values fit. + +__:jigsaw:  Usability__ + +API is mostly same as `encoding/json` plus interfaces that simplify concurrency for CBOR options. Encoding and decoding modes can be created at startup and reused by any goroutines. -#### CBOR Security +Presets include Core Deterministic Encoding, Preferred Serialization, CTAP2 Canonical CBOR, etc. + +__πŸ“†  Extensibility__ -Decoder has configurable limits that defend against malicious inputs. Duplicate map key detection is supported. +Features include CBOR [extension points](https://www.rfc-editor.org/rfc/rfc8949.html#section-7.1) (e.g. CBOR tags) and extensive settings. API has interfaces that allow users to create custom encoding and decoding without modifying this library. + +

+ +
+ +### Secure Decoding with Configurable Settings + +`fxamacker/cbor` has configurable limits, etc. that defend against malicious CBOR data. By contrast, `encoding/gob` is [not designed to be hardened against adversarial inputs](https://pkg.go.dev/encoding/gob#hdr-Security). -Decoding 10 bytes of malicious CBOR data directly into `[]byte` with default settings: +
Example decoding with encoding/gob πŸ’₯ fatal error (out of memory)

+ +```Go +// Example of encoding/gob having "fatal error: runtime: out of memory" +// while decoding 181 bytes. +package main +import ( + "bytes" + "encoding/gob" + "encoding/hex" + "fmt" +) + +// Example data is from https://github.com/golang/go/issues/24446 +// (shortened to 181 bytes). +const data = "4dffb503010102303001ff30000109010130010800010130010800010130" + + "01ffb80001014a01ffb60001014b01ff860001013001ff860001013001ff" + + "860001013001ff860001013001ffb80000001eff850401010e3030303030" + + "30303030303030303001ff3000010c0104000016ffb70201010830303030" + + "3030303001ff3000010c000030ffb6040405fcff00303030303030303030" + + "303030303030303030303030303030303030303030303030303030303030" + + "30" + +type X struct { + J *X + K map[string]int +} + +func main() { + raw, _ := hex.DecodeString(data) + decoder := gob.NewDecoder(bytes.NewReader(raw)) + + var x X + decoder.Decode(&x) // fatal error: runtime: out of memory + fmt.Println("Decoding finished.") +} +``` + +

+ +
+ +`fxamacker/cbor` is fast at rejecting malformed CBOR data. E.g. attempts to +decode 10 bytes of malicious CBOR data to `[]byte` (with default settings): | Codec | Speed (ns/op) | Memory | Allocs | | :---- | ------------: | -----: | -----: | -| fxamacker/cbor 2.5.0 | 43.95n Β± 5% | 32 B/op | 2 allocs/op | -| ugorji/go 1.2.11 | 5353261.00n Β± 4% | 67111321 B/op | 13 allocs/op | +| fxamacker/cbor 2.5.0 | 44 Β± 5% | 32 B/op | 2 allocs/op | +| ugorji/go 1.2.11 | 5353261 Β± 4% | 67111321 B/op | 13 allocs/op |
Benchmark details

@@ -62,40 +131,88 @@ Latest comparison used: - go1.19.6, linux/amd64, i5-13600K (DDR4) - go test -bench=. -benchmem -count=20 +

+
-#### Design and Feature Highlights +### Smaller Encodings with Struct Tags -Design balances tradeoffs between speed, security, memory, encoded data size, usability, etc. +Struct tags (`toarray`, `keyasint`, `omitempty`) reduce encoded size of structs. -
Highlights

+

Example encoding 3-level nested Go struct to 1 byte CBOR

-__πŸš€  Speed__ +https://go.dev/play/p/YxwvfPdFQG2 -Encoding and decoding is fast without using Go's `unsafe` package. Slower settings are opt-in. Default limits allow very fast and memory efficient rejection of malformed CBOR data. +```Go +// Example encoding nested struct (with omitempty tag) +// - encoding/json: 18 byte JSON +// - fxamacker/cbor: 1 byte CBOR +package main -__πŸ”’  Security__ +import ( + "encoding/hex" + "encoding/json" + "fmt" -Decoder has configurable limits that defend against malicious inputs. Duplicate map key detection is supported. By contrast, `encoding/gob` is [not designed to be hardened against adversarial inputs](https://pkg.go.dev/encoding/gob#hdr-Security). + "github.com/fxamacker/cbor/v2" +) -Codec passed multiple confidential security assessments in 2022. No vulnerabilities found in subset of codec in a [nonconfidential security assessment](https://github.com/veraison/go-cose/blob/v1.0.0-rc.1/reports/NCC_Microsoft-go-cose-Report_2022-05-26_v1.0.pdf) prepared by NCC Group for Microsoft Corporation. +type GrandChild struct { + Quux int `json:",omitempty"` +} -__πŸ—œοΈ  Data Size__ +type Child struct { + Baz int `json:",omitempty"` + Qux GrandChild `json:",omitempty"` +} -Struct tags (`toarray`, `keyasint`, `omitempty`) automatically reduce size of encoded structs. Encoding optionally shrinks float64β†’32β†’16 when values fit. +type Parent struct { + Foo Child `json:",omitempty"` + Bar int `json:",omitempty"` +} -__:jigsaw:  Usability__ +func cb() { + results, _ := cbor.Marshal(Parent{}) + fmt.Println("hex(CBOR): " + hex.EncodeToString(results)) -API is mostly same as `encoding/json` plus interfaces that simplify concurrency for CBOR options. Encoding and decoding modes can be created at startup and reused by any goroutines. + text, _ := cbor.Diagnose(results) // Diagnostic Notation + fmt.Println("DN: " + text) +} -Presets include Core Deterministic Encoding, Preferred Serialization, CTAP2 Canonical CBOR, etc. +func js() { + results, _ := json.Marshal(Parent{}) + fmt.Println("hex(JSON): " + hex.EncodeToString(results)) -__πŸ“†  Extensibility__ + text := string(results) // JSON + fmt.Println("JSON: " + text) +} -Features include CBOR [extension points](https://www.rfc-editor.org/rfc/rfc8949.html#section-7.1) (e.g. CBOR tags) and extensive settings. API has interfaces that allow users to create custom encoding and decoding without modifying this library. +func main() { + cb() + fmt.Println("-------------") + js() +} +``` + +Output (DN is Diagnostic Notation): +``` +hex(CBOR): a0 +DN: {} +------------- +hex(JSON): 7b22466f6f223a7b22517578223a7b7d7d7d +JSON: {"Foo":{"Qux":{}}} +``` + +

+Example using different struct tags together: + +![alt text](https://github.com/fxamacker/images/raw/master/cbor/v2.3.0/cbor_struct_tags_api.svg?sanitize=1 "CBOR API and Go Struct Tags") + +API is mostly same as `encoding/json`, plus interfaces that simplify concurrency for CBOR options. + ## Quick Start __Install__: `go get github.com/fxamacker/cbor/v2` and `import "github.com/fxamacker/cbor/v2"`. @@ -112,22 +229,22 @@ Package level functions only use this library's default settings. They provide the "default mode" of encoding and decoding. ```go -// API matches encoding/json. +// API matches encoding/json for Marshal, Unmarshal, Encode, Decode, etc. b, err = cbor.Marshal(v) // encode v to []byte b err = cbor.Unmarshal(b, &v) // decode []byte b to v -encoder = cbor.NewEncoder(w) // create encoder with io.Writer w decoder = cbor.NewDecoder(r) // create decoder with io.Reader r -err = encoder.Encode(v) // encode v to a CBOR data item err = decoder.Decode(&v) // decode a CBOR data item to v -// v2.5.0 added new functions that can return remaining bytes. +// v2.5.0 added new functions that return remaining bytes. // UnmarshalFirst decodes first CBOR data item and returns remaining bytes. -// Unlike Unmarshal, extraneous data is not treated as an error by UnmarshalFirst. rest, err = cbor.UnmarshalFirst(b, &v) // decode []byte b to v // DiagnoseFirst translates first CBOR data item to text and returns remaining bytes. text, rest, err = cbor.DiagnoseFirst(b) // decode []byte b to Diagnostic Notation text + +// NOTE: Unmarshal returns ExtraneousDataError if there are remaining bytes, +// but new funcs UnmarshalFirst and DiagnoseFirst do not. ``` __IMPORTANT__: πŸ‘‰ CBOR settings allow trade-offs between speed, security, encoding size, etc. @@ -177,7 +294,75 @@ Default mode and custom modes automatically apply struct tags. Struct tags (`toarray`, `keyasint`, `omitempty`) reduce encoded size of structs. -
Example using struct tags

+

Example encoding 3-level nested Go struct to 1 byte CBOR

+ +https://go.dev/play/p/YxwvfPdFQG2 + +```Go +// Example encoding nested struct (with omitempty tag) +// - encoding/json: 18 byte JSON +// - fxamacker/cbor: 1 byte CBOR +package main + +import ( + "encoding/hex" + "encoding/json" + "fmt" + + "github.com/fxamacker/cbor/v2" +) + +type GrandChild struct { + Quux int `json:",omitempty"` +} + +type Child struct { + Baz int `json:",omitempty"` + Qux GrandChild `json:",omitempty"` +} + +type Parent struct { + Foo Child `json:",omitempty"` + Bar int `json:",omitempty"` +} + +func cb() { + results, _ := cbor.Marshal(Parent{}) + fmt.Println("hex(CBOR): " + hex.EncodeToString(results)) + + text, _ := cbor.Diagnose(results) // Diagnostic Notation + fmt.Println("DN: " + text) +} + +func js() { + results, _ := json.Marshal(Parent{}) + fmt.Println("hex(JSON): " + hex.EncodeToString(results)) + + text := string(results) // JSON + fmt.Println("JSON: " + text) +} + +func main() { + cb() + fmt.Println("-------------") + js() +} +``` + +Output (DN is Diagnostic Notation): +``` +hex(CBOR): a0 +DN: {} +------------- +hex(JSON): 7b22466f6f223a7b22517578223a7b7d7d7d +JSON: {"Foo":{"Qux":{}}} +``` + +

+ +
+ +
Example using several struct tags

![alt text](https://github.com/fxamacker/images/raw/master/cbor/v2.3.0/cbor_struct_tags_api.svg?sanitize=1 "CBOR API and Go Struct Tags") @@ -267,12 +452,14 @@ Default limits may need to be increased for systems handling very large data (e. ## Status -v2.5.0 was released on Sunday, August 13, 2023 with new features and important bug fixes. It is fuzz tested and production quality. +v2.5.0 was released on Sunday, August 13, 2023 with new features and important bug fixes. It is fuzz tested and production quality after extended beta [v2.5.0-beta](https://github.com/fxamacker/cbor/releases/tag/v2.5.0-beta) (Dec 2022) -> [v2.5.0](https://github.com/fxamacker/cbor/releases/tag/v2.5.0) (Aug 2023). __IMPORTANT__: πŸ‘‰ Before upgrading from v2.4 or older release, please read the notable changes highlighted in the release notes. v2.5.0 is a large release with bug fixes to error handling for extraneous data in `Unmarshal`, etc. that should be reviewed before upgrading. See [v2.5.0 release notes](https://github.com/fxamacker/cbor/releases/tag/v2.5.0) for list of new features, improvements, and bug fixes. +See ["Version and API Changes"](https://github.com/fxamacker/cbor#versions-and-api-changes) section for more info about version numbering, etc. +