car: complete spec

Closes #229

README.md

@@ -58,6 +58,7 @@ This layer does not define data structures or types, although many codecs may co
 | [Specification: DAG-JSON](block-layer/codecs/DAG-JSON.md) | [block-layer/codecs/DAG-JSON.md](block-layer/codecs/DAG-JSON.md) |
 | [Specification: IPLD Selectors](selectors/selectors.md) | [selectors/selectors.md](selectors/selectors.md) |
 | [Specification: Graphsync](block-layer/graphsync/graphsync.md) | [block-layer/graphsync/graphsync.md](block-layer/graphsync/graphsync.md) |
+| [Specification: Content Addressable aRchives (CAR / .car)](block-layer/content-addressable-archives.md) | [block-layer/content-addressable-archives.md](block-layer/content-addressable-archives.md) |
 
 ### Data Model Layer (Layer 1)
 

block-layer/content-addressable-archives.md

@@ -0,0 +1,150 @@
+# Specification: Content Addressable aRchives (CAR / .car)
+
+**Status: Draft**
+
+## Summary
+
+The CAR format (Content Addressable aRchives) can be used to store content addressable objects in the form of IPLD block data as a sequence of bytes; typically in a file with a `.car` filename extension.
+
+The CAR format is intended as a serialized representation of any IPLD DAG (graph) as the concatenation of its blocks, plus a header that describes the graphs in the file (via root CIDs). The requirement for the blocks in a CAR to form coherent DAGs is not strict, so the CAR format may also be used to store any arbitrary list of IPLD blocks.
+
+In addition to the binary block data, storage overhead for the CAR format consists of:
+
+ * A header block encoded as [DAG-CBOR](codecs/dag-cbor.md) containing the format version and an array of root CIDs
+ * A CID for each block preceding its binary data
+ * A compressed integer prefixing each block, including the header block, indicating the total length of that block including CID
+
+This diagram shows how IPLD blocks, their root CID, and a header combine to form a CAR.
+
+![Content Addressable aRchive Diagram](content-addressable-archives.png)
+
+## Format Description
+
+The CAR format comprises a sequence of length-prefixed IPLD block data, where the first block in the CAR is the Header encoded as CBOR, and the remaining blocks form the Data component of the CAR and are each additionally prefixed with their CIDs. The length prefix of each block in a CAR is encoded as a "varint"—an unsigned [LEB128](https://en.wikipedia.org/wiki/LEB128) integer. This integer specifies the number of remaining bytes for that block entry—excluding the bytes used to encode the integer, but including the CID for non-header blocks.
+
+```
+|--------- Header --------| |---------------------------------- Data -----------------------------------|
+
+[ varint | dag-cbor block ] [ varint | CID | block ] [ varint | CID | block ] [ varint | CID | block ] …
+```
+
+### Header
+
+The first bytes of the CAR format hold a varint, this unsigned integer specifies the number of bytes beyond the varint itself that contain the _Header_ block. This Header block is a byte array DAG-CBOR (CBOR with tag 42 for CIDs) encoded object holding the version number and array of roots. As an [IPLD Schema](../schemas/):
+
+```ipldsch
+type CarHeader struct {
+  version Int
+  roots [&Any]
+}
+```
+
+#### Constraints
+
+* The `version` is always a value of `1`. Future iterations of this specification may make use of `version` to introduce variations of the format.
+* The `roots` array must contain **zero or more** CIDs, each of which must be present somewhere in the remainder of the CAR.
+
+An empty `roots` array is valid, but indicates that the CAR does not necessarily contain blocks forming a coherent DAG. Such a CAR may, for instance, form part of a collection of archives that, when combined, contain an entire DAG.
+
+### Data
+
+Immediately following the Header block, **zero or more** IPLD blocks are concatenated to form the _Data_ section of the CAR format. Each block is encoded into a _Section_ by the concatenation of the following values:
+
+1. Length in bytes of the combined CID and data in this Section, encoded as a varint
+2. CID of the block in this Section, encoded in the raw byte form of the CID
+3. Binary data of the block
+
+#### Length
+
+Each Section begins with a varint representation of an unsigned integer indicating the number of bytes containing the remainder of the section.
+
+#### CID
+
+Following the Length, the CID of the block is included in raw byte form. A decoder reading a Section must decode the CID according to CID byte encoding rules, which don't provide a stable length. See https://github.com/multiformats/cid for details on the encoding of a CID. CIDv0 and CIDv1 are currently supported.
+
+**CID byte decoding summary**
+
+_See the [CID specification](https://github.com/multiformats/cid) for full details._
+
+A CIDv0 is indicated by a first byte of `0x12` followed by `0x20` which specifies a 32-byte (`0x20`) length SHA2-256 ([`0x12`](https://github.com/multiformats/multicodec/blob/master/table.csv)) digest.
+
+Failure to find `0x12, 0x20` indicates a CIDv1 which is decoded by reading:
+
+1. Version as an unsigned varint (should be `1`)
+2. Codec as an unsigned varint (valid according to the [multicodec table](https://github.com/multiformats/multicodec/blob/master/table.csv))
+3. The raw bytes of a [multihash](https://github.com/multiformats/multihash)
+
+Reading the multihash requires a partial decode in order to determine the length:
+
+```
+| hash function code (varint) | digest size (varint) | digest |
+```
+
+The first two bytes of a multihash are varints, where the second varint is an unsigned integer indicating the length of the remaining portion of the multihash. Therefore, a manual decode requires two varint reads and then copying the bytes of those varints in addition to the number of bytes indicated by the second varint into a byte array.
+
+#### Data
+
+The remainder of a Section, after length-prefix and CID, comprises the raw byte data of the IPLD block. The encoded block may be any IPLD block format as specified by the codec in the CID. Typical codecs will be [DAG-PB](codecs/dag-pb.md), [DAG-CBOR](codecs/dag-cbor.md) or [RAW](https://github.com/ipld/specs/issues/223).
+
+## Additional Considerations
+
+### Performance
+
+Some considerations regarding performance:
+
+* **Streaming**: the CAR format is ideal for dumping blocks via streaming reads as the Header can be loaded first and minimal state is required for ongoing parsing.
+* **Individual block reads**: as the CAR format contains no index information, reads require either a partial scan to discover the location of a required block or an external index must be maintained and referenced for a seek and partial read of that data. See below regarding indexing.
+* **DAG traversal**: without an external index, traversal of a DAG specified by a "root" CID is not possible without dumping all blocks into a more convenient data store or by partial scans to find each block as required, which will likely be too inefficient to be practical.
+* **Modification**: CARs may be appended after initial write as there is no constraint in the Header regarding total length. Care must be taken in appending if a CAR is intended to contain coherent DAG data.
+
+### Security and Verifiability
+
+The roots specified by the Header of a CAR must appear somewhere in its Data section, however there is no requirement that the roots define entire DAGs, nor that all blocks in a CAR must be part of DAGs described by the root CIDs in the Header. Therefore, the roots must not be used alone to determine or differentiate the contents of a CAR.
+
+The CAR format contains no internal means, beyond the IPLD block formats and their CIDs, to verify or differentiate contents. Where such a requirement exists, this must be performed externally, such as creating a digest of the entire CAR.
+
+### Indexing and Seeking Reads
+
+The CAR format contains no internal indexing, any indexing must be stored externally to a CAR. However, such indexing is possible and makes seeking reads practical. An index storing byte offset (of section start or block data start) and length (of section or block data), keyed by CID, will enable a single block read by seeking to the offset and reading the block data. The format of any such index is not specified here and is left up to CAR format parsing implementations.
+
+### Padding
+
+The CAR format contains no specified means of padding to achieve specific total archive sizes or internal byte offset alignment of block data. Because it is not a requirement that each block be part of a coherent DAG under one of the roots of the CAR, dummy block entries may be used to achieve padding. Such padding should also account for the size of the length-prefix varint and the CID for a section. All sections must be valid and dummy entries should still decode to valid IPLD blocks.
+
+### Duplicate Blocks
+
+The possibility of duplicate blocks in a single CAR (such as for padding) is currently not specified.
+
+## Implementations
+
+### Go
+
+https://github.com/ipfs/go-car
+
+As used in Filecoin for genesis block sharing. Supports creation via a DAG walk from a datastore:
+
+```go
+WriteCar(ctx context.Context, ds format.DAGService, roots []cid.Cid, w io.Writer) (error)
+```
+
+And writing to a data store via `Put(block)` operations:
+
+```go
+LoadCar(s Store, r io.Reader) (*CarHeader, error)
+```
+
+### JavaScript
+
+https://github.com/rvagg/js-datastore-car
+
+Wraps in [Datastore](https://github.com/ipfs/interface-datastore) interface with various modes for reading and writing to support different use-cases—including streaming reading and writing:
+
+```js
+async CarDatastore.readBuffer(buffer)
+async CarDatastore.readFileComplete(file)
+async CarDatastore.readStreamComplete(stream)
+async CarDatastore.readStreaming(stream)
+async CarDatastore.writeStream(stream)
+```
+
+Also supports an `indexer()` that parses a file or stream and yields block index data including CID, offset and length, in addition to a `readRaw()` to read individual blocks according to their index data.