Specifying graph builder's JSON format

[wanderer]

There might be a usecase for making the JSON format that ipld-graph-builer uses a specification. It would be usefully for doing things like partial or full dumps and imports of graphs. Also it could be used in ethereum for state imports. Related

@cdetrio @diasdavid

[daviddias]

I believe there are some indeed :) We've been discussing and evaluating the possibility of having several layouts in addition to the several chunkers options that we currently support on the fly.

Currently, unixfs-engine, which is a graph-builder in itself, supports 3 different layouts for files: balanced, trickle and flat, and two layouts for directories: sharding and non sharding. Plus, files can have several chunkers (tar, zip, fixed-size, etc). This project is currently known as DEX (Data Importers / Exporters) and several people showed interest on it, it matches super well with IPLD transformations too.

Notes of these discussions:

• Data Importers notes - IPLD Importers - endeavor notes ipfs/notes#144
• Moar notes - IPLD Data Importing - Set of Importers ipfs/js-ipfs#41 (comment)
• The name bikeshed (with motivations behind the project) Need a name for the data importers project -> DEX ipfs/notes#204
• related Small Languages for IPLD things (like IPRS) ipfs/notes#229

//cc @pgte @jbenet @nicola

[wanderer]

cool thank @diasdavid so i think the above issues are a bit different (let me know if im wrong here!). We already know the DAG structure, so we don't need to chuck the file or anything. We are assuming that data has already been merklized and turned into a DAG.

The goal here:

1. to have a JSON dump that is human readable.
2. to have full OR partially imports for example, lets say I have 90% of the graph already and just need to patch in a small update
   example:

update.json
{
   "alreadyHaveThisBranch": {
       "/":   "zdpuB3eZFjLEhDGiCaawbLyu56QfHGBNmr1zmZTkakCBpDFfE"
    },
   "newStuff": {
      "options": {
           "format": "dag-cbor",
          "hashAlg": "sha2-256"
       }
      "/": {
          "some": {
             "new": "data"
           }
       }
   }
}

[rvagg]

closing for archival