From 75d5e406a89df38f8d8be682e88e24bea84a2f00 Mon Sep 17 00:00:00 2001 From: nmalhotra Date: Thu, 20 Dec 2018 11:52:35 -0500 Subject: [PATCH 1/6] docs/code-flow : Add code flow documentation for add cmd. License: MIT Signed-off-by: Nitish Malhotra --- docs/add-code-flow.md | 97 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 docs/add-code-flow.md diff --git a/docs/add-code-flow.md b/docs/add-code-flow.md new file mode 100644 index 00000000000..221d93a3877 --- /dev/null +++ b/docs/add-code-flow.md @@ -0,0 +1,97 @@ +https://github.com/ipfs/go-ipfs/blob/master/core/commands/add.go#L208-L213 + +The goal of this document is to capture the code flow for adding a file (see the `coreapi` package) using the IPFS CLI, in the process exploring some datastructures and packages like `ipld.Node` (aka `dagnode`, `FSNode`, `MFS`, etc. + +# `UnixfsAPI.Add()` +*Entrypoint into the `Unixfs` package* + +https://github.com/ipfs/go-ipfs/blob/master/core/coreapi/unixfs.go#L78-L86 + +The `UnixfsAPI.Add()` acts on the input data or files, to build a _merkledag_ node (in essence it is the entire tree represented by the root node) and adds it to the _blockstore_. +Within the function, a new `Adder` is created with the configured `Blockstore` and __DAG service__`. + + +# `Adder.AddAllAndPin(files.File)` +*Entrypoint* + +https://github.com/ipfs/go-ipfs/blob/master/core/coreunix/add.go#L427-L431 + +The more interesting stuff happens in the `Adder.AddAllAndPin(files)` function. + +https://github.com/ipfs/go-ipfs/blob/master/core/coreunix/add.go#L467-L476 + +Let's focus on the simple case of a single file, handled by `Adder.addFileNode(path, file)` which redirects the case to `Adder.addFile(pathm files.File)`. + +## `Adder.addFile(path, files.File)` +*Create the _DAG_ and add to `MFS`* + +The `addFile(file)` method is responsible taking the data and converting it into a __DAG__ tree, followed by adding the root of the DAG tree in to the `MFS`. + +https://github.com/ipfs/go-ipfs/blob/master/core/coreunix/add.go#L508-L521 + +There are two main methods to focus on - + +### `Adder.add(io.Reader)` +*Create and return the **root** __DAG__ node* + +https://github.com/ipfs/go-ipfs/blob/master/core/coreunix/add.go#L115-L137 + +This method converts the input _data_ (`io.Reader`) to a `DAG` tree. This is done by splitting the data into _chunks_ using the `Chunker` **(HELP: elaborate on chunker types ?)** and organizing them in a `DAG` (with a *trickle* or *balanced* layout). The method returns the **root** of the __DAG__, formatted as an `ipld.Node`. + +### `Adder.addNode(ipld.Node, path)` +*Add **root** __DAG__ node to the `MFS`* + +https://github.com/ipfs/go-ipfs/blob/master/core/coreunix/add.go#L365-L399 + +Now that we have the **root** node of the `DAG`, this needs to be added to the `MFS` file system. +The `MFS` **root** is first fetched (or created, if doesn't already exist) by invoking `mfsRoot()`. +Assuming the directory structure already exists in the MFS file system, (if it doesn't exist it will be created using `mfs.Mkdir()` function by passing in the `MFS` **root**), the **root** __DAG__ node is added to the `MFS` File system within the `mfs.PutNode()` function. + +#### `[MFS] PutNode(mfs.Root, path, ipld.Node)` +*Insert node at path into given `MFS`* + +https://github.com/ipfs/go-mfs/blob/master/ops.go#L101-L113 + +In this the path is used to determine the `MFS` `Directory`, which is first lookup up in the `MFS` using `lookupDir()` function. This is followed by adding the **root** __DAG__ node (`ipld.Node`) in to the found `Directory` using `directory.AddChild()` method. + +#### - `directory.AddChild(filename, ipld.Node)` +*Add **root** __DAG__ node , as filename, under this directory* + +https://github.com/ipfs/go-mfs/blob/master/dir.go#L381-L402 + +Within this method the node is added to the __DAG service__ of the `Directory` object using the `dserv.Add()` method [HELP NEEDED]. +This is subsequently followed by adding the **root** __DAG__ node by creating a `directory.child{}` object with the given name, within in the `directory.addUnixFSChild(directory.child{name, ipld.Node})` method. + +#### -- `directory.addUnixFSChild(child)` +*Switch to HAMT (if configured) and add child to inner UnixFS Directory* + +https://github.com/ipfs/go-mfs/blob/master/dir.go#L406-L425 + +Here the transition of the UnixFS directory to __HAMT__ implemetation is done, if configured, wherein if the directory is of type `BasicDirectory`, it is converted to a __HAMT__ implementation. +The node is then added as a child to the inner `UnixFS` directory using the `directory.AddChild()` method. +Note: This is not to be confused with the `directory.AddChild(filename, ipld.Node)`, as this operates on the inner `UnixFS` `Directory` object only. + +#### --- (inner)`Directory.AddChild(ctx, name, ipld.Node)` + +This method vastly differs based on the inner `Directory` implementation (Basic vs HAMT). Let's focus on the `BasicDirectory` implementation to keep things simple. + +https://github.com/ipfs/go-unixfs/blob/master/io/directory.go#L142-L147 + +> IMPORTANT +> It should be noted that the inner `Directory` of the `UnixFS` package, encasulates a node object of type `ProtoNode`, which is a different format as compared to the `ipld.Node` we have been working on throughout this document. + +This method first attempts to remove any old links (`ProtoNode.RemoveNodeLink(name)`) to the `ProtoNode` prior to adding a link to the newly added `ipld/Node`, using `ProtoNode.AddNodeLink(name, ipld.Node)`. + +https://github.com/ipfs/go-merkledag/blob/master/node.go#L99-L112 + +The `AddNodeLink()` method is where an `ipld.Link` is created with the `ipld.Node`'s `CID` and size in the `ipld.MakeLink(ipld.Node)` method, and is then appended to the `ProtoNode`'s links in the `ProtoNode.AddRawLink(name)` method. + +--- + +## `adder.Finalize()` +*Fetch and return the __DAG__ **root** from the `MFS` and `UnixFS` directory* + +https://github.com/ipfs/go-ipfs/blob/master/core/coreunix/add.go#L199-L244 + +The whole process ends with `adder.Finalize()` which returns the `ipld.Node` from the `UnixFS` `Directory`. +**(HELP: Do we need to elaborate ?)** \ No newline at end of file From e229bc27c0472b1673bee308b3364e92975ce371 Mon Sep 17 00:00:00 2001 From: nmalhotra Date: Sun, 6 Jan 2019 18:48:40 -0500 Subject: [PATCH 2/6] Cleanup and resolve docs - Cleaned up some of the language. - Added versioned links to the code. - Provided examples License: MIT Signed-off-by: Nitish Malhotra --- docs/add-code-flow.md | 124 +++++++++++++++++++++--------------------- 1 file changed, 63 insertions(+), 61 deletions(-) diff --git a/docs/add-code-flow.md b/docs/add-code-flow.md index 221d93a3877..fa3bf92f400 100644 --- a/docs/add-code-flow.md +++ b/docs/add-code-flow.md @@ -1,97 +1,99 @@ -https://github.com/ipfs/go-ipfs/blob/master/core/commands/add.go#L208-L213 +# IPFS : The `Add` command demystified + +https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/commands/add.go#L208-L213 The goal of this document is to capture the code flow for adding a file (see the `coreapi` package) using the IPFS CLI, in the process exploring some datastructures and packages like `ipld.Node` (aka `dagnode`, `FSNode`, `MFS`, etc. -# `UnixfsAPI.Add()` -*Entrypoint into the `Unixfs` package* +**Try this yourself** +> +> ``` +> # Convert a file to the IPFS format. +> echo "Hello World" > new-file +> ipfs add new-file +> added QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u new-file +> 12 B / 12 B [=========================================================] 100.00% +> +> # Add a file to the MFS. +> NEW_FILE_HASH=$(ipfs add new-file -Q) +> ipfs files cp /ipfs/$NEW_FILE_HASH /new-file +> +> # Get information from the file in MFS. +> ipfs files stat /new-file +> # QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u +> # Size: 12 +> # CumulativeSize: 20 +> # ChildBlocks: 0 +> # Type: file +> +> # Retrieve the contents. +> ipfs files read /new-file +> # Hello World +> ``` + +--- -https://github.com/ipfs/go-ipfs/blob/master/core/coreapi/unixfs.go#L78-L86 +# **[`UnixfsAPI.Add()`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreapi/unixfs.go#L31)** - *Entrypoint into the `Unixfs` package* The `UnixfsAPI.Add()` acts on the input data or files, to build a _merkledag_ node (in essence it is the entire tree represented by the root node) and adds it to the _blockstore_. Within the function, a new `Adder` is created with the configured `Blockstore` and __DAG service__`. +## **[`adder.AddAllAndPin(files)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L403)** - *Entrypoint to the `Add` logic* + encapsulates a lot of the underlying functionality that will be investigated in the following sections. -# `Adder.AddAllAndPin(files.File)` -*Entrypoint* - -https://github.com/ipfs/go-ipfs/blob/master/core/coreunix/add.go#L427-L431 - -The more interesting stuff happens in the `Adder.AddAllAndPin(files)` function. - -https://github.com/ipfs/go-ipfs/blob/master/core/coreunix/add.go#L467-L476 +Our focus will be on the simplest case, a single file, handled by `Adder.addFileNode(path, file)` handled by `Adder.addFile(pathm files.File)`. -Let's focus on the simple case of a single file, handled by `Adder.addFileNode(path, file)` which redirects the case to `Adder.addFile(pathm files.File)`. +- **[`adder.addFile(path, files.File)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L450)** - *Create the _DAG_ and add to `MFS`* -## `Adder.addFile(path, files.File)` -*Create the _DAG_ and add to `MFS`* + The `addFile(file)` method takes the data and converts it into a __DAG__ tree and adds the root of the tree in to the `MFS`. -The `addFile(file)` method is responsible taking the data and converting it into a __DAG__ tree, followed by adding the root of the DAG tree in to the `MFS`. + https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L508-L521 -https://github.com/ipfs/go-ipfs/blob/master/core/coreunix/add.go#L508-L521 + There are two main methods to focus on - -There are two main methods to focus on - + 1. **[`adder.add(io.Reader)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L115)** - *Create and return the **root** __DAG__ node* -### `Adder.add(io.Reader)` -*Create and return the **root** __DAG__ node* + This method converts the input data (`io.Reader`) to a __DAG__ tree, by splitting the data into _chunks_ using the `Chunker` and organizing them in to a __DAG__ (with a *trickle* or *balanced* layout. See [balanced](https://github.com/ipfs/go-unixfs/blob/6b769632e7eb8fe8f302e3f96bf5569232e7a3ee/importer/balanced/builder.go) for more info). -https://github.com/ipfs/go-ipfs/blob/master/core/coreunix/add.go#L115-L137 + The method returns the **root** `ipld.Node` of the __DAG__. -This method converts the input _data_ (`io.Reader`) to a `DAG` tree. This is done by splitting the data into _chunks_ using the `Chunker` **(HELP: elaborate on chunker types ?)** and organizing them in a `DAG` (with a *trickle* or *balanced* layout). The method returns the **root** of the __DAG__, formatted as an `ipld.Node`. + 2. **[`adder.addNode(ipld.Node, path)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L366)** - *Add **root** __DAG__ node to the `MFS`* -### `Adder.addNode(ipld.Node, path)` -*Add **root** __DAG__ node to the `MFS`* + Now that we have the **root** node of the `DAG`, this needs to be added to the `MFS` file system. + Fetch (or create, if doesn't already exist) the `MFS` **root** using `mfsRoot()`. -https://github.com/ipfs/go-ipfs/blob/master/core/coreunix/add.go#L365-L399 + > NOTE: The `MFS` **root** is an ephemeral root, created and destroyed solely for the `add` functionality. -Now that we have the **root** node of the `DAG`, this needs to be added to the `MFS` file system. -The `MFS` **root** is first fetched (or created, if doesn't already exist) by invoking `mfsRoot()`. -Assuming the directory structure already exists in the MFS file system, (if it doesn't exist it will be created using `mfs.Mkdir()` function by passing in the `MFS` **root**), the **root** __DAG__ node is added to the `MFS` File system within the `mfs.PutNode()` function. + Assuming the directory already exists in the MFS file system, (if it doesn't exist it will be created using `mfs.Mkdir()`), the **root** __DAG__ node is added to the `MFS` File system using the `mfs.PutNode()` function. -#### `[MFS] PutNode(mfs.Root, path, ipld.Node)` -*Insert node at path into given `MFS`* + - **[MFS] [`PutNode(mfs.Root, path, ipld.Node)`](https://github.com/ipfs/go-mfs/blob/master/ops.go#L101)** - *Insert node at path into given `MFS`* -https://github.com/ipfs/go-mfs/blob/master/ops.go#L101-L113 + The `path` param is used to determine the `MFS Directory`, which is first looked up in the `MFS` using `lookupDir()` function. This is followed by adding the **root** __DAG__ node (`ipld.Node`) in to this `Directory` using `directory.AddChild()` method. -In this the path is used to determine the `MFS` `Directory`, which is first lookup up in the `MFS` using `lookupDir()` function. This is followed by adding the **root** __DAG__ node (`ipld.Node`) in to the found `Directory` using `directory.AddChild()` method. + - **[MFS] Add Child To `UnixFS`** + - **[`directory.AddChild(filename, ipld.Node)`](https://github.com/ipfs/go-mfs/blob/master/dir.go#L374)** - *Add **root** __DAG__ node under this directory* -#### - `directory.AddChild(filename, ipld.Node)` -*Add **root** __DAG__ node , as filename, under this directory* + Within this method the node is added to the `Directory`'s __DAG service__ using the `dserv.Add()` method, followed by adding the **root** __DAG__ node with the given name, in the `directory.addUnixFSChild(directory.child{name, ipld.Node})` method. -https://github.com/ipfs/go-mfs/blob/master/dir.go#L381-L402 + - **[MFS] [`directory.addUnixFSChild(child)`](https://github.com/ipfs/go-mfs/blob/master/dir.go#L374)** - *Add child to inner UnixFS Directory* -Within this method the node is added to the __DAG service__ of the `Directory` object using the `dserv.Add()` method [HELP NEEDED]. -This is subsequently followed by adding the **root** __DAG__ node by creating a `directory.child{}` object with the given name, within in the `directory.addUnixFSChild(directory.child{name, ipld.Node})` method. + The node is then added as a child to the inner `UnixFS` directory using the `(BasicDirectory).AddChild()` method. -#### -- `directory.addUnixFSChild(child)` -*Switch to HAMT (if configured) and add child to inner UnixFS Directory* + > NOTE: This is not to be confused with the `directory.AddChild(filename, ipld.Node)`, as this operates on the `UnixFS` `BasicDirectory` object only. -https://github.com/ipfs/go-mfs/blob/master/dir.go#L406-L425 + - **[UnixFS] [`(BasicDirectory).AddChild(ctx, name, ipld.Node)`](https://github.com/ipfs/go-unixfs/blob/master/io/directory.go#L142)** - *Add child to `BasicDirectory`* -Here the transition of the UnixFS directory to __HAMT__ implemetation is done, if configured, wherein if the directory is of type `BasicDirectory`, it is converted to a __HAMT__ implementation. -The node is then added as a child to the inner `UnixFS` directory using the `directory.AddChild()` method. -Note: This is not to be confused with the `directory.AddChild(filename, ipld.Node)`, as this operates on the inner `UnixFS` `Directory` object only. + > IMPORTANT: It should be noted that the `BasicDirectory` struct of the `UnixFS` package, encasulates a node object of type `ProtoNode`, which is a different format from the `ipld.Node` we have been working on throughout this document. -#### --- (inner)`Directory.AddChild(ctx, name, ipld.Node)` + This method first attempts to remove any old links (`ProtoNode.RemoveNodeLink(name)`) to the `ProtoNode` prior to adding a link to the newly added `ipld.Node`, using `ProtoNode.AddNodeLink(name, ipld.Node)`. -This method vastly differs based on the inner `Directory` implementation (Basic vs HAMT). Let's focus on the `BasicDirectory` implementation to keep things simple. + - **[Merkledag] [`AddNodeLink()`](https://github.com/ipfs/go-unixfs/blob/master/io/directory.go#L142)** -https://github.com/ipfs/go-unixfs/blob/master/io/directory.go#L142-L147 + The `AddNodeLink()` method is where an `ipld.Link` is created with the `ipld.Node`'s `CID` and size in the `ipld.MakeLink(ipld.Node)` method, and is then appended to the `ProtoNode`'s links in the `ProtoNode.AddRawLink(name)` method. -> IMPORTANT -> It should be noted that the inner `Directory` of the `UnixFS` package, encasulates a node object of type `ProtoNode`, which is a different format as compared to the `ipld.Node` we have been working on throughout this document. - -This method first attempts to remove any old links (`ProtoNode.RemoveNodeLink(name)`) to the `ProtoNode` prior to adding a link to the newly added `ipld/Node`, using `ProtoNode.AddNodeLink(name, ipld.Node)`. - -https://github.com/ipfs/go-merkledag/blob/master/node.go#L99-L112 - -The `AddNodeLink()` method is where an `ipld.Link` is created with the `ipld.Node`'s `CID` and size in the `ipld.MakeLink(ipld.Node)` method, and is then appended to the `ProtoNode`'s links in the `ProtoNode.AddRawLink(name)` method. - ---- +- **[`adder.Finalize()`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L200)** - *Fetch and return the __DAG__ **root** from the `MFS` and `UnixFS` directory* -## `adder.Finalize()` -*Fetch and return the __DAG__ **root** from the `MFS` and `UnixFS` directory* + The `Finalize` method returns the `ipld.Node` from the `UnixFS` `Directory`. -https://github.com/ipfs/go-ipfs/blob/master/core/coreunix/add.go#L199-L244 +- **[`adder.PinRoot()`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L171)** - *Pin all files under the `MFS` **root*** -The whole process ends with `adder.Finalize()` which returns the `ipld.Node` from the `UnixFS` `Directory`. -**(HELP: Do we need to elaborate ?)** \ No newline at end of file + The whole process ends with `PinRoot` recursively pinning all the files under the `MFS` **root** \ No newline at end of file From 9e0872338f2ff088a12fe6e87c98439ab554829c Mon Sep 17 00:00:00 2001 From: nmalhotra Date: Thu, 10 Jan 2019 20:36:45 -0500 Subject: [PATCH 3/6] Updated the code reference links - go-mfs linked to v0.1.18 - go-unifx linked to v1.1.16 However, - go-merkledag has been linked to v1.1.13 which is not the right version. v1.1.15 has not been released and cannot be referenced License: MIT Signed-off-by: Nitish Malhotra --- docs/add-code-flow.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/add-code-flow.md b/docs/add-code-flow.md index fa3bf92f400..8ccdd3f7950 100644 --- a/docs/add-code-flow.md +++ b/docs/add-code-flow.md @@ -65,29 +65,31 @@ Our focus will be on the simplest case, a single file, handled by `Adder.addFile Assuming the directory already exists in the MFS file system, (if it doesn't exist it will be created using `mfs.Mkdir()`), the **root** __DAG__ node is added to the `MFS` File system using the `mfs.PutNode()` function. - - **[MFS] [`PutNode(mfs.Root, path, ipld.Node)`](https://github.com/ipfs/go-mfs/blob/master/ops.go#L101)** - *Insert node at path into given `MFS`* + - **[MFS] [`PutNode(mfs.Root, path, ipld.Node)`](https://github.com/ipfs/go-mfs/blob/v0.1.18/ops.go#L86)** - *Insert node at path into given `MFS`* The `path` param is used to determine the `MFS Directory`, which is first looked up in the `MFS` using `lookupDir()` function. This is followed by adding the **root** __DAG__ node (`ipld.Node`) in to this `Directory` using `directory.AddChild()` method. - **[MFS] Add Child To `UnixFS`** - - **[`directory.AddChild(filename, ipld.Node)`](https://github.com/ipfs/go-mfs/blob/master/dir.go#L374)** - *Add **root** __DAG__ node under this directory* + - **[`directory.AddChild(filename, ipld.Node)`](https://github.com/ipfs/go-mfs/blob/v0.1.18/dir.go#L350)** - *Add **root** __DAG__ node under this directory* Within this method the node is added to the `Directory`'s __DAG service__ using the `dserv.Add()` method, followed by adding the **root** __DAG__ node with the given name, in the `directory.addUnixFSChild(directory.child{name, ipld.Node})` method. - - **[MFS] [`directory.addUnixFSChild(child)`](https://github.com/ipfs/go-mfs/blob/master/dir.go#L374)** - *Add child to inner UnixFS Directory* + - **[MFS] [`directory.addUnixFSChild(child)`](https://github.com/ipfs/go-mfs/blob/v0.1.18/dir.go#L375)** - *Add child to inner UnixFS Directory* The node is then added as a child to the inner `UnixFS` directory using the `(BasicDirectory).AddChild()` method. > NOTE: This is not to be confused with the `directory.AddChild(filename, ipld.Node)`, as this operates on the `UnixFS` `BasicDirectory` object only. - - **[UnixFS] [`(BasicDirectory).AddChild(ctx, name, ipld.Node)`](https://github.com/ipfs/go-unixfs/blob/master/io/directory.go#L142)** - *Add child to `BasicDirectory`* + - **[UnixFS] [`(BasicDirectory).AddChild(ctx, name, ipld.Node)`](https://github.com/ipfs/go-unixfs/blob/v1.1.16/io/directory.go#L137)** - *Add child to `BasicDirectory`* > IMPORTANT: It should be noted that the `BasicDirectory` struct of the `UnixFS` package, encasulates a node object of type `ProtoNode`, which is a different format from the `ipld.Node` we have been working on throughout this document. This method first attempts to remove any old links (`ProtoNode.RemoveNodeLink(name)`) to the `ProtoNode` prior to adding a link to the newly added `ipld.Node`, using `ProtoNode.AddNodeLink(name, ipld.Node)`. - - **[Merkledag] [`AddNodeLink()`](https://github.com/ipfs/go-unixfs/blob/master/io/directory.go#L142)** + - **[Merkledag] [`AddNodeLink()`](https://github.com/ipfs/go-merkledag/blob/v1.1.13/node.go#L99)** + **HELP: The go-merkledag version is supposed to v1.1.15 for this example, however that version has not been added to the [releases](https://github.com/ipfs/go-merkledag/releases)** + The `AddNodeLink()` method is where an `ipld.Link` is created with the `ipld.Node`'s `CID` and size in the `ipld.MakeLink(ipld.Node)` method, and is then appended to the `ProtoNode`'s links in the `ProtoNode.AddRawLink(name)` method. - **[`adder.Finalize()`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L200)** - *Fetch and return the __DAG__ **root** from the `MFS` and `UnixFS` directory* From d304a2faec4d9f700ab8d2e17da228fd8be8b9bf Mon Sep 17 00:00:00 2001 From: nmalhotra Date: Thu, 10 Jan 2019 22:27:36 -0500 Subject: [PATCH 4/6] Fixed the indentation of the sections - Moved AddAllAndPin as a subsection of UnixfsAPI.Add() License: MIT Signed-off-by: Nitish Malhotra --- docs/add-code-flow.md | 70 ++++++++++++++++++++++--------------------- 1 file changed, 36 insertions(+), 34 deletions(-) diff --git a/docs/add-code-flow.md b/docs/add-code-flow.md index 8ccdd3f7950..3cc7cbf0f2e 100644 --- a/docs/add-code-flow.md +++ b/docs/add-code-flow.md @@ -32,70 +32,72 @@ The goal of this document is to capture the code flow for adding a file (see the --- -# **[`UnixfsAPI.Add()`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreapi/unixfs.go#L31)** - *Entrypoint into the `Unixfs` package* +# Code Flow + +**[`UnixfsAPI.Add()`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreapi/unixfs.go#L31)** - *Entrypoint into the `Unixfs` package* The `UnixfsAPI.Add()` acts on the input data or files, to build a _merkledag_ node (in essence it is the entire tree represented by the root node) and adds it to the _blockstore_. Within the function, a new `Adder` is created with the configured `Blockstore` and __DAG service__`. -## **[`adder.AddAllAndPin(files)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L403)** - *Entrypoint to the `Add` logic* - encapsulates a lot of the underlying functionality that will be investigated in the following sections. +- **[`adder.AddAllAndPin(files)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L403)** - *Entrypoint to the `Add` logic* + encapsulates a lot of the underlying functionality that will be investigated in the following sections. -Our focus will be on the simplest case, a single file, handled by `Adder.addFileNode(path, file)` handled by `Adder.addFile(pathm files.File)`. + Our focus will be on the simplest case, a single file, handled by `Adder.addFileNode(path, file)` handled by `Adder.addFile(pathm files.File)`. -- **[`adder.addFile(path, files.File)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L450)** - *Create the _DAG_ and add to `MFS`* + - **[`adder.addFile(path, files.File)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L450)** - *Create the _DAG_ and add to `MFS`* - The `addFile(file)` method takes the data and converts it into a __DAG__ tree and adds the root of the tree in to the `MFS`. + The `addFile(file)` method takes the data and converts it into a __DAG__ tree and adds the root of the tree in to the `MFS`. - https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L508-L521 + https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L508-L521 - There are two main methods to focus on - + There are two main methods to focus on - - 1. **[`adder.add(io.Reader)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L115)** - *Create and return the **root** __DAG__ node* + 1. **[`adder.add(io.Reader)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L115)** - *Create and return the **root** __DAG__ node* - This method converts the input data (`io.Reader`) to a __DAG__ tree, by splitting the data into _chunks_ using the `Chunker` and organizing them in to a __DAG__ (with a *trickle* or *balanced* layout. See [balanced](https://github.com/ipfs/go-unixfs/blob/6b769632e7eb8fe8f302e3f96bf5569232e7a3ee/importer/balanced/builder.go) for more info). + This method converts the input data (`io.Reader`) to a __DAG__ tree, by splitting the data into _chunks_ using the `Chunker` and organizing them in to a __DAG__ (with a *trickle* or *balanced* layout. See [balanced](https://github.com/ipfs/go-unixfs/blob/6b769632e7eb8fe8f302e3f96bf5569232e7a3ee/importer/balanced/builder.go) for more info). - The method returns the **root** `ipld.Node` of the __DAG__. + The method returns the **root** `ipld.Node` of the __DAG__. - 2. **[`adder.addNode(ipld.Node, path)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L366)** - *Add **root** __DAG__ node to the `MFS`* + 2. **[`adder.addNode(ipld.Node, path)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L366)** - *Add **root** __DAG__ node to the `MFS`* - Now that we have the **root** node of the `DAG`, this needs to be added to the `MFS` file system. - Fetch (or create, if doesn't already exist) the `MFS` **root** using `mfsRoot()`. + Now that we have the **root** node of the `DAG`, this needs to be added to the `MFS` file system. + Fetch (or create, if doesn't already exist) the `MFS` **root** using `mfsRoot()`. - > NOTE: The `MFS` **root** is an ephemeral root, created and destroyed solely for the `add` functionality. + > NOTE: The `MFS` **root** is an ephemeral root, created and destroyed solely for the `add` functionality. - Assuming the directory already exists in the MFS file system, (if it doesn't exist it will be created using `mfs.Mkdir()`), the **root** __DAG__ node is added to the `MFS` File system using the `mfs.PutNode()` function. + Assuming the directory already exists in the MFS file system, (if it doesn't exist it will be created using `mfs.Mkdir()`), the **root** __DAG__ node is added to the `MFS` File system using the `mfs.PutNode()` function. - - **[MFS] [`PutNode(mfs.Root, path, ipld.Node)`](https://github.com/ipfs/go-mfs/blob/v0.1.18/ops.go#L86)** - *Insert node at path into given `MFS`* + - **[MFS] [`PutNode(mfs.Root, path, ipld.Node)`](https://github.com/ipfs/go-mfs/blob/v0.1.18/ops.go#L86)** - *Insert node at path into given `MFS`* - The `path` param is used to determine the `MFS Directory`, which is first looked up in the `MFS` using `lookupDir()` function. This is followed by adding the **root** __DAG__ node (`ipld.Node`) in to this `Directory` using `directory.AddChild()` method. + The `path` param is used to determine the `MFS Directory`, which is first looked up in the `MFS` using `lookupDir()` function. This is followed by adding the **root** __DAG__ node (`ipld.Node`) in to this `Directory` using `directory.AddChild()` method. - - **[MFS] Add Child To `UnixFS`** - - **[`directory.AddChild(filename, ipld.Node)`](https://github.com/ipfs/go-mfs/blob/v0.1.18/dir.go#L350)** - *Add **root** __DAG__ node under this directory* + - **[MFS] Add Child To `UnixFS`** + - **[`directory.AddChild(filename, ipld.Node)`](https://github.com/ipfs/go-mfs/blob/v0.1.18/dir.go#L350)** - *Add **root** __DAG__ node under this directory* - Within this method the node is added to the `Directory`'s __DAG service__ using the `dserv.Add()` method, followed by adding the **root** __DAG__ node with the given name, in the `directory.addUnixFSChild(directory.child{name, ipld.Node})` method. + Within this method the node is added to the `Directory`'s __DAG service__ using the `dserv.Add()` method, followed by adding the **root** __DAG__ node with the given name, in the `directory.addUnixFSChild(directory.child{name, ipld.Node})` method. - - **[MFS] [`directory.addUnixFSChild(child)`](https://github.com/ipfs/go-mfs/blob/v0.1.18/dir.go#L375)** - *Add child to inner UnixFS Directory* + - **[MFS] [`directory.addUnixFSChild(child)`](https://github.com/ipfs/go-mfs/blob/v0.1.18/dir.go#L375)** - *Add child to inner UnixFS Directory* - The node is then added as a child to the inner `UnixFS` directory using the `(BasicDirectory).AddChild()` method. + The node is then added as a child to the inner `UnixFS` directory using the `(BasicDirectory).AddChild()` method. - > NOTE: This is not to be confused with the `directory.AddChild(filename, ipld.Node)`, as this operates on the `UnixFS` `BasicDirectory` object only. + > NOTE: This is not to be confused with the `directory.AddChild(filename, ipld.Node)`, as this operates on the `UnixFS` `BasicDirectory` object only. - - **[UnixFS] [`(BasicDirectory).AddChild(ctx, name, ipld.Node)`](https://github.com/ipfs/go-unixfs/blob/v1.1.16/io/directory.go#L137)** - *Add child to `BasicDirectory`* + - **[UnixFS] [`(BasicDirectory).AddChild(ctx, name, ipld.Node)`](https://github.com/ipfs/go-unixfs/blob/v1.1.16/io/directory.go#L137)** - *Add child to `BasicDirectory`* - > IMPORTANT: It should be noted that the `BasicDirectory` struct of the `UnixFS` package, encasulates a node object of type `ProtoNode`, which is a different format from the `ipld.Node` we have been working on throughout this document. + > IMPORTANT: It should be noted that the `BasicDirectory` struct of the `UnixFS` package, encasulates a node object of type `ProtoNode`, which is a different format from the `ipld.Node` we have been working on throughout this document. - This method first attempts to remove any old links (`ProtoNode.RemoveNodeLink(name)`) to the `ProtoNode` prior to adding a link to the newly added `ipld.Node`, using `ProtoNode.AddNodeLink(name, ipld.Node)`. + This method first attempts to remove any old links (`ProtoNode.RemoveNodeLink(name)`) to the `ProtoNode` prior to adding a link to the newly added `ipld.Node`, using `ProtoNode.AddNodeLink(name, ipld.Node)`. - - **[Merkledag] [`AddNodeLink()`](https://github.com/ipfs/go-merkledag/blob/v1.1.13/node.go#L99)** + - **[Merkledag] [`AddNodeLink()`](https://github.com/ipfs/go-merkledag/blob/v1.1.13/node.go#L99)** - **HELP: The go-merkledag version is supposed to v1.1.15 for this example, however that version has not been added to the [releases](https://github.com/ipfs/go-merkledag/releases)** - - The `AddNodeLink()` method is where an `ipld.Link` is created with the `ipld.Node`'s `CID` and size in the `ipld.MakeLink(ipld.Node)` method, and is then appended to the `ProtoNode`'s links in the `ProtoNode.AddRawLink(name)` method. + **HELP: The go-merkledag version is supposed to v1.1.15 for this example, however that version has not been added to the [releases](https://github.com/ipfs/go-merkledag/releases)** + + The `AddNodeLink()` method is where an `ipld.Link` is created with the `ipld.Node`'s `CID` and size in the `ipld.MakeLink(ipld.Node)` method, and is then appended to the `ProtoNode`'s links in the `ProtoNode.AddRawLink(name)` method. -- **[`adder.Finalize()`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L200)** - *Fetch and return the __DAG__ **root** from the `MFS` and `UnixFS` directory* + - **[`adder.Finalize()`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L200)** - *Fetch and return the __DAG__ **root** from the `MFS` and `UnixFS` directory* - The `Finalize` method returns the `ipld.Node` from the `UnixFS` `Directory`. + The `Finalize` method returns the `ipld.Node` from the `UnixFS` `Directory`. -- **[`adder.PinRoot()`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L171)** - *Pin all files under the `MFS` **root*** + - **[`adder.PinRoot()`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L171)** - *Pin all files under the `MFS` **root*** The whole process ends with `PinRoot` recursively pinning all the files under the `MFS` **root** \ No newline at end of file From 7dbdb0e813c565532d5b0526d2b9cc4b367d0d5f Mon Sep 17 00:00:00 2001 From: nmalhotra Date: Mon, 14 Jan 2019 21:53:26 -0500 Subject: [PATCH 5/6] Document cleanup and additional references - Cleaned up the document. - Added additional references to documents. License: MIT Signed-off-by: Nitish Malhotra --- docs/add-code-flow.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/add-code-flow.md b/docs/add-code-flow.md index 3cc7cbf0f2e..ba96cfea91f 100644 --- a/docs/add-code-flow.md +++ b/docs/add-code-flow.md @@ -1,8 +1,6 @@ # IPFS : The `Add` command demystified -https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/commands/add.go#L208-L213 - -The goal of this document is to capture the code flow for adding a file (see the `coreapi` package) using the IPFS CLI, in the process exploring some datastructures and packages like `ipld.Node` (aka `dagnode`, `FSNode`, `MFS`, etc. +The goal of this document is to capture the code flow for adding a file (see the `coreapi` package) using the IPFS CLI, in the process exploring some datastructures and packages like `ipld.Node` (aka `dagnode`), `FSNode`, `MFS`, etc. **Try this yourself** > @@ -42,11 +40,11 @@ Within the function, a new `Adder` is created with the configured `Blockstore` a - **[`adder.AddAllAndPin(files)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L403)** - *Entrypoint to the `Add` logic* encapsulates a lot of the underlying functionality that will be investigated in the following sections. - Our focus will be on the simplest case, a single file, handled by `Adder.addFileNode(path, file)` handled by `Adder.addFile(pathm files.File)`. + Our focus will be on the simplest case, a single file, handled by `Adder.addFile(file files.File)`. - - **[`adder.addFile(path, files.File)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L450)** - *Create the _DAG_ and add to `MFS`* + - **[`adder.addFile(file files.File)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L450)** - *Create the _DAG_ and add to `MFS`* - The `addFile(file)` method takes the data and converts it into a __DAG__ tree and adds the root of the tree in to the `MFS`. + The `addFile(file)` method takes the data and converts it into a __DAG__ tree and adds the root of the tree into the `MFS`. https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L508-L521 @@ -80,18 +78,16 @@ Within the function, a new `Adder` is created with the configured `Blockstore` a The node is then added as a child to the inner `UnixFS` directory using the `(BasicDirectory).AddChild()` method. - > NOTE: This is not to be confused with the `directory.AddChild(filename, ipld.Node)`, as this operates on the `UnixFS` `BasicDirectory` object only. + > NOTE: This is not to be confused with the `directory.AddChild(filename, ipld.Node)`, as this operates on the `UnixFS` `BasicDirectory` object. - **[UnixFS] [`(BasicDirectory).AddChild(ctx, name, ipld.Node)`](https://github.com/ipfs/go-unixfs/blob/v1.1.16/io/directory.go#L137)** - *Add child to `BasicDirectory`* - > IMPORTANT: It should be noted that the `BasicDirectory` struct of the `UnixFS` package, encasulates a node object of type `ProtoNode`, which is a different format from the `ipld.Node` we have been working on throughout this document. + > IMPORTANT: It should be noted that the `BasicDirectory` object uses the `ProtoNode` type object which is an implementation of the `ipld.Node` interface, seen and used throughout this document. Ideally the `ipld.Node` should always be used, unless we need access tp specific functions from `ProtoNode` (like `Copy()`) that are not available in the interface. This method first attempts to remove any old links (`ProtoNode.RemoveNodeLink(name)`) to the `ProtoNode` prior to adding a link to the newly added `ipld.Node`, using `ProtoNode.AddNodeLink(name, ipld.Node)`. - - **[Merkledag] [`AddNodeLink()`](https://github.com/ipfs/go-merkledag/blob/v1.1.13/node.go#L99)** + - **[Merkledag] [`AddNodeLink()`](https://github.com/ipfs/go-merkledag/blob/v1.1.15/node.go#L99)** - **HELP: The go-merkledag version is supposed to v1.1.15 for this example, however that version has not been added to the [releases](https://github.com/ipfs/go-merkledag/releases)** - The `AddNodeLink()` method is where an `ipld.Link` is created with the `ipld.Node`'s `CID` and size in the `ipld.MakeLink(ipld.Node)` method, and is then appended to the `ProtoNode`'s links in the `ProtoNode.AddRawLink(name)` method. - **[`adder.Finalize()`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L200)** - *Fetch and return the __DAG__ **root** from the `MFS` and `UnixFS` directory* @@ -100,4 +96,8 @@ Within the function, a new `Adder` is created with the configured `Blockstore` a - **[`adder.PinRoot()`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L171)** - *Pin all files under the `MFS` **root*** - The whole process ends with `PinRoot` recursively pinning all the files under the `MFS` **root** \ No newline at end of file + The whole process ends with `PinRoot` recursively pinning all the files under the `MFS` **root** + +# Also see +1. https://github.com/ipfs/go-ipfs/#development +2. [Concept document about files](https://github.com/ipfs/docs/issues/133) \ No newline at end of file From 269550bcbaf7779a4fdfe2b0eca075becce040e9 Mon Sep 17 00:00:00 2001 From: nmalhotra Date: Tue, 15 Jan 2019 21:03:50 -0500 Subject: [PATCH 6/6] Added link to doc in README.md - Fixed typo. - Moved concept document link to top of page. - Added link to this doc to README.md License: MIT Signed-off-by: Nitish Malhotra --- README.md | 1 + docs/add-code-flow.md | 17 ++++++++--------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 1fb35a06a67..4dfb83806d3 100644 --- a/README.md +++ b/README.md @@ -403,6 +403,7 @@ Some places to get you started on the codebase: - libp2p: https://github.com/libp2p/go-libp2p - DHT: https://github.com/libp2p/go-libp2p-kad-dht - PubSub: https://github.com/libp2p/go-libp2p-pubsub +- [IPFS : The `Add` command demystified](https://github.com/ipfs/go-ipfs/tree/master/docs/add-code-flow.md) ### CLI, HTTP-API, Architecture Diagram diff --git a/docs/add-code-flow.md b/docs/add-code-flow.md index ba96cfea91f..a13c7177d40 100644 --- a/docs/add-code-flow.md +++ b/docs/add-code-flow.md @@ -2,6 +2,11 @@ The goal of this document is to capture the code flow for adding a file (see the `coreapi` package) using the IPFS CLI, in the process exploring some datastructures and packages like `ipld.Node` (aka `dagnode`), `FSNode`, `MFS`, etc. +## Concepts +- [Files](https://github.com/ipfs/docs/issues/133) + +--- + **Try this yourself** > > ``` @@ -28,9 +33,7 @@ The goal of this document is to capture the code flow for adding a file (see the > # Hello World > ``` ---- - -# Code Flow +## Code Flow **[`UnixfsAPI.Add()`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreapi/unixfs.go#L31)** - *Entrypoint into the `Unixfs` package* @@ -82,7 +85,7 @@ Within the function, a new `Adder` is created with the configured `Blockstore` a - **[UnixFS] [`(BasicDirectory).AddChild(ctx, name, ipld.Node)`](https://github.com/ipfs/go-unixfs/blob/v1.1.16/io/directory.go#L137)** - *Add child to `BasicDirectory`* - > IMPORTANT: It should be noted that the `BasicDirectory` object uses the `ProtoNode` type object which is an implementation of the `ipld.Node` interface, seen and used throughout this document. Ideally the `ipld.Node` should always be used, unless we need access tp specific functions from `ProtoNode` (like `Copy()`) that are not available in the interface. + > IMPORTANT: It should be noted that the `BasicDirectory` object uses the `ProtoNode` type object which is an implementation of the `ipld.Node` interface, seen and used throughout this document. Ideally the `ipld.Node` should always be used, unless we need access to specific functions from `ProtoNode` (like `Copy()`) that are not available in the interface. This method first attempts to remove any old links (`ProtoNode.RemoveNodeLink(name)`) to the `ProtoNode` prior to adding a link to the newly added `ipld.Node`, using `ProtoNode.AddNodeLink(name, ipld.Node)`. @@ -96,8 +99,4 @@ Within the function, a new `Adder` is created with the configured `Blockstore` a - **[`adder.PinRoot()`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L171)** - *Pin all files under the `MFS` **root*** - The whole process ends with `PinRoot` recursively pinning all the files under the `MFS` **root** - -# Also see -1. https://github.com/ipfs/go-ipfs/#development -2. [Concept document about files](https://github.com/ipfs/docs/issues/133) \ No newline at end of file + The whole process ends with `PinRoot` recursively pinning all the files under the `MFS` **root** \ No newline at end of file