Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce buzhash chunker #6701

Merged
merged 4 commits into from
Oct 7, 2019
Merged

Introduce buzhash chunker #6701

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
67f9837
Introduce buzhash chunker
Oct 7, 2019
6f4f9b4
Add test for buzhash
Oct 7, 2019
76d2ad4
Add buzhash to `ipfs add` docs
Oct 7, 2019
4391ef9
fix(sharness): spelling
Stebalien Oct 7, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions core/commands/add.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import (
"github.com/ipfs/go-ipfs/core/commands/cmdenv"

cmds "github.com/ipfs/go-ipfs-cmds"
"github.com/ipfs/go-ipfs-files"
files "github.com/ipfs/go-ipfs-files"
coreiface "github.com/ipfs/interface-go-ipfs-core"
"github.com/ipfs/interface-go-ipfs-core/options"
mh "github.com/multiformats/go-multihash"
Expand Down Expand Up @@ -80,9 +80,9 @@ how to break files into blocks. Blocks with same content can
be deduplicated. Different chunking strategies will produce different
hashes for the same file. The default is a fixed block size of
256 * 1024 bytes, 'size-262144'. Alternatively, you can use the
Rabin fingerprint chunker for content defined chunking by specifying
rabin-[min]-[avg]-[max] (where min/avg/max refer to the desired
chunk sizes in bytes), e.g. 'rabin-262144-524288-1048576'.
Buzhash or Rabin fingerprint chunker for content defined chunking by
specifying buzhash or rabin-[min]-[avg]-[max] (where min/avg/max refer
to the desired chunk sizes in bytes), e.g. 'rabin-262144-524288-1048576'.

The following examples use very small byte sizes to demonstrate the
properties of the different chunkers on a small file. You'll likely
Expand Down Expand Up @@ -120,7 +120,7 @@ You can now check what blocks have been created by:
cmds.BoolOption(trickleOptionName, "t", "Use trickle-dag format for dag generation."),
cmds.BoolOption(onlyHashOptionName, "n", "Only chunk and hash - do not write to disk."),
cmds.BoolOption(wrapOptionName, "w", "Wrap files with a directory object."),
cmds.StringOption(chunkerOptionName, "s", "Chunking algorithm, size-[bytes] or rabin-[min]-[avg]-[max]").WithDefault("size-262144"),
cmds.StringOption(chunkerOptionName, "s", "Chunking algorithm, size-[bytes], rabin-[min]-[avg]-[max] or buzhash").WithDefault("size-262144"),
cmds.BoolOption(pinOptionName, "Pin this object when adding.").WithDefault(true),
cmds.BoolOption(rawLeavesOptionName, "Use raw blocks for leaf nodes. (experimental)"),
cmds.BoolOption(noCopyOptionName, "Add the file using filestore. Implies raw-leaves. (experimental)"),
Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ require (
github.com/ipfs/go-filestore v0.0.2
github.com/ipfs/go-fs-lock v0.0.1
github.com/ipfs/go-ipfs-blockstore v0.1.0
github.com/ipfs/go-ipfs-chunker v0.0.1
github.com/ipfs/go-ipfs-chunker v0.0.3
github.com/ipfs/go-ipfs-cmds v0.1.1
github.com/ipfs/go-ipfs-config v0.0.11
github.com/ipfs/go-ipfs-ds-help v0.0.1
Expand Down
2 changes: 2 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,8 @@ github.com/ipfs/go-ipfs-blocksutil v0.0.1 h1:Eh/H4pc1hsvhzsQoMEP3Bke/aW5P5rVM1IW
github.com/ipfs/go-ipfs-blocksutil v0.0.1/go.mod h1:Yq4M86uIOmxmGPUHv/uI7uKqZNtLb449gwKqXjIsnRk=
github.com/ipfs/go-ipfs-chunker v0.0.1 h1:cHUUxKFQ99pozdahi+uSC/3Y6HeRpi9oTeUHbE27SEw=
github.com/ipfs/go-ipfs-chunker v0.0.1/go.mod h1:tWewYK0we3+rMbOh7pPFGDyypCtvGcBFymgY4rSDLAw=
github.com/ipfs/go-ipfs-chunker v0.0.3 h1:UuXhKoxvxl/vGhie+WXFRZYCwpZjbKF2SzD1Tvxif1I=
github.com/ipfs/go-ipfs-chunker v0.0.3/go.mod h1:RkGJorerOQNTDPgmX7HtJ5YzVQqaIYdzI/hrCHty5Kc=
github.com/ipfs/go-ipfs-cmds v0.1.1 h1:H9/BLf5rcsULHMj/x8gC0e5o+raYhqk1OQsfzbGMNM4=
github.com/ipfs/go-ipfs-cmds v0.1.1/go.mod h1:k1zMXcOLtljA9iAnZHddbH69yVm5+weRL0snmMD/rK0=
github.com/ipfs/go-ipfs-config v0.0.11 h1:5/4nas2CQXiKr2/MLxU24GDGTBvtstQIQezuk7ltOQQ=
Expand Down
10 changes: 10 additions & 0 deletions test/sharness/t0040-add-and-cat.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,16 @@ test_add_cat_file() {
test_expect_code 1 ipfs add -Q --chunker rabin-12-512-1024 mountdir/hello.txt
'

test_expect_success "ipfs add --chunker buzhash suceeds" '
ipfs add --chunker buzhash mountdir/hello.txt >actual
'

test_expect_success "ipfs add --chunker buzhahs output looks good" '
Kubuxu marked this conversation as resolved.
Show resolved Hide resolved
Stebalien marked this conversation as resolved.
Show resolved Hide resolved
HASH="QmVr26fY1tKyspEJBniVhqxQeEjhF78XerGiqWAwraVLQH" &&
echo "added $HASH hello.txt" >expected &&
test_cmp expected actual
'

test_expect_success "ipfs add on hidden file succeeds" '
echo "Hello Worlds!" >mountdir/.hello.txt &&
ipfs add mountdir/.hello.txt >actual
Expand Down