benchmarks

[dignifiedquire]

As discussed in #455 this adds a simple shell script to run ipfs-whatever benchmarks agains js-ipfs.

Missing currently

• top level aliascat for files.cat
• top level alias add for files.add

[daviddias]

update?

[dignifiedquire]

Depends on #493

[daviddias]

There @dignifiedquire #493

Thank you @victorbjelkholm :)

[daviddias]

🛎

[dignifiedquire]

Investigating using https://github.com/haadcode/js-ipfs-command-benchmark instead

[daviddias]

1 > 0, this PR seems ready and gives us some ground up. Plus it is the benchmarks that go-ipfs uses.

We can then evolve into @haadcode's tool.

[dignifiedquire]

Doesn't look promising, not even patch works (which worked before)

-- running benchmarks
2016/11/28 21:49:10 checking patch operations per second...
2016/11/28 21:51:10 Post http://127.0.0.1:5001/api/v0/object/patch/set-data?arg=QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn&encoding=json&stream-channels=true: EOF
-- cleaning up

[dignifiedquire]

Got something working, before = js, after = go@0.4.4

Results          Before  After  % Change
PatchOpsPerSec   0.00    0.00   NaN%
DirAddOpsPerSec  0.00    0.00   NaN%
DirAddOpsStdev   0.00    0.00   NaN%
Add10MBTime      0.23    0.11   -53.09%
Add10MBStdev     0.04    0.00   -90.45%
Cat1MBTime       15.30   29.31  91.56%
Cat1MBStdev      3.98    2.86   -28.22%
MkGraphTime      0.00    0.00   NaN%
TravGraphTime    0.00    0.00   NaN%
TravGraphStdev   0.00    0.00   NaN%

note need to run go-ipfs with --offline flag for fast results

[dignifiedquire]

Commands that still need fixing

• PatchOpsPerSec
• DirAddOpsPerSec
• DirAddOpsStdev
• MkGraphTimek
• TravGraphTime
• TravGraphStdev

[dignifiedquire]

Looks like the current blocker is: whyrusleeping/ipfs-whatever#6

[dignifiedquire]

The missing object is created in go-ipfs using object new unixfs-dir but calling this on js-ipfs results in a different object. :(

[dignifiedquire]

Implemented object.new with template option, this needs

• feat(object): add template option to object.new ipfs-inactive/interface-js-ipfs-core#103
• feat(object): add template support to object.new ipfs-inactive/js-ipfs-http-client#476

[dignifiedquire]

Getting this error now:

Debug: internal, implementation, error
    RangeError: Uncaught error: Could not decode varint
    at Object.read [as decode] (/Users/dignifiedquire/opensource/ipfs/js-ipfs-lerna/packages/js-ipfs/node_modules/protocol-buffers/node_modules/varint/decode.js:17:13)
    at Object.decode (/Users/dignifiedquire/opensource/ipfs/js-ipfs-lerna/packages/js-ipfs/node_modules/protocol-buffers/encodings.js:45:22)
    at Object.decode (eval at line.toFunction (/Users/dignifiedquire/opensource/ipfs/js-ipfs-lerna/packages/js-ipfs/node_modules/generate-function/index.js:1:1), <anonymous>:28:25)
    at Object.deserialize (/Users/dignifiedquire/opensource/ipfs/js-ipfs-lerna/packages/js-ipfs/node_modules/ipld-dag-pb/src/util.js:32:28)
    at pull.asyncMap (/Users/dignifiedquire/opensource/ipfs/js-ipfs-lerna/packages/js-ipfs/node_modules/ipld-resolver/src/index.js:150:18)
    at /Users/dignifiedquire/opensource/ipfs/js-ipfs-lerna/packages/js-ipfs/node_modules/pull-stream/throughs/async-map.js:28:13
    at /Users/dignifiedquire/opensource/ipfs/js-ipfs-lerna/packages/js-ipfs/node_modules/pull-stream/sources/values.js:21:7
    at Function.read.resolve (/Users/dignifiedquire/opensource/ipfs/js-ipfs-lerna/packages/js-ipfs/node_modules/pull-defer/source.js:20:13)
    at released (/Users/dignifiedquire/opensource/ipfs/js-ipfs-lerna/packages/js-ipfs/node_modules/ipfs-repo/src/stores/blockstore.js:79:18)
    at /Users/dignifiedquire/opensource/ipfs/js-ipfs-lerna/packages/js-ipfs/node_modules/lock/index.js:12:22

On

2016/12/10 21:40:42 generating graph...
object/patch/add-link: An internal server error occurred

[dignifiedquire]

It seems the protobuf it throws on is

12320a221220364473f88918c1391b046c716baf85519a787f1ee5ccd52202b9d7b1a70b280a12013018808080f489e0e9d2b4b70412320a221220364473f88918c1391b046c716baf85519a787f1ee5ccd52202b9d7b1a70b280a12013118808080f489e0e9d2b4b7040a

[daviddias]

@dignifiedquire can you make this commit -- 0835f08 -- be a separate PR, it is kind of annoying that tests fail because that specific commit is on a branch.

[dignifiedquire]

I found the issue why js-ipfs is throwing up.

It turns out ipfs-whatever generates an object graph where the size cumulative size of the root node is larger than the uint64 value used to encode this size into protobufs. In JavaScript the protocol-buffers module throws when decoding such a value 1.0466891133557776×10^22 in this example.

This "works" in go-ipfs, but there are issues as well calling object stat on the root hash of this graph results in

> ipfs object stat QmWra8edcqDmHi5TEdM3MW1GTXmpqam4bdXDW13XxxzHtA
NumLinks: 10
BlockSize: 524
LinksSize: 522
DataSize: 2
CumulativeSize: -806169813684932772

which is not entirely correct I would say

The code generating these graphs is here: https://github.com/whyrusleeping/ipfs-whatever/blob/master/main.go#L154-L166

cc @diasdavid @whyrusleeping

[dignifiedquire]

And the last issue is that we don't have a resolver to resole paths like <hash>/prop currently so the graph traversal fails.

[Kubuxu]

It is known: ipfs/kubo#3440

I don't really know why why it is sign negative, probably bad conversion somewhere.
I also don't know who to proceed, we might want to detect the overflow and cap it, but I am not sure about this issue at all.

[dignifiedquire]

Thanks @Kubuxu for the reference, not sure how to proceed with this either. Will bring it up later today on the go-ipfs sprint

[victorb]

And the last issue is that we don't have a resolver to resole paths like <hash>/prop currently so the graph traversal fails.

This has been added now: #875

[daviddias]

thanks for bringing this up @victorbjelkholm. @dignifiedquire would you be able to finish this now?

[daviddias]

@dignifiedquire any hopes that you will find the time to ship this?