dag: Support multiple files in put #4254
Conversation
License: MIT Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
| test_expect_success "dag put multiple files" ' | ||
| printf {\"foo\":\"bar\"} > a.json && | ||
| printf {\"foo\":\"baz\"} > b.json && | ||
| ipfs dag put a.json b.json > dag_put_out |
There was a problem hiding this comment.
Awesome!
Will test this week how many files it can get at once w/o choking. Thanks a lot for adding this!
core/commands/dag/dag.go
Outdated
| if !ok { | ||
| return nil, fmt.Errorf("expected a different object in marshaler") | ||
| fmt.Println(reflect.TypeOf(res.Output())) |
core/commands/dag/dag.go
Outdated
| @@ -100,52 +97,86 @@ into an object of the specified format. | |||
| defer n.Blockstore.PinLock().Unlock() | |||
There was a problem hiding this comment.
Given that you add the blocks/pin in a go routine, this needs to go somewhere else (i.e., in the go routine). Also, if you want to improve this slightly, it's probably best not to hold this lock while waiting on the client (either receiving or sending data).
License: MIT Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
magik6k
force-pushed
the
feat/dag-batch
branch
from
629cef5
to
03cf65b
Compare
|
Fixed |
core/commands/dag/dag.go
Outdated
| defer n.Blockstore.PinLock().Unlock() | ||
| } | ||
|
|
||
| b := n.DAG.Batch() |
There was a problem hiding this comment.
We should put all dag nodes being added into the same batch here, i would move this batch creation up above the start of the for loop.
There was a problem hiding this comment.
Are you sure it is right thing to do? What if I start adding 100GB of DAGs?
There was a problem hiding this comment.
That was my thinking there, though we might just say that users should care about their batch sizes.
There was a problem hiding this comment.
The DAG batch is size limited, it will autoflush if you put too many items into it
core/commands/dag/dag.go
Outdated
|
|
||
| cid = nds[0].Cid() | ||
| if dopin { | ||
| n.Pinning.PinWithMode(cid, pin.Recursive) |
There was a problem hiding this comment.
Also, since we're batching, we should just gather up all the things we're going to pin into an array, pin them all at once at the end, then only call Pinning.Flush once
License: MIT Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
See #4234