Skip to content
This repository has been archived by the owner on Feb 12, 2024. It is now read-only.

Commit

Permalink
feat: pin API (#1045)
Browse files Browse the repository at this point in the history
* revert: default assets are not added when running on a browser

refactor: change pin.flush logging message

* feat(test): add tests for failure cases of normalizeHashes

fix: don't need to cast the object.get result with toJSON

revert: use interface-datastore.Key for datastore pin storage
The proper change would be that datastore-level automatically casts operations into Keys

fix: do not invoke callback within a try/catch

feat(test): make cli pin tests more robust
By using files that aren't added on IPFS initialization. Still needs work on files.rm (direct) and ipfs ls (indirect).

fix: remove commented code, traced test failures to pin-set
Got to go for the night, though, so will checkpoint here and address tomorrow.

feat: parseIpfsPath now throws errors for consistency

feat: resolveIpfsPaths error message lists the relative path that failed

feat: use follow.bind instead of mutating the links
Also decided not show relative paths. Less human friendly but probably cleaner otherwise.

refactor: resolveIpfsPaths -> resolvePaths

feat: promisify resolvePaths

test: change parseIpfsPath failure tests to use try/catch

docs: edit resolvePath doc

revert: accidentally deleted commands/pin.js

* feat: jsipfs pin improvements (#1249)

* initial sweep through to understand how pin works. Did make some changes but mostly minor.

* refactor pb schema to it's own file

* fix: don't pin files during files.add if opts.pin === false

* feat: add some http qs parsing, http route/resources cleanup, cleanup core/utils.parseIpfsPath

* feat: expand pin tests. \nFirst draft. still needs some further work.

* feat: Add logging for entry/exit of pins: add/rm/flush/load. Clean some documentation.

* feat: add --pin to files.add, fix: improper pin option parsing in core.

* feat: Use ipfs.files.add to add init-docs instead of directly using the unix-fs importer.

* feat(tests): Add tests for cli --pin option. I know this should be more of an integration test. Should be written in /core. Maybe talk with Victor about testing different layers

* feat: use isIPFS to valiate a multihash.

* fix: add some changes missed during rebase, syntax fixes, etc

I think my original rebase for this branch 2 weeks ago might have
changed history for the intervening commits, indirectly causing some of
these missed changes. or I just rebase onto the wrong oldparent.

fix: some onlyHash and pin tests broke after merging
onlyHash and pin interact: shouldn't pin when --only-hash.

fix: trim output for 'pin ls when no hash is passed'

test: indirect pins supersede direct pins: turns out we had a bug

feat: add expectTimeout test utility

feat: promisify some additional pin utils

* test: initial work testing the core/pin.js implementation
I think I'll end up moving most tests here.

test: add tests for pin.ls and pin.rm
Based tests on other pin fixtures, need to migrate the isPinned* tests to them as well.

fix: direct pins are now deleted by a default pin.rm(hash)

test: prepare for pin.add tests
'indirect supersedes direct' test exposes a bug in pin.ls

feat: switch away from multihashes for isPinned* tests

test: impl pin.add tests

fix: add fixture files only once

test: add test for a potential bug, clean isPinned* tests

refactor: remove a test that's no longer needed

fix: pin.ls, indirect pins should supersede direct pins

test: naive pin.load, pin.flush tests

feat: remove most pin cli tests as functionality is tested in pin core tests

refactor: rename solarSystem

* refactor: move pin http-api tests to http-api/inject

fix: attempt to find a way to use http-api/inject test structure for pin tests

test: fix pin.rm http-api tests

test: fix pin.add http-api tests

docs: docs and cleanup of http-api pin tests

refactor: renaming

fix: lint errors

fix: resolvePaths tests are failing on CI, it might be long ops, testing a timeout bump

fix: add files explicitly before testing resolvePaths

fix: remove mocha.only from resolvePaths. let's hope tests pass, they are passing CI now

fix: rename test/core/utils.spec.js -> utils.js so it's not run during browser tests

* test: first draft of pin-set tests
Need to leave computer, this is a checkpoint.

test: add sanity test for walkItems and hasChild, clean others
These tests are more descriptive than really pushing the impl. I'd love others' thoughts on what else should be hit and how. I also need to compare go's pinset impl against ours

fix: stop daemons

feat: documentation and multihash buffer handling for dag.get

fix: lint

* feat: simplify root dagnode generation for storeItems base case

* feat: rename vars, fix _depth default value, add docs

fix: pinset.hasChild buffer check

feat: hardcode expected length for flush/load tests

* feat: parallelize pin.isPinnedWithType

* refactor: refactor pinset.storeItems

* fix: re-add pin interface tests
I must have missed a commit during a rebase.

* fix: lint

* feat: docs, rename resolvePaths, pin.getIndirectKeys now uses eachLimit

* chore: rebase a month of changes, resolve minor issues from that

fix: yarg arugment naming for pin cli commands

fix: convert file multihashes to a b58 string

fix: another way of checking for CID-ness

fix: lint

fix: toB58String handles non-buffers

fix: key-exchange core tests now shutdown daemon.

* chore: update big.js version

* revert: do not pin content added with a non-default hash algorithm

* revert: internalKey recording

* refactor: use lodash.flattenDeep

refactor: pinset.hasChild -> pinset.hasDescendent

fix: invoke someCb if we've seen the hash before

refactor: async patterns in dag._getRecursive

refactor: pinset.hasDescendant

refactor: pinset.storeItems async patterns

refactor: pinset.loadSet and pin.walkItem async patterns

docs: add link to go-ipfs' fanout bin implementation

refactor: async patterns of pin.load/flush

refactor: lint

refactor: privatize internal pin key storage

refactor: change encapsulation of ipfs.pin, fix resulting issues

fix: lint

fix: 'files add --pin=false' test was giving a false positive

refactor: use is-ipfs to check CID-ability of a string

refactor: remove last instance of 'once' in the pin code

* refactor: do not expose pinTypes

They're simple enough, documented elsewhere, and not used by any exposed functionality.

* fix: do not destructure node callback results
  • Loading branch information
daviddias authored and alanshaw committed Jun 19, 2018
1 parent 4e51a69 commit 2a5cc5e
Show file tree
Hide file tree
Showing 34 changed files with 2,117 additions and 27 deletions.
7 changes: 5 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@
},
"dependencies": {
"async": "^2.6.0",
"big.js": "^5.0.3",
"big.js": "^5.1.2",
"binary-querystring": "~0.1.2",
"bl": "^1.2.2",
"boom": "^7.2.0",
Expand All @@ -98,13 +98,15 @@
"debug": "^3.1.0",
"file-type": "^7.7.1",
"filesize": "^3.6.1",
"fnv1a": "^1.0.1",
"fsm-event": "^2.1.0",
"get-folder-size": "^1.0.1",
"glob": "^7.1.2",
"hapi": "^16.6.2",
"hapi-set-header": "^1.0.2",
"hoek": "^5.0.3",
"human-to-milliseconds": "^1.0.0",
"interface-datastore": "^0.4.1",
"ipfs-api": "^22.0.0",
"ipfs-bitswap": "~0.20.0",
"ipfs-block": "~0.7.1",
Expand Down Expand Up @@ -137,6 +139,7 @@
"libp2p-websocket-star": "~0.8.0",
"libp2p-websockets": "~0.12.0",
"lodash.flatmap": "^4.5.0",
"lodash.flattendeep": "^4.4.0",
"lodash.get": "^4.4.2",
"lodash.set": "^4.3.2",
"lodash.sortby": "^4.7.0",
Expand Down Expand Up @@ -218,7 +221,7 @@
"Jade Meskill <jade.meskill@gmail.com>",
"Johannes Wikner <johannes.wikner@gmail.com>",
"Jon Schlinkert <dev@sellside.com>",
"Jonathan <jkrone@vt.edu>",
"Jonathan Krone <jonathankrone@gmail.com>",
"João Antunes <j.goncalo.antunes@gmail.com>",
"João Santos <joaosantos15@users.noreply.github.com>",
"Kevin Wang <kevin@fossa.io>",
Expand Down
8 changes: 7 additions & 1 deletion src/cli/commands/files/add.js
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,11 @@ module.exports = {
type: 'boolean',
default: false,
describe: 'Write no output'
},
pin: {
type: 'boolean',
default: true,
describe: 'Pin this object when adding'
}
},

Expand All @@ -188,7 +193,8 @@ module.exports = {
rawLeaves: argv.rawLeaves,
onlyHash: argv.onlyHash,
hashAlg: argv.hash,
wrapWithDirectory: argv.wrapWithDirectory
wrapWithDirectory: argv.wrapWithDirectory,
pin: argv.pin
}

// Temporary restriction on raw-leaves:
Expand Down
15 changes: 15 additions & 0 deletions src/cli/commands/pin.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
'use strict'

module.exports = {
command: 'pin',

description: 'Pin and unpin objects to local storage.',

builder (yargs) {
return yargs
.commandDir('pin')
},

handler (argv) {
}
}
29 changes: 29 additions & 0 deletions src/cli/commands/pin/add.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
'use strict'

const print = require('../../utils').print

module.exports = {
command: 'add <ipfsPath...>',

describe: 'Pins object to local storage.',

builder: {
recursive: {
type: 'boolean',
alias: 'r',
default: true,
describe: 'Recursively pin the object linked to by the specified object(s).'
}
},

handler (argv) {
const recursive = argv.recursive
const type = recursive ? 'recursive' : 'direct'
argv.ipfs.pin.add(argv.ipfsPath, { recursive: recursive }, (err, results) => {
if (err) { throw err }
results.forEach((res) => {
print(`pinned ${res.hash} ${type}ly`)
})
})
}
}
43 changes: 43 additions & 0 deletions src/cli/commands/pin/ls.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
'use strict'

const print = require('../../utils').print

module.exports = {
// bracket syntax with '...' tells yargs to optionally accept a list
command: 'ls [ipfsPath...]',

describe: 'List objects pinned to local storage.',

builder: {
type: {
type: 'string',
alias: 't',
default: 'all',
choices: ['direct', 'indirect', 'recursive', 'all'],
describe: 'The type of pinned keys to list.'
},
quiet: {
type: 'boolean',
alias: 'q',
default: false,
describe: 'Write just hashes of objects.'
}
},

handler: (argv) => {
const paths = argv.ipfsPath
const type = argv.type
const quiet = argv.quiet

argv.ipfs.pin.ls(paths, { type }, (err, results) => {
if (err) { throw err }
results.forEach((res) => {
let line = res.hash
if (!quiet) {
line += ` ${res.type}`
}
print(line)
})
})
}
}
28 changes: 28 additions & 0 deletions src/cli/commands/pin/rm.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
'use strict'

const print = require('../../utils').print

module.exports = {
command: 'rm <ipfsPath...>',

describe: 'Removes the pinned object from local storage.',

builder: {
recursive: {
type: 'boolean',
alias: 'r',
default: true,
describe: 'Recursively unpin the objects linked to by the specified object(s).'
}
},

handler: (argv) => {
const recursive = argv.recursive
argv.ipfs.pin.rm(argv.ipfsPath, { recursive: recursive }, (err, results) => {
if (err) { throw err }
results.forEach((res) => {
print(`unpinned ${res.hash}`)
})
})
}
}
1 change: 1 addition & 0 deletions src/core/boot.js
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ module.exports = (self) => {

series([
(cb) => self._repo.open(cb),
(cb) => self.pin._load(cb),
(cb) => self.preStart(cb),
(cb) => {
self.log('initialized')
Expand Down
25 changes: 25 additions & 0 deletions src/core/components/dag.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
const promisify = require('promisify-es6')
const CID = require('cids')
const pull = require('pull-stream')
const mapAsync = require('async/map')
const flattenDeep = require('lodash.flattendeep')

module.exports = function dag (self) {
return {
Expand Down Expand Up @@ -33,6 +35,12 @@ module.exports = function dag (self) {
} else {
path = '/'
}
} else if (Buffer.isBuffer(cid)) {
try {
cid = new CID(cid)
} catch (err) {
return callback(err)
}
}

self._ipld.get(cid, path, options, callback)
Expand Down Expand Up @@ -73,6 +81,23 @@ module.exports = function dag (self) {
self._ipld.treeStream(cid, path, options),
pull.collect(callback)
)
}),

// TODO - use IPLD selectors once they are implemented
_getRecursive: promisify((multihash, callback) => {
// gets flat array of all DAGNodes in tree given by multihash

self.dag.get(new CID(multihash), (err, res) => {
if (err) { return callback(err) }

mapAsync(res.value.links, (link, cb) => {
self.dag._getRecursive(link.multihash, cb)
}, (err, nodes) => {
// console.log('nodes:', nodes)
if (err) return callback(err)
callback(null, flattenDeep([res.value, nodes]))
})
})
})
}
}
23 changes: 20 additions & 3 deletions src/core/components/files.js
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,9 @@ function prepareFile (self, opts, file, callback) {
}

waterfall([
(cb) => opts.onlyHash ? cb(null, file) : self.object.get(file.multihash, opts, cb),
(cb) => opts.onlyHash
? cb(null, file)
: self.object.get(file.multihash, opts, cb),
(node, cb) => {
const b58Hash = cid.toBaseEncodedString()

Expand Down Expand Up @@ -87,6 +89,19 @@ function normalizeContent (opts, content) {
})
}

function pinFile (self, opts, file, cb) {
// Pin a file if it is the root dir of a recursive add or the single file
// of a direct add.
const pin = 'pin' in opts ? opts.pin : true
const isRootDir = !file.path.includes('/')
const shouldPin = pin && isRootDir && !opts.onlyHash && !opts.hashAlg
if (shouldPin) {
return self.pin.add(file.hash, err => cb(err, file))
} else {
cb(null, file)
}
}

class AddHelper extends Duplex {
constructor (pullStream, push, options) {
super(Object.assign({ objectMode: true }, options))
Expand Down Expand Up @@ -130,7 +145,8 @@ module.exports = function files (self) {
}

let total = 0
let prog = opts.progress || (() => {})

const prog = opts.progress || noop
const progress = (bytes) => {
total += bytes
prog(total)
Expand All @@ -141,7 +157,8 @@ module.exports = function files (self) {
pull.map(normalizeContent.bind(null, opts)),
pull.flatten(),
importer(self._ipld, opts),
pull.asyncMap(prepareFile.bind(null, self, opts))
pull.asyncMap(prepareFile.bind(null, self, opts)),
pull.asyncMap(pinFile.bind(null, self, opts))
)
}

Expand Down
1 change: 1 addition & 0 deletions src/core/components/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ exports.swarm = require('./swarm')
exports.ping = require('./ping')
exports.pingPullStream = require('./ping-pull-stream')
exports.pingReadableStream = require('./ping-readable-stream')
exports.pin = require('./pin')
exports.files = require('./files')
exports.bitswap = require('./bitswap')
exports.pubsub = require('./pubsub')
Expand Down
23 changes: 9 additions & 14 deletions src/core/components/init-assets.js
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
'use strict'

const path = require('path')
const fs = require('fs')
const glob = require('glob')
const importer = require('ipfs-unixfs-engine').importer
const pull = require('pull-stream')
const file = require('pull-file')
const CID = require('cids')
Expand All @@ -15,23 +13,20 @@ module.exports = function addDefaultAssets (self, log, callback) {

pull(
pull.values([initDocsPath]),
pull.asyncMap((val, cb) => glob(path.join(val, '/**/*'), cb)),
pull.asyncMap((val, cb) =>
glob(path.join(val, '/**/*'), { nodir: true }, cb)
),
pull.flatten(),
pull.map((element) => {
pull.map(element => {
const addPath = element.substring(index + 1)

if (fs.statSync(element).isDirectory()) { return }

return { path: addPath, content: file(element) }
}),
// Filter out directories, which are undefined from above
pull.filter(Boolean),
importer(self._ipld),
pull.through((el) => {
if (el.path === 'init-docs') {
const cid = new CID(el.multihash)
self.files.addPullStream(),
pull.through(file => {
if (file.path === 'init-docs') {
const cid = new CID(file.hash)
log('to get started, enter:\n')
log(`\t jsipfs files cat /ipfs/${cid.toBaseEncodedString()}/readme\n`)
log(`\tjsipfs files cat /ipfs/${cid.toBaseEncodedString()}/readme\n`)
}
}),
pull.collect((err) => {
Expand Down
4 changes: 3 additions & 1 deletion src/core/components/init.js
Original file line number Diff line number Diff line change
Expand Up @@ -85,16 +85,18 @@ module.exports = function init (self) {
return cb(null, true)
}

self.log('adding assets')
const tasks = [
// add empty unixfs dir object (go-ipfs assumes this exists)
(cb) => self.object.new('unixfs-dir', cb)
]

if (typeof addDefaultAssets === 'function') {
// addDefaultAssets is undefined on browsers.
// See package.json browser config
tasks.push((cb) => addDefaultAssets(self, opts.log, cb))
}

self.log('adding assets')
parallel(tasks, (err) => {
if (err) {
cb(err)
Expand Down
Loading

0 comments on commit 2a5cc5e

Please sign in to comment.