npm on IPFS

[jbenet]

• ipfs-blob-store
• ipfs-daemon-ctrl control whatever daemon is installed
• plug in ipfs-blob-store to reginabox
• figure if we can 302 publish
• mirror the registry
• bundle it all into one module
• make physical nodes

cc @bengl

[whyrusleeping]

just idle thought? Or discussion?

[jbenet]

@whyrusleeping notes from discussion in person :)

[bengl]

As mentioned in the first checkbox, one blocker for this is that someone needs to make a ipfs-blob-store adhering to abstract-blob-store. I'm working on this on and off in my spare time. Slowly. If someone beats me to it, yay!

[seidtgeist]

@jbenet Hmm... After you explained the ipfs docker mount hack yesterday, what if we mount ipfs on node_modules and require('hash')? Why am I even asking and not just trying this? brb

[whyrusleeping]

@ehd how did that work out?

[seidtgeist]

@whyrusleeping It works :)

$ cat add.js
'use strict';

module.exports = function(a, b) {
  return a + b;
};

$ ipfs add add.js
added QmXctmnj6tEMSJsb433jKwcdGwuBy4uGMnqva5Nkvp5E7x add.js

$ cat prog.js
'use strict';

var add = require('QmXctmnj6tEMSJsb433jKwcdGwuBy4uGMnqva5Nkvp5E7x');

console.log(add(1, 2));

$ node prog.js
3

Two things:

• I'd rather not rely on the filesystem or fuse, that's doable
• I'd rather require names instead of SHAs and move the mapping to metadata

[whyrusleeping]

@ehd thats great! what would the ideal interface be for you to want to use ipfs for js modules?

[seidtgeist]

Ideally we'd have a simple bootstrapping js/node program in ipfs that can
load other things from ipfs using require. Reminds me of the StackStream
opcodes actually ☺️ Anyway, without a file system the ideal
delivery mechanism would be the true ipfs network, so baking a simple ipfs
client into a runnable JavaScript bundle sounds like an idea worth trying.
Initially this could be done by just using the global or local ipfs http
gateway for loading code. I'm not sure whether a dynamic module loader
(require on demand) or a dynamic bundler (compile program, then start
program) would be more viable. Whatever runs in <script src="browser-fetchable-address-on-ipfs"/> or vanilla node I guess.

Another related idea we've bounced: storing computational results in ipfs.
Say you have blobs for js-compile and js-source-with-links; you can load
both and run js-compile(js-source-with-links), then store the result and
use that as the above entry point. It's basically how all browser side JS
already works, just that it's very sad how we store build results in dumb
files without linking to how they've been conceived (creation metadata! cc
@jbenet).

Side note: Ironically I'm in a place without wifi and just looked up
stackstream on my mac's browser history, but of course I can't access the
already cached page. Idea: any ipfs browser/browser extension should pin
history items. Immediate benefit over any other real world browser.

[jbenet]

Update on the list above: @krl made https://github.com/ipfs/node-ipfsd-ctl

---

@ehd all of that sounds awesome!

Ideally we'd have a simple bootstrapping js/node program in ipfs that can
load other things from ipfs using require.

Yeah! like a special require that uses ipfs directly. Could patch npm to make requires that start with "/ipfs/..." require straight out of ipfs.

Btw, what @bengl and I want to do above with reginabox will work with stock npm and require.

Reminds me of the StackStream
opcodes actually ☺️ Anyway, without a file system the ideal
delivery mechanism would be the true ipfs network, so baking a simple ipfs
client into a runnable JavaScript bundle sounds like an idea worth trying.

that sounds awesome

Initially this could be done by just using the global or local ipfs http
gateway for loading code. I'm not sure whether a dynamic module loader
(require on demand) or a dynamic bundler (compile program, then start
program) would be more viable. Whatever runs in <script src="browser-fetchable-address-on-ipfs"/> or vanilla node I guess.

@krl's https://github.com/ipfs/node-ipfsd-ctl can make ephemeral ipfs nodes using the go-ipfs package, so you dont even have to install ipfs beforehand. it's huge (cause go-ipfs is huge) but once we have node-ipfs this will be a drop-in replacement.

Another related idea we've bounced: storing computational results in ipfs.
Say you have blobs for js-compile and js-source-with-links; you can load
both and run js-compile(js-source-with-links), then store the result and
use that as the above entry point. It's basically how all browser side JS
already works, just that it's very sad how we store build results in dumb
files without linking to how they've been conceived (creation metadata! cc
@jbenet).

Absolutely! And-- i want to make a way to do browserify with partial results separated out. THOUGH, rabin fingerprinting will get us 80% of the way there without the worries anyway.

Side note: Ironically I'm in a place without wifi and just looked up
stackstream on my mac's browser history, but of course I can't access the
already cached page. Idea: any ipfs browser/browser extension should pin
history items. Immediate benefit over any other real world browser.

Yeahhh i really want this. wonder what the overhead + pains of doing this might be... hard to get right because people dont hash things or use cache headers correctly :/

[bengl]

Could patch npm to make requires that start with "/ipfs/..." require straight out of ipfs.

You'd have to patch node. All the require logic is self-contained in node, not npm.

There are also many lifecycle scripts and compilation steps and all kinds of other things that are done to modules during the install process. In some modules, the results of these scripts are different depending on where they are in the node_modules tree.

I think it's much more practical to keep a strict separation between the source of packages (ipfs/ipns, in an ideal world) and the current install directory. Requiring them directly might work for JS-only modules with no lifecycle scripts. But then you're not supporting a huge chunk of available modules.

I'm only really talking about node here, but for people using frontend build pipelines that depend on npm, the same problems can arise.

So yeah, I'm not sure grabbing node dependencies directly from ipfs at run-time is realistic.

[jbenet]

@bengl ah thanks! makes a lot of sense.

i guess we could do it to specific packages people decide to do this with (i.e. people explicitly publish them + require them with hashes, so they take on the responsibility of not making all the lifecycle stuff happen?) not sure.

[bengl]

Yeah for sure. ipfs-pm/ipfspm/ippm. You'd pretty much have to make a clear separation from npm at that point, I'd think.

I still would like to see the ipfs leveraged as the package source, as we discussed before. Should be easy once ipfs-blob-store exists :)

[jbenet]

@bengl maybe we can hack that (ipfs-blob-store) out this week. I'm in bay area atm. (cc @krl @travisperson @mappum @diasdavid in case they're inspired)

[travisperson]

Definitely, I took a look at the spec today. @bengl do you have any of the work you've made towards it published anywhere?

[bengl]

So really we're looking at potentially two different use-cases for ipfs-blob-store. Might be better off as two different modules. I'm not sure.

Use Case 1: Raw Blob Store

In this case, we're really just plopping blobs onto IPFS and getting their IPFS paths back in the metadata for later retrieval. So in createWriteStream, you wouldn't provide a key in the opts as that's what you'd be getting back from IPFS (i.e. the IPFS path). In createReadStream, the key you'd pass in would be the IPFS path, and it would stream back the blob at that path.

Use Case 2: IPNS Mutable File System Thing

In this case (which could use the previous case under the hood), in the constructor for the blob store, we'd provide all the necessary data to read and write to an IPNS namespace (keypair? just the hash? Sorry, my IPNS knowledge is rather limited). Here, createWriteStream would use the key option from the args to determine where to put the blob, and the metadata returned would be kind of irrelevant, but would have the same key. Behind the scenes it would update IPNS properly. So the key would be something like /path/in/namepsace. Similarly for createReadStream.

---

Unfortunately I don't spend as much time on this as I'd like, so I'm unfamiliar with how things actually work in IPNS-land right now. Sorry!

Here's the abstract-blob-store API reference.

[jbenet]

@bengl yep, you've got it right :)

We'd want a third module: unixfs-blob-store which would read an ipfs object as unixfs and stream out the full file (or compute StackStream)

[seidtgeist]

Pardon my late replies, I’m on a vacation until Thursday.

You’d have to patch node. All the require logic is self-contained in node, not npm.

That’s true. But you can change the meaning of require through compilation, e.g. like webpack does. The JavaScript bootloader could do this when loading modules from ipfs.

But then you’re not supporting a huge chunk of available modules.

That’s unfortunately true, but I think we can do lots of useful stuff without modules that rely on native compilation. Most compilation I see when running npm install is (dev)dependencies installing native file watchers. If native dependencies are a must, there could be pre-compiled binaries per runtime that could be grabbed from ipfs. Most backend or hybrid JS code I’ve written runs in both node and browsers and doesn’t depend on native dependencies. I could do lots of interesting stuff without them. Maybe @grncdr can also comment on that.

[travisperson]

First stab at ipfs-blob-store

https://github.com/ipfs/ipfs-blob-store

Currently uses raw blocks, and is only content addressable. It does not use ipns at the moment, passes 45 of the tests (out of a bunch more, I don't fully understand tape).

[jbenet]

i think it's time to resurrect this effort.

cc @mappum @daviddias @RichardLitt

[whyrusleeping]

npm uses tar right? ipfs tar is so ready.

[daviddias]

👍

[bengl]

👍 💥

[daviddias]

Looking for some clarifications:

• Do we need https://github.com/yahoo/reginabox for this? Or should I be just looking at https://github.com/davglass/registry-static , as it already creates npm endpoints
• Is registry-static ready to handle publishes? (not fully getting it from docs)
• Is there a way to proactively cache the whole npm built into registry-static?

this is so awesome!

[bengl]

• Do we need https://github.com/yahoo/reginabox for this? Or should I be just looking at https://github.com/davglass/registry-static , as it already creates npm endpoints

Just registry-static. I maintain it as part of my day job, so LMK if you need anything there. reginabox is just a fancily-pre-packaged version of registry-static.

• Is registry-static ready to handle publishes? (not fully getting it from docs)

No. registry-static is not actually equivalent to an npm registry. It's only useful for mirroring from the public registry (or any other registry that supports couchdb change feeds). For a full-on registry replacement for internal or alternative usage, you need to either run your own couchdb, or something like cnpm.

• Is there a way to proactively cache the whole npm built into registry-static?

Yes, that's the first thing registry-static does when you start it up (but it takes like 2 days).

[hackergrrl]

Hi. Trying to make sure I grok the big picture here, in terms of what an ipfs-npm request looks like. Please correct me as I go. ;)

1. User installs the ipfs-npm command, or whatever we call it
2. User runs ipfs-npm foobar
3. I'm not clear on how look-up happens, to map a module name to an IPFS address. One option is to use IPNS, and claim that "all ipfs-npm modules are published to this pubkey. This means all clients know that to get foobar the path will always be /ipns/<static pubkey we control>/foobar. Do we want that kind of control? It would mean centralizing the publish operation.
4. Easy part: we grab the module tarball from IPFS and install.

How does ipfs-blob-store fit in? Is this what reginabox uses to retrieve the blobs?

[hackergrrl]

@whyrusleeping helped me better understand the intent here. Looks like I was getting ahead of myself; it sounds like the first goal is to opt for decentralized before distributed: let users run their own npm registry that uses IPFS as the blob store rather than the local fs.

Lingering Q: how will our reginabox map module names to IPFS addresses?

[whyrusleeping]

(decentralized first, then distributed: https://ipfs.io/ipfs/QmZpfhN5rucQC4kx7Gu5udS8FXxsxMddiDCvG8WFjG8SMv)

[daviddias]

Just PR'ed ipfs-blob-store using mfs (mutable file system) ipfs-shipyard/ipfs-blob-store#5

@bengl I'm being unable to attach a custom blob-store to registry-static, looking at the --help menu, the --blobstore option doesn't appear (for ref posted the help menu below) and when I run something like » registry-static -d npm.ipfs.io -o /npm-ipfs --blobstore=registry-ipfs-blob-store.js, I get this error:

module.js:338
    throw err;
    ^

Error: Cannot find module 'registry-ipfs-blob-store.js'
    at Function.Module._resolveFilename (module.js:336:15)
    at Function.Module._load (module.js:286:25)
    at Module.require (module.js:365:17)
    at require (module.js:384:17)
    at Object.<anonymous> (/usr/local/lib/node_modules/registry-static/lib/args.js:108:25)
    at Module._compile (module.js:434:26)
    at Object.Module._extensions..js (module.js:452:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Module.require (module.js:365:17)

Although I have the module exporting the blob store:

$ ls -al                                                                                                                                                                                                                                                                         
total 464
# ...
-rw-r--r--   1 david  staff      84 Oct 12 19:36 registry-ipfs-blob-store.js
# ...

$ cat registry-ipfs-blob-store.js
var ipfsBlobStore = require('ipfs-blob-store')
module.exports = ipfsBlobStore().mfs

Could you give me a hint on how to do it? Thank you :)

note: mfs is only available on IPFS dev0.4.0

$ registry-static --help
registry-static@2.0.0

Examples:
  registry-static -d my.registry.com -o /var/www/registry
  registry-static -c /path/to/myconfig.json


Options:
  --help, -h    Show help
  --config      Use a config file
  --domain, -d  The domain replacer                                 [required]
  --dir, -o     The output directory                                [required]
  --registry    The registry to mirror from                                     [default: "http://registry.npmjs.org/"]
  --limit, -l   Limit the number of concurrent downloads                        [default: 10]
  --user, -u    Set the user that this process should change to
  --group, -g   Set the group that this process should change to
  --hooks       Provide a hooks module.
  --clean       Clear the sequance file
  --check       Crawl the tarballs and check all of their shasums
  --sync        Crawl the json files and look for outdated index files
  --report      Used with --check, write a json report for all missing files
  --log         The file to log the output to
  --quiet       Turn down the logger
  --index       An index.json for the whole registry                            [default: "/usr/local/lib/node_modules/registry-static/defaults/index.json"]
  --error       A 404 file                                                      [default: "/usr/local/lib/node_modules/registry-static/defaults/404.json"]
  --replport    Port to listen on for REPL
  --one         Sync only the provided package, right now, then quit
  --spawn, -s                                                                   [default: 20]
  --tmp, -t                                                                     [default: "/var/folders/_y/h419_yvx6tb6wjkgjr6gp0p40000gn/T"]

[jbenet]

(The root ipns name would point to another ipns key, the current signing key from a rotation, etc. that way can revoke it by moving ipns -> ipns ptr at a given time, bounded only by record lifetimes)

—
Sent from Mailbox

On Tue, Nov 3, 2015 at 9:55 AM, Stephan Seidt notifications@github.com
wrote:

Had an idea, been wondering whether that's being tried or pursued:
I'd like to have a PEER that offers a FILE of PACKAGE at VERSION as such:

/ipns/PEER/npm/PACKAGE/VERSION/path/to/FILE

Would something like this make sense? In a way, it would be useful to have a good way to access npm packages canonically, and have a trusted authority for feelgood-immutable npm@version names to ipfs objects.

Reply to this email directly or view it on GitHub:
#2 (comment)

[daviddias]

@ehd that is similar to what we are doing, where we have one PEER that has the entire npm by a similar folder structure (the one used in registry-static) and then the others, by copying that root path /npm have access to the folder structure and respective hashes, which then can be server by 1 or more nodes.

[seidtgeist]

I wouldn't count too much on specific IPNS names yet until we have easy ways of delegating record signing through auth agents, so we can keep important keys more secure and isolated from the Internet.

(The root ipns name would point to another ipns key, the current signing key from a rotation, etc. that way can revoke it by moving ipns -> ipns ptr at a given time, bounded only by record lifetimes)

Sounds great. Can't wait.

@ehd that is similar to what we are doing, where we have one PEER that has the entire npm by a similar folder structure (the one used in registry-static) and then the others, by copying that root path /npm have access to the folder structure and respective hashes, which then can be server by 1 or more nodes.

What do you mean by "the others"?

Here's an interesting project by @mjackson: http://npmcdn.com – It's a heroku app with a CDN in front, serving files out of npm packages.

Idea: Use trusted computation instead of trusted names for this problem (Distributed request network?)

f = hash of a javascript function (could be any lang with sufficient metadata or linking (unikernel!))
x = hash of an input

Someone else executes it for me, caches the result. I trust them to provide y = f x

If I paid them to do this for me, and they provided a binding legal contract: Very nice and trustworthy. You know who's got at providing a useful and stable environment for running code? CI services. There probably is a nicer blockchainy/ethereum smart contract way, but I'm not well versed here.

Something like npmcdn would be trivial with this: f is the lookup function, x is {name, version, pathToFile}, y is the computation result.

/ipcompute/funhash/inputhash -> result?

There are many things we could use that work like npmcdn

Benefit:

• Anyone could do the computation
• Code, inputs and outputs hosted on IPFS
• Reproducible/verifiable
• Fault-tolerant (have multiple nodes computing & caching)

Q:

• Is this just something on top of ipns? pub-sub? req-res? notifications?

[daviddias]

@ehd like Amazon lambda/WebTask for the network? That is a field that is very very exciting and something we keep going back, it is definitely a valid idea.

Now when it comes to npm on IPFS, what we want is to be able to install the modules from the network, which is far less ambitious that having a universal way to do computation in a P2P network (btw I did work towards that goal in my MSc thing that I got to share at DTN last May, we can talk more or open a issue about it :):)).

The main challenge was, without forcing users to adopt a new way to install their modules, how could we offer the same npm install module-name experience? Well, the answer to that was IPNS and mfs.

Quick update on state of things

• got reginabox to use a blob-store driver for storage https://github.com/diasdavid/reginabox
• read only mode (avoid forcing the download of all the modules, useful if ipfs-blob-store is being used and the modules are already in the network)
• ipfs-blob-store
• fat storage node ready to glob the entire npm
• rebase ipfs 0.3.9 onto 0.4.0 so that the new js-ipfs-api works again (which is a dependency of ipfs-blob-store)

[seidtgeist]

@ehd like Amazon lambda/WebTask for the network? That is a field that is very very exciting and something we keep going back, it is definitely a valid idea.

Yeah, it's so cool! I'll try to visualize my idea on the weekend.

The main challenge was, without forcing users to adopt a new way to install their modules, how could we offer the same npm install module-name experience? Well, the answer to that was IPNS and mfs.

And that's awesome. It'll be really cool and useful to have the that for npm end users and also the idea above. It relies on some system serving all the archives and metadata, and using npm on ipfs instead of the registry brings those benefits with it.

[daviddias]

The moment is near, you hear! :D

So, right now, using IPFS 0.4.0, with mfs, we can, using the new shiny js-ipfs-api and ipfs-blob-store, put the npm registry on IPFS and serve modules from there. To make this happen (and please replicate it so it works for everyone).

1. git clone git@github.com:diasdavid/reginabox.git
2. cd reginabox && npm i && npm link
3. cd .. && mkdir custom-blob-store
4. Create a index.js file with module.exports = require('ipfs-blob-store')().mfs && npm i ipfs-blob-store
5. Install IPFS dev0.4.0 branch and ipfs daemon
6. run reginabox mirror npm-registry <full-path-to-custom-blob-store>/index.js

This is to download the entire npm into an IPFS Node, once our fat Node has finished that, you will be able to do:

1. everything till step 5
2. run reginabox mirror npm-registry <full-path-to-custom-blob-store>/index.js true for fetch modules from the network instead of trying to pull everything down from npm first

@bengl, I wanted to make this setup more nice, without all of the installing steps, any suggestions? Would it be good to have all of this setting up be part of reginabox, so that devs only had to do npm i regianbox -g once and run the cli?

[dignifiedquire]

@diasdavid how about adding another module reginabox-cli which you can install globally and does all that work, without polluting the actual reginabox module?

[dignifiedquire]

So it would more like this:

$ npm i -g reginabox-cli
$ ipfs daemon 
$ reginabox mirror npm-registry
# internet connection lighting up because of all the traffic

[daviddias]

@dignifiedquire, regianbox is already a CLI :) so I guess we just need the changes on https://github.com/diasdavid/reginabox to go to https://github.com/yahoo/reginabox

[dignifiedquire]

hmmmmmmmm now that you say it like that cough ignore my comment, but it would still be nice to put all that manual work you mentioned in there or somewhere else to automate it more

[dignifiedquire]

It has begun

/Volumes/npm/custom-blob-store
❯ ls npm-registry |wc -l                                                                     
     185

[travisperson]

👍

[ion1]

@dignifiedquire

It has begun

[bengl]

I'd suggest that a module that just implements the registry API by talking to IPFS would be sufficient, rather than having everyone use reginabox. reginabox is supposed to mean registry-in-a-box. Most users aren't going to need the whole registry mirrored, and thus have no need to run registry-static (the main component in reginabox, and probably the module this should have been based on in the first place).

That way users could do something like:

$ npm i -g ipfs-reg-mirror # or something like that

and then start that and use it as a registry in their .npmrc. I don't think we're likely to accept patches to reginabox that disable the actual mirroring when run with the mirror command, because that's its primary purpose.

reginabox also carries with it a bunch of code for discovering peers on your local network, which users likely don't need since that's kind of the point of having IPFS.

So, in summary, I'd say: a small module implementing only the http server that serves from ipfs-blob-store is all that an average user needs. Not reginabox.

Also, the fat server that's doing the following of the public registry doesn't need reginabox either, only registry-static.

[daviddias]

@bengl ok, sounds good, starting https://github.com/diasdavid/registry-mirror :)

[dignifiedquire]

@diasdavid @bengl registry-static is pretty slow :( not really using the full connection or cpu available. I'm only at 10% after 20h even though my connection could have downloaded 800gb in this time according to my calcs

[bengl]

@dignifiedquire No excuses, but here's an explanation of what's going on: registry-static uses the couchdb replication feed from skimdb.npmjs.com, and is limited by that. On each new piece of replication feed data, it goes through and processes all the metadata and tarballs for one package and all its versions, and then only upon completion moves on to the next piece of data from the replication feed. There are lots of tiny little files, and this has impacts on HTTP performance.

That being said, PRs to have more efficient pipelining of HTTP requests in registry-static would be considered. 😄

[dignifiedquire]

@bengl created a PR to follow-registry to improve performance :)

[daviddias]

Published the first version https://github.com/diasdavid/registry-mirror Wanna give it a try? Now it is only registry-mirror --ipfs --clone` and good to go :)

[balupton]

I think replicating the entire registry into ipfs is the wrong approach here, as the popularity of npm will make the replication into another system a constant catchup.

The use case really seems to speed up npm in places of poor internet connectivity to the npm registry — australia, conferences, islands, etc.

The usage can be opted into. Essentially one only needs to replicate the files into ipfs that they are using. Essentially one could alias "npm install" to be "install from ipfs if there, otherwise install from npm then publish to ipfs" — one would need a ipns space to alias package names and versions to directory hashes, however I imagine the registry also has the hashes so would be easy enough to verify (is this npm install alias publishing the same version to hash link as to what the npm registry indicates). An alias could also be creates for "npm publish" to publish to ipfs after publishing to the registry.

While not comprehensive would allow moderare opting in abilities.

[perguth]

It's really time for NPM on IPFS: How one developer just broke Node, Babel and thousands of projects...

[balupton]

Perhaps @rstacruz from http://github.com/rstacruz/pnpm may be interested in this?

pnpm's architecture works really well for my ideal here, as pnpm I believe downloads each package to user directory, then symlinks the package directories to the relevant node_modules directory.

Combining this with IPFS, this user node_modules directory could be added to IPFS. Making all ipfs-pnpm users seeders of the packages they have installed - the ideal.

This works for hosting files, but doesn't work for hosting meta data and publishing. How is that planned to be addressed? Specifically publishing.

[hsanjuan]

This shipped: https://github.com/ipfs-shipyard/npm-on-ipfs