ipfs drive

[daviddias]

@Gozala started implementing an ipfs drive feature which is essentially a mix between mfs + ipns (for syncing). I really like the name drive for the API and it makes total sense that the API not only creates a virtual folder but also supports the sync between multiple nodes.

Phasing out mfs in favor of ipfs drive would simplify and answer a ton of questions from our users that do not get at all what mfs is (not documented).

@whyrusleeping @pgte thoughts?

[ghost]

Potentially relevant: ipfs name follow - ipfs/kubo#4435 & ipfs/kubo#4462

[pgte]

@diasdavid @Gozala interesting! Is the API for ipfs drive defined?

[Gozala]

I actually did not knew about mfs and while I did mention wanting to solve ipns part, that was not necessarily part of the ipfs-drive in my mind you’d mount a drive onto ipns node that would update pointer to latest address.

In terms of API and code I just have a small piece written down https://github.com/Gozala/ipfs-drive/blob/master/src/Drive.js as I end up in the rabbit hole of figuring out all of ipfs & type annotating it :) it is in my head though & I’m happy to write it down if that would help. Mostrly I was thinking to change along the lines of how immutable data structures change mutable interface of their mutable counterparts

[victorb]

I think this should much rather be a part of the already existing files api. We could add additional method for sync which would ensure that two root-nodes are the same. Something like this:

ipfs files sync /ipfs/Qmasdfkla /my-public-files

Adding a flag for doing this automatically via the daemon, in the background

ipfs files sync --watch /ipfs/Qmasdfkla /my-public-files

Optionally, the sync command could accept a IPNS hash and you'll always have the latest version. Not entirely sure about syncing two different peers though, you would probably watch each others IPNS addresses.

Phasing out mfs in favor of ipfs drive would simplify and answer a ton of questions from our users that do not get at all what mfs is (not documented).

Changing a internal implementation (MFS) for a user-land command feels confusing. You're suggesting we rename the files API to drive? As I understand, it would still use MFS under-the-hood.

[magik6k]

For files sync there is already an issue in go-ipfs: ipfs/kubo#4614.

Phasing out mfs in favor of ipfs drive would simplify and answer a ton of questions from our users that do not get at all what mfs is (not documented).

What would be the difference other than name and auto-publish to ipns (which we could theoretically allow with symlinks to /ipns)?

[whyrusleeping]

Phasing out mfs in favor of ipfs drive would simplify and answer a ton of questions from our users that do not get at all what mfs is (not documented).

Thats like phasing out ext4 because you want to use a new desktop file browser.

[daviddias]

@whyrusleeping perhaps the word "phasing" didn't communicate the intention well. I just meant that using a name such as drive will be more intuitive for users than mfs, the Mutable File System.

[whyrusleeping]

but we only call it mfs internally, where it describes the mutable file system structures that back the ipfs files api. I'm still not following.

[lidel]

FYSA there is somehow related design discussion on how UI for ipfs.files "File Browser" should look like: ipfs/ipfs-gui#9 (It touches areas such as reusable, platform-agnostic web-based File Browser GUI and native OS integration).

[daviddias]

There isn't a single event that goes by that I don't get at least 50 requests for a thing that looks like this Drive API.

Phasing out mfs in favor of ipfs drive

I really just meant a rename to something more familiar to users and then evolve what it is.

[lidel]

Yeah I must say I agree at this point.

The UX problem we have is that a lot of features are hidden behind terms like "MFS" or "Files API", plus (based on conversations I had) the / of MFS is really confusing in context of having /ipfs/ and /ipns/ namespaces available to commands like ipfs files cp.
Not to mention that ipfs.files.add actually does not add anything to your MFS so people use ipfs.files.write which confuses them even further, as it produces trickle-dag and a different hash (should be .append or something).

Seen people having the same questions over and over in the past six months. I think changing the way we communicate those concepts could remove a tons of confusion and improve onboarding of new users.

ipfs drive is a very good name because it imports a familiar mental model: technical people immediately get "a drive" is a separate root / and in that context a lot of vague operations become super clear to newcomers.

It means just renaming files to drive + keeping files alias would bring some added value (IMO).

---

Click to expan additional notes

• we should prototype and investigate UX and find things that look odd
  • should we rename ipfs files to ipfs drive? or move it under ipfs drive files? etc
• what if we allow user to have additional "drives" in a fashion similar to how additional keys can be added in ipfs key** (commands defaulting to default drive unless --key-id is passed)

Some CLI ideas:

# mount MFS into userspace + watch for changes
ipfs drive mount /home/user/ipfs-drive 

# sync with other peer (one time or continuous, or bi-directional) 
ipfs drive sync [--watch [--bidirectional]] <peerID>[:/optional/source/path] [/optional/destination/path]

# personal pet peeve: 'cp' should only work for things that are already in 'drive', adding /ipfs/* is.. adding
ipfs drive add /home/user/file.jpg [/optional/drive/destination/file-from-local-fs.jpg]
ipfs drive add /ipfs/Qmdsfsdf/test.file [/optional/drive/destination/test-from-ipfs.bin]

[lidel]

Dropping some additional notes/ideas around MFS and ipfs drive:

• Start fresh with ipfs drive namespace
  • Enables us to design intuitive API and rename/tweak behavior of things like add, write, cp without creating problems move from ipfs.files.add to ipfs.add did
• Current MFS becomes the root of the default "drive"
• Objective I: User can create multiple drives and define different privacy controls for each
  • e.g. disable reproviding or have a drive encrypted using one of keys
  • it would be also a neat abstraction for mounting directories from local filesystem
• Objective II: ipfs drive API becomes the easiest way of updating IPNS
  • The idea is to hide IPNS publishing dance behind an abstraction of "Published Drive"
    • It could be implicit: "every published IPNS website can be managed using drive API"
    • or explicit: "every drive can be marked for seamless IPNS publishing"
  • When data in a drive changes, IPNS record get updated automatically
    • I believe this is the killer feature of ipfs drive. It would not only simplify IPNS publishing, but provide simple abstraction to build UIs on top of it.
• Objective III: Shared Drives
  • In the future, when something like IPNS multiwriter is a reality, we would have abstraction of "published drive" ready in place to integrate bi-directional sync.

(cc https://github.com/ipfs/interface-js-ipfs-core/issues/284, ipfs/kubo#1607)

[mikeal]

Enables us to design intuitive API and rename/tweak

Do we have to move to a new namespace to do this? @alanshaw mentioned that he plans on doing this to the existing namespace in js-ipfs.

[Gozala]

I'd like to provide few suggestions:

• As @lidel already pointed out there probably will be more the one drive, ideally API would be designed with that in mind.
• Dat uses hyperdrive filesystem abstraction over it's storage layer, which is also more or less what Beakers DatArchive API is. IMO it provides a very intuitive mental model & was inspiration for the drive API mentioned in the beginning of the thread. It might make sense to aim for API compatibility with DatArchive where it makes sense & propose changes to beaker team where it doesn't.
• IPFS has tendency to backing all the APIs into core, it has benefits as users get the whole thing but on the flip side it means that library grows in size and alternative APIs are second class. It might be worth considering breaking out MFS / Drive so they can be pulled-in as necessary and better one could become standard. (For example Dat hyperdb is an example of very different API over same dat storage layer being created)
• One reason people find Dat more intuitive is because it creates familiar mental model. You create a site / archive that has a unique URL and changes are replicated to peers. That is to suggest that seamless IPNS integration is the key element of this. I would go as far to suggest that each drive probably should just have URI in from of /ipns/${key}/
• Lately I've being tinkering with data feeds over IPFS ideally it should be possible to mount feed+virtualization e.g. automerge as drive. Maybe there should be notion of volumes - as in mounted drive creates a volume. Mounted data feed also creates a volume.