[2021 Theme Proposal] NFS daemon for IPFS(-Cluster)

[RubenKelevra]

Note, this is part of the 2021 IPFS project planning process - feel free to add other potential 2021 themes for the IPFS project by opening a new issue or discuss this proposed theme in the comments, especially other example workstreams that could fit under this theme for 2021. Please also review others’ proposed themes and leave feedback here!

Theme description

Currently the ways to read and write data to a IPFS-cluster are kind of difficult. You cannot mount it and put files on it, which would probably follow the principle of the least surprise. Creating a way to mount a Cluster-Pinset via NFS or any other folder stored on IPFS, would make it much easier to read and write data. Not only locally but with NFS you can use the fail over between different members of the same cluster.

Hypothesis

NFS is a portable, robust and mature protocol which enables us to mount not only files and folders but also boot devices and attach hard-drive-(images) in a reliable way from a cluster of servers.

Using it as the transport layer on the network side makes sense to enable us to write and read with applications natively to IPFS without having to bother with portability.

A separate NFS-Server daemon would connect via the IPFS-API to a node and optional via the Cluster-API to a cluster.

Vision statement

With IPFS you can access the same data from everywhere in the world, making it a perfect solution to store large quantities of data inside a cluster which cannot be hold on a single physical location. NFS then allows us to boot operating systems from IPFS streamed from a different location, while blocks which are already local doesn't have to be transported and access files inside an operating system like any other drive.

Why focus this year

Netflix and the IPFS-Team improved the performance of IPFS significantly, making it possible to stream operating system images in real time on demand from a different location, caching it locally for further requests on the data.

Example workstreams

We need to look around if there's already a good NFS implementation we can adapt to not serve local files, but make requests to the IPFS(-Cluster)-API.

Other resources

I've mentioned this idea previously:

ipfs-cluster/ipfs-cluster#1146

[vorburger]

#90

[RubenKelevra]

@vorburger yeah, pretty related. :)

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[RubenKelevra]

@2color do I now have to bump my roadmap proposals once a month to keep them open, or what's the idea here?

[2color]

With the current direction of IPFS, there isn't a single team developing IPFS nor a singular roadmap. The updated readme in this repo should better reflect the current state of things. The short version is that there's currently a focus on specs and standards as a means to encourage innovation while maintaining interoperability.

As for bumping specific issues, I think it should be directed at a specific team.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[djdv]

@RubenKelevra
I have a standalone utility which mounts Go file systems via various host APIs (like FUSE, WinFSP, 9P, et al.).
And I've implemented a few of the IPFS APIs as guests for it already. (Files API example, an old IPFS example)

If you or anyone else would be interested, I could implement NFS as a host for it too.
So that a node's IPFS, IPNS, and Files APIs could be exposed as NFS servers.
(This includes non-local IPFS nodes as long as you have API access to it.)

I'm not familiar with the cluster APIs specifically, but I have a good amount of experience mapping APIs to Go's fs.FS and 9P, so I can imagine constructing a Go file system that maps file system operations to cluster API calls.
E.g. writing a multihash to some synthetic mounted file which triggers the cluster to pin that hash.
Like: fs mount fuse cluster -credentials=j:coolpassword /mnt/cluster; "Qm..." > /mnt/cluster/add.
You could host the same mapping as an NFS server as well.

If any of that sounds good just let me know and I'll put effort into it.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.