Datastore config file format notes

[tv42]

Current ~/.ipfs/config has

  "Datastore": {
    "Type": "leveldb",
    "Path": "/home/jdoe/.ipfs/datastore"
  },

Which is actually a lie; since ipfs/kubo@24daeec the real config would look more like the following imaginary config:

  "Datastore": {
    "Type": "mount",
    "Mountpoints": [
        // TODO this format cannot mount the same thing in two locations
        {
        "Path": "/blocks",
        "Mount": {
            "Type": "flatfs",
        "Path": "/home/jdoe/.ipfs/blocks"
        }
    },
        {
        "Path": "/",
        "Mount": {
            "Type": "leveldb",
        "Path": "/home/jdoe/.ipfs/datastore"
        }
    }
     ]
  }

Or, if one wants the flexibility to mount the same backend in multiple places in the tree, alternative 1:

  "Datastore": {
    "Type": "mount",
    "Backends": {
        "default": {
        "Type": "leveldb",
        "Path": "/home/jdoe/.ipfs/blocks"
    },
    "blocks": {
        "Type": "flatfs",
        "Path": "/home/jdoe/.ipfs/blocks"
    },
    },
    // order matters, first match is used
    "Mountpoints": [
      {
        Path: "/blocks",
        Backend: "blocks",
        // True is the default value; being able to mount same
        // backend multiple times is pointless without being able
        // to set this to false, but maybe that's not a real need,
        // and the other variant without named backends is fine.
        TrimPrefix: true
      },
      {
        Path: "/",
        Backend: "default",
      }
    ]
  }

alternative 2:

  "Datastore": {
    "Type": "mount",
    "Backends": {
        "default": {
        "Type": "leveldb",
        "Path": "/home/jdoe/.ipfs/blocks"
    },
    "blocks": {
        "Type": "flatfs",
        "Path": "/home/jdoe/.ipfs/blocks"
    },
    "prefixedBlocks": {
        "Type": "addPrefix",
        "Prefix": "/blocks",
        "Backend": "blocks"
    },
    },
    // order matters, first match is used
    "Mountpoints": [
        {
        Path: "/blocks",
        Backend: "prefixedBlocks",
    },
        {
        Path: "/",
        Backend: "leveldb",
    }
     ]
  }

Please don't interpret me as suggesting all this work should actually done; I think the mount functionaliy is a transition pain that, if I had realized how little there is outside of /blocks, I would have completely avoided. I'm merely using this to introduce the Datastore config, and wanted to point out how it's not really accurate anymore anyway.

Now, with that out of the way, S3 integration might happen as simply as writing support for the following:

  "Datastore": {
    "Type": "s3",
    "Region": "s3-us-west-1",
    "Bucket": "my-ipfs-data",
    "Prefix": "example/",
    ... auth from env, file, strings ...
  },

With caching:

  "Datastore": {
    "Type": "cacheWritethrough",
    "MaxSize": "200M",
    "Cache": {
        "Type": "flatfs",
        "Path": "/home/jdoe/.ipfs/cache"
    },
    "Store": {
        "Type": "s3",
        "Region": "s3-us-west-1",
        "Bucket": "my-ipfs-data",
        "Prefix": "example/",
        ... auth from env, file, strings ...
    }
  }

Note how I ripped out the leveldb part wholly. It really isn't used enough to justify its existence.

[hsanjuan]

This issue is a bit old and I think the problem was resolved (users are able to configure different mount points and wrap datastores on each others with the Spec/child format. So I am going to close it.