ERROR bitswap: Error writing block to datastore: sync /ipfs/blocks: invalid argument bitswap.go:303

[starrychloe]

20:35:43.128 ERROR    bitswap: Error writing block to datastore: sync /ipfs/blocks: invalid argument bitswap.go:303
20:35:43.922 ERROR  providers: error adding new providers: sync /ipfs/datastore: invalid argument providers.go:258
20:35:45.664 ERROR  providers: error adding new providers: sync /ipfs/datastore: invalid argument providers.go:258

and it just repeats...

Version information:

[vagrant@localhost go-ipfs]$ ./ipfs version --all
go-ipfs version: 0.4.6-
Repo version: 5
System version: amd64/linux
Golang version: go1.8

Type:

Error during running. Must be a bug somewhere.

Priority:

P4

Description:

IPFS daemon leaves errors in the console as it runs. I don't know what causes it, but clicking on big, old IPFS links seems to cause it. I can view /ipfs/datastore/ and /ipfs/blocks fine. I can access the cat pic http://127.0.0.1:8080/ipfs/QmW2WQi7j6c7UgJTarActp7tDNikE4B2qXtFCfLPdsgaTQ/cat.jpg fine. These links seem to cause the error to print:
http://127.0.0.1:8080/ipfs/QmcYKHpcBJNL9uaxxTZJGuk9QYBfS3cDuYKoELPhfdmRqn
http://127.0.0.1:8080/ipfs/QmbzpQmRULESivRvpP9SZL2vsu1dGkmczwVKCWzD2oXdMP
http://127.0.0.1:8080/ipfs/QmRcsML93Bwi8V9DiGhX4Zs9ypDgHSCWCpAMxd7cpHcS1f

[whyrusleeping]

Invalid argument is a new one.

After a small amount of research i see that some filesystems (particularly on virtualbox for some reason?) do not support calling fsync on directories. This might be a lead here.

What filesystem are you using?

[starrychloe]

I'm using vboxsf.

[vagrant@localhost ~]$ mount | grep ipfs
ipfs on /media/sf_ipfs type vboxsf (rw,nodev,relatime)
ipfs on /ipfs type vboxsf (rw,nodev,relatime)

The host (Windows 8.1) is using NTFS.

[Kubuxu]

You should be able to work it around by switching NoSync in config to true.

[whyrusleeping]

@Kubuxu NoSync still does a minimal amount of sync calls IIRC. Maybe youre right though?

[Kubuxu]

not sure, but it might help

[whyrusleeping]

@starrychloe any update here? does this still happen?

[starrychloe]

Yes, it seems to help. No invalid argument errors in the console.

[vagrant@localhost go-ipfs]$ grep NoSync /ipfs/config
    "NoSync": true,

[eginez]

hey all I'd like to draw attention to this bug again. This still happening for setups that have point the IPFS_PATH to some network mounted drive. The NoSync config is not really helping. A typical use case of this are raspberrypi devices running the ipfs. Typically pis don't come with meaningful storage thus drives are attached to it over common protocols(smb, nfs, cifs)

More details of the failure:

pi@raspberrypi:~/src $ ipfs version
ipfs version 0.4.13

pi@raspberrypi:~/src $ IPFS_LOGGING=debug ipfs init
19:20:35.934 DEBUG   cmd/ipfs: config path is /mnt/ext-dsk/ipfs main.go:257
19:20:35.938  INFO   cmd/ipfs: IPFS_PATH /mnt/ext-dsk/ipfs main.go:301
19:20:35.940 DEBUG   cmd/ipfs: calling pre-command hooks... main.go:295
19:20:35.941 DEBUG       lock: Checking lock lock.go:32
19:20:35.949 DEBUG       lock: File doesn't exist: /mnt/ext-dsk/ipfs/repo.lock lock.go:34
19:20:35.950  INFO   cmd/ipfs: checking if daemon is running... init.go:66
19:20:35.951 DEBUG   cmd/ipfs: executing command locally main.go:342
initializing IPFS node at /mnt/ext-dsk/ipfs
generating 2048-bit RSA keypair...done
peer identity: QmT7o2Mhh1KAkKwCVGFGru6saus3WdpYnwNVKEVbekNAA9
Error: sync /mnt/ext-dsk/ipfs/datastore: invalid argument

pi@raspberrypi:~/src $ ipfs daemon
Initializing daemon...
Adjusting current ulimit to 2048...
Successfully raised file descriptor limit to 2048.
19:22:23.899 ERROR   cmd/ipfs: error from node construction: failure writing to dagstore: sync /mnt/ext-dsk/ipfs/blocks: invalid argument daemon.go:320
Error: failure writing to dagstore: sync /mnt/ext-dsk/ipfs/blocks: invalid argument

pi@raspberrypi:~/src $ ipfs config show |grep Datastore -A 10
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "NoSync": true,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",

Here is another instance of the problem: https://unix.stackexchange.com/questions/329894/is-it-possible-to-mount-ipfs-folder-to-an-external-drive

Perhaps I can lend a hand fixing the issue, pointing me in the right direction would help get me started

[whyrusleeping]

@eginez Does the issue persist if you init with ipfs init --profile=badgerds ?

[eginez]

@whyrusleeping unfortunately yes I get the same error Error: While syncing directory: /mnt/ext-dsk/ipfs/badgerds.: sync /mnt/ext-dsk/ipfs/badgerds: invalid argument

[eginez]

Alright I think I got to the bottom of it. I think there is a bug while parsing the config file around these lines: https://github.com/ipfs/go-ipfs/blob/master/repo/fsrepo/fsrepo.go#L398.

The NoSync config is being ignored. Presumably in favor of sync inside the Spec map. Setting sync to false fixes the fs syncing issues and the pi seems to be happy.

We should either warn/fail on the presence of NoSync and let the user know the setting has move or override sync with NoSync preserve backwards compat. I'll submit a PR time permitting