Error: garbage collection aborted: could not retrieve some links

[obo20]

Version information:

go-ipfs version: 0.4.19-
Repo version: 7
System version: amd64/linux
Golang version: go1.11.5

Type:

• "bug":
• "panic":

Description:

When attempting to garbage collect on a node, I'm receiving the following error output:
Error: could not retrieve links for QmU1aLywufB7MwaXfFyj2Qy1v6w93fkA9xR34Kz1kZv3vX: merkledag: not found
Error: could not retrieve links for QmNturQjFAhXF3N9f4pLcFtoowdHh2f43wfirrBH7fyuDr: merkledag: not found
Error: could not retrieve links for QmR2LMpncATAdWhVYGehXpQs32CS7UXRNvaRMh6M2PgTmf: merkledag: not found
Error: could not retrieve links for QmVcV9kLxbUYxf9vbG5eSDQXoFQ7JECCqDJhumhswJZP7R: merkledag: not found
Error: could not retrieve links for QmRquTQaRxyARq2LGPQtbaw2d8H74uL5YHaDksk1y8vDpA: merkledag: not found
Error: could not retrieve links for Qmch5dhdJeCrFxDXHuLDM27vK8VD2j5WUJnwnmqAKagSvG: merkledag: not found
Error: could not retrieve links for QmQT42D8ptwFeShqsczdUw8b8XZ9B4DFEQZ1mRqsyjkWxW: merkledag: not found
Error: could not retrieve links for QmPhtoB7mJUnYTk6bz6tGmrbk4jJEfRsZ72i9dyZWzPR82: merkledag: not found
Error: could not retrieve links for QmdDsNhWyfj2uV2ejyC8tokPRV2wM8kbCERLo4hUK6aVsx: merkledag: not found
Error: could not retrieve links for QmUKMojL2S2oov4Q6CER3uWK7uUUYw77JmvRGT3TyLTtKx: merkledag: not found
Error: could not retrieve links for QmYGKfvuTYuvLWTeuQBeX4Yymfw9KYe43txhEYavVDFBdK: merkledag: not found
Error: could not retrieve links for QmP2hiUxQtes9Sip7TqenTBnFsqnH7akfJEv5o1UBes3Au: merkledag: not found
Error: garbage collection aborted: could not retrieve some links
Error: encountered errors during gc run

When I ran ipfs repo verify everything worked fine and I received "verify complete, all blocks validated"

When I ran ipfs pin ls QmU1aLywufB7MwaXfFyj2Qy1v6w93fkA9xR34Kz1kZv3vX (This is the first broken link) I received:

QmU1aLywufB7MwaXfFyj2Qy1v6w93fkA9xR34Kz1kZv3vX indirect through QmYGCUXNzo1hZVhJKd2efhu3nJHy3He5o9QjFpmgy8TJwg

When I ran ipfs object stat QmYGCUXNzo1hZVhJKd2efhu3nJHy3He5o9QjFpmgy8TJwg I received:
NumLinks: 5 BlockSize: 248 LinksSize: 222 DataSize: 26 CumulativeSize: 1109580

I have not been able to successfully retrieve QmYGCUXNzo1hZVhJKd2efhu3nJHy3He5o9QjFpmgy8TJwg from the node however. My node recognizes obviously recognizes it as pinned though.

I also have not been able to run a successful ipfs pin ls with any of the other broken hashes. Only QmU1aLywufB7MwaXfFyj2Qy1v6w93fkA9xR34Kz1kZv3vX

Upon running ipfs pin verify --verbose the following pins error out:

QmVep5xUUHfmwdbAuApzE8R2H5NyUJw8npYRfxr54AZoHy broken
  QmP2hiUxQtes9Sip7TqenTBnFsqnH7akfJEv5o1UBes3Au: merkledag: not found
  QmNturQjFAhXF3N9f4pLcFtoowdHh2f43wfirrBH7fyuDr: merkledag: not found
QmQN7hP8xzm5vFApADSEsSkvc5kEmBeAPeQ2eNLEC9M3tQ broken
  QmP2hiUxQtes9Sip7TqenTBnFsqnH7akfJEv5o1UBes3Au: merkledag: not found
QmaLNW6NgX5eD7XBHnCfUHQMczyPHMADFNbgL7wVGF1z7K broken
  QmU1aLywufB7MwaXfFyj2Qy1v6w93fkA9xR34Kz1kZv3vX: merkledag: not found
  QmR2LMpncATAdWhVYGehXpQs32CS7UXRNvaRMh6M2PgTmf: merkledag: not found
QmUpFLqiffiRdTLJUXpsbq6ZcaMmzHYKQcJSSAeK3kT2ad broken
  Qmch5dhdJeCrFxDXHuLDM27vK8VD2j5WUJnwnmqAKagSvG: merkledag: not found
  QmQT42D8ptwFeShqsczdUw8b8XZ9B4DFEQZ1mRqsyjkWxW: merkledag: not found
QmdWeszMyxnQ8122gbTJenuXXwtwseNih3apopHZJZohNz broken
  QmVcV9kLxbUYxf9vbG5eSDQXoFQ7JECCqDJhumhswJZP7R: merkledag: not found
  QmRquTQaRxyARq2LGPQtbaw2d8H74uL5YHaDksk1y8vDpA: merkledag: not found
QmYGCUXNzo1hZVhJKd2efhu3nJHy3He5o9QjFpmgy8TJwg broken
  QmU1aLywufB7MwaXfFyj2Qy1v6w93fkA9xR34Kz1kZv3vX: merkledag: not found
  QmNturQjFAhXF3N9f4pLcFtoowdHh2f43wfirrBH7fyuDr: merkledag: not found
QmbrSGHVhEK6uA7ofo3gdsxunF7Ntv8PE3iSGp81N1QqW6 broken
  QmYGKfvuTYuvLWTeuQBeX4Yymfw9KYe43txhEYavVDFBdK: merkledag: not found
QmRxcG7NSeNJEeBc2pgbcvprFpBp1tbDwLvmuw4WcuHUjX broken
  QmPhtoB7mJUnYTk6bz6tGmrbk4jJEfRsZ72i9dyZWzPR82: merkledag: not found
  QmdDsNhWyfj2uV2ejyC8tokPRV2wM8kbCERLo4hUK6aVsx: merkledag: not found
  QmUKMojL2S2oov4Q6CER3uWK7uUUYw77JmvRGT3TyLTtKx: merkledag: not found

First: Is there a way to recover from something like this?'
Second: Is there anything that can trigger such a state?

[Kubuxu]

First: there is: remove broken pins (ipfs pin rm) but you will lose that content.

Second: internally in go-ipfs, no it shouldn't be possible but if the machine encountered power loss, filesystem corruption or disk failure it would be possible.

Are you running default datastore config or for example badger? Can you share the output of ipfs config Datastore.

[obo20]

internally in go-ipfs, no it shouldn't be possible but if the machine encountered power loss, filesystem corruption or disk failure it would be possible.

Could you elaborate on this? If the node encountered power loss during pinning or adding of content? Or just in general? My database is showing these records as all having been successfully pinned, and that doesn't happen until the pinning process completes on IPFS's side.

• My datastore is not badger. Here's my setup:

  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "24h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "1000GB"
  },

[Kubuxu]

We ensure that we send all the content to the disk before we add the pin to the pinset. In case of the power failure (and well-behaved filesystem), it should all be there or there shouldn't be a pin for it. It depends on the operating system and filesystem, whether in case of a power failure this content survives.

You can try running ipfs refs -r on hashes listed as broken to hopefully recover them from the network.

[Stebalien]

Fix in #6288, we think.