add pinning support to the urlstore

[Stebalien]

fixes #5833

[parkan]

err, actually, I don't understand why on line 110 we have test_must_fail ipfs cat $HASH2 > /dev/null

HASH2 is added w/o pin=false and isn't pin rm'd until line 114, so it should still be in the repo after the gc on line 90

[Stebalien]

HASH2 is added w/o pin=false and isn't pin rm'd until line 114, so it should still be in the repo after the gc on line 90

In these tests, we:

1. Add the files to IPFS.
2. Add them again with a new chunking algorithm by pointing the urlstore at the local gateway.
3. Unpin and GC the original files (line 91).

That means catting both files should fail (because the underlying URLs are no longer valid).

The actual test case is the ipfs filestore verify. We get errors for HASH2 because HASH2 is pinned (but HASH2a, the backing file, was garbage collected on line 91). However, we don't get errors for HASH1 because we told IPFS to not pin it (so the entire file was garbage collected on line 91, leaving no dangling references).

[parkan]

ah yes that makes sense, the urlstore retrieve should indeed fail if the underlying URL is missing 😄

the one test I would add is checking that a urlstore file added w/o pin=false does survive a gc, since that was the case that triggered this path -- I don't think we cover that currently?

maybe something like

test_expect_success "get large pinned file via urlstore after gc" '
  rm -f file3.actual &&
  ipfs repo gc > /dev/null
  ipfs get $HASH3 -o file3.actual &&
  test_cmp file3 file3.actual
'

doing so with a large file would additionally verify that internal nodes are not GC'd, which was a concern of why's

[parkan]

(on a side note, I wonder if the output of ipfs filestore verify should be porcelained such that it can be piped to ipfs pin rm, so you can easily perform the "unpin all unreachable *store items" operation)

[Stebalien]

maybe something like

Technically, it already tests that as ipfs filestore verify would print out no errors if the file wasn't pinned (it would have been GCed). However, I was being lazy. We should definitely be more explicit.

(on a side note, I wonder if the output of ipfs filestore verify should be porcelained such that it can be piped to ipfs pin rm, so you can easily perform the "unpin all unreachable *store items" operation)

I believe the correct way to do that right now is ipfs pin verify -q | ipfs pin rm. ipfs filestore verify will print out missing filestore blocks but pins will likely point to parent objects, not the missing blocks themselves.

[parkan]

great, the tests are explicit and descriptive now! 😄