GC should skip over raw blocks

[Stebalien]

Currently, when GC traverses the pin set, it loads all blocks, even raw blocks. This means:

1. We need to perform unnecessary work (fetching raw blocks from disk).
2. When using the filestore, if the backing data disappears, the associated leaf blocks will return errors on read. At the moment, this will completely break GC.

We should simply avoid loading raw blocks when garbage collecting (or at least just check for existence instead of reading the block). This will:

1. Improve GC performance significantly when working with large amounts of data with raw leaves.
2. Make GC continue to work even if the filestore gets corrupted.

[RubenKelevra]

How about adding three flags to the GC operation to determine how the GC should handle filestore and urlstore?

1. --filestore[/urlstore]=ignore

   Just skip all filestore/URL store entries completely and don't check if they are accessible.

2. --filestore[/urlstore]=keep

   Keep all entries which are still accessible, regardless if they are pinned or not.

   Warn about non-accessible files/urls which are part of a pin/file.

3. --filestore[/urlstore]=drop - default

   Drop entries from IPFS if they are not pinned/in the MFS

[hsanjuan]

Is this work meant to be scheduled soon or would we rather wait to doing a larger GC revamp (possibly after raw-multihashes and pinning system changes)? I have tentatively tagged as P3.

[Stebalien]

P3 seems right given that it won't help with performance in many cases. My main interest is actually allowing GC to progress even when filestore blocks are missing.

[poonai]

The GC should avoid fetching raw blocks if the given CID's codec is raw. Is this right?

If it's right. I would like to raise a PR for this.

[Stebalien]

Exactly. PRs welcome!

[Stebalien]

This idiot (@Stebalien <-- this one) made bad assumptions and filed an issue without actually verifying the issue. Apparently, we don't load raw blocks when garbage collecting.

[Stebalien]

@balajijinnah thank you for implementing this, and again, I apologize for not verifying the issue first.