Implement new index type that also includes mutltihash code

[masih]

Implement a new CARv2 index that contains enough information to
reconstruct the multihashes of the data payload, since CarIndexSorted
only includes multihash digests. The new index builds on top of the
existing IndexSorted by adding an additional layer of grouping the
multi-width indices by their multihash code.

Note, this index intentionally ignores
any given record with multihash.IDENTITY CID hash.

Add a test that asserts offsets for the same CID across sorted index and
new multihash sorted index are consistent.

Add tests that assert marshal unmarshalling of the new index type is as
expected, and it does not load records with multihash.IDENTITY digest.

Relates to:

• Add CARv2 MultihashIndexSorted codec multiformats/multicodec#227

Fixes #214

[masih]

@willscott This index implementation uses a nested map of length -> multihash.Code -> singleWidthIndex. I am not sure if we ever need to support hash functions that produce variable length digests. IDENTITY is an obvious one but that's a special case which we will be skipping over as captured in #215. If not the storage structure could be simplified.

I'd love to get your feedback whenever you get a chance. 🍻

[willscott]

@willscott This index implementation uses a nested map of length -> multihash.Code -> singleWidthIndex. I am not sure if we ever need to support hash functions that produce variable length digests. IDENTITY is an obvious one but that's a special case which we will be skipping over as captured in #215. If not the storage structure could be simplified.

I'd love to get your feedback whenever you get a chance. 🍻

Unless we can think of cases where we wouldn't be able to get away with the simpler case, lets see if we can get away with it?

[ribasushi]

@masih @willscott one of these days when I carve some time to build what I was planning to build before joining PL, I am definitely going to use multihash-level truncated full-sha3-512 ( can go into the weeds why if interested ). So you won't be able to index my content e.g. ( whether this is a showstopper or not: 🤷 )

[ribasushi]

multihash-level truncated full-sha3-512

To be precise: 0x01551420{{32bytes}}

[willscott]

that means the car will have to be read through block by block to calculate the list of full multi-hashes to provide for indexing, rather than being able to crib it off the index of the car. Eventually, it sounds like we should be able to store the computed announcement for indexer nodes if we do have to do the extra work of computation to generate the announcement. that'll probably happen after MVP (although @masih already wrote the walking code once 😅 )

[willscott]

@rvagg anything we should be aware of in putting together a PR for the additional index multicodec format over in multicodecs?

[rvagg]

nope, should be pretty straightforward, just add a 0x0401 and it should get merged in short order

[masih]

nope, should be pretty straightforward, just add a 0x0401 and it should get merged in short order

thanks @rvagg, going to open a PR for it just now if we are happy with 0x0401 value

[masih]

Self-review

[masih]

Postponing the optimisation here to reduce the size of this PR