Bump Repo Version and Provide Migration before strict Cid code lands

[kevina]

Right now #4751 is providing a stop-gap measure to avoid allowing insecure hashes. Things won't break, but the insecure hashes will become useless. Once the strict Cid code (ipfs/go-cid#40) lands things will likely break badly if there are any insecure hashes. I recommend we bump the repo version and provide a migration tool that just checks and aborts in there are any CID's with insecure hashes.

I also suggest, though not require, we provide a tool to automatically convert the insure hashes to a secure hash so that data is not lost. The same tool can just offer to delete them instead.

[whyrusleeping]

This is a good idea, a migration is likely the best way forward.

[kevina]

@whyrusleeping I would like to go ahead and write this. I am thinking if the migration tool finds invalid Cid's and stores them in a file in the repo (and possible list then to stdout if there are not too many). (I want to store the list somewhere because it is very easy to lose the output of the migration command.)
It then instructs the user to remove them somehow. It will suggest command to remove them with the previous IPFS version but warn of the obvious data lose. If that is not adequate I would like to suggest the user contact us somehow because I want to know about cases where insure hashes are already used. I can then help the user on a case by case bases. If necessary I can write a tool that could convert the Cid's to a secure hash.

[kevina]

If we go ahead with this repo migration I would like to combine this migration with one that will remove all id-hashes from the datastore to make #4910 easier.

[kevina]

cc @Stebalien @Kubuxu

[kevina]

@Stebalien @whyrusleeping told me you are also working on a DHT datastore migration, what the status of that? I would like to go ahead and write the code that checks for invalid hashes and also removed identity hashes but don't want to clash with what you are doing.

[Stebalien]

This is blocked on #5007. That should be done and ready to merge (once I get a final sign off). That should finally unblock literally everything.

[Stebalien]

If we go ahead with this repo migration I would like to combine this migration with one that will remove all id-hashes from the datastore to make #4910 easier.

If you do that, we should probably land #4910 at the same time. I'd rather not temporarily break ID hashed content (although..., in practice we'll break absolutely nothing because nobody's using them..., maybe not an issue).

[kevina]

If you do that, we should probably land #4910 at the same time. I'd rather not temporarily break ID hashed content (although..., in practice we'll break absolutely nothing because nobody's using them..., maybe not an issue).

That is my hope. Except for the addition of thirdparty (which I intent to fix) I am hoping there is nothing controversial about it which should allow it to go in fairly quickly.

[kevina]

See ipfs/fs-repo-migrations#77 for an initial implementation of the migration.