Unify repository layout

[fd0]

As discussed in #940, I think we'll need to standardize the repository layout to what the local/sftp backends use. I'd like to start moving towards that direction. The rationale is that only having one layout regardless of the backend easily allows moving data around and having restic access the repo, so it "just works". The local layout is a good compromise between number of subdirectories and the number of files. We will probably drop the restriction of 16MiB per pack file in the near future, this should scale much better for very large repositories.

The to-do list is roughly as follows:

• Document the design decision in the design.rst file
• Write code to auto-detect the repository layout for the s3 backend (s3legacy or local)
• Add a migrate command (or something similar) to convert an s3 repo to the default format
• Change the default layout for s3 to local

The local/sftp backends are already able to auto-detect the layout and just use that.

(I may add more list items).

At the moment I'd like to keep the REST protocol and server at it is, @zcalusic made sure in the server implementation that the on-disk format is the local layout.

[fd0]

I've just merged #966 which adds support for the default layout to the s3 backend. The other issues will be addressed in a new PR. The default layout for the s3 backend is still s3legacy.

[fd0]

Done. default is now the default layout for the s3 backend, and there's a migrate command.

[middelink]

Will users get a nice message when they access a legacy S3 repo telling them to run migrate? Or is it all below the surface and they will never know till you drop s3_legacy support?

[fd0]

Hm, I haven't planned that in such detail. We'll mention it for sure in the changelog and when we phase out s3legacy for a few releases we'll probably display warnings. That should do it. Also, documentation for the migrate command is not done yet.