Add support for rsync and/or scp

[chrispcampbell]

Enhancement Suggestion

Add support for using rsync and/or scp (or similar) to move files between the local Rome cache and a remote (privately hosted) file store.

For example:

[Cache]
local = ~/Library/Caches/Rome
remote_scp = username@remotehost.com:/some/remote/rome-cache
# OR
remote_rsync = username@remotehost.com:/some/remote/rome-cache

Current and suggested behavior

Currently only Amazon's S3 is supported as remote storage. Supporting rsync and/or scp would allow for using existing in-house storage infrastructure.

Why would the enhancement be useful to most users

Supporting alternative remote copy mechanisms would allow users to get all the benefits of Rome's remote caching support without relying on a cloud storage provider.

[tmspzz]

Hi,

Thanks for your suggestion. I will look into how to make this possible. Probably starting with scp.

[tmspzz]

@chrispcampbell so this is more tricky than I though as it requires to support listing as well which does not come out of the box with scp.

So there is really 2 alternatives here:

• write an scp client that supports so ( I think this might be interesting but an uphill battle)
• treat any error from scp while listing as "file not found"

[chrispcampbell]

@blender Thanks for looking into this. Yeah, I'm not sure there's a way to list via scp. However, rsync does support this, so maybe that would be a better option? For example:

# List all files in a directory
% rsync --list-only user@hostname:/some/dir/
-rwxrwxrwx          1,081 2018/01/03 18:26:23 one.txt
-rwxrwxrwx          1,081 2018/01/03 18:26:23 two.txt

# Test presence of a certain file (when present, error code is zero)
% rsync --list-only user@hostname:/some/dir/one.txt
-rwxrwxrwx          1,081 2018/01/03 18:26:23 one.txt
% echo $?
0

# Check for another file that's not present (error code is non-zero)
% rsync --list-only user@hostname:/some/dir/three.txt
rsync: link_stat "/some/dir/three.txt" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1708) [Receiver=3.1.1]
% echo $?
23

[tmspzz]

@chrispcampbell

It's probably easier with rsync, but at the end of the day it's still hard to tell if something went wrong while running the command (scp, or rsych) or if the file is actually missing.

I've done more exploration around scp so it'll try with that somehow.

My plan so far is to make remote caches mutually exclusive for now. This means you can't have a remote cache on S3 and an scp cache.

In the mean time, if you can mount the scp directory via FUSE, rome can use it as local cache and all operations are supported. I know of people using dropbox as local cache...

[tmspzz]

@chrispcampbell Thanks to @BalestraPatrick you can now use whatever backend you want

[tmspzz]

You can try the pre-release at: https://github.com/blender/Rome/releases/tag/v0.22.0.59