-
-
Notifications
You must be signed in to change notification settings - Fork 901
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding encryption to sync command #12
Conversation
Hi, thanks for your work. I'm keen to merge such a functionality, however the overhead of calling HEAD every time seems to be too huge. Many people run s3cmd on buckets with millions of files and some others (me) run it in on a remote South Pacific island with a high latency to S3 datacentres. I would much prefer to store the attributes locally, for example in .s3cmd.info in every directory which could be a Python Pickle file, or sqlite3 database file or something like that. Alternatively store the attributes in xattr on filesystems that support it (most current Linux/Unix filesystems do). That way s3cmd would need to make the HEAD calls only if the required attributes couldn't be found locally. How does that sound? Are you ok to implement that? Thanks! |
I'll look at the idea. I haven't done much Python, but I'll dig into what I also didn't like the HEAD call solution either, but I couldn't come up On Mon, Nov 14, 2011 at 10:06 PM, Michal Ludvig <
|
I'm going to close this request and work on the new way of doing it. I also shouldn't have been working in master anyway, so I'm going to refactor my git branches so this won't cause problems in the future. I'll re request a pull when I'm finished. |
I know this ticket is closed now. But it looks like implementing this similar was is not a good idea: md5 of original, unencrypted file is stored in x-amz-meta-s3tools-orig_md5 headers ? http://www.cs.jhu.edu/~astubble/dss/winzip.pdf |
I added encryption to the sync command by storing extra metadata for encrypted files, the original file's md5 and the original file's size. There needs to be a HEAD call on every resource now for a sync, but it shouldn't happen on a straight get or put.