-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Question rather than issue #17
Comments
Hi, It should just work fine also if the (or some of) the files are smaller than the split size. The chunk size only gives a maximum size for each chunk in the multipart upload, but if there is just one chunk and that is less then the max should not break it. Btw, I changed the code in the meantime to support multiple files, so you can also use wildcards now to invoke the script. The wildcard expressions are expanded already by the shell, so you can test what they cover by just trying them with Best, |
Why the maximum value of the split-size is 2^22 which means 4TB, whilst the maximum chunk accepted by glacier is 4GB (2^12)? |
Sorry, that is a typo in the documentation, it should be indeed 12.. |
Btw, the advantage of a split size smaller than the file size is that the multiple parts are uploaded in parallell. If you specify multiple files in the |
What if I have 8TB of data to upload and total available space on local hard drive for cache/split 250GB? Will the script cleanup after each upload completed? |
It will be also great if script could write to the log file. It can be txt not json necessarily. |
That should be fine, it caches only the parts that are currently uploaded on disk, i.e. at most 4GB*(#of parallel uploads) and it will clean up after completion of each part (in case of error there might be some data left in tmp folders, but they should be cleaned on a restart of the operating system). The number of parallel uploads for a single invocation of |
Thanks. I am testing it now. |
Only from curiosity, did it work properly or did you run into any problems? If it worked fine I'll tag the current state as a release :) |
It is running currently (500GB out of 8TB so far). It is slow because source data is mounted using s3fs onto ec2 instance, and then files are being chunked to 2GB chunks. |
Then fingers crossed ;) Btw, if you already have the data in S3(?) there might be easier options to get it into glacier. I'm not really an expert on this, but found for example this question. |
What will happen if I run your script with option to split to 4GB chunks against a folder containing large files >4GB and files <4GB? Will it fail on 'small' files and continue splitting large ones? Let's say that I have following folder structure:
/data/files/x/
/data/files/y/
/data/files/
all of them contains files from 300MB to 100GB big. Ideally I'd like to run script in tmux/screen session and check once a day.
The text was updated successfully, but these errors were encountered: