Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added multipart upload, added optional encryption/decryption. #24

Closed
wants to merge 85 commits into from
Closed

Added multipart upload, added optional encryption/decryption. #24

wants to merge 85 commits into from

Commits on Mar 1, 2013

  1. .gitignore added.

    Piotr Kaleta committed Mar 1, 2013
    Configuration menu
    Copy the full SHA
    7ba14fa View commit details
    Browse the repository at this point in the history

  2. Multipart upload added.

    Piotr Kaleta committed Mar 1, 2013
    Configuration menu
    Copy the full SHA
    68af3c1 View commit details
    Browse the repository at this point in the history

  3. Importing default part size from boto.

    Piotr Kaleta committed Mar 1, 2013
    Configuration menu
    Copy the full SHA
    6a90f7f View commit details
    Browse the repository at this point in the history

  4. Added option to specify number of threads in multipart upload.

    Piotr Kaleta committed Mar 1, 2013
    Configuration menu
    Copy the full SHA
    4be17d7 View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2013

  1. Using ConcurrentUploader directly to upload file to glacier.

    Piotr Kaleta committed Mar 12, 2013
    Configuration menu
    Copy the full SHA
    ec13d93 View commit details
    Browse the repository at this point in the history

Commits on Mar 13, 2013

  1. First shot at GPG encryption/decryption of transferred messages.

    Piotr Kaleta committed Mar 13, 2013
    Configuration menu
    Copy the full SHA
    6f7174c View commit details
    Browse the repository at this point in the history

  2. Fixed decryption when retrieving file.

    Piotr Kaleta committed Mar 13, 2013
    Configuration menu
    Copy the full SHA
    389a1b8 View commit details
    Browse the repository at this point in the history

  3. Fixed missing encryptor.

    Piotr Kaleta committed Mar 13, 2013
    Configuration menu
    Copy the full SHA
    85cc6b4 View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2013

  1. Add some docstrings.

    Piotr Kaleta committed Mar 15, 2013
    Configuration menu
    Copy the full SHA
    88e499f View commit details
    Browse the repository at this point in the history

  2. Added option to disable encryption/decryption.

    Piotr Kaleta committed Mar 15, 2013
    Configuration menu
    Copy the full SHA
    724f65f View commit details
    Browse the repository at this point in the history

  3. Using command line option to decide whether to encrypt/decrypt or not.

    Piotr Kaleta committed Mar 15, 2013
    Configuration menu
    Copy the full SHA
    806c31e View commit details
    Browse the repository at this point in the history

  4. Updated readme.

    Piotr Kaleta committed Mar 15, 2013
    Configuration menu
    Copy the full SHA
    7f30654 View commit details
    Browse the repository at this point in the history

  5. Added setup.py

    Piotr Kaleta committed Mar 15, 2013
    Configuration menu
    Copy the full SHA
    d5af2ed View commit details
    Browse the repository at this point in the history

  6. Fix package names in setup file.

    Piotr Kaleta committed Mar 15, 2013
    Configuration menu
    Copy the full SHA
    720bdbd View commit details
    Browse the repository at this point in the history

  7. Fixed setup.py one more time.

    Piotr Kaleta committed Mar 15, 2013
    Configuration menu
    Copy the full SHA
    e8a25bb View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2013

  1. Merge public master with counsyl encryption additions. 

    The Counsyl fork from Nov 2012 adds encryption to upload/retrieve.

Also adds setup.py.

Conflicts:
	glacier.py
    @sakoht
    sakoht committed Nov 22, 2013
    Configuration menu
    Copy the full SHA
    dbae589 View commit details
    Browse the repository at this point in the history

  2. remove comments from merge

    @sakoht
    sakoht committed Nov 22, 2013
    Configuration menu
    Copy the full SHA
    2af4c6f View commit details
    Browse the repository at this point in the history

  3. fix handling of file name vs obj

    @sakoht
    sakoht committed Nov 22, 2013
    Configuration menu
    Copy the full SHA
    724d692 View commit details
    Browse the repository at this point in the history

  4. fix handling of file name vs obj

    @sakoht
    sakoht committed Nov 22, 2013
    Configuration menu
    Copy the full SHA
    b51724a View commit details
    Browse the repository at this point in the history

Commits on Nov 23, 2013

  1. set DEFAULT_NUM_THREADS to 10 as boto does internally

    @sakoht
    sakoht committed Nov 23, 2013
    Configuration menu
    Copy the full SHA
    bfcdaa4 View commit details
    Browse the repository at this point in the history

  2. Ensure archive multipart_upload passes the multipart flag.

    @sakoht
    sakoht committed Nov 23, 2013
    Configuration menu
    Copy the full SHA
    d4088e6 View commit details
    Browse the repository at this point in the history

  3. Cache encrypted file for retries until upload completes. 

    Also add logging.  Before merging reduce from info to debug.
    @sakoht
    sakoht committed Nov 23, 2013
    Configuration menu
    Copy the full SHA
    635fbb4 View commit details
    Browse the repository at this point in the history

Commits on Nov 24, 2013

  1. cleanup of new code around encryption temp files

    @sakoht
    sakoht committed Nov 24, 2013
    Configuration menu
    Copy the full SHA
    30a86c0 View commit details
    Browse the repository at this point in the history

  2. fix temporarily disabled check around --encryption-dir

    @sakoht
    sakoht committed Nov 24, 2013
    Configuration menu
    Copy the full SHA
    a72a504 View commit details
    Browse the repository at this point in the history

  3. remove extra argument checking: not operating cleanly

    @sakoht
    sakoht committed Nov 24, 2013
    Configuration menu
    Copy the full SHA
    20dd737 View commit details
    Browse the repository at this point in the history

Commits on Nov 25, 2013

  1. return to default gnupg key handling (use homedir, override with env)

    @sakoht
    sakoht committed Nov 25, 2013
    Configuration menu
    Copy the full SHA
    f11cf78 View commit details
    Browse the repository at this point in the history

Commits on Dec 2, 2013

  1. Remove --encryption-directory.

    @sakoht
    sakoht committed Dec 2, 2013
    Configuration menu
    Copy the full SHA
    d3bb117 View commit details
    Browse the repository at this point in the history

  2. add whitespace around operator

    @sakoht
    sakoht committed Dec 2, 2013
    Configuration menu
    Copy the full SHA
    b578431 View commit details
    Browse the repository at this point in the history

  3. whitespace around operator

    @sakoht
    sakoht committed Dec 2, 2013
    Configuration menu
    Copy the full SHA
    e5db4d8 View commit details
    Browse the repository at this point in the history

Commits on Dec 4, 2013

  1. Merge pull request basak#6 from dev/101_setup_aws_glacier_for_uploadi… 

    …ng_batches

merge mainline changes with Counsyl internal encryption additions
    @sakoht
    sakoht committed Dec 4, 2013
    Configuration menu
    Copy the full SHA
    1b99931 View commit details
    Browse the repository at this point in the history

Commits on Dec 10, 2013

  1. Add koality

    Chris Crutchfield committed Dec 10, 2013
    Configuration menu
    Copy the full SHA
    0643e9a View commit details
    Browse the repository at this point in the history

  2. add mock

    Chris Crutchfield committed Dec 10, 2013
    Configuration menu
    Copy the full SHA
    a6c3aed View commit details
    Browse the repository at this point in the history

Commits on Dec 11, 2013

  1. add MIT license

    @sakoht
    sakoht committed Dec 11, 2013
    Configuration menu
    Copy the full SHA
    0deb231 View commit details
    Browse the repository at this point in the history

  2. make mutipart_upload a parameter to upload instead of a separate command

    @sakoht
    sakoht committed Dec 11, 2013
    Configuration menu
    Copy the full SHA
    8bd36ad View commit details
    Browse the repository at this point in the history

  3. Merge pull request basak#9 from dev/add_koality 

    Add koality
    Chris Crutchfield committed Dec 11, 2013
    Configuration menu
    Copy the full SHA
    5bba053 View commit details
    Browse the repository at this point in the history

  4. set type for archive upload file parameter

    @sakoht
    sakoht committed Dec 11, 2013
    Configuration menu
    Copy the full SHA
    3319d20 View commit details
    Browse the repository at this point in the history

  5. restore support for stdin streaming for "-"

    @sakoht
    sakoht committed Dec 11, 2013
    Configuration menu
    Copy the full SHA
    857d450 View commit details
    Browse the repository at this point in the history

Commits on Dec 12, 2013

  1. renamed callback to "sortable_date"

    @sakoht
    sakoht committed Dec 12, 2013
    Configuration menu
    Copy the full SHA
    8f248c2 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into 7_sync_with_public_project

    @sakoht
    sakoht committed Dec 12, 2013
    Configuration menu
    Copy the full SHA
    5ae273a View commit details
    Browse the repository at this point in the history

  3. Remove codequality from koality

    Chris Crutchfield committed Dec 12, 2013
    Configuration menu
    Copy the full SHA
    d14d4da View commit details
    Browse the repository at this point in the history

  4. test archive upload w/ encrypt, multi-part

    @sakoht
    sakoht committed Dec 12, 2013
    Configuration menu
    Copy the full SHA
    e0cc3ad View commit details
    Browse the repository at this point in the history

  5. Merge branch 'master' into 7_sync_with_public_project

    @sakoht
    sakoht committed Dec 12, 2013
    Configuration menu
    Copy the full SHA
    de61cf4 View commit details
    Browse the repository at this point in the history

  6. Get glacier_test.py working. (Failing on master.) 

    TODO: extend to cover encryption and multipart upload.
    @sakoht
    sakoht committed Dec 12, 2013
    Configuration menu
    Copy the full SHA
    dc688a2 View commit details
    Browse the repository at this point in the history

  7. add multipart upload unit test

    @sakoht
    sakoht committed Dec 12, 2013
    Configuration menu
    Copy the full SHA
    e9f1736 View commit details
    Browse the repository at this point in the history

  8. test cleanup

    @sakoht
    sakoht committed Dec 12, 2013
    Configuration menu
    Copy the full SHA
    2627cc1 View commit details
    Browse the repository at this point in the history

  9. remove koality.yml

    @sakoht
    sakoht committed Dec 12, 2013
    Configuration menu
    Copy the full SHA
    3839287 View commit details
    Browse the repository at this point in the history

  10. warn the user that multipart uploads do not support streaming from stdin

    @sakoht
    sakoht committed Dec 12, 2013
    Configuration menu
    Copy the full SHA
    863a74b View commit details
    Browse the repository at this point in the history

  11. gnupg test keys

    @sakoht
    sakoht committed Dec 12, 2013
    Configuration menu
    Copy the full SHA
    1f29e41 View commit details
    Browse the repository at this point in the history

  12. rename --multi-part to --multipart

    @sakoht
    sakoht committed Dec 12, 2013
    Configuration menu
    Copy the full SHA
    1958366 View commit details
    Browse the repository at this point in the history

  13. restore koality.yml

    @sakoht
    sakoht committed Dec 12, 2013
    Configuration menu
    Copy the full SHA
    07a6348 View commit details
    Browse the repository at this point in the history

Commits on Dec 13, 2013

  1. Updated setup.py

    @chrisgilmerproj
    chrisgilmerproj committed Dec 13, 2013
    Configuration menu
    Copy the full SHA
    c3b1564 View commit details
    Browse the repository at this point in the history

  2. Fix missing comma

    @chrisgilmerproj
    chrisgilmerproj committed Dec 13, 2013
    Configuration menu
    Copy the full SHA
    c8c58fb View commit details
    Browse the repository at this point in the history

Commits on Dec 21, 2013

  1. Merge branch 'master' into 7_sync_with_public_project

    @sakoht
    sakoht committed Dec 21, 2013
    Configuration menu
    Copy the full SHA
    426e6b2 View commit details
    Browse the repository at this point in the history

  2. leave stubs for additional tests

    @sakoht
    sakoht committed Dec 21, 2013
    Configuration menu
    Copy the full SHA
    f10aa31 View commit details
    Browse the repository at this point in the history

  3. Add ConcurrentDownloader, rename --multipart to --concurrent. 

    The regular uploader and downloader automatically do multipart
upload/download as needed, but do not do it concurrently.  I renamed
the --multipart flag to --concurrent to be clearer about the
feature it adds.
    @sakoht
    sakoht committed Dec 21, 2013
    Configuration menu
    Copy the full SHA
    d7bc036 View commit details
    Browse the repository at this point in the history

  4. Merge branch '7_sync_with_public_project' of github.counsyl.com:dev/g… 

    …lacier-cli into 7_sync_with_public_project

Conflicts:
	glacier.py
	glacier_test.py
    @sakoht
    sakoht committed Dec 21, 2013
    Configuration menu
    Copy the full SHA
    d7c5c5e View commit details
    Browse the repository at this point in the history

  5. fix docstring

    @sakoht
    sakoht committed Dec 21, 2013
    Configuration menu
    Copy the full SHA
    81b6d89 View commit details
    Browse the repository at this point in the history

  6. add additional spaces for pep8

    @sakoht
    sakoht committed Dec 21, 2013
    Configuration menu
    Copy the full SHA
    232e5ec View commit details
    Browse the repository at this point in the history

  7. clarify code around temp file naming

    @sakoht
    sakoht committed Dec 21, 2013
    Configuration menu
    Copy the full SHA
    e68a102 View commit details
    Browse the repository at this point in the history

  8. make multipart downloads use the concurrent downloader by default

    @sakoht
    sakoht committed Dec 21, 2013
    Configuration menu
    Copy the full SHA
    c765a16 View commit details
    Browse the repository at this point in the history

Commits on Dec 31, 2013

  1. rename --multi-part to --concurrent in test case

    @sakoht
    sakoht committed Dec 31, 2013
    Configuration menu
    Copy the full SHA
    f866664 View commit details
    Browse the repository at this point in the history

Commits on Jan 2, 2014

  1. remove code to explictly delete tmp dir

    @sakoht
    sakoht committed Jan 2, 2014
    Configuration menu
    Copy the full SHA
    ce133df View commit details
    Browse the repository at this point in the history

  2. test combinations of encryption and concurrency with uploading

    @sakoht
    sakoht committed Jan 2, 2014
    Configuration menu
    Copy the full SHA
    e1a9114 View commit details
    Browse the repository at this point in the history

  3. test combinations of --encrypt and --concurrent when uploading

    @sakoht
    sakoht committed Jan 2, 2014
    Configuration menu
    Copy the full SHA
    8232f81 View commit details
    Browse the repository at this point in the history

  4. update setup.py to wrap duplicate requirements{,-dev}.txt

    @sakoht
    sakoht committed Jan 2, 2014
    Configuration menu
    Copy the full SHA
    06f5f38 View commit details
    Browse the repository at this point in the history

  5. update koality.yml to run the test case correctly

    @sakoht
    sakoht committed Jan 2, 2014
    Configuration menu
    Copy the full SHA
    96b5fa9 View commit details
    Browse the repository at this point in the history

  6. update tmpdir check to work on Mac, not expect /tmp

    @sakoht
    sakoht committed Jan 2, 2014
    Configuration menu
    Copy the full SHA
    6c2ead1 View commit details
    Browse the repository at this point in the history

Commits on Jan 3, 2014

  1. Configure the test case to use pre-generated testing GPNUPG keys.

    @sakoht
    sakoht committed Jan 3, 2014
    Configuration menu
    Copy the full SHA
    efeee9f View commit details
    Browse the repository at this point in the history

  2. updated random seet for gnupg_test_home

    @sakoht
    sakoht committed Jan 3, 2014
    Configuration menu
    Copy the full SHA
    287fdab View commit details
    Browse the repository at this point in the history

Commits on Jan 7, 2014

  1. fix indentation bug preventing final output after decrypt

    @sakoht
    sakoht committed Jan 7, 2014
    Configuration menu
    Copy the full SHA
    e21681f View commit details
    Browse the repository at this point in the history

  2. remove commented-out code

    @sakoht
    sakoht committed Jan 7, 2014
    Configuration menu
    Copy the full SHA
    24b741a View commit details
    Browse the repository at this point in the history

  3. verify that incorrect combinations of new args raise exceptions

    @sakoht
    sakoht committed Jan 7, 2014
    Configuration menu
    Copy the full SHA
    9c81466 View commit details
    Browse the repository at this point in the history

Commits on Jan 14, 2014

  1. removed extra line

    @sakoht
    sakoht committed Jan 14, 2014
    Configuration menu
    Copy the full SHA
    59f31bd View commit details
    Browse the repository at this point in the history

  2. shorten new lines that are too long

    @sakoht
    sakoht committed Jan 14, 2014
    Configuration menu
    Copy the full SHA
    929abb1 View commit details
    Browse the repository at this point in the history

  3. shorten new lines that are too long

    @sakoht
    sakoht committed Jan 14, 2014
    Configuration menu
    Copy the full SHA
    2afd841 View commit details
    Browse the repository at this point in the history

  4. Merge pull request basak#10 from dev/7_sync_with_public_project 

    7 sync with public project

I did a final test run after the cosmetic updates and things still look work.  Next I'll push to @piotr's public fork and update the PR to the original author.
    @sakoht
    sakoht committed Jan 14, 2014
    Configuration menu
    Copy the full SHA
    bffdeb6 View commit details
    Browse the repository at this point in the history

  5. rename --multipart-size to --part-size

    @sakoht
    sakoht committed Jan 14, 2014
    Configuration menu
    Copy the full SHA
    9211c64 View commit details
    Browse the repository at this point in the history

  6. update README.md to reflect new args

    @sakoht
    sakoht committed Jan 14, 2014
    Configuration menu
    Copy the full SHA
    d8bd041 View commit details
    Browse the repository at this point in the history

  7. fix typo in README.md update

    @sakoht
    sakoht committed Jan 14, 2014
    Configuration menu
    Copy the full SHA
    e13c63f View commit details
    Browse the repository at this point in the history

  8. update [] around optional --concurrent arg

    @sakoht
    sakoht committed Jan 14, 2014
    Configuration menu
    Copy the full SHA
    12b2c37 View commit details
    Browse the repository at this point in the history

  9. Merge pull request basak#12 from dev/11_rename_mulitpart_size_to_part… 

    …_size

11 rename --multipart-size to --part-size
    @sakoht
    sakoht committed Jan 14, 2014
    Configuration menu
    Copy the full SHA
    62ef6cb View commit details
    Browse the repository at this point in the history

Commits on Jan 21, 2014

  1. allow glacier-cli to use newer sqlalchemy, but not require it

    @sakoht
    sakoht committed Jan 21, 2014
    Configuration menu
    Copy the full SHA
    a3b68ef View commit details
    Browse the repository at this point in the history

Commits on Feb 5, 2014

  1. Update requirements-dev.txt

    Chris Crutchfield committed Feb 5, 2014
    Configuration menu
    Copy the full SHA
    7dd297b View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2014

  1. Update requirements-dev.txt

    Chris Crutchfield committed Feb 7, 2014
    Configuration menu
    Copy the full SHA
    9a70cf2 View commit details
    Browse the repository at this point in the history

Commits on Feb 14, 2014

  1. Update version to 0.1.1.alpha

    Chris Crutchfield committed Feb 14, 2014
    Configuration menu
    Copy the full SHA
    58721b4 View commit details
    Browse the repository at this point in the history