Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RF/ENH: download - reimplementation to support original and upcoming UI/API #134

Merged
merged 39 commits into from
Aug 12, 2020

Commits on Aug 5, 2020

  1. Configuration menu
    Copy the full SHA
    e06cbac View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    9871ece View commit details
    Browse the repository at this point in the history
  3. ENH: support parsing "dandiapi" style URLs from unversioned current D…

    …ANDI archive + publish development
    yarikoptic committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    888b2e4 View commit details
    Browse the repository at this point in the history
  4. BF(TST): there was one not fixed earlier test with incorrect assert o…

    …f tuple elements
    
    It became obvious after RFing to use a helper function
    yarikoptic committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    d647032 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    b1539aa View commit details
    Browse the repository at this point in the history
  6. RF+ENH: introduce "dandiapi" type of server to download from

    Common code from girder.py was moved into download.py etc.
    Found these changes uncomitted after a long period of time, committing
    before progressing forward with development
    yarikoptic committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    d10fa77 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    2c1cc94 View commit details
    Browse the repository at this point in the history
  8. NF: IteratorWithAggregation - to help receiving/displaying total whil…

    …e going through individual elements
    
    Probably overdone/ugly and should be easier/better with async. TODO REDO later
    yarikoptic committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    764237b View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    cb4c80f View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    55774be View commit details
    Browse the repository at this point in the history
  11. ENH: harmonize records from girder to API (+modified), both downloade…

    …rs work
    
    See referenced in the diff issues about "modified" etc.
    Also I removed block of code to fetch metadata for an asset if missing - API
    now returns asset records with metadata
    yarikoptic committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    188530c View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    e2bbac1 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    d291eb9 View commit details
    Browse the repository at this point in the history
  14. ENH: mess of changes to get all downloads working (no parallel yet)

    Download of released datasets is broken, independent issue:
    dandi/dandi-publish#112
    yarikoptic committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    84fb8c4 View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    c56d948 View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2020

  1. Configuration menu
    Copy the full SHA
    fcfa310 View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2020

  1. Configuration menu
    Copy the full SHA
    afea1f0 View commit details
    Browse the repository at this point in the history
  2. Merge remote-tracking branch 'origin/master' into enh-download-api

    * origin/master:
      Test upload of file second time failing with `existing="error"`
      Delete a straggler comment
      Put more of local_docker inside `try`
      Test upload function
      Move upload code to a function
      Document Sourcegraph in DEVELOPMENT.md
      Remove DANDI_DEVEL testing setting
      Test register() function directly instead of cmd_register
      Rename local-docker-compose instance to local-docker-tests
      Test register with & without a dandiset.yaml file
      Add a comment
      Allow local-docker and local-docker-compose to exist side by side
      Split off register code into a Python function
      Skip local_docker tests on Windows
      Assert that stdout from register is empty
      Got register test to work
      Include docker-compose.yml in sdist and wheel
      Unfinished smoke test for `register`
      Use an unscoped key for local testing
      [GH-27] Define a `local_docker` fixture
    yarikoptic committed Aug 7, 2020
    Configuration menu
    Copy the full SHA
    aa170d0 View commit details
    Browse the repository at this point in the history
  3. BF: add None for publish/api in local-docker-tests + TODO comments fo…

    …r issues which would add them
    yarikoptic committed Aug 7, 2020
    Configuration menu
    Copy the full SHA
    c3081e7 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    e4938cf View commit details
    Browse the repository at this point in the history
  5. ENH(TST): make sleeps shorter, but keep last outter sleep long

    With the hope that it would make test faster and also avoid sporadic
    test failures on some runs on CI where it does manage to get to the 2nd
    element before exception is raised:
    
        >       assert got == [0]
        E       assert [0, 1] == [0]
        E         Left contains one more item: 1
        E         Full diff:
        E         - [0]
        E         + [0, 1]
    yarikoptic committed Aug 7, 2020
    Configuration menu
    Copy the full SHA
    c9d3cf2 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    51b0743 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    d61295d View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2020

  1. Merge remote-tracking branch 'origin/master' into enh-download-api

    * origin/master:
      ENH: test code paths for when DANDI_LOG_GIRDER is set when testing with 3.6
      TST: test that our progress reporter does not blow if tqdm blows
      register(): Adjust dandi_instance argument
      Remove uses of "known_instances" in register & upload tests
      ENH(code,TST): girder - sleep at least 1 sec between retries, test retries loop
      BF: just issue a warning and no progress indication if tqdm pbar fails to initialize
      Apply review suggestions
      BF(workaround): do not fail test workflow if codecov fails
      local_docker_compose: Get Girder address from known_instances
      Add "instance" key to local_docker_compose return value
      Include "www." in ABOUT_URL
      Add Dockerized redirector to Docker Compose
      Add fixture for setting DANDI_API_KEY from local_docker_compose
      Make local_docker_compose return instance_id
    
    Conflicts:
    	dandi/consts.py - took master version but added the last None for
                          the publish/API with an issue referenced
    	dandi/girder.py - just took +1 for random.random() since we no
                          longer use that code path here
    	dandi/tests/test_download.py - RFed test to test retries on
                          getDownloaderAsIterator instead for Girder
    yarikoptic committed Aug 10, 2020
    Configuration menu
    Copy the full SHA
    0708d1a View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2020

  1. Configuration menu
    Copy the full SHA
    a941485 View commit details
    Browse the repository at this point in the history

Commits on Aug 12, 2020

  1. Merge remote-tracking branch 'origin/master' into enh-download-api

    * origin/master: (24 commits)
      Remove duplicate "requests" dependency
      get_instance(): Immediately return known instances that lack redirector URLs
      Raise helpful error if /server-info returns invalid version
      Make redirector report http://localhost:8081 as Girder URL
      Test uploading an unregistered dandiset
      Assert a certain code path is unreachable
      Test that lock_dandiset() is called when uploading
      Tests of lock_dandiset()
      Remove unused monkeypatches
      Configure flake8 to run on test code as well
      Detect nonexistent Dandisets when uploading
      ENH: upload - add locking of the dandiset
      Make pytest ignore DeprecationWarnings from responses
      Fetch all commits in test workflow
      Replace some occurrences of known_instances with get_instance()
      Test get_instance()
      CliVersionTooLowError → CliVersionTooOldError
      Move CLI version exceptions to dandi/exceptions.py
      Use semantic_version instead of packaging
      [GH-184] Add a get_instance() function
      ...
    
     Conflicts:
    	dandi/download.py
    	dandi/girder.py
    	dandi/tests/test_download.py - conflicted with a linting change
    	dandi/tests/test_utils.py - both added tests
    
     No conflicts but needed changes:
        dandi/utils.py -- needed to add dandiapi into returned dandi_instance
    yarikoptic committed Aug 12, 2020
    Configuration menu
    Copy the full SHA
    19e3d77 View commit details
    Browse the repository at this point in the history
  2. BF: making linting happy

    yarikoptic committed Aug 12, 2020
    Configuration menu
    Copy the full SHA
    aed98c5 View commit details
    Browse the repository at this point in the history
  3. RF: "dandiapi" -> "api" and "DANDI API" to match /server-info simple …

    …"api"
    
    I left module and class still with dandiapi names
    yarikoptic committed Aug 12, 2020
    Configuration menu
    Copy the full SHA
    327881e View commit details
    Browse the repository at this point in the history
  4. BF: moved lock_dandiset into correct position (incorrect merge side e…

    …ffect)
    
    It did not rebase easily into merge commit so I decided to keep it as is
    yarikoptic committed Aug 12, 2020
    Configuration menu
    Copy the full SHA
    a4f2a3d View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    64d9952 View commit details
    Browse the repository at this point in the history
  6. RF: do not duplicate defaults within kwargs in @click entrypoint for …

    …download
    
    Which also removes a typoed default value - duplication is evil!
    yarikoptic committed Aug 12, 2020
    Configuration menu
    Copy the full SHA
    4798e49 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    09b2196 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    f367313 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    a45b550 View commit details
    Browse the repository at this point in the history
  10. RF: dandiapi - remove not used include_metadata option

    Originally it was thought that may be resolution for
    dandi/dandi-publish#78
    would make metadata return optional, but it is always returned,
    so no need for the option
    yarikoptic committed Aug 12, 2020
    Configuration menu
    Copy the full SHA
    515538e View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    096c8dd View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    5abd195 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    e52d728 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    20718e2 View commit details
    Browse the repository at this point in the history