Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Threaded MultipackDistributedDataloader with prefetched samples #759

Merged
merged 23 commits into from
Oct 26, 2023
Merged

Threaded MultipackDistributedDataloader with prefetched samples #759

merged 23 commits into from
Oct 26, 2023

Commits on Oct 21, 2023

  1. Multithreading implementation [WIP]

    @casper-hansen
    casper-hansen committed Oct 21, 2023
    Configuration menu
    Copy the full SHA
    099eed0 View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2023

  1. Added benchmarking

    @casper-hansen
    casper-hansen committed Oct 22, 2023
    Configuration menu
    Copy the full SHA
    eda9848 View commit details
    Browse the repository at this point in the history

  2. 35% increased throughput

    @casper-hansen
    casper-hansen committed Oct 22, 2023
    Configuration menu
    Copy the full SHA
    8cb60b6 View commit details
    Browse the repository at this point in the history

  3. Memory pinning

    @casper-hansen
    casper-hansen committed Oct 22, 2023
    Configuration menu
    Copy the full SHA
    2454b6a View commit details
    Browse the repository at this point in the history

  4. Start threads in init

    @casper-hansen
    casper-hansen committed Oct 22, 2023
    Configuration menu
    Copy the full SHA
    55fc5ab View commit details
    Browse the repository at this point in the history

  5. Correct print of samples

    @casper-hansen
    casper-hansen committed Oct 22, 2023
    Configuration menu
    Copy the full SHA
    b9f2e12 View commit details
    Browse the repository at this point in the history

  6. Sleep if queue is full

    @casper-hansen
    casper-hansen committed Oct 22, 2023
    Configuration menu
    Copy the full SHA
    5d9959a View commit details
    Browse the repository at this point in the history

  7. Remove pin_memory (worse)

    @casper-hansen
    casper-hansen committed Oct 22, 2023
    Configuration menu
    Copy the full SHA
    6c32064 View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2023

  1. Simplify logic to one thread

    @casper-hansen
    casper-hansen committed Oct 23, 2023
    Configuration menu
    Copy the full SHA
    121f996 View commit details
    Browse the repository at this point in the history

  2. Remove benchmark

    @casper-hansen
    casper-hansen committed Oct 23, 2023
    Configuration menu
    Copy the full SHA
    d6890bd View commit details
    Browse the repository at this point in the history

  3. Use deque for constant speed

    @casper-hansen
    casper-hansen committed Oct 23, 2023
    Configuration menu
    Copy the full SHA
    ac65ba9 View commit details
    Browse the repository at this point in the history

  4. Formatting

    @casper-hansen
    casper-hansen committed Oct 23, 2023
    Configuration menu
    Copy the full SHA
    f3f1ba0 View commit details
    Browse the repository at this point in the history

  5. Formatting

    @casper-hansen
    casper-hansen committed Oct 23, 2023
    Configuration menu
    Copy the full SHA
    2d6bd64 View commit details
    Browse the repository at this point in the history

  6. Formatting

    @casper-hansen
    casper-hansen committed Oct 23, 2023
    Configuration menu
    Copy the full SHA
    4ff2903 View commit details
    Browse the repository at this point in the history

  7. Formatting

    @casper-hansen
    casper-hansen committed Oct 23, 2023
    Configuration menu
    Copy the full SHA
    e026a54 View commit details
    Browse the repository at this point in the history

  8. Rollback to use queue

    @casper-hansen
    casper-hansen committed Oct 23, 2023
    Configuration menu
    Copy the full SHA
    2bb3a8e View commit details
    Browse the repository at this point in the history

  9. Fix multi-epoch training

    @casper-hansen
    casper-hansen committed Oct 23, 2023
    Configuration menu
    Copy the full SHA
    d5d91db View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2023

  1. Merge branch 'main' into multithreading

    @casper-hansen
    casper-hansen committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    1dc3235 View commit details
    Browse the repository at this point in the history

  2. Add num epochs arg

    @casper-hansen
    casper-hansen committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    4ee1584 View commit details
    Browse the repository at this point in the history

  3. Start thread in __iter__

    @casper-hansen
    casper-hansen committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    c564819 View commit details
    Browse the repository at this point in the history

  4. Formatting

    @casper-hansen
    casper-hansen committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    62bd93b View commit details
    Browse the repository at this point in the history

  5. Use is_alive correctly

    @casper-hansen
    casper-hansen committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    78823d0 View commit details
    Browse the repository at this point in the history

  6. Simplify loading thread

    @casper-hansen
    casper-hansen committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    e1bd784 View commit details
    Browse the repository at this point in the history