Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

P2P shuffle deduplicates data and can be run several times #7486

Merged
merged 54 commits into from
Jan 27, 2023
Merged

P2P shuffle deduplicates data and can be run several times #7486

merged 54 commits into from
Jan 27, 2023

Commits on Jan 4, 2023

  1. Sign every compute task with a unique counter to correlated responses

    @fjetter
    fjetter committed Jan 4, 2023
    Configuration menu
    Copy the full SHA
    a6d9cc7 View commit details
    Browse the repository at this point in the history

  2. Fix attempt before dispatching to threadpool

    @fjetter
    fjetter committed Jan 4, 2023
    Configuration menu
    Copy the full SHA
    4ef8975 View commit details
    Browse the repository at this point in the history

Commits on Jan 9, 2023

  1. Fix tests

    @hendrikmakait
    hendrikmakait committed Jan 9, 2023
    Configuration menu
    Copy the full SHA
    7ecab11 View commit details
    Browse the repository at this point in the history

  2. Check for sentinel

    @hendrikmakait
    hendrikmakait committed Jan 9, 2023
    Configuration menu
    Copy the full SHA
    68dfeeb View commit details
    Browse the repository at this point in the history

  3. Minor

    @hendrikmakait
    hendrikmakait committed Jan 9, 2023
    Configuration menu
    Copy the full SHA
    e4b2674 View commit details
    Browse the repository at this point in the history

  4. Run ID

    @hendrikmakait
    hendrikmakait committed Jan 9, 2023
    Configuration menu
    Copy the full SHA
    b1af340 View commit details
    Browse the repository at this point in the history

  5. Fix dummy

    @hendrikmakait
    hendrikmakait committed Jan 9, 2023
    Configuration menu
    Copy the full SHA
    8e0f23c View commit details
    Browse the repository at this point in the history

Commits on Jan 10, 2023

  1. Fix freeing of outdated keys

    @hendrikmakait
    hendrikmakait committed Jan 10, 2023
    Configuration menu
    Copy the full SHA
    8552568 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into sign-tasks-with-run-id

    @hendrikmakait
    hendrikmakait committed Jan 10, 2023
    Configuration menu
    Copy the full SHA
    061c3e8 View commit details
    Browse the repository at this point in the history

  3. Improve stimulus handling

    @hendrikmakait
    hendrikmakait committed Jan 10, 2023
    Configuration menu
    Copy the full SHA
    4272a41 View commit details
    Browse the repository at this point in the history

  4. More verbose story

    @hendrikmakait
    hendrikmakait committed Jan 10, 2023
    Configuration menu
    Copy the full SHA
    062c019 View commit details
    Browse the repository at this point in the history

  5. Fix

    @hendrikmakait
    hendrikmakait committed Jan 10, 2023
    Configuration menu
    Copy the full SHA
    fc3b690 View commit details
    Browse the repository at this point in the history

  6. Retore old condition

    @hendrikmakait
    hendrikmakait committed Jan 10, 2023
    Configuration menu
    Copy the full SHA
    6ea5f78 View commit details
    Browse the repository at this point in the history

  7. Improve conditions

    @hendrikmakait
    hendrikmakait committed Jan 10, 2023
    Configuration menu
    Copy the full SHA
    f9e67b1 View commit details
    Browse the repository at this point in the history

  8. Return to old-style checks to avoid unintended side-effects when chan… 

    …ging stimulus_task_finished
    @hendrikmakait
    hendrikmakait committed Jan 10, 2023
    Configuration menu
    Copy the full SHA
    15e80cf View commit details
    Browse the repository at this point in the history

Commits on Jan 11, 2023

  1. Naming

    @hendrikmakait
    hendrikmakait committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    f3008ff View commit details
    Browse the repository at this point in the history

  2. Cleanup

    @hendrikmakait
    hendrikmakait committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    399733a View commit details
    Browse the repository at this point in the history

  3. Fix tests

    @hendrikmakait
    hendrikmakait committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    215ab59 View commit details
    Browse the repository at this point in the history

  4. Reduce complexity

    @hendrikmakait
    hendrikmakait committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    4c4663c View commit details
    Browse the repository at this point in the history

  5. Remove register_complete

    @hendrikmakait
    hendrikmakait committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    83aed2a View commit details
    Browse the repository at this point in the history

Commits on Jan 12, 2023

  1. Fix test

    @hendrikmakait
    hendrikmakait committed Jan 12, 2023
    Configuration menu
    Copy the full SHA
    880321e View commit details
    Browse the repository at this point in the history

Commits on Jan 16, 2023

  1. Drop tombstones

    @hendrikmakait
    hendrikmakait committed Jan 16, 2023
    Configuration menu
    Copy the full SHA
    4133bfe View commit details
    Browse the repository at this point in the history

  2. Simplify test

    @hendrikmakait
    hendrikmakait committed Jan 16, 2023
    Configuration menu
    Copy the full SHA
    7ded915 View commit details
    Browse the repository at this point in the history

  3. Test

    @hendrikmakait
    hendrikmakait committed Jan 16, 2023
    Configuration menu
    Copy the full SHA
    df70bef View commit details
    Browse the repository at this point in the history

  4. Add runs

    @hendrikmakait
    hendrikmakait committed Jan 16, 2023
    Configuration menu
    Copy the full SHA
    9165c60 View commit details
    Browse the repository at this point in the history

  5. Run ID in shuffle_fail

    @hendrikmakait
    hendrikmakait committed Jan 16, 2023
    Configuration menu
    Copy the full SHA
    3608a7f View commit details
    Browse the repository at this point in the history

  6. Fail shuffle over batched comm

    @hendrikmakait
    hendrikmakait committed Jan 16, 2023
    Configuration menu
    Copy the full SHA
    3ab5432 View commit details
    Browse the repository at this point in the history

  7. Simplify

    @hendrikmakait
    hendrikmakait committed Jan 16, 2023
    Configuration menu
    Copy the full SHA
    060201a View commit details
    Browse the repository at this point in the history

Commits on Jan 17, 2023

  1. Deduplicate

    @hendrikmakait
    hendrikmakait committed Jan 17, 2023
    Configuration menu
    Copy the full SHA
    9350946 View commit details
    Browse the repository at this point in the history

  2. Test deduplication

    @hendrikmakait
    hendrikmakait committed Jan 17, 2023
    Configuration menu
    Copy the full SHA
    02357da View commit details
    Browse the repository at this point in the history

Commits on Jan 18, 2023

  1. Run IDs

    @hendrikmakait
    hendrikmakait committed Jan 18, 2023
    Configuration menu
    Copy the full SHA
    30bb6bc View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into deduplicate-in-shuffle

    @hendrikmakait
    hendrikmakait committed Jan 18, 2023
    Configuration menu
    Copy the full SHA
    769a177 View commit details
    Browse the repository at this point in the history

  3. Fix test

    @hendrikmakait
    hendrikmakait committed Jan 18, 2023
    Configuration menu
    Copy the full SHA
    f20bb73 View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2023

  1. Refactor get_shuffle

    @hendrikmakait
    hendrikmakait committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    e51aed8 View commit details
    Browse the repository at this point in the history

  2. Fix test_bad_disk

    @hendrikmakait
    hendrikmakait committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    48b3b4b View commit details
    Browse the repository at this point in the history

  3. Stricter test_handle_stale_barrier

    @hendrikmakait
    hendrikmakait committed Jan 19, 2023
    Configuration menu
    Copy the full SHA
    b67d794 View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2023

  1. Merge branch 'main' into deduplicate-in-shuffle

    @hendrikmakait
    hendrikmakait committed Jan 23, 2023
    Configuration menu
    Copy the full SHA
    dc9d4e4 View commit details
    Browse the repository at this point in the history

  2. Remove eq

    @hendrikmakait
    hendrikmakait committed Jan 23, 2023
    Configuration menu
    Copy the full SHA
    5b88753 View commit details
    Browse the repository at this point in the history

  3. Refresh on stale shuffle run in _get_shuffle_run

    @hendrikmakait
    hendrikmakait committed Jan 23, 2023
    Configuration menu
    Copy the full SHA
    6d184d6 View commit details
    Browse the repository at this point in the history

Commits on Jan 24, 2023

  1. Add test for shuffle lifecycle

    @hendrikmakait
    hendrikmakait committed Jan 24, 2023
    Configuration menu
    Copy the full SHA
    71eec80 View commit details
    Browse the repository at this point in the history

  2. Formatting

    @hendrikmakait
    hendrikmakait committed Jan 24, 2023
    Configuration menu
    Copy the full SHA
    d6789a3 View commit details
    Browse the repository at this point in the history

  3. Add test for replacing stale shuffle run

    @hendrikmakait
    hendrikmakait committed Jan 24, 2023
    Configuration menu
    Copy the full SHA
    78b5e44 View commit details
    Browse the repository at this point in the history

  4. Typo

    @hendrikmakait
    hendrikmakait committed Jan 24, 2023
    Configuration menu
    Copy the full SHA
    19d6e23 View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2023

  1. Early deletion

    @hendrikmakait
    hendrikmakait committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    20ea2d5 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into deduplicate-in-shuffle

    @hendrikmakait
    hendrikmakait committed Jan 25, 2023
    Configuration menu
    Copy the full SHA
    f1bf7fb View commit details
    Browse the repository at this point in the history

Commits on Jan 26, 2023

  1. Avoid sizeof(pa.Table)

    @hendrikmakait
    hendrikmakait committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    4e81316 View commit details
    Browse the repository at this point in the history

  2. Update distributed/shuffle/_worker_extension.py 

    Co-authored-by: Florian Jetter <fjetter@users.noreply.github.com>
    @hendrikmakait @fjetter
    hendrikmakait and fjetter authored Jan 26, 2023
    Configuration menu
    Copy the full SHA
    999cd0b View commit details
    Browse the repository at this point in the history

  3. Fix typing

    @hendrikmakait
    hendrikmakait committed Jan 26, 2023
    Configuration menu
    Copy the full SHA
    43e0df2 View commit details
    Browse the repository at this point in the history

Commits on Jan 27, 2023

  1. Add documentation

    @hendrikmakait
    hendrikmakait committed Jan 27, 2023
    Configuration menu
    Copy the full SHA
    db20854 View commit details
    Browse the repository at this point in the history

  2. Naming and parametrization

    @hendrikmakait
    hendrikmakait committed Jan 27, 2023
    Configuration menu
    Copy the full SHA
    4fcc3fe View commit details
    Browse the repository at this point in the history

  3. Improve test docs

    @hendrikmakait
    hendrikmakait committed Jan 27, 2023
    Configuration menu
    Copy the full SHA
    ab72722 View commit details
    Browse the repository at this point in the history

  4. Add test for shuffle instance repeat

    @hendrikmakait
    hendrikmakait committed Jan 27, 2023
    Configuration menu
    Copy the full SHA
    e96beec View commit details
    Browse the repository at this point in the history

  5. Await result in test

    @hendrikmakait
    hendrikmakait committed Jan 27, 2023
    Configuration menu
    Copy the full SHA
    53bac39 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into deduplicate-in-shuffle

    @hendrikmakait
    hendrikmakait committed Jan 27, 2023
    Configuration menu
    Copy the full SHA
    1f24c5d View commit details
    Browse the repository at this point in the history