Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ADD] support Distributed Data Parallel #137

Merged
merged 32 commits into from
Mar 10, 2023
Merged

[ADD] support Distributed Data Parallel #137

merged 32 commits into from
Mar 10, 2023

Commits on Jan 29, 2023

  1. initial data_parallel code based on colossalai code but it needs to b… 

    …e updated more. (remove unnecssary code)
    dongsung kim committed Jan 29, 2023
    Configuration menu
    Copy the full SHA
    f04471e View commit details
    Browse the repository at this point in the history

Commits on Feb 12, 2023

  1. take out the code from _coloddp, _coloddp will be removed soon

    dongsung kim committed Feb 12, 2023
    Configuration menu
    Copy the full SHA
    b15e42c View commit details
    Browse the repository at this point in the history

Commits on Feb 18, 2023

  1. initialization for colossalAI integration

    @jinwonkim93
    jinwonkim93 committed Feb 18, 2023
    Configuration menu
    Copy the full SHA
    0aa7020 View commit details
    Browse the repository at this point in the history

Commits on Feb 20, 2023

  1. working code

    @jinwonkim93
    jinwonkim93 committed Feb 20, 2023
    Configuration menu
    Copy the full SHA
    3eba4a1 View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2023

  1. Change to oslo interface

    @jinwonkim93
    jinwonkim93 committed Feb 22, 2023
    Configuration menu
    Copy the full SHA
    f89e22a View commit details
    Browse the repository at this point in the history

  2. remove temp testcode

    @jinwonkim93
    jinwonkim93 committed Feb 22, 2023
    Configuration menu
    Copy the full SHA
    d45e029 View commit details
    Browse the repository at this point in the history

  3. change docstrings

    @jinwonkim93
    jinwonkim93 committed Feb 22, 2023
    Configuration menu
    Copy the full SHA
    80f0970 View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2023

  1. Merge branch 'data_parallel' into distributed_data_parallel

    @nijkah
    nijkah committed Mar 1, 2023
    Configuration menu
    Copy the full SHA
    315e5f8 View commit details
    Browse the repository at this point in the history

Commits on Mar 2, 2023

  1. reformat all files

    @jinwonkim93
    jinwonkim93 committed Mar 2, 2023
    Configuration menu
    Copy the full SHA
    e9f8b0b View commit details
    Browse the repository at this point in the history

  2. [Refactor] Refactor backward interface in DP (#141) 

    ## Title
Refactor backward in DP

## Description
Followed
https://github.com/KKIEEK/oslo/blob/3ca6b1aa0d87688af891f12b22837d89847680e9/oslo/torch/nn/parallel/data_parallel/distributed_data_parallel.py#L96.
And committed as KKIEEK for the code ownership.

Co-authored-by: KKIEEK <ethan9867@gmail.com>
    @nijkah @KKIEEK
    nijkah and KKIEEK committed Mar 2, 2023
    Configuration menu
    Copy the full SHA
    b49092c View commit details
    Browse the repository at this point in the history

  3. [Clean] Clean DDP Code (#142) 

    ## Title
Deleted legacy code.

## Description
Only newly added code written by jinwonkim93 remains.

---------

Co-authored-by: KKIEEK <ethan9867@gmail.com>
    @nijkah @KKIEEK
    nijkah and KKIEEK committed Mar 2, 2023
    Configuration menu
    Copy the full SHA
    20c0d10 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into distributed_data_parallel

    @nijkah
    nijkah committed Mar 2, 2023
    Configuration menu
    Copy the full SHA
    5079773 View commit details
    Browse the repository at this point in the history

  5. Update oslo/torch/nn/parallel/data_parallel/data_parallel.py 

    Co-authored-by: Junhwa Song <ethan9867@gmail.com>
    @nijkah @KKIEEK
    nijkah and KKIEEK committed Mar 2, 2023
    Configuration menu
    Copy the full SHA
    f9097f0 View commit details
    Browse the repository at this point in the history

  6. [Clean] Remove unused code

    @jinwonkim93
    jinwonkim93 committed Mar 2, 2023
    Configuration menu
    Copy the full SHA
    28e474a View commit details
    Browse the repository at this point in the history

  7. [Clean] Merge _DistirbutedDataParallelWrapper to _DistributedDataPara… 

    …llel
    @jinwonkim93
    jinwonkim93 committed Mar 2, 2023
    Configuration menu
    Copy the full SHA
    5401fa1 View commit details
    Browse the repository at this point in the history

  8. [Fix] fix forward max recursion

    @jinwonkim93
    jinwonkim93 committed Mar 2, 2023
    Configuration menu
    Copy the full SHA
    fe9ff2f View commit details
    Browse the repository at this point in the history

  9. [Clean] Remove parameters

    @jinwonkim93
    jinwonkim93 committed Mar 2, 2023
    Configuration menu
    Copy the full SHA
    1c581c8 View commit details
    Browse the repository at this point in the history

  10. Update oslo/torch/nn/parallel/data_parallel/data_parallel.py 

    Co-authored-by: Junhwa Song <ethan9867@gmail.com>
    @nijkah @KKIEEK
    nijkah and KKIEEK committed Mar 2, 2023
    Configuration menu
    Copy the full SHA
    cc0e6a8 View commit details
    Browse the repository at this point in the history

  11. [Refactor] Remove zero_grad from forward

    @jinwonkim93
    jinwonkim93 committed Mar 2, 2023
    Configuration menu
    Copy the full SHA
    7494c9c View commit details
    Browse the repository at this point in the history

Commits on Mar 3, 2023

  1. [Fix] Support long tensor for DDP backward (#146) 

    Related to #137
For now, our implementation of DDP does not support long tensor input,
so I fixed it.

---------

Co-authored-by: Hakjin Lee <nijkah@gmail.com>
    @KKIEEK @nijkah
    KKIEEK and nijkah committed Mar 3, 2023
    Configuration menu
    Copy the full SHA
    a5e0c9c View commit details
    Browse the repository at this point in the history

  2. [Add] Add copyright

    @jinwonkim93
    jinwonkim93 committed Mar 3, 2023
    Configuration menu
    Copy the full SHA
    894d66d View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2023

  1. [Refactor] move zero_grad to parallelize

    @jinwonkim93
    jinwonkim93 committed Mar 4, 2023
    Configuration menu
    Copy the full SHA
    4b14253 View commit details
    Browse the repository at this point in the history

  2. [Refactor] refactor backward

    @KKIEEK @jinwonkim93
    KKIEEK authored and jinwonkim93 committed Mar 4, 2023
    Configuration menu
    Copy the full SHA
    8b72f4d View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2023

  1. [Feat] support gloo backend

    @jinwonkim93
    jinwonkim93 committed Mar 5, 2023
    Configuration menu
    Copy the full SHA
    313d325 View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2023

  1. [Fix] fix conflict

    @jinwonkim93
    jinwonkim93 committed Mar 8, 2023
    Configuration menu
    Copy the full SHA
    50814c3 View commit details
    Browse the repository at this point in the history

  2. [Comment] change backward comment

    @jinwonkim93
    jinwonkim93 committed Mar 8, 2023
    Configuration menu
    Copy the full SHA
    b2282de View commit details
    Browse the repository at this point in the history

  3. [Fix] fix conflict

    @jinwonkim93
    jinwonkim93 committed Mar 8, 2023
    Configuration menu
    Copy the full SHA
    32dd163 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into distributed_data_parallel

    @jinwonkim93
    jinwonkim93 committed Mar 8, 2023
    Configuration menu
    Copy the full SHA
    8a61b92 View commit details
    Browse the repository at this point in the history

  5. [Style] change __all__ to __ALL__

    @jinwonkim93
    jinwonkim93 committed Mar 8, 2023
    Configuration menu
    Copy the full SHA
    78f003a View commit details
    Browse the repository at this point in the history

  6. [Style] change __all__ to __ALL__

    @jinwonkim93
    jinwonkim93 committed Mar 8, 2023
    Configuration menu
    Copy the full SHA
    520eb53 View commit details
    Browse the repository at this point in the history

  7. [Style] change zero __all__ to __ALL__

    @jinwonkim93
    jinwonkim93 committed Mar 8, 2023
    Configuration menu
    Copy the full SHA
    2408bf0 View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2023

  1. Update oslo/torch/nn/parallel/data_parallel/zero/sharded_optim/bookke… 

    …eping/__init__.py

Co-authored-by: Junhwa Song <ethan9867@gmail.com>
    @jinwonkim93 @KKIEEK
    jinwonkim93 and KKIEEK committed Mar 9, 2023
    Configuration menu
    Copy the full SHA
    697653b View commit details
    Browse the repository at this point in the history