feat(python): separate algorithm declaration and implementation #246

Tengxu-Sun · 2021-10-06T14:10:36Z

refactor(python): support distributed algorithm instance reuse for multi models

…lti models

bagua/torch_api/algorithms/async_model_average.py

bagua/torch_api/algorithms/bytegrad.py

bagua/torch_api/algorithms/decentralized.py

tests/torch_api/test_multi_models.py

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

bagua/torch_api/algorithms/bytegrad.py

bagua/torch_api/algorithms/decentralized.py

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

tests/torch_api/test_multi_models.py

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

…n.py Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

bagua/torch_api/algorithms/decentralized.py

bagua/torch_api/algorithms/gradient_allreduce.py

bagua/torch_api/algorithms/q_adam.py

tests/torch_api/test_multi_models.py

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

bagua/torch_api/algorithms/decentralized.py

bagua/torch_api/algorithms/gradient_allreduce.py

bagua/torch_api/algorithms/q_adam.py

tests/torch_api/test_multi_models.py

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

bagua/torch_api/algorithms/decentralized.py

bagua/torch_api/algorithms/gradient_allreduce.py

bagua/torch_api/algorithms/q_adam.py

tests/torch_api/test_multi_models.py

NOBLES5E · 2021-10-14T08:25:53Z

bagua/torch_api/algorithms/async_model_average.py

@@ -29,17 +29,10 @@ def __init__(
        warmup_steps: int = 0,
    ):
        """
-        Create an instance of the
+        Instance implementation of the


Implementation of

bagua/torch_api/algorithms/gradient_allreduce.py

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

NOBLES5E · 2021-10-18T02:21:40Z

bagua/torch_api/algorithms/bytegrad.py

@@ -8,10 +8,10 @@
 from typing import List


-class ByteGradAlgorithm(Algorithm):
+class ByteGradAlgorithm_Implementation(Algorithm):


Don't add underscore _. Similar for other places.

bagua/torch_api/algorithms/gradient_allreduce.py

…nto algorithm_check

wangraying · 2021-10-19T05:51:05Z

bagua/torch_api/algorithms/async_model_average.py

@@ -21,15 +22,15 @@ class _AsyncInternalState(IntEnum):
    ABORT = 1


-class AsyncModelAverageAlgorithm(Algorithm):
+class AsyncModelAverageAlgorithmImplementation(Algorithm):


I think it's better to keep the inheritance relationship between **Algorithm and Algorithm. The Algorithm class has a method reify, and different **Algorithm overwrites it.

Add another base class AlgorithmImpl for all implementations of algorithms, and its derived classes **AlgorithmImpl overwrites init_tensors, init_forward_pre_hook ..., etc.

This seems more natural for our current with_bagua method, which accept an Algorithm as its second argument.

And AsyncModelAverageAlgorithmImplementation seems too long, better named as AsyncModelAverageAlgorithmImpl, it is in accord with Java naming notation. :)

bagua/torch_api/algorithms/base.py

wangraying · 2021-10-19T07:41:44Z

bagua/torch_api/algorithms/base.py

+
+class AlgorithmImpl:
+    """
+    This is the base class that all Bagua algorithms implementation inherit.


I guess all Bagua algorithm implementations ? @NOBLES5E

wangraying · 2021-10-19T07:46:31Z

bagua/torch_api/algorithms/base.py

+    and return a corresponding algorithm instance.
+    """
+
+    def reify(self):


Proposal:
remove It provides a method that accept the specified distributed algorithm parameters and return a corresponding algorithm instance.

and just add doc for reify: "Create an algorithm instance" ?

refactor(python): support distributed algorithm instance reuse for mu…

a36a6ea

…lti models

pr-triage bot added the PR: unreviewed label Oct 6, 2021

Tengxu-Sun marked this pull request as draft October 6, 2021 14:10

github-actions bot reviewed Oct 6, 2021

View reviewed changes

pr-triage bot added PR: draft and removed PR: unreviewed labels Oct 6, 2021

Tengxu-Sun and others added 3 commits October 6, 2021 22:12

Update bagua/torch_api/algorithms/async_model_average.py

9e71d33

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Update bagua/torch_api/algorithms/bytegrad.py

b29d911

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Update bagua/torch_api/algorithms/decentralized.py

e5620ca

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

github-actions bot reviewed Oct 6, 2021

View reviewed changes

Tengxu-Sun and others added 6 commits October 6, 2021 22:12

Update tests/torch_api/test_multi_models.py

8334548

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Update tests/torch_api/test_multi_models.py

1f27cae

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Update tests/torch_api/test_multi_models.py

20ed8ca

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Update tests/torch_api/test_multi_models.py

37c5774

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Update tests/torch_api/test_multi_models.py

2c30d74

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Update bagua/torch_api/algorithms/decentralized.py

ea3dea6

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

github-actions bot reviewed Oct 6, 2021

View reviewed changes

tests/torch_api/test_multi_models.py Outdated Show resolved Hide resolved

tests/torch_api/test_multi_models.py Outdated Show resolved Hide resolved

tests/torch_api/test_multi_models.py Outdated Show resolved Hide resolved

Tengxu-Sun and others added 6 commits October 6, 2021 22:13

Update bagua/torch_api/algorithms/decentralized.py

5996c55

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Update bagua/torch_api/algorithms/q_adam.py

55388f5

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Update bagua/torch_api/algorithms/gradient_allreduce.py

b0ba5b2

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Update bagua/torch_api/algorithms_implementation/__init__.py

ac230df

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Update bagua/torch_api/algorithms_implementation/q_adam_implementatio…

87188b0

…n.py Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Update tests/torch_api/test_multi_models.py

fdad05b

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

github-actions bot reviewed Oct 6, 2021

View reviewed changes

Tengxu-Sun and others added 2 commits October 6, 2021 22:16

Update tests/torch_api/test_multi_models.py

2650954

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Update tests/torch_api/test_multi_models.py

c661c32

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

github-actions bot reviewed Oct 6, 2021

View reviewed changes

Tengxu-Sun and others added 2 commits October 6, 2021 22:17

Update tests/torch_api/test_multi_models.py

4c1659d

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Update tests/torch_api/test_multi_models.py

9232149

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

github-actions bot reviewed Oct 6, 2021

View reviewed changes

NOBLES5E reviewed Oct 14, 2021

View reviewed changes

change docs

359b82d

Tengxu-Sun requested a review from NOBLES5E October 14, 2021 10:54

merge master and resolve confict file

700494e

github-actions bot reviewed Oct 17, 2021

View reviewed changes

bagua/torch_api/algorithms/gradient_allreduce.py Outdated Show resolved Hide resolved

suntengxu and others added 3 commits October 17, 2021 11:27

fix error

3d21af4

retrigger checks

e3a6156

Update bagua/torch_api/algorithms/gradient_allreduce.py

431bb3d

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

NOBLES5E reviewed Oct 18, 2021

View reviewed changes

Update async_model_average.py

5d0ccdc

github-actions bot reviewed Oct 18, 2021

View reviewed changes

bagua/torch_api/algorithms/gradient_allreduce.py Outdated Show resolved Hide resolved

github-actions bot reviewed Oct 18, 2021

View reviewed changes

bagua/torch_api/algorithms/gradient_allreduce.py Outdated Show resolved Hide resolved

suntengxu and others added 3 commits October 18, 2021 10:33

delete underscore

79eaaad

Update async_model_average.py

cff7466

Merge branch 'algorithm_check' of https://github.com/BaguaSys/bagua i…

3887036

…nto algorithm_check

Tengxu-Sun requested a review from NOBLES5E October 18, 2021 04:47

wangraying reviewed Oct 19, 2021

View reviewed changes

suntengxu added 2 commits October 19, 2021 14:39

refactor the inheritance relationship

ee60440

rename the instance

a852f08

github-actions bot reviewed Oct 19, 2021

View reviewed changes

bagua/torch_api/algorithms/base.py Show resolved Hide resolved

fix error

de0c7e0

Tengxu-Sun requested a review from wangraying October 19, 2021 07:35

wangraying reviewed Oct 19, 2021

View reviewed changes

fix docs

83c4f6f

NOBLES5E changed the title ~~refactor(python): support distributed algorithm instance reuse for multi models~~ feat(python): separate algorithm declaration and implementation Oct 19, 2021

NOBLES5E merged commit 6be1399 into master Oct 19, 2021

pr-triage bot removed the PR: unreviewed label Oct 19, 2021

NOBLES5E deleted the algorithm_check branch October 19, 2021 08:32

pr-triage bot added the PR: merged label Oct 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(python): separate algorithm declaration and implementation #246

feat(python): separate algorithm declaration and implementation #246

Tengxu-Sun commented Oct 6, 2021

NOBLES5E Oct 14, 2021

Tengxu-Sun Oct 14, 2021

NOBLES5E Oct 18, 2021

Tengxu-Sun Oct 18, 2021

wangraying Oct 19, 2021

wangraying Oct 19, 2021 •

edited

Loading

Tengxu-Sun Oct 19, 2021

wangraying Oct 19, 2021

NOBLES5E Oct 19, 2021

wangraying Oct 19, 2021 •

edited

Loading

Tengxu-Sun Oct 19, 2021

feat(python): separate algorithm declaration and implementation #246

feat(python): separate algorithm declaration and implementation #246

Conversation

Tengxu-Sun commented Oct 6, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangraying Oct 19, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangraying Oct 19, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangraying Oct 19, 2021 •

edited

Loading

wangraying Oct 19, 2021 •

edited

Loading