refactor: new module wrapping design and algorithm implementation design

[NOBLES5E]

Goal: To support easier implementation of new algorithms like OneBitAdam. Decouple bucketing policy from wrapper.

[todo]

previous buckets and hooks need to be cleared before reinit

bagua/bagua/torch_api/distributed_dev.py

Lines 13 to 18 in 71a759f

	# TODO: previous buckets and hooks need to be cleared before reinit
	pass
	def init_buckets(self, module, optimizer) -> List:
	pass

---

This comment was generated by todo based on a TODO comment in 71a759f in #24. cc @BaguaSys.

[todo]

document this

bagua/bagua/torch_api/distributed_dev.py

Lines 34 to 39 in 71a759f

	# TODO: document this
	if hasattr(module, "_ddp_params_and_buffers_to_ignore"):
	parameters_to_ignore = module._ddp_params_and_buffers_to_ignore
	else:
	parameters_to_ignore = []
	module_states = []

---

This comment was generated by todo based on a TODO comment in 71a759f in #24. cc @BaguaSys.

[todo]

broadcast optimizer parameters

bagua/bagua/torch_api/distributed_dev.py

Lines 61 to 66 in 71a759f

	# TODO: broadcast optimizer parameters
	self.init_algorithm()
	def init_algorithm(self):
	self.buckets = self.algorithm.init_buckets(self.module, self.optimizer)

---

This comment was generated by todo based on a TODO comment in 71a759f in #24. cc @BaguaSys.

[github-actions]

[blackfmt] _{reported by reviewdog 🐶}

Suggested change

	class BaguaTensor(torch.Tensor):
	class BaguaTensor(torch.Tensor):

[github-actions]

[blackfmt] _{reported by reviewdog 🐶}

Suggested change

[todo]

@shjwudp merge with service module

bagua/bagua/autotune/__init__.py

Lines 1 to 4 in 1cfccfe

	# TODO: @shjwudp merge with service module
	import copy
	import collections
	import logging

---

This comment was generated by todo based on a TODO comment in 1cfccfe in #24. cc @BaguaSys.

[todo]

@ganshaoduo check if this should be used

bagua/bagua/torch_api/algorithms/onebit_adam.py

Lines 192 to 197 in 1cfccfe

	# weight_decay = group["weight_decay"] # TODO: @ganshaoduo check if this should be used
	beta1, beta2 = group["betas"]
	eps = group["eps"]
	for param_id, param in enumerate(group["params"]):
	state = self.state[param]

---

This comment was generated by todo based on a TODO comment in 1cfccfe in #24. cc @BaguaSys.

[todo]

remove parameter group logic

bagua/bagua/torch_api/distributed.py

Lines 276 to 281 in 1cfccfe

	] # TODO: remove parameter group logic
	)
	self._bagua_autotune_client.register_models( # TODO: @shjwudp rename to register tensors
	autotune_tensor_list, bagua_tensor_group_info
	).json() # TODO: @shjwudp error check

---

This comment was generated by todo based on a TODO comment in 1cfccfe in #24. cc @BaguaSys.

[todo]

@shjwudp check whether these are still needed

bagua/bagua/torch_api/utils.py

Lines 261 to 264 in 1cfccfe

	# score = np.mean(score_list) # TODO: @shjwudp check whether these are still needed
	# std = np.std(score_list)
	return np.mean(score_list), np.std(score_list), score_list.tolist()

---

This comment was generated by todo based on a TODO comment in 1cfccfe in #24. cc @BaguaSys.

[todo]

@shjwudp merge with service module

bagua/bagua/autotune/__init__.py

Lines 1 to 4 in 96cb6fe

	# TODO: @shjwudp merge with service module
	import copy
	import collections
	import logging

---

This comment was generated by todo based on a TODO comment in 96cb6fe in #24. cc @BaguaSys.

[todo]

@ganshaoduo check if this should be used

bagua/bagua/torch_api/algorithms/onebit_adam.py

Lines 192 to 197 in 96cb6fe

	# weight_decay = group["weight_decay"] # TODO: @ganshaoduo check if this should be used
	beta1, beta2 = group["betas"]
	eps = group["eps"]
	for param_id, param in enumerate(group["params"]):
	state = self.state[param]

---

This comment was generated by todo based on a TODO comment in 96cb6fe in #24. cc @BaguaSys.

[todo]

remove parameter group logic

bagua/bagua/torch_api/distributed.py

Lines 276 to 281 in 96cb6fe

	] # TODO: remove parameter group logic
	)
	self._bagua_autotune_client.register_models( # TODO: @shjwudp rename to register tensors
	autotune_tensor_list, bagua_tensor_group_info
	).json() # TODO: @shjwudp error check

---

This comment was generated by todo based on a TODO comment in 96cb6fe in #24. cc @BaguaSys.

[todo]

@shjwudp check whether these are still needed

bagua/bagua/torch_api/utils.py

Lines 261 to 264 in 96cb6fe

	# score = np.mean(score_list) # TODO: @shjwudp check whether these are still needed
	# std = np.std(score_list)
	return np.mean(score_list), np.std(score_list), score_list.tolist()

---

This comment was generated by todo based on a TODO comment in 96cb6fe in #24. cc @BaguaSys.