Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature Request] Support Stochastic Weight Averaging (SWA) for improved stability #321

Open
1 task done
pchalasani opened this issue Nov 27, 2022 · 2 comments · May be fixed by #320
Open
1 task done

[Feature Request] Support Stochastic Weight Averaging (SWA) for improved stability #321

pchalasani opened this issue Nov 27, 2022 · 2 comments · May be fixed by #320
Labels
enhancement New feature or request

Comments

@pchalasani
Copy link
Contributor

🚀 Feature

Stochastic Weight Averaging (SWA) is a recently proposed technique can potentially help improve training stability in DRL. There is now a new implementation in torchcontrib. Quoting/paraphrasing from their page:

a simple procedure that improves generalization in deep learning over Stochastic Gradient Descent (SGD) at no additional cost, and can be used as a drop-in replacement for any other optimizer in PyTorch. SWA has a wide range of applications and features, [...] including [...] improve the stability of training as well as the final average rewards of policy-gradient methods in deep reinforcement learning.

See the PyTorch SWA page for more.

Motivation

SWA might help improve training stability as well as final reward in some DRL scenarios. It may also alleviate sensitivity to random-seed initialization.

Pitch

See above :)

Alternatives

No response

Additional context

See the PyTorch SWA page for more.

Checklist

  • I have checked that there is no similar issue in the repo
@pchalasani pchalasani added the enhancement New feature or request label Nov 27, 2022
@pchalasani pchalasani linked a pull request Nov 27, 2022 that will close this issue
Draft
13 tasks
@pchalasani pchalasani changed the title [Feature Request] request title [Feature Request] Support Stochastic Weight Averaging (SWA) for improved stability Nov 27, 2022
@araffin
Copy link
Member

araffin commented Nov 29, 2022 •
edited
Loading

Hello,

can potentially help improve training stability in DRL

do you have experimental results to back this claim?

In the paper linked in the blog post, results are on A2C/DDPG only (which have usually weaker results compared to PPO/TD3/SAC) and they used only 3 random seeds, which is not enough to account for noise in the results.

Torch contrib is also now archived and didn't receive any update for almost 3 years (https://github.com/pytorch/contrib).

EDIT: SWA seems to be directly in pytorch now https://pytorch.org/docs/stable/optim.html#stochastic-weight-averaging

@pchalasani
Copy link
Contributor Author

Thanks, I did not know SWA is in main PyTorch. I will look into it.
As for empirical evidence, I'll continue experimenting and report back.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support for Stoch Wt Avg (SWA) closes #321 pchalasani/rl-baselines3-zoo
2 participants
@pchalasani @araffin