Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Native Amp Support #1336

Closed
mcarilli opened this issue Apr 2, 2020 · 2 comments · Fixed by #1561
Closed

Native Amp Support #1336

mcarilli opened this issue Apr 2, 2020 · 2 comments · Fixed by #1561
Labels
duplicate This issue or pull request already exists feature Is an improvement or enhancement help wanted Open to be worked on

Comments

@mcarilli
Copy link

mcarilli commented Apr 2, 2020 •
edited
Loading

Native automatic mixed precision support (torch.cuda.amp) is finally merged:
https://pytorch.org/docs/master/amp.html
https://pytorch.org/docs/master/notes/amp_examples.html
Apex Amp has many known pain points (extension builds, forward/backward compatibilty, DataParallel support, flaky checkpointing, i don’t even know if it can be hacked to handle double backward/gradient penalty, others…). torch.cuda.amp repairs all these, the interface is more flexible and intuitive, and the tighter integration brings more future performance optimizations into scope.

If you want to talk about adding torch.cuda.amp to Lightning, with an eye towards it becoming the true source of mixed precision and replacing Apex, message me on Pytorch slack anytime. I pinged you there as well, but I’m not sure if you monitor it habitually.
@mcarilli mcarilli added feature Is an improvement or enhancement help wanted Open to be worked on labels Apr 2, 2020
@github-actions
Copy link
Contributor

github-actions bot commented Apr 2, 2020

Hi! thanks for your contribution!, great first issue!

@mcarilli mcarilli changed the title Native Amp Support Native Amp (torch.cuda.amp) Support Apr 2, 2020
@mcarilli mcarilli changed the title Native Amp (torch.cuda.amp) Support Native Amp Support Apr 2, 2020
@mcarilli
Copy link
Author

mcarilli commented Apr 2, 2020

Closing and starting a duplicate issue to get #1337.

@mcarilli mcarilli closed this as completed Apr 2, 2020
@Borda Borda added the duplicate This issue or pull request already exists label Apr 2, 2020
@williamFalcon williamFalcon mentioned this issue Apr 22, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
duplicate This issue or pull request already exists feature Is an improvement or enhancement help wanted Open to be worked on
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

support for native amp Lightning-AI/pytorch-lightning
2 participants
@Borda @mcarilli