-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
**gather_all_tensors_if_available** share the same underlying storage for all GPUs #3253
Comments
@ShomyLiu good catch, would you be up for sending a PR? Please, note that the function is not used anywhere yet, but are there for future changes to the metric package. |
@SkafteNicki it's my pleasure for a PR. I will finish this as soon as possible. Yeah, it's a new function to wrap the |
@ShomyLiu Yes, I agree that it is a common use case. |
@SkafteNicki Hi, I have sent a PR jus now for your review #3319 |
* Fix: gather_all_tensors cross GPUs in metrics * add a test case for gather_all_tensors_ddp in #3253
🐛 Bug
Hi, one of new features in #2528
gather_all_tensors_if_available
has alist
copy bug, and this would lead that tensors in all GPUs are the wrongly same as one GPU, since they share the same storage:https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/metrics/converters.py#L304
change into:
The text was updated successfully, but these errors were encountered: