multi-gpu training triggers CUDA out of memory error

[griff4692]

Hi -

I am running into issues when going from single to multi-gpu training. Specifically, if I switch the line

pl.Trainer(gpus=1, precision=16, distributed_backend='ddp')

to

pl.Trainer(gpus=4, precision=16, distributed_backend='ddp')

I get the dreaded CUDA out of memory error. Is there any reason why the parallelism causes the GPU to receive more data?

[github-actions]

Hi! thanks for your contribution!, great first issue!

[justusschock]

Hi, what are your outputs of the validation_step? If there are any large tensors, it's likely they get synced back to root GPU by #2434 . We're working on that.

cc @williamFalcon ^^

[griff4692]

Hi - I actually haven't implemented the validation step yet. this just occurs on the training side

[justusschock]

what is your gpu consumption on a single gpu (used/available)?

[griff4692]

On single gpu, I am using 5/11 GB. The problem seems to be that when I switch over to multiple GPUs, there is an explosion of processes created on the first GPU. Any ideas what could be causing this?

[griff4692]

Fixed it!. I was calling .to('cuda') on my input tensors in my Dataset __get__item function which caused all the data to be uploaded to the first GPU. Removed that and solved the problem.

[williamFalcon]

@justusschock does that mean we should add back the all reduce for val?

[justusschock]

No, there were other issues with that as well :D Let's just keep it out for now.

[MuhammadWaleedUsman]

Fixed it!. I was calling .to('cuda') on my input tensors in my Dataset __get__item function which caused all the data to be uploaded to the first GPU. Removed that and solved the problem.

I have the same issue but couldnt solve it by removing .to('cuda'). when i do this i get error:
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

[ahmadikalkhorani]

I have the same issue. When I use 2 nodes everything seems fine. However, when I try to increase the number of nodes it causes CUDA out-of-memory error!