CUDA memory leak

[awsaf49]

I'm getting CUDA out of memory for training v5 on 1024 size Image with batch_size 4. I have trained v5 on 1024 image_size with batch_size 4 before without any error. But now I'm getting this error.
But the weird fact is while training, GPU usage is around 10GB and it trains perfectly. But while validating it throws the error.

[glenn-jocher]

@awsaf49 I reviewed the code here, it seems like testing batch sizes were inadvertently affected by PR #2125, I've pushed a fix for this in #2587, though note this should only affect Multi-GPU training memory issues during testing.

[awsaf49]

But I was using a single GPU also unfortunately it wasn't solved by this change.

[glenn-jocher]

@awsaf49 thanks for the info! If you believe you have a reproducible issue, we suggest you close this issue and raise a new one using the 🐛 Bug Report template, providing screenshots and a minimum reproducible example to help us better understand and diagnose your problem. Thank you!