About Loss NaN

[liangleikun]

Hello, Thanks for your nice work! And I have some problems.
I trained on my own dataset. After some epochs of training, loss would become Nan. I added the minimum in Focalloss, but the wh_loss and off_loss also are Nan and inf. Could you give me some advice. Thanks.

[yijingru]

Hi, I met this problem before. I guess the unnormalized data distribution in the head layers would result in unstable training. Can you add a batch normalization after the head convolutional layers (e.g. Conv+BN+ReLU+Conv) to see if it helps?

[liangleikun]

Thank you very much. I have added a batch normalization in the head convolutional layers. It is useful.

[yijingru]

Thanks for letting me know. I add this information to the new version.

[huangmanba1]

Hi, After I added a batch normalization in the head convolutional layers , the loss still become Nan, what should I do

[yijingru]

Hi, After I added a batch normalization in the head convolutional layers , the loss still become Nan, what should I do

How large is the batch size? The other solution I think would be (1) increasing the batch size or (2) decreasing the learning rate. Empirically I stop at about 40 epochs using the default learning rate.

[18804601171]

@yijingru my batch size is 8 and I use 2 gpus, In addition, I add batch batch normalization,but loss is still become nan, what should I do?

[rush9838465]

I noticed that the Focalloss log function input parameter pred is sometimes 0. (sigmoid output).
So I changed the following code, Seems to be working:
pos_loss = torch.log(pred+0.0000001) * torch.pow(1 - pred, 2) * pos_inds

[Fly-dream12]

I have tried this but the hm loss is still nan. @yijingru

[navidasj96]

main reason that cause this problem is images without any object in it, by deleting those images after spliting main images with scale .5 and 1 this problem will solve and decreasing learning rate and adding that batch normalization layer didnt help me without deleting that images

[yijingru]

main reason that cause this problem is images without any object in it, by deleting those images after spliting main images with scale .5 and 1 this problem will solve and decreasing learning rate and adding that batch normalization layer didnt help me without deleting that images

Thanks a lot for the effort to clear this issue! It's good to know the cause of the NAN loss. I will also share your comment in README.md. Thanks!