Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Training process not stable #12

Open
yzheng97 opened this issue Aug 28, 2019 · 1 comment
Open

Training process not stable #12

yzheng97 opened this issue Aug 28, 2019 · 1 comment

Comments

@yzheng97
Copy link

Durintg training, the loc_loss ocassionly became a very large value (As shown below). I checked that the calculation of loc_loss would not encounter the denominator=0 situation. I wonder why.

counter:[21000], loss:9098183573504.00000000, loc_loss:9098183573504.00000000, conf_loss:0.00971856, conf_pos_loss:0.00420601, conf_neg_loss:0.00551255, reg_loss:32.57967377

counter:[22000], loss:285012888911872.00000000, loc_loss:285012888911872.00000000, conf_loss:0.01419698, conf_pos_loss:0.00380920, conf_neg_loss:0.01038779, reg_loss:34.89893341

@ZongxuPan
Copy link
Owner

Durintg training, the loc_loss ocassionly became a very large value (As shown below). I checked that the calculation of loc_loss would not encounter the denominator=0 situation. I wonder why.

counter:[21000], loss:9098183573504.00000000, loc_loss:9098183573504.00000000, conf_loss:0.00971856, conf_pos_loss:0.00420601, conf_neg_loss:0.00551255, reg_loss:32.57967377

counter:[22000], loss:285012888911872.00000000, loc_loss:285012888911872.00000000, conf_loss:0.01419698, conf_pos_loss:0.00380920, conf_neg_loss:0.01038779, reg_loss:34.89893341

Since everyone using this code could encounter his/her specific issues, completely relying on us to solving these issues is unpractical, in part because the information is not enough in part because we have not so much time or effort, so we refer the users to the Q&A part in README, especially the second Q&A for your own data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants