-
Notifications
You must be signed in to change notification settings - Fork 44
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Could you share distill log for each dataset ? #30
Comments
Sorry I did not take the logs out from my internship, so I don't really have it anymore. Maybe you can specify what kind of issues you are encountering? |
Emmmmm, I never encountered such issue before. Make sure that your data
preprocessing is correct, and the codes you changed do not divide by 0 or
so.
…On Thu, Jun 1, 2023, 5:37 AM Li Jiahui ***@***.***> wrote:
I am distilling on my own data.
Occasionally I get 'WARNING:tensorboardX.x2num:NaN or Inf found in input
tensor.
' then the loss is Nan as well. Restart then NaN would disappear ...
I have no idea how to explore this bug .....
[image: image]
<https://user-images.githubusercontent.com/21051247/242471283-39d08935-d716-4636-b460-bac19a23e3b8.png>
—
Reply to this email directly, view it on GitHub
<#30 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADNJ3IXU57USQ4NADIXD4Y3XJAE6HANCNFSM6AAAAAAYVKLH5E>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
@wengmingao I will investigate this from my end and share the information to you later. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
As title, for nuScenes, ScanNet, Matterport, it would be great if you can share the logs of distilling, which can help me to debug the training process ...
The text was updated successfully, but these errors were encountered: