Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

how to do resume of training? should i need to train for 300 epoch to get quantised ,model? #3

Open
akashAD98 opened this issue Jun 23, 2022 · 4 comments

Comments

@akashAD98
Copy link

akashAD98 commented Jun 23, 2022

doing resume=True in train.py solved the problem

@akashAD98 akashAD98 reopened this Jun 23, 2022
@akashAD98 akashAD98 changed the title how to do resume of training? how to do resume of training? should i need to train for 300 epoch to get quantised ,model? Jun 23, 2022
@akashAD98
Copy link
Author

akashAD98 commented Jun 23, 2022

should i need to complete all 300 epoch training to get a quantized model? bcz I'm getting an error when i try to export 150 epoch model, & when i resume this training, it adds extra epochs.
here you can see i tried it for 300 epochs but when i stopped training & started resuming training its showed 389 epochs

image

@akashAD98
Copy link
Author

akashAD98 commented Jun 24, 2022

getting this issue while converting to onnx, whats wrong here? should i need to do continuous training without stop?

image

@akashAD98
Copy link
Author

even training is not completed ,only 2 epochs remaining ,its giving cuda out of memory ,

image

image

@dnth
Copy link
Owner

dnth commented Jun 24, 2022

Hi @akashAD98

The quantization only happens at the last 2 epochs of the training. This is specified in the recipe file. So if you halt the training before the quantization epoch, you will not get a quantized model.

The screenshot below shows where you can change the quantization epoch.

image

num_epochs is the total number of training epochs.

quantization_start_epoch is the exact epoch where quantization begins.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants