Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix export of all quantized transformer models #1654

Merged
merged 9 commits into from
Jul 7, 2023

Conversation

eldarkurtic
Copy link
Contributor

Our current transformers export-to-onnx pipeline doesn't work for quantized models. To reproduce: try to export any quantized model with the latest versions of sparseml and neuralmagic/transformers.

There are two reasons:

  1. we are passing in resolved_archive_file=[] at which causes error in HF library here: https://github.com/neuralmagic/transformers/blob/0798c9e3b743a7e5c552f943a1a7d52ff63bbffb/src/transformers/modeling_utils.py#L3150 . We can fix this by passing None instead of [].
  2. the function self.model._load_pretrained_model from transformers returns 6 items (see https://github.com/neuralmagic/transformers/blob/0798c9e3b743a7e5c552f943a1a7d52ff63bbffb/src/transformers/modeling_utils.py#L3323), and in our interface we only accept 5
    _, missing, unexpected, _, _ = self.model._load_pretrained_model(
    which also raises an issue while unpacking the returned tuple from their function.

bfineran
bfineran previously approved these changes Jul 6, 2023
Copy link
Member

@bfineran bfineran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great catch @eldarkurtic this should also fix our broken tests on main. Thank you!

@eldarkurtic
Copy link
Contributor Author

eldarkurtic commented Jul 6, 2023

Accidentally pushed changes from this PR (#1630) as well to enable --trust_remote_code, so we can close that PR by merging this one into main.

@eldarkurtic eldarkurtic requested review from bfineran, a team, KSGulin and robertgshaw2-neuralmagic and removed request for a team July 6, 2023 20:36
@bfineran
Copy link
Member

bfineran commented Jul 7, 2023

failing tests are unrelated - investigating separately but unable to reproduce outside of GHA. Merging this PR to unblock functionality

@bfineran bfineran merged commit 4ec5133 into neuralmagic:main Jul 7, 2023
9 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants