Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CLIP]: Sparseml onnx export pathways for open_clip models #1626

Merged
merged 11 commits into from
Jun 23, 2023

Conversation

dsikka
Copy link
Contributor

@dsikka dsikka commented Jun 14, 2023

Summary:

  • The current onnx export functionality using OpenClip requires torch nightly and opset = 14
  • This PR provides an example of how to use sparseml's onnx export functionality to export the visual and text branches of the CLIP model, using OpenClip
  • OpenClip wraps everything in a torch.nn.Module object for the visual branch (ie everything required for the forward pass) but does not do this for the text branch. For the sake of completeness, the TextModel class was added (see clip_models.py) which wraps the appropriate steps needed in the forward pass. This is then exported
  • Furthermore, for captioning, OpenClip provides thee CoCa models, which requires an additional text-decoder and also uses a separate interface compared to the other models. As such, there is a special is_coca case provided in the scripts

Test Plan:

  • Ran this locally (requires openclip and torch nightly to be installed) and created a convnext_onnx directory with the clip_text.onnx and clip_visual.onnx exports
 python clip_onnx_export.py --model convnext_base_w_320 --pretrained laion_aesthetic_s13b_b82k --export-path convnext_onnx
  • Also ran a CoCa-specific command which created the default clip_onnx directory with the visual and text exports, along with the text decoder export clip_text_decoder.onnx
python clip_onnx_export.py --model coca_ViT-L-14 --pretrained laion2b_s13b_b90k

@dsikka dsikka marked this pull request as ready for review June 15, 2023 17:21
@dsikka dsikka requested review from KSGulin and bfineran June 15, 2023 17:21
@dsikka dsikka changed the title CLIP: Sparseml onnx export pathways for open_clip models [CLIP]: Sparseml onnx export pathways for open_clip models Jun 16, 2023
bfineran
bfineran previously approved these changes Jun 22, 2023
Copy link
Member

@bfineran bfineran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM pending comments

integrations/clip/clip_models.py Outdated Show resolved Hide resolved
integrations/clip/clip_onnx_export.py Show resolved Hide resolved
src/sparseml/pytorch/base.py Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
bfineran
bfineran previously approved these changes Jun 23, 2023
integrations/clip/clip_onnx_export.py Show resolved Hide resolved
integrations/clip/clip_onnx_export.py Outdated Show resolved Hide resolved
integrations/clip/clip_onnx_export.py Show resolved Hide resolved
KSGulin
KSGulin previously approved these changes Jun 23, 2023
@dsikka dsikka dismissed stale reviews from KSGulin and bfineran via 980b6bc June 23, 2023 15:18
Copy link
Member

@rahul-tuli rahul-tuli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@dsikka dsikka merged commit 5a3697f into main Jun 23, 2023
10 checks passed
@dsikka dsikka deleted the clip_export_example branch June 23, 2023 15:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants