Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable SD XL ONNX export and ONNX Runtime inference #1168

Merged
merged 40 commits into from
Jul 17, 2023
Merged

Enable SD XL ONNX export and ONNX Runtime inference #1168

merged 40 commits into from
Jul 17, 2023

Conversation

echarlaix
Copy link
Collaborator

@echarlaix echarlaix commented Jul 6, 2023 •
edited
Loading

Add Stable Diffusion XL ONNX export and pipelines to enable ONNX Runtime inference for text-to-image and image-to-image

Export using the CLI :

optimum-cli export onnx --model stabilityai/stable-diffusion-xl-base-0.9 --task stable-diffusion-xl ./sd_xl_onnx

For inference :

from optimum.onnxruntime import ORTStableDiffusionXLPipeline

model_id = "stabilityai/stable-diffusion-xl-base-0.9"
pipeline = ORTStableDiffusionXLPipeline.from_pretrained(model_id, export=True)

@HuggingFaceDocBuilderDev
Copy link

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

@echarlaix echarlaix mentioned this pull request Jul 7, 2023
Merged
@echarlaix echarlaix marked this pull request as ready for review July 10, 2023 17:23
@echarlaix echarlaix changed the title Add SD XL export Enable SD XL ONNX export and ONNX Runtime inference Jul 10, 2023
fxmarty
fxmarty reviewed Jul 11, 2023
View reviewed changes
Copy link
Contributor

@fxmarty fxmarty left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks for working on it! Probably we could have export / run tests & doc as well in the PR?

Edit: not sure what happened, I was not seeing the right diff and I see there are tests already!

optimum/onnxruntime/modeling_diffusion.py Outdated Show resolved Hide resolved
optimum/onnxruntime/modeling_diffusion.py Outdated Show resolved Hide resolved
fxmarty
fxmarty reviewed Jul 11, 2023
View reviewed changes
tests/exporters/onnx/test_exporters_onnx_cli.py Outdated Show resolved Hide resolved
optimum/exporters/tasks.py Show resolved Hide resolved
optimum/exporters/tasks.py Show resolved Hide resolved
optimum/onnxruntime/modeling_diffusion.py Outdated Show resolved Hide resolved
optimum/pipelines/diffusers/watermark.py Outdated Show resolved Hide resolved
optimum/utils/normalized_config.py Outdated Show resolved Hide resolved
optimum/pipelines/diffusers/pipeline_stable_diffusion_xl_img2img.py Outdated Show resolved Hide resolved
michaelbenayoun
michaelbenayoun reviewed Jul 12, 2023
View reviewed changes
optimum/exporters/onnx/__main__.py Outdated Show resolved Hide resolved
optimum/exporters/onnx/__main__.py Outdated Show resolved Hide resolved
optimum/exporters/onnx/convert.py Outdated Show resolved Hide resolved
optimum/exporters/onnx/model_configs.py Outdated Show resolved Hide resolved
optimum/exporters/onnx/model_configs.py Outdated Show resolved Hide resolved
optimum/exporters/onnx/model_configs.py Outdated Show resolved Hide resolved
optimum/exporters/onnx/model_configs.py Outdated Show resolved Hide resolved
optimum/exporters/onnx/utils.py Outdated Show resolved Hide resolved
optimum/pipelines/diffusers/pipeline_stable_diffusion_xl.py Outdated Show resolved Hide resolved
@echarlaix echarlaix requested a review from regisss July 13, 2023 10:27
fxmarty
fxmarty approved these changes Jul 17, 2023
View reviewed changes
Copy link
Contributor

@fxmarty fxmarty left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, awesome!

(just left taste comments)

tests/exporters/onnx/test_onnx_export.py Show resolved Hide resolved
tests/onnxruntime/test_stable_diffusion_pipeline.py Show resolved Hide resolved
optimum/exporters/onnx/__main__.py Show resolved Hide resolved
michaelbenayoun
michaelbenayoun approved these changes Jul 17, 2023
View reviewed changes
Copy link
Member

@michaelbenayoun michaelbenayoun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @echarlaix

tests/exporters/onnx/test_exporters_onnx_cli.py Show resolved Hide resolved
@echarlaix echarlaix merged commit a9ffe07 into main Jul 17, 2023
65 of 66 checks passed
@echarlaix echarlaix deleted the sd-XL branch July 17, 2023 16:39
baskrahmer pushed a commit to baskrahmer/optimum that referenced this pull request Jul 22, 2023
@echarlaix @baskrahmer
Enable SD XL ONNX export and ONNX Runtime inference (huggingface#1168)
c5252a3 
* add stable diffusion XL export

* fix style

* fix test model name

* fix style

* remove clip with projection from test

* change model name

* fix style

* remove need create pretrainedconfig

* fix style

* fix dummy input generation

* add saving second tokenzier when exporting a SD XL model

* fix style

* add SD XL pipeline

* fix style

* add test

* add watermarker

* fix style

* add watermark

* add test

* set default height width stable diffusion pipeline

* enable img2img task

* fix style

* enable to only have the second tokenizer and text encoder

* add test

* fix cli export

* adapt test for batch size > 1
@NeusZimmer
Copy link

Any option to run fp16 without cuda for O4 optimization on AMD?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fxmarty fxmarty fxmarty approved these changes

@michaelbenayoun michaelbenayoun michaelbenayoun approved these changes

@patrickvonplaten patrickvonplaten Awaiting requested review from patrickvonplaten

@JingyaHuang JingyaHuang Awaiting requested review from JingyaHuang

@regisss regisss Awaiting requested review from regisss

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@echarlaix @HuggingFaceDocBuilderDev @NeusZimmer @fxmarty @michaelbenayoun