Add Scheduler.from_pretrained and better scheduler changing #1286

patrickvonplaten · 2022-11-14T21:17:26Z

🚨🚨 Scheduler.from_pretrained & Better scheduler switching 🚨🚨

This PR was inspired by @keturn's feedback here: #1268

It deprecates SchedulerClass.from_config(<path>) and instead improves the usability of from_config to change schedulers.

1. Allow to change schedulers with the from_config API. It should become as easy as:

from diffusers import DiffusionPipeline, DDIMScheduler

pipeline = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5")

pipeline.scheduler = DDIMScheduler.from_config(pipeline.scheduler.config)

1. Add a from_pretrained(...) API to the schedulers.

Schedulers should now be loaded with:

from diffusers import DDIMScheduler

ddim = DDIMScheduler.from_pretrained("google/ddpm-cifar10-32")

Doing:

from diffusers import DDIMScheduler

ddim = DDIMScheduler.from_config("google/ddpm-cifar10-32")

is deprecated.

…main

HuggingFaceDocBuilder · 2022-11-14T21:21:36Z

The documentation is not available anymore as the PR was closed or merged.

…add_scheduler_save_pretrained

docs/source/using-diffusers/schedulers.mdx

patrickvonplaten · 2022-11-15T13:01:03Z

Also ping @keturn for some feedback since the PR originated from our discussions

patrickvonplaten · 2022-11-15T13:02:13Z

All slow tests are passing - ran them locally (except ONNX because they take forever 😅 - but we'll seen when merged on main :-) )

pcuenca

Some small typos.

docs/source/using-diffusers/schedulers.mdx

pcuenca

Very awesome! Changes look good, but there are many files touched so I'll try to test a checkout for real later.

docs/source/api/configuration.mdx

docs/source/api/pipelines/stable_diffusion.mdx

docs/source/quicktour.mdx

docs/source/using-diffusers/loading.mdx

src/diffusers/schedulers/scheduling_utils.py

src/diffusers/schedulers/scheduling_utils_flax.py

tests/test_pipelines.py

patil-suraj

Amazing PR, thanks a lot for working on this :)

The PR looks good to me, also tried it out and works super well. My main comment are

Whether we should add _get_compatibles in ConfigMixin or SchedulerMixin. As ConfigMixin is also used by models, and this compatibility notion does not exist for models.
Not sure if _COMPATIBLE_STABLE_DIFFUSION_SCHEDULERS is a good name, since as said in the comment below schedulers can be compatible with each other irrespective of the model.

Apart from this everything looks good!

docs/source/using-diffusers/schedulers.mdx

patil-suraj · 2022-11-15T14:39:19Z

docs/source/using-diffusers/schedulers.mdx

+# first we need to login with our access token
+login()


Is this function available in previous hub versions ?

login() is till in the hf_hub RC, let's wait until a full release and then add it with huggingface_hub>=0.11.0 in the dependencies.
Until then we still need to use the CLI

Sounds good

docs/source/using-diffusers/schedulers.mdx

src/diffusers/configuration_utils.py

patil-suraj · 2022-11-15T14:53:54Z

src/diffusers/schedulers/scheduling_ddim.py

@@ -24,7 +24,7 @@

 from ..configuration_utils import ConfigMixin, register_to_config
 from ..utils import BaseOutput
-from .scheduling_utils import SchedulerMixin
+from .scheduling_utils import _COMPATIBLE_STABLE_DIFFUSION_SCHEDULERS, SchedulerMixin


Not sure if _COMPATIBLE_STABLE_DIFFUSION_SCHEDULERS is the right name. Think schedulers can be compatible with each other irrespective of the model. For example, DDIM and DDPM will always be compatible and could be interchanged for sampling. Maybe we could call this _COMPATIBLE_SCHEDULERS ?

Think there will be other "groups" of schedulers soon so just _COMPATIBLE_SCHEDULERS is a bit too generic. Will leave for now, we can always adapt down the road

src/diffusers/configuration_utils.py

patil-suraj · 2022-11-15T14:57:39Z

src/diffusers/schedulers/scheduling_dpmsolver_multistep_flax.py

+    [`SchedulerMixin`] provides general loading and saving functionality via the [`SchedulerMixin.save_pretrained`] and
+    [`~SchedulerMixin.from_pretrained`] functions.


We don't need _COMPATIBLE_STABLE_DIFFUSION_SCHEDULERS here ?

Actually yeah let's add it right away for Flax as well - why not!

src/diffusers/schedulers/scheduling_utils.py

anton-l

Overall a very clear upgrade, and the tests are solid, thank you @patrickvonplaten!

anton-l · 2022-11-15T15:16:31Z

docs/source/using-diffusers/schedulers.mdx

+# first we need to login with our access token
+login()


login() is till in the hf_hub RC, let's wait until a full release and then add it with huggingface_hub>=0.11.0 in the dependencies.
Until then we still need to use the CLI

docs/source/using-diffusers/schedulers.mdx

tests/test_pipelines.py

docs/source/api/configuration.mdx

Co-authored-by: Pedro Cuenca <pedro@huggingface.co> Co-authored-by: Anton Lozhkov <anton@huggingface.co> Co-authored-by: Suraj Patil <surajp815@gmail.com>

patrickvonplaten · 2022-11-15T17:15:07Z

Applied all changes - thanks a bunch for the review! Merging!

…ace#1286) * add conversion script for vae * uP * uP * more changes * push * up * finish again * up * up * up * up * finish * up * uP * up * Apply suggestions from code review Co-authored-by: Pedro Cuenca <pedro@huggingface.co> Co-authored-by: Anton Lozhkov <anton@huggingface.co> Co-authored-by: Suraj Patil <surajp815@gmail.com> * up * up Co-authored-by: Pedro Cuenca <pedro@huggingface.co> Co-authored-by: Anton Lozhkov <anton@huggingface.co> Co-authored-by: Suraj Patil <surajp815@gmail.com>

patrickvonplaten added 4 commits November 14, 2022 19:43

add conversion script for vae

ec7c8d3

Merge branch 'main' of https://github.com/huggingface/diffusers into …

e4ffadc

…main

Merge branch 'main' of https://github.com/huggingface/diffusers into …

d5ab55e

…main

uP

864d1d9

patrickvonplaten changed the title ~~Add scheduler save pretrained~~ Add scheduler from pretrained Nov 14, 2022

uP

28d2472

patrickvonplaten added 5 commits November 14, 2022 21:31

more changes

2f00458

push

5f389e3

up

fc9acb1

finish again

04e671c

up

d005419

patrickvonplaten mentioned this pull request Nov 14, 2022

set_scheduler method #1247

Closed

patrickvonplaten added 5 commits November 15, 2022 09:22

Merge branch 'main' of https://github.com/huggingface/diffusers into …

425dda0

…add_scheduler_save_pretrained

up

144a587

up

21b84e5

up

0a79f00

finish

e3ec5f1

patrickvonplaten changed the title ~~Add scheduler from pretrained~~ Add Scheduler.from_pretrained and better scheduler changing Nov 15, 2022

patrickvonplaten added 3 commits November 15, 2022 12:52

up

f1a68ba

uP

e596aae

up

4d7f90d

patrickvonplaten commented Nov 15, 2022

View reviewed changes

docs/source/using-diffusers/schedulers.mdx Show resolved Hide resolved

patrickvonplaten requested review from pcuenca, patil-suraj and anton-l and removed request for pcuenca November 15, 2022 13:02

patrickvonplaten linked an issue Nov 15, 2022 that may be closed by this pull request

[RFC] Proposal to change API of ConfigMixin and SchedulerMixin. #1268

Closed

pcuenca reviewed Nov 15, 2022

View reviewed changes

patil-suraj approved these changes Nov 15, 2022

View reviewed changes

anton-l approved these changes Nov 15, 2022

View reviewed changes

patrickvonplaten commented Nov 15, 2022

View reviewed changes

docs/source/api/configuration.mdx Outdated Show resolved Hide resolved

patrickvonplaten commented Nov 15, 2022

View reviewed changes

docs/source/api/configuration.mdx Outdated Show resolved Hide resolved

patrickvonplaten and others added 3 commits November 15, 2022 16:45

Apply suggestions from code review

f998ace

Co-authored-by: Pedro Cuenca <pedro@huggingface.co> Co-authored-by: Anton Lozhkov <anton@huggingface.co> Co-authored-by: Suraj Patil <surajp815@gmail.com>

up

22dbe58

up

741e89f

patrickvonplaten merged commit a052019 into main Nov 15, 2022

patrickvonplaten deleted the add_scheduler_save_pretrained branch November 15, 2022 17:17

skirsten mentioned this pull request Nov 16, 2022

[Flax] Fix loading scheduler from subfolder #1319

Merged

leopoldmaillard mentioned this pull request Dec 13, 2022

Question about noise schedule parametrization & samplers #1656

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Scheduler.from_pretrained and better scheduler changing #1286

Add Scheduler.from_pretrained and better scheduler changing #1286

patrickvonplaten commented Nov 14, 2022 •

edited

Loading

HuggingFaceDocBuilder commented Nov 14, 2022 •

edited

Loading

patrickvonplaten commented Nov 15, 2022

patrickvonplaten commented Nov 15, 2022

pcuenca left a comment

pcuenca left a comment

patil-suraj left a comment

patil-suraj Nov 15, 2022

anton-l Nov 15, 2022

patrickvonplaten Nov 15, 2022

patil-suraj Nov 15, 2022

patrickvonplaten Nov 15, 2022

patil-suraj Nov 15, 2022

patrickvonplaten Nov 15, 2022

anton-l left a comment •

edited

Loading

anton-l Nov 15, 2022

patrickvonplaten commented Nov 15, 2022

		[`SchedulerMixin`] provides general loading and saving functionality via the [`SchedulerMixin.save_pretrained`] and
		[`~SchedulerMixin.from_pretrained`] functions.

Add Scheduler.from_pretrained and better scheduler changing #1286

Add Scheduler.from_pretrained and better scheduler changing #1286

Conversation

patrickvonplaten commented Nov 14, 2022 • edited Loading

🚨🚨 Scheduler.from_pretrained & Better scheduler switching 🚨🚨

HuggingFaceDocBuilder commented Nov 14, 2022 • edited Loading

patrickvonplaten commented Nov 15, 2022

patrickvonplaten commented Nov 15, 2022

pcuenca left a comment

Choose a reason for hiding this comment

pcuenca left a comment

Choose a reason for hiding this comment

patil-suraj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anton-l left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickvonplaten commented Nov 15, 2022

patrickvonplaten commented Nov 14, 2022 •

edited

Loading

HuggingFaceDocBuilder commented Nov 14, 2022 •

edited

Loading

anton-l left a comment •

edited

Loading