Add support for multiple controlnet

[JingyaHuang]

What does this PR do?

• Export of multiple controlnets

optimum-cli export neuron --inline-weights-neff --model jyoung105/stable-diffusion-v1-5 --task stable-diffusion --auto_cast matmul --auto_cast_type bf16 --batch_size 1 --num_images_per_prompt 1 --controlnet_ids lllyasviel/control_v11p_sd15_openpose lllyasviel/control_v11f1p_sd15_depth --height 512 --width 512 sd15-512x512-bf16-openpose-depth

• Support for multiple ControlNets for stable diffusion

import numpy as np
import torch
from PIL import Image

from controlnet_aux import OpenposeDetector
from transformers import pipeline
from diffusers import UniPCMultistepScheduler
from diffusers.utils import load_image
from optimum.neuron import NeuronStableDiffusionControlNetPipeline


# OpenPose+Depth ControlNet
model_id = "sd15-512x512-bf16-openpose-depth"

# Load ControlNet images

# 1. openpose
image_0 = load_image("https://huggingface.co/lllyasviel/control_v11p_sd15_openpose/resolve/main/images/input.png")
processor = OpenposeDetector.from_pretrained('lllyasviel/ControlNet')
openpose_image = processor(image_0)

# 2. depth
image_1 = load_image("https://huggingface.co/lllyasviel/control_v11p_sd15_depth/resolve/main/images/input.png")
depth_estimator = pipeline('depth-estimation')
image_1 = depth_estimator(image_1)['depth']
image_1 = np.array(image_1)
image_1 = image_1[:, :, None]
image_1 = np.concatenate([image_1, image_1, image_1], axis=2)
depth_image = Image.fromarray(image_1)

images = [openpose_image.resize((512, 512)), depth_image.resize((512, 512))]

# Inference
pipe = NeuronStableDiffusionControlNetPipeline.from_pretrained(model_id)
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
prompt = "a giant in a fantasy landscape, best quality"
negative_prompt = "monochrome, lowres, bad anatomy, worst quality, low quality"

image = pipe(prompt=prompt, image=images).images[0]
image.save('inf2_out.png')

• Support for multiple ControlNets for SDXL

• Export

optimum-cli export neuron -m stabilityai/stable-diffusion-xl-base-1.0 --task stable-diffusion-xl --batch_size 1 --height 1024 --width 1024 --controlnet_ids diffusers/controlnet-canny-sdxl-1.0-small thibaud/controlnet-openpose-sdxl-1.0 --num_images_per_prompt 1 sdxl_neuron_canny_openpose/

• Inference
  Theoretically yes, but easily get oom without parallelism which will be supported with future NxD integration for inference.

• Tests

• Documentation

• Some changes to fix CIs

It seems that the CIs failed after we bump the transformers/optimum versions:

• Deprecating use_auth_token since the CIs failed with recent changes in Optimum: Deprecated use_auth_token optimum#1837
• Fix generation mixing following: Generate: store special token tensors under a unique variable name transformers#31980

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[mludvig]

Hi @JingyaHuang that's great, it works! Thanks a lot for providing the fix 🎉

[dacorvo]

LGTM, thank you for the pull-request !

[dacorvo]

nit: I actually think we should pass the token here, so that it can be omitted later.

cc @Wauplin

[JingyaHuang]

that sounds good, I just did it the way it was to pass the CIs, I will leave it to you if you want to change! (don't want to wait for CIs again hhh

[michaelbenayoun]

LGTM