-
Background The paper discusses the significant challenges existing methods face in the text-to-image diffusion models, particularly in generating images that align with image conditional controls. The researchers propose a new method called ControlNet++, designed to improve controllable generation by explicitly optimizing pixel-level cycle consistency between generated images and conditional controls.
-
Existing Work Existing approaches often involve retraining diffusion models from scratch, which demands significant computational resources and large public datasets, the latter of which are scarce. To improve controllability, studies have attempted fine-tuning pre-trained text-to-image models or introducing trainable modules like ControlNet. However, these existing methods still fail to control image generation precisely and lack a clear strategy for improvement, with generated images significantly deviating from input conditions.
- Introduced ControlNet++
-
Challenge 1: Lack of precise controllability Achieving a high degree of controllability with generated images aligning with input conditions is a notable challenge. ControlNet++ addresses this issue by employing pre-trained discriminative models to extract the corresponding condition from the generated images and then optimizing the consistency loss between the input conditional control and the extracted condition. This method embraces the concept of cycle consistency, improving performance by explicitly optimizing controllability at the pixel level.
-
Challenge 2: Efficiency and resource constraints Implementing pixel-level loss directly can lead to efficiency issues, requiring storage of gradients at every time step, resulting in substantial time and GPU memory consumption. To circumvent these costs, ControlNet++ introduces an efficient reward strategy. It disrupts the consistency between input images and conditions, then uses single-step denoised images to fine-tune reward and reconstruct consistency, thus avoiding time and memory overhead from image sampling.
-
ControlNet++ was extensively tested, showing significant improvements in controllability across various conditional controls. For instance, compared to ControlNet, ControlNet++ achieved improvements in mIoU, SSIM, and RMSE of 7.9%, 13.4%, and 7.6%, respectively, for segmentation masks, line-art edges, and depth conditions. These results demonstrate that ControlNet++ can produce more accurately controllable generated images across various conditions and provides a unified and public evaluation of controllability for further research.
ControlNet++ significantly improves controllability across a range of conditional controls by optimizing pixel-level consistency between the generated images and the conditions, while the efficient reward fine-tuning strategy reduces the time and memory costs associated with image sampling.