CharacterFactory: Sampling Consistent Characters with GANs for Diffusion Models

🤗[Gradio Demo]🔥   [Paper]   [Project Page]

News

• [2024.05.16]: Release a Gradio Demo! Thanks HuggingFace and AK!
• [2024.05.15]: Release training and inference codes!

More results can be found in our Project Page and Paper.

---

Getting Started

Installation

• Requirements (Only need 2.5GB VRAM for training):

  conda create -n IDEGAN python=3.8.5
  conda activate IDEGAN
  pip install -r requirements.txt

• Download pretrained models: Stable Diffusion v2-1_512.

• Set the paths of pretrained SD-2.1 models as default in the Line106 of train.py or command with

  --pretrained_model_name_or_path **your SD-2.1 path**

Train

We have already provided the pretrained weights in training_weight.

python train.py --experiment_name="normal_GAN"

The trained IDE-GAN model will be saved in training_weight.

Test

First set your SD-2.1 path in the test files.

• test.ipynb: Generate a random character with text prompts.

• test_create_many_characters.ipynb: Generate many characters with text prompts.

The results will be generated in test_results/{index}/.

• If you want to test with ControlNet, ModelScopeT2V and LucidDreamer, you could find the related codes in StableIdentity.

TODOs

• Release training and inference code
• Huggingface demo

Citation

@article{wang2024characterfactory,
  title={CharacterFactory: Sampling Consistent Characters with GANs for Diffusion Models},
  author={Wang, Qinghe and Li, Baolu and Li, Xiaomin and Cao, Bing and Ma, Liqian and Lu, Huchuan and Jia, Xu},
  journal={arXiv preprint arXiv:2404.15677},
  year={2024}
}