A diffusers based implementation of HyperDreamBooth
the code is based on the HyperDreamBooth implementation in the LyCORIS project.
If you have any idea on these todos, just open issues or PRs. Thx!
- Implement in-run validation
- Implement section 4
- Implement better dreambooth training (like mask the face during training)
- Implement a more general hypernetwork (Already in LyCORIS)
This section is a brief introduction of HyperDreamBooth, I will split hyperdreambooth into 4 sections. And this projection will have 4 corresponding scripts
Before we start to train the hypernetwork, we should train the lora(lilora) on each identites(instance) first. In this implementation, we took a batch of instances and then do a inner training loop to get the pre optimized weights.
Just train it. send the image into hypernetwork, get the weights, apply to unet. Calc the loss based on diffusion loss and weight loss.
Use the image of the identity you want to train on, generate the weight from hypernetwork.
resume from the generated weight, do a few step training (for about 20~50step)