Skip to content

Jeff-LiangF/streamv2v

Repository files navigation

StreamV2V

English | 中文 | 日本語

Looking Backward: Streaming Video-to-Video Translation with Feature Banks
Feng Liang, Akio Kodaira, Chenfeng Xu, Masayoshi Tomizuka, Kurt Keutzer, Diana Marculescu

arXiv Project Page Huggingface demo

Highlight

Our StreamV2V could perform real-time video-2-video translation on one RTX 4090 GPU. Check the video and try it by yourself!

Video

For functionality, our StreamV2V supports face swap (e.g., to Elon Musk or Will Smith) and video stylization (e.g., to Claymation or doodle art). Check the video and reproduce the results!

Video

Although StreamV2V is designed for the vid2vid task, it could seamlessly integrate with the txt2img application. Compared with per-image StreamDiffusion, StreamV2V continuously generates images from texts, providing a much smoother transition. Check the video and try it by yourself!

Video

Installation

Please see the installation guide.

Getting started

Please see getting started instruction.

Realtime camera demo on GPU

Please see the demo with camera guide.

Continuous txt2img

Please see the demo continuous txt2img.

LICENSE

StreamV2V is licensed under a UT Austin Research LICENSE.

Acknowledgements

Our StreamV2V is highly dependended on the open-source community. Our code is copied and adapted from < StreamDiffusion with LCM-LORA. Besides the base SD 1.5 model, we also use a variaty of LORAs from CIVITAI.

Citing StreamV2V 🙏

If you use StreamV2V in your research or wish to refer to the baseline results published in the paper, please use the following BibTeX entry.

@article{liang2024looking,
  title={Looking Backward: Streaming Video-to-Video Translation with Feature Banks},
  author={Liang, Feng and Kodaira, Akio and Xu, Chenfeng and Tomizuka, Masayoshi and Keutzer, Kurt and Marculescu, Diana},
  journal={arXiv preprint arXiv:2405.15757},
  year={2024}
}

@article{kodaira2023streamdiffusion,
  title={StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation},
  author={Kodaira, Akio and Xu, Chenfeng and Hazama, Toshiki and Yoshimoto, Takanori and Ohno, Kohei and Mitsuhori, Shogo and Sugano, Soichi and Cho, Hanying and Liu, Zhijian and Keutzer, Kurt},
  journal={arXiv preprint arXiv:2312.12491},
  year={2023}
}

Contributors