Skip to content

Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models

License

Notifications You must be signed in to change notification settings

luosiallen/Diff-Foley

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

26 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

[NeurIPS 2023] Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models

Offical implementation of the NeurIPS 2023 paper: Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models.

Project Page: https://diff-foley.github.io

To-Do:

  • Evaluation Tool ☑️
  • Stage1 CAVP Training Code ☑️
  • Stage2 LDM Training Code ☑️
  • Environment Setting
  • Diff-Foley Inference Code ☑️
  • Diff-Foley Pretrained Model ☑️

News

  • (🔥New) 2023/11/5 Diff-Foley Inference Pipeline is released! See the 'Inference Usages'.
  • (🔥New) 2023/11/5 Diff-Foley Pretrained Model is released! Download from Hugging Face 🤗 here.
  • Including: Stage1-CAVP, Stage2-LDM, Double-Guidance Classifier !!

Inference Usages:

  1. Open the diff_foley_inference.ipynb in inference folder.
  2. Download the pretrained model foler diff_foley_ckpt from Hugging Face 🤗 here and place it under inference folder.
  3. Run the diff_foley_inference.ipynb.

Diff-Foley

BibTeX

@misc{luo2023difffoley, 
title={Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models}, 
author={Simian Luo and Chuanhao Yan and Chenxu Hu and Hang Zhao}, 
year={2023}, 
eprint={2306.17203}, 
archivePrefix={arXiv}, 
primaryClass={cs.SD} 
}

About

Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages