Skip to content
/ EVP Public

Code for paper 'Audio-Driven Emotional Video Portraits'.

Notifications You must be signed in to change notification settings

jixinya/EVP

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

14 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Audio-Driven Emotional Video Portraits [CVPR2021]

Xinya Ji, Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, Feng Xu

[Project] [Paper]

visualization

Given an audio clip and a target video, our Emotional Video Portraits (EVP) approach is capable of generating emotion-controllable talking portraits and change the emotion of them smoothly by interpolating at the latent space.

Installation

We train and test based on Python3.6 and Pytorch. To install the dependencies run:

pip install -r requirements.txt

Testing

  • Download the pre-trained models and data under the following link: google-drive (we release results of two target person: M003 and M030), unzip the test.zip and put the file in corresponding places.

  • Step1 : audio2landmark

    The emotion of predicted landmark motions can be manipulated by the emotion features (recommanded):

    python audio2lm/test.py --config config/target_test.yaml --audio path/to/audio --condition feature --emo_feature path/to/feature
    

    or by the emotional audio of the target person:

    python audio2lm/test.py --config config/target_test.yaml --audio path/to/audio --condition feature --emo_audio path/to/emo_audio
    

    The results will be stored in results/target.mov

  • Step2 : landmark2video

    A parametric 3D face model and the corresponding fitting algorithm should be used here to regress the geometry, expression and pose parameters of the predicted landmarks and the target video. Here we release some parameters of the testing results.

    lm2video/data/target/3DMM/3DMM: images and landmark positions of the video

    lm2video/data/target/3DMM/target_test: parameters of target's video

    lm2video/data/target/3DMM/target_test_pose: pose parameters of video

    lm2video/data/target/3DMM/test_results: parameters of predicted landmarks

    Here we use vid2vid to generate video from edgemaps:

    1. Generate the testing data by running:

      python lm2video/lm2map.py
      

      and copy the results in lm2video/results/ to vid2vid/datasets/face/.

    2. Replace the face_dataset.py and base_options.py in vid2vid to lm2video/face_dataset.py and lm2video/base_options.py, the 106 keypoint version.

    3. Copy lm2video/data/target/latest_net_G0.pth to vid2vid/checkpoints/target/ , lm2video/test_target.sh to vid2vid/scripts/face and run:

      bash ./scripts/face/test_target.sh
      

Training

  • Download the pre-trained models and data under the following link: google-drive (we release data of M030), unzip the train.zip and put the file in corresponding places.

  • Step1 : emotion_pretrain:

  1. Generate the trainig data(MFCC) from the raw audio:

    python emotion_pretrain/code/mfcc_preprocess.py
    
  2. The emotion classification for MFCC:

     python emotion_pretrain/code/train.py
    
  • Step2 : disentanglement
  1. Use DTW to align the audio:

    python disentanglement/dtw/MFCC_dtw.py
    
  2. Cross-reconstruction for disentanglement:

     python disentanglement/code/train_content+cla.py
    
  • Step3 : landmark
  1. Generate the data for training:

    python landmark/code/preprocess.py
    
  2. Training the Audio-to-Landmark module:

     python landmark/code/train.py
    

Citation

@article{ji2021audio,
  title={Audio-Driven Emotional Video Portraits},
  author={Ji, Xinya and Zhou, Hang and Wang, Kaisiyuan and Wu, Wayne and Loy, Chen Change and Cao, Xun and Xu, Feng},
  journal={arXiv preprint arXiv:2104.07452},
  year={2021}
}

About

Code for paper 'Audio-Driven Emotional Video Portraits'.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published