Non-Adversarial Video Synthesis with Generative Latent Nearest Neighbors - VGLANN

Abstract

Generative video models can impact many applications (e.g., future prediction) in video understanding and simulation. We propose a generative non-adversarial network for video based on GLANN model (Non-Adversarial Image Synthesis with Generative Latent Nearest Neighbors), with a spatio-temporal convolutional architecture that untangles the scene’s foreground from the background which is based on MIT model (Generating Videos with Scene Dynamics). The experiments showed that the model provides good but unsatisfactory results.

Read More - Non-Adversarial Video Synthesis with Generative Latent Nearest Neighbors

This repo contains PyTorch code.

Requirements

• Python 3+
• Pytorch

Files

• glo.py - Laplacian Loss

• icp.py - IMLE model

• model_video_orig.py - MIT model for generating videos (two stream model foreground and background).

• modelvideo.py - Expanded MIT model for generating videos (two stream model foreground and background).

• perceptual_loss_video.py - perceptual losses (resnext 101, resnet 50 - 3D model for action recognition, pretrained model on the Kinetics dataset).

• resnet.py - resnet 50 for perceptual loss (forward method has been changed).

• resnext.py - resnext 101 for perceptual loss (forward method has been changed).

• vae.py - VAE model

• vgg_metric.py - VGG perceptual loss

• train_vae.py - training and generating a video via vae model (mapping with vae from noise space to a latent space), using the weights og GLO model. run this file with lab_vae.sh

• train_gmm - training and generating a video via gmm model (mapping with vae from noise space to a latent space), using the weights og GLO model. run this file with lab_gmm.sh

• tester_icp - training and generating a video via imle model (mapping with vae from noise space to a latent space), using the weights og GLO model. run this file with lab_imle.sh

• tester_video - training GLO model, run this file with GLANN.sh

• glo_video_trainer.py - training GLO.

• calc_fvd.py - calculate FVD score

• frechet_video_distance.py - FVD program. copyright to google research.

Getting Started

1. download the data from MIT github (link in the paper). and use the data_loader.py for preprocces the data

2. run tester_video.py or GLANN.sh. this will train the glo model and you should choose your hyperparams in this file.

3. run tester_icp.py / train_vae.py / train_gmm.py for training this models respectively. those files generate a video

---

Example (Optional)

run tester_video.py 
run train_vae.py 

---

Some Examples