Skip to content
/ VDT Public

[ICLR2024] The official implementation of paper "VDT: General-purpose Video Diffusion Transformers via Mask Modeling", by Haoyu Lu, Guoxing Yang, Nanyi Fei, Yuqi Huo, Zhiwu Lu, Ping Luo, Mingyu Ding.

License

Notifications You must be signed in to change notification settings

RERV/VDT

Repository files navigation

VDT

[ICLR2024] The official implementation of paper "VDT: General-purpose Video Diffusion Transformers via Mask Modeling", by Haoyu Lu, Guoxing Yang, Nanyi Fei, Yuqi Huo, Zhiwu Lu, Ping Luo, Mingyu Ding.

Introduction

This work introduces Video Diffusion Transformer (VDT), which pioneers the use of transformers in diffusion-based video generation. It features transformer blocks with modularized temporal and spatial attention modules, allowing separate optimization of each component and leveraging the rich spatial-temporal representation inherited from transformers.

  1. It excels at capturing temporal dependencies to produce temporally consistent video frames and even simulate the physics and dynamics of 3D objects over time.
  2. It facilitates flexible conditioning information, e.g., simple concatenation in the token space, effectively unifying different token lengths and modalities.
  3. Pairing with our proposed spatial-temporal mask modeling mechanism, it becomes a general-purpose video diffuser for harnessing a range of tasks, including unconditional generation, video prediction, interpolation, animation, and completion, etc

Extensive experiments on video generation, prediction, and dynamics modeling (i.e., physics-based QA) tasks have been conducted to demonstrate the effectiveness of VDT in various scenarios, including autonomous driving, human action, and physics-based simulation.

Release

2024-05-05: Release spatial-temporal mask modeling code and inference code.
2024-01-27: Our VDT has been accepted by ICLR2024.
2023-05-22: We propose Video Diffusion Transformer (VDT) model and release checkpoint and inference code.

Getting Started

  • Python3, PyTorch>=1.8.0, torchvision>=0.7.0 are required for the current codebase.
  • To install the other dependencies, run
    conda env create -f environment.yml
    conda activate VDT

Checkpoint

We now provide checkpoint for Sky Time-Lapse unified generation. You can download it from here.

Inference

We provide inference ipynb on Sky Time-Lapse unified generation (predict, backward, unconditional, single-frame, arbitrary interpolation, spatial_temporal). To sample results, you can first download the checkpoint, then run inference.ipynb, have fun!

Acknowledgement

Our codebase is built based on DiT, BEiT, SlotFormer and MVCD. We thank the authors for the nicely organized code!

About

[ICLR2024] The official implementation of paper "VDT: General-purpose Video Diffusion Transformers via Mask Modeling", by Haoyu Lu, Guoxing Yang, Nanyi Fei, Yuqi Huo, Zhiwu Lu, Ping Luo, Mingyu Ding.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published