Skip to content

Novel View Synthesis with multiplane/multilayer representation: CVPR2022, WACV2023

License

Notifications You must be signed in to change notification settings

SamsungLabs/MLI

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

64 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

SIMPLI - Self-improving Multiplane-to-layer Images for Novel View Synthesis



Google Colab

WACV 2023

Logo

We suggest to use collab to run our inference code. To see the resulted geometry use viewer incorporated into the notebook. We demonstrate results at our project page.

Build docker image

After downloading the repository, use docker/Dockerfile to create an image that sets up all dependencies.

docker build -t nvs_torch_image ./docker

nvidia-docker container run \
    -it --shm-size=16G \
    -u $(id -u ${USER}):$(id -g ${USER}) \
    --rm \
    -v $(pwd):/home/Develop/nvs_torch  \
    nvs_torch_image:latest

Setup env

cd /home/Develop/nvs_torch
export PYTHONPATH=/home/Develop/nvs_torch/
export TORCH_EXTENSIONS_DIR=tmp
export MPLCONFIGDIR=tmp

SIMPLI model already in the repo.

Train model with sword dataset

dataset link - https://samsunglabs.github.io/StereoLayers/sword/ You need to load the full dataset as the small test data. Then set the correct paths to data in the config file.

torchrun \
  --standalone \ # if all GPUs are on a single node
  --nproc_per_node=1 \ # set number of GPUs here
  bin/train.py \
  --config configs/tblock4_train.yaml \
  --output-path train_outputs

Prepare validation data

render_val_dataset.py script below expects a different data format (check out lib/datasets/val_dataset.py for details). First, process your data with configs from configs/val_data: sword_val.yaml is for rendering the exact copies of the validation dataset, sword_val_spiral.yaml is for generating views along novel trajectories. Possible trajectories are listed in lib/modules/cameras/trajectory_generators.py. The following script prepares data for validation format.

python bin/val_utils/generate_val_dataset.py \
        --config configs/val_data/sword_val_spiral.yaml \
        --output-path sample_val_spiral

Render predefined Dataset

Weights for rendering the model must be in checkpoints subfolder of the same folder as the config.

python bin/val_utils/render_val_dataset.py \
       --config pretrained/stereo_layers/stereo_layers.yaml \
       --val-dataset datasets/sword_sample/ \ #folder of the data in validation format
       --iteration 400000 \ #iteration of the checkpoint, must be in its file name
       --output-path outputs

Run Rendering with custom functions

python bin/render.py \
       --config pretrained/stereo_layers/stereo_layers.yaml \
       --checkpoints-path pretrained/stereo_layers/checkpoints \
       --iteration 400000 \ 
       --output-path outputs

About

Novel View Synthesis with multiplane/multilayer representation: CVPR2022, WACV2023

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published