Project Page | Latest arXiv | Colab Notebook | LLFF Released Data | BLEFF Data | ckpts
Zirui Wang¹, Shangzhe Wu², Weidi Xie², Min Chen³, Victor Adrian Prisacariu¹.
¹Active Vision Lab (AVL) + ²Visual Geometry Group (VGG) + ³e-Research Centre, University of Oxford.
Some BLEFF-related info discussed in issues:
- #34: training config.
- #37: coordinate system.
- #38: loading script.
- #41: non-normalised rotation matrix issue.
Release our checkpoints on LLFF dataset.
Update our arXiv paper with
- A breaking point analysis for the camera parameter estimation. In short, our method can tolerate ±20 degrees of rotation variance and ±20% of translation variance.
- A customised dataset, which we named Blender Forward Facing (BLEFF). We will provide the dataloader file to load this data soon.
Initial code release, corresponding to arXiv paper v3.
We provide 3 training targets in this repository, under the tasks
directory:
tasks/nerfmm/train.py
: This is our main training script for the NeRF-LLFF dataset, which estimates camera poses, focal lenghts and a NeRF jointly and monitors the absolute trajectory error (ATE) between our estimation of camera parameters and COLMAP estimation during training. This target can also start training from a COLMAP initialisation and refine the COLMAP camera parameters.tasks/refine_nerfmm/train.py
: This is the training script that refines a pretrained nerfmm system.tasks/any_folder/train.py
: This is a training script that takes a folder that contains forward-facing images and trains with our nerfmm system without making any comparison with COLMAP. It is similar to what we offer in our CoLab notebook and we treat thisany_folder
target as a playgraound, where users can try novel view synthesis by just providing an image folder and do not care how the camera parameter estimation compares with COLMAP.
For each target, we provide relevant utilities to evaluate our system. Specifically,
- for the
nerfmm
target, we provide three utility files:eval.py
to evaluate image rendering quality on validation splits with PSNR, SSIM and LPIPS, i.e, results in Table 1.spiral.py
to render novel views using a spiral camera trajectory, i.e. results in Figure 1.vis_learned_poses.py
to visualise our camera parameter estimation with COLMAP estimation in 3D. It also computes ATE between them, i.e. E1 in Table 2.
- for the
refine_nerfmm
target, all utilities innerfmm
target above are compatible withrefine_nerfmm
target, since it just refines a pretrained nerfmm system. - for the
any_folder
target, it has its ownspiral.py
andvis_learned_poses.py
utilities, as it does not compare with COLMAP. It does not have aeval.py
file as this target is treated as a playground and does not split images to train/validation sets. It only provides novel view synthesis results via thespiral.py
file.
We provide a environment.yml
file to set up a conda
environment:
git clone https://github.com/ActiveVisionLab/nerfmm.git
cd nerfmm
conda env create -f environment.yml
Generally, our code should be able to run with any pytorch >= 1.1
.
(Optional) Install open3d
for visualisation. You might need a physical monitor to install this lib.
pip install open3d
We use the LLFF dataset with two small structural changes:
- We remove their
image_4
andimage_8
folder and downsample images to any desirable resolution during data loadingdataloader/with_colmap.py
, by calling PyTorch'sinterpolate
function. - We explicitly generate two txt files for train/val image ids. i.e. take every 8th image as the validation set, as in the official NeRF train/val split. The only difference is that we store them as txt files while NeRF split them during data loading. The file produces these two txt files is
utils/split_dataset.py
.
In addition to the NeRF-LLFF dataset, we provide two demo scenes to demonstrate how to use the any_folder
target.
We pack the re-structured LLFF data and our data to a tar ball (~1.8G), to get it, run:
wget https://www.robots.ox.ac.uk/~ryan/nerfmm2021/nerfmm_release_data.tar.gz
tar -xzvf path/to/the/tar.gz
There are mainly two reasons that motivate us to create BLEFF:
- We need to evaluate both the camera parameter estimation accuracy and image rendering quality at the same time.
- To facilitate the analysis of the robustness of our method, a dataset with progressively increasing pose perturbation levels is required.
To that end, we introduce a synthetic dataset BLEFF, containing 14 path-traced scenes, with each rendered in multiple levels of rotation and translation perturbations. Those scenes are modified and rendered with open-source blender files on blendswap and the license info can be found in our supplementary file.
Usage of our data:
wget https://www.robots.ox.ac.uk/~ryan/nerfmm2021/BLEFF.tar.gz
tar -xzvf path/to/the/tar.gz
We show how to:
- train a
nerfmm
from scratch, i.e. initialise camera poses with identity matrices and focal lengths with image resolution:python tasks/nerfmm/train.py \ --base_dir='path/to/nerfmm_release/data' \ --scene_name='LLFF/fern'
- train a
nerfmm
from COLMAP initialisation:This command initialises apython tasks/nerfmm/train.py \ --base_dir='path/to/nerfmm_release/data' \ --scene_name='LLFF/fern' \ --start_refine_pose_epoch=1000 \ --start_refine_focal_epoch=1000
nerfmm
target with COLMAP parameters, trains with them for 1000 epochs, and starts refining those parameters after 1000 epochs. - train a
nerfmm
from a pretrainednerfmm
:This command initialises apython tasks/refine_nerfmm/train.py \ --base_dir='path/to/nerfmm_release/data' \ --scene_name='LLFF/fern' --start_refine_epoch=1000 \ --ckpt_dir='path/to/a/dir/contains/nerfmm/ckpts'
refine_nerfmm
target with a set of pretrainednerfmm
parameters, trains with them for 1000 epochs, and starts refining those parameters after 1000 epochs. - train an
any_folder
from scratch given an image folder:This command trains anpython tasks/any_folder/train.py \ --base_dir='path/to/nerfmm_release/data' \ --scene_name='any_folder_demo/desk'
any_folder
target using a provided demo scenedesk
.
(Optional) set a symlink to the downloaded data:
mkdir data_dir # do it in this nerfmm repo
cd data_dir
ln -s /path/to/downloaded/data ./nerfmm_release_data
cd ..
this can simplify the above training commands, for example:
python tasks/nerfmm/train.py
Call eval.py
in nerfmm
target:
python tasks/nerfmm/eval.py \
--base_dir='path/to/nerfmm_release/data' \
--scene_name='LLFF/fern' \
--ckpt_dir='path/to/a/dir/contains/nerfmm/ckpts'
This file can be used to evaluate a checkpoint trained with nerfmm
or refine_nerfmm
target. For some scenes, you might need to tweak with --opt_eval_lr
option to get the best results. Common values for opt_eval_lr
are 0.01 / 0.005 / 0.001 / 0.0005 / 0.0001. The default value is 0.001. Overall, it finds validation poses that can produce highest PSNR on validation set while freezing NeRF and focal lengths. We do this because the learned camera pose space is different from the COLMAP estimated camera pose space.
Call spiral.py
in each target. The spiral.py
in nerfmm
is compatible with refine_nerfmm
target:
python spiral.py \
--base_dir='path/to/nerfmm_release/data' \
--scene_name='LLFF/fern' \
--ckpt_dir='path/to/a/dir/contains/nerfmm/ckpts'
Call vis_learned_poses.py
in each target. The vis_learned_poses.py
in nerfmm
is compatible with refine_nerfmm
target:
python vis_learned_poses.py \
--base_dir='path/to/nerfmm_release/data' \
--scene_name='LLFF/fern' \
--ckpt_dir='path/to/a/dir/contains/nerfmm/ckpts'
Shangzhe Wu is supported by Facebook Research. Weidi Xie is supported by Visual AI (EP/T028572/1).
The authors would like to thank Tim Yuqing Tang for insightful discussions and proofreading.
During our NeRF implementation, we referenced several open sourced NeRF implementations, and we thank their contributions. Specifically, we referenced functions from nerf and nerf-pytorch, and borrowed/modified code from nerfplusplus and nerf_pl. We especially appreciate the detailed code comments and git issue answers in nerf_pl.
@article{wang2021nerfmm,
title={Ne{RF}$--$: Neural Radiance Fields Without Known Camera Parameters},
author={Zirui Wang and Shangzhe Wu and Weidi Xie and Min Chen and Victor Adrian Prisacariu},
journal={arXiv preprint arXiv:2102.07064},
year={2021}
}