Pengchong Hu · Zhizhong Han
Table of Contents
Please install all dependencies by following the instrutions here. You can use anaconda to finish the installation easily.
You can bulid a conda environment called df-prior
. Note that for linux users, you need to install libopenexr-dev
first before building the environment.
git clone https://github.com/MachinePerceptionLab/Attentive_DFPrior.git
cd Attentive_DFPrior
sudo apt-get install libopenexr-dev
conda env create -f environment.yaml
conda activate df-prior
Please download the Replica dataset generated by the authors of iMAP into ./Datasets/Replica
folder.
bash scripts/download_replica.sh # Released by authors of NICE-SLAM
Please follow the data downloading procedure on ScanNet website, and extract color/depth frames from the .sens
file using this code.
[Directory structure of ScanNet (click to expand)]
DATAROOT is ./Datasets
by default. If a sequence (sceneXXXX_XX
) is stored in other places, please change the input_folder
path in the config file or in the command line.
DATAROOT
└── scannet
└── scans
└── scene0000_00
└── frames
├── color
│ ├── 0.jpg
│ ├── 1.jpg
│ ├── ...
│ └── ...
├── depth
│ ├── 0.png
│ ├── 1.png
│ ├── ...
│ └── ...
├── intrinsic
└── pose
├── 0.txt
├── 1.txt
├── ...
└── ...
To run our code, you first need to generate the TSDF volume and corresponding bounds. We provide the generated TSDF volume and bounds for Replica and ScanNet: replica_tsdf_volume.tar, scannet_tsdf_volume.tar.
You also can generate the TSDF volume and corresponding bounds by using the following code:
CUDA_VISIVLE_DEVICES=0 python get_tsdf.py configs/Replica/room0.yaml --space 1 # For Replica
CUDA_VISIVLE_DEVICES=0 python get_tsdf.py configs/ScanNet/scene0050_00.yaml --space 10 # For ScanNet
You can run DF-Prior by using the following code:
CUDA_VISIVLE_DEVICES=0 python -W ignore run.py configs/Replica/room0.yaml # For Replica
CUDA_VISIVLE_DEVICES=0 python -W ignore run.py configs/ScanNet/scene0050.yaml # For ScanNet
The mesh for evaluation is saved as $OUTPUT_FOLDER/mesh/final_mesh_eval_rec.ply
, where the unseen regions are culled using all frames.
To evaluate the average trajectory error. Run the command below with the corresponding config file:
python src/tools/eval_ate.py configs/Replica/room0.yaml
To evaluate the reconstruction error in Replica, first download the ground truth Replica meshes where unseen region have been culled.
bash scripts/download_cull_replica_mesh.sh # Released by authors of NICE-SLAM
Then run the command below. The 2D metric requires rendering of 1000 depth images, which will take some time (~9 minutes). Use -2d
to enable 2D metric. Use -3d
to enable 3D metric.
# assign any output_folder and gt mesh you like, here is just an example
OUTPUT_FOLDER=output/Replica/room0
GT_MESH=cull_replica_mesh/room0.ply
python src/tools/eval_recon.py --rec_mesh $OUTPUT_FOLDER/mesh/final_mesh_eval_rec.ply --gt_mesh $GT_MESH -2d -3d
We also provide code to cull the mesh given camera poses. Here we take culling of ground truth mesh of Replica room0 as an example.
python src/tools/cull_mesh.py --input_mesh Datasets/Replica/room0_mesh.ply --traj Datasets/Replica/room0/traj.txt --output_mesh cull_replica_mesh/room0.ply
To evaluate the reconstruction error in ScanNet, first download the ground truth ScanNet meshes.zip into ./Datasets/scannet
folder. Then run the command below.
python src/tools/evaluate_scannet.py configs/ScanNet/scene0050.yaml
We also provide our reconstructed meshes in Replica and ScanNet for evaluation purposes: meshes.zip.
We adapt codes from some awesome repositories, including NICE-SLAM, NeuralRGBD, tsdf-fusion, manhattan-sdf, MonoSDF. Thanks for making the code available. We also thank Zihan Zhu of NICE-SLAM, for prompt responses to our inquiries regarding the details of their methods.
If you find our code or paper useful, please cite
@inproceedings{Hu2023LNI-ADFP,
title = {Learning Neural Implicit through Volume Rendering with Attentive Depth Fusion Priors},
author = {Hu, Pengchong and Han, Zhizhong},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2023}
}