The human hand is the main medium through which we interact with our surroundings. Hence, its digitization is of uttermost importance, with direct applications in VR/AR, gaming, and media production amongst other areas. While there are several works modeling the geometry of hands, little attention has been paid to capturing photo-realistic appearance. Moreover, for applications in extended reality and gaming, real-time rendering is critical. We present the first neural- implicit approach to photo-realistically render hands in real-time. This is a challenging problem as hands are textured and undergo strong articulations with pose-dependent effects. However, we show that this aim is achievable through our carefully designed method. This includes training on a low- resolution rendering of a neural radiance field, together with a 3D-consistent super-resolution module and mesh-guided sampling and space canonicaliza- tion. We demonstrate a novel application of perceptual loss on the image space, which is critical for learning details accurately. We also show a live demo where we photo-realistically render the human hand in real-time for the first time, while also modeling pose- and view-dependent appearance effects. We ablate all our design choices and show that they optimize for rendering speed and quality.
Clone the repository and install the dependencies:
git clone https://github.com/amundra15/livehand.git
cd livehand
conda create --name livehand python=3.10
conda activate livehand
pip install -r requirements.txt
pip install --editable .
The codebase supports heirarchical sampling (by setting N_importance > 0 in the config file). Install torchsearchsorted to support this. You DO NOT need this installation to run the default livehand method.
We provide our pre-processed data for the test/Capture1
split of the InterHand2.6M dataset here. After downloading it, you can proceed to the Usage section.
To prepare the dataset for other splits, follow the instructions below.
Download the InterHand2.6M dataset and run the following commands:
cd preprocessing/InterHands2.6M
python prepare_dataset.py --input_dir path/to/dataset --output_dir path/to/output/dir
cd preprocessing/InterHands2.6M
# Set the input and output directories in run_prepare_dataset_cluster.sh
sbatch run_prepare_dataset_cluster.sh
Note: Currently, the script has been configured to process the test/Capture0 sequence of the InterHands2.6M dataset. You can modify the script to process other sequences as well (change the variables 'mode' and 'allowed_capture_ids').
Next, run the following command to generate the poses_bounds.npy files:
# Set the metashape license path and the preprocessing output_dir from above in run_metashape.sh
./run_metashape.sh
You can download the pretrained model for test/Capture0 sequence from here. Set the 'exp_folder' variable in scripts/render_experiments.sh to the path/containing/trained/model/checkpoint (NOTE: it supports pattern matching to specify multiple experiments). Then run the following command to render the experiments:
./scripts/render_experiments.sh
Set the following variables in the config file (configs/test_capture0):
'expname': name of the experiment
'data_fol': path/to/training/data
'val_data_fol': path/to/validation/data
'basedir': path/to/save/checkpoints
Then run the following command to train a new model:
./scripts/test_capture0.sh
If you find this code useful for your research, please cite our paper:
@misc{mundra2023livehand,
title={LiveHand: Real-time and Photorealistic Neural Hand Rendering},
author={Akshay Mundra and Mallikarjun {B R} and Jiayi Wang and Marc Habermann and Christian Theobalt and Mohamed Elgharib},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2023},
}
This codebase adapts modules from NeRF-PyTorch, NeuralActor and EG3D. Please consider citing them as well.
@misc{lin2020nerfpytorch,
title={NeRF-pytorch},
author={Yen-Chen, Lin},
publisher = {GitHub},
journal = {GitHub repository},
howpublished={\url{https://github.com/yenchenlin/nerf-pytorch/}},
year={2020}
}
@article{liu2021neural,
title={Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control},
author={Lingjie Liu and Marc Habermann and Viktor Rudnev and Kripasindhu Sarkar and Jiatao Gu and Christian Theobalt},
year={2021},
journal = {ACM Trans. Graph.(ACM SIGGRAPH Asia)}
}
@inproceedings{Chan2021,
author = {Eric R. Chan and Connor Z. Lin and Matthew A. Chan and Koki Nagano and Boxiao Pan and Shalini De Mello and Orazio Gallo and Leonidas Guibas and Jonathan Tremblay and Sameh Khamis and Tero Karras and Gordon Wetzstein},
title = {Efficient Geometry-aware {3D} Generative Adversarial Networks},
booktitle = {arXiv},
year = {2021}
}
LiveHand is under CC-BY-NC 4.0 license. The license also applies to the pre-trained models.