Skip to content

Offical pytorch implementation of "HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network", CVPR 2022.

Notifications You must be signed in to change notification settings

namepllet/HandOccNet

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

26 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network

Introduction

This repository is the offical Pytorch implementation of HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network (CVPR 2022). Below is the overall pipeline of HandOccNet. overall pipeline

Quick demo

  • Install PyTorch and Python >= 3.7.4 and run sh requirements.sh.
  • Download snapshot_demo.pth.tar from here and place at demo folder.
  • Prepare input.jpg at demo folder.
  • Download MANO_RIGHT.pkl from here and place at common/utils/manopth/mano/models.
  • Go to demo folder and edit bbox in here.
  • Run python demo.py --gpu 0 if you want to run on gpu 0.
  • You can see hand_bbox.png, hand_image.png, and output.obj.
  • Run python demo_fitting.py --gpu 0 --depth 0.5 if you want to get the hand mesh's translation from the camera. The depth argument is initialization for the optimization.
  • You can see fitting_input_3d_mesh.json that contains the translation and MANO parameters, fitting_input_3dmesh.obj, fitting_input_2d_prediction.png, and fitting_input_projection.png.

Directory

Root

The ${ROOT} is described as below.

${ROOT}  
|-- data  
|-- demo
|-- common  
|-- main  
|-- output  
  • data contains data loading codes and soft links to images and annotations directories.
  • demo contains demo codes.
  • common contains kernel codes for HandOccNet.
  • main contains high-level codes for training or testing the network.
  • output contains log, trained models, visualized outputs, and test result.

Data

You need to follow directory structure of the data as below.

${ROOT}  
|-- data  
|   |-- HO3D
|   |   |-- data
|   |   |   |-- train
|   |   |   |   |-- ABF10
|   |   |   |   |-- ......
|   |   |   |-- evaluation
|   |   |   |-- annotations
|   |   |   |   |-- HO3D_train_data.json
|   |   |   |   |-- HO3D_evaluation_data.json
|   |-- DEX_YCB
|   |   |-- data
|   |   |   |-- 20200709-subject-01
|   |   |   |-- ......
|   |   |   |-- annotations
|   |   |   |   |--DEX_YCB_s0_train_data.json
|   |   |   |   |--DEX_YCB_s0_test_data.json

Pytorch MANO layer

  • For the MANO layer, I used manopth. The repo is already included in common/utils/manopth.
  • Download MANO_RIGHT.pkl from here and place at common/utils/manopth/mano/models.

Output

You need to follow the directory structure of the output folder as below.

${ROOT}  
|-- output  
|   |-- log  
|   |-- model_dump  
|   |-- result  
|   |-- vis  
  • Creating output folder as soft link form is recommended instead of folder form because it would take large storage capacity.
  • log folder contains training log file.
  • model_dump folder contains saved checkpoints for each epoch.
  • result folder contains final estimation files generated in the testing stage.
  • vis folder contains visualized results.

Running HandOccNet

Start

  • Install PyTorch and Python >= 3.7.4 and run sh requirements.sh.
  • In the main/config.py, you can change settings of the model including dataset to use and input size and so on.

Train

In the main folder, set trainset in config.py (as 'HO3D' or 'DEX_YCB') and run

python train.py --gpu 0-3

to train HandOccNet on the GPU 0,1,2,3. --gpu 0,1,2,3 can be used instead of --gpu 0-3.

Test

Place trained model at the output/model_dump/.

In the main folder, set testset in config.py (as 'HO3D' or 'DEX_YCB') and run

python test.py --gpu 0-3 --test_epoch {test epoch}  

to test HandOccNet on the GPU 0,1,2,3 with {test epoch}th epoch trained model. --gpu 0,1,2,3 can be used instead of --gpu 0-3.

  • For the HO3D dataset, pred{test epoch}.zip will be generated in output/result folder. You can upload it to the codalab challenge and see the results.
  • Our trained model can be downloaded from here

Results

Here I report the performance of the HandOccNet.

Reference

@InProceedings{Park_2022_CVPR_HandOccNet,  
author = {Park, JoonKyu and Oh, Yeonguk and Moon, Gyeongsik and Choi, Hongsuk and Lee, Kyoung Mu},  
title = {HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network},  
booktitle = {Conference on Computer Vision and Pattern Recognition (CVPR)},  
year = {2022}  
}  

Acknowledgements

For this project, we relied on research codes from:

About

Offical pytorch implementation of "HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network", CVPR 2022.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published