Skip to content

dfki-av/DeepLiDARFlow

Repository files navigation

DeepLiDARFlow

This is the source code of DeepLiDARFlow for dense scene flow estimation using monocular camera and sparse LiDAR.

Requirements

The model is trained and tested on:

  • Windows 10 Pro and/or Ubuntu 18.04
  • Python 3.7.8
    • gast==0.2.2 (to resolve incompatibilities between python version and tensorflow)
    • imageio==2.5.0
    • matplotlib==3.2.1
    • numpy==1.16.4
    • tensorflow-gpu==1.14
    • tqdm==4.31.1
  • All required libraries can be installed by running pip install -r requirements.txt.

Data Sets

  • The original form of FlyingThings3D (FT3D) is used for pre-training and KITTI 2015 scene-flow is used for fine-tuning. All mappings have been done.
  • For KITTI data set, downloading the de-warped disparities 1 to the view at time t+1 is required by running sh download_preprocessed_kitti.sh. Then, the folder should be placed in the KITTI data set (training) folder.

Training and Fine-tuning

  • Running sh run_train.sh will train the model from scratch. As a prerequisite, the correct path of FlyingThings3D data set should be provided for --data_path flag.
  • Running sh run_finetune.sh will fine-tune the model. As a prerequisites, the correct path of KITTI data set should be provided for --data_path flag and the pre-trained model path on FlyingThings3D should be provided for --pretrained_model flag.
  • During the training, best weights will be stored in the model directory and make a JSON file named best_checkpoints. If the best_checkpoints is available and the flag--best_checkpoint is True, the training will continue with the best checkpoint.

Maximum Epochs and Learning Rate Strategy

  • For pre-training on FlyingThings3D, the maximum epochs number is defined with 600 epochs. The initial learning rate is 0.0001 and it's reduced to 0.00001 in case of over-fitting.
  • For fine-tuning on KITTI, the model needs to be fine-tuned with 200 epochs. First 100 epochs are trained using 0.0001 learning rate, then it's reduced to 0.00001 for the rest epochs.
  • If the training or fine-tuning will be continued after changing learning rates or maximum epochs number, the --best_checkpoint flag should be activated to call the best weights for continuing the training.

Evaluation and Inference

  • Running sh run_evaluation.sh will evaluate the whole frames in the test split of FlyingThings3D and the mapped frames of KITTI in 'KITT_test.txt'.
    • The number of LiDAR samples can be estimated by setting the --samples flag [default: 5000].
    • The data set type (either 'KITTI' or 'FT3D' should be provided for the flag --dataset.
    • The path of the data set should be given for --data_path.
    • The model path should be given for --model_path [default: 'model/DeepLiDARFlow'].
      The software can take the suitable model based on data set type --dataset.
  • Running sh run_inference.sh can evaluate the model on two selected frames inside 'images' folder (one is KITTI example and another is FT3D).
    • 5000 of LiDAR points are considered with these frames.
    • The example flag --ex can be specified either by '1' for KITTI frame or '2' for FT3D frame.
    • The model path should be given for --model_path [default: 'model/DeepLiDARFlow'] and the software can take the suitable model based on image resolution.

Model Weights

  • The execution of download_models.sh can download the weights for both data sets (FT3D/KITTI) inside the 'model' folder:
	model/DeepLiDARFlow-FT3D
	model/DeepLiDARFlow-KITTI

Citation

If you find the code or the paper useful, consider citing us:

@inproceedings{DeepLiDARFlow2020,
  title={{DeepLiDARFlow: A Deep Learning Architecture For Scene Flow Estimation Using Monocular Camera and Sparse LiDAR}},
  author={Rishav and Battrawy, Ramy and Schuster, Ren{\'e} and Wasenm{\"u}ller, Oliver and Stricker, Didier},
  booktitle={IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year={2020},
}

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published