Let It Flow: Simultaneous Optimization of 3D Flow and Object Clustering

Patrik Vacek¹    David Hurych²    Karel Zimmermann¹    Tomas Svoboda<sup>1

1</sup>CTU Deparment of Cybernetics    ²Valeo AI   

✨ Performance

• 🚀 Scalability: Let It Flow does not require training or supervision as an optimization-based method and works on out-of-distribution data.
• 🌈 Real Scene Flow: Let It Flow performs consistently accross real world benchmark datasets and not only on StereoKITTI. Improvements are observable even on new dynamic/object class error metrics.
• ⚖️ Generalizability: Let It Flow does not have to adjust the parameter config for accurate results on the tested datasets.

Mechanism

Installation

Tested with Python==3.10, PyTorch3d, PyTorch Scatter See install.sh for installation of libraries or run it directly:

conda create -n let-it-flow python=3.10
conda activate let-it-flow

conda install pytorch=1.13.0 torchvision pytorch-cuda=11.6 -c pytorch -c nvidia
conda install -c fvcore -c iopath -c conda-forge fvcore iopath

git clone https://github.com/facebookresearch/pytorch3d.git
cd pytorch3d && python3 -m pip install -e .
cd ..

conda install pytorch-scatter -c pyg

Dataset

Download and unpack the Argoverse2 preprocessed sequence samples into the dataset/argoverse2/ path.

tar -xvf argoverse2.tgz datasets/argoverse2/

Set up the config.yaml to include path to the argoverse2 folder and path to store the results.

Inference

To infer the model on Argoverse2 dataset sequences, please run the following:

python run_optimization.py

The script will store the outputs (and inputs with Ground truth) into the store_path for metric calculation.

Generation of EPE metrics

After the model inference (takes about 2 hours on GPU), generate metrics of Let It Flow (lif) by running:

python compute_metrics.py lif

Result will print the End-Point-Error per class annotated in Argoverse2 and also overall ThreewayEPE.

Citation

If you find our work helpful, please consider citing our paper:

@misc{vacek2024lif,
      title={Let It Flow: Simultaneous Optimization of 3D Flow and Object Clustering}, 
      author={Patrik Vacek and David Hurych and Tomáš Svoboda and Karel Zimmermann},
      year={2024},
      eprint={2404.08363},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

License

This work is under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Acknowledgement

This work is developed based on the open-source repositories and baselines: KISS-ICP, NSFP, MBNSFP, Chodosh

❤️ We thank to the contributors from the above open-source repositories.