This project is aimed for 3d reconstruction(sparse up-to scale cloud + camera positions) using camera images. The focus of the whole project is on symbiosis between learning-based approaches and time-proven classic algorithms. Inspired by existing open-source approaches: Colmap, AliceVision, OpenMVG.
At the moment there is cpp a wrapper over SuperPoint(https://github.com/magicleap/SuperPointPretrainedNetwork), also a cpp wrapper over SuperGlue matcher(https://github.com/magicleap/SuperGluePretrainedNetwork)
Fountain dataset. Taken from kaggle's image matching 2023 competition.
There is also corresponding cloud_fountain.ply file. It might be opened via Meshlab(renders colored pointcloud) or CloudCompare(renders uncolored pointcloud) or any other cloud viewer. Camera poses are visualized as green points.
- 2 feature extractors: SuperPoint, ORB + BaseClass for possible extension
- 2 feature matchers: SuperGlue, Flann + BaseClass for possible extension
- Feature filtering(based on epipolar geometry)
- Initial img pair choice and initial triangulation
- Separate class for end2end reconstruction
- Img matcher(which for now assumes all images are matched)
- 3d visualization of reconstructed cloud(up to scale) + camera positions
- incremental reconstruction via pnp + BA
- colored pointcloud, saved in *.ply format
- use openmp/tbb to make it faster
- code refactoring + add comments
- Add saving of intermediate steps(feature detection, image matching, feature matching) to avoid their computation in case of changes only in sfm part
- image matcher(apply some image retrieval, FAISS, metric learning)
- use monocular depth estimating networks(for example https://github.com/isl-org/MiDaS) to obtain real scale and dense reconstruction
It is currently added for feature extraction + feature matching + matches saving. Here are some results on sift + flann for 100 images(duration in seconds):
- Openmp 4 threads + collapse(on double for): (7 + 76 + 171)
- Without openmp : (7 + 180 + 406)
- install OpenCV 4.2+ https://docs.opencv.org/4.x/d7/d9f/tutorial_linux_install.html
- install libTorch https://pytorch.org/, choose Language: C++/Java, only CPU platform tested at the moment
- install PCL https://pointclouds.org/downloads/
- clone this repo and change path to libtorch in CMakeLists.txt "CMAKE_PREFIX_PATH"
mkdir build && cd build
cmake .. && make
- run
./reconstruct