An evaluation of Deep Learning based stereo dense matching dataset shift from aerial images and a large scale stereo dataset

This Github will provide the detail information of our paper An evaluation of Deep Learning based stereo dense matching dataset shift from aerial images and a large scale stereo dataset.

Because the length limit of International Journal of Applied Earth Observation and Geoinformation is 8000 words, the original paper can be found on Arxiv, if you have any questions about this long version paper, please contact me.

This is the Github repository for the stereo dense matching benchmark for AI4GEO project.

In order to discuss the transferability of deep learning methods on aerial dataset, we produce 6 aerial dataset covers 4 different area.

All the data and model can be found On Zenodo

History

This work is an extension of our previous work, and the old version dataset is already published. In the ISPRS Conress 2022 in Nice, we presented an extension work as a poster, and the slide and the poster is provided.

Introduction

For stereo dense matching, there are many famous benchmark dataset in Robust Vision, for example, KITTI stereo and middlebury stereo. With the development of machine learning, especially deep learning, these methods usually need a lot of training data(or ground truth). For photogrammetry community, as far as we know, it is not easy to find these training data. We will publish our data as ground truth. The data is produced from original image and LiDAR dataset. To be noticed, the image and LiDAR should be well-registered.

Global information of dataset

For each dataset, the global information of the dataset is listed follow:

Dataset	Color	GSD(cm)	LiDAR($pt/m^2$)	Origin orientation	ICP refined	Outlier remove	Difficulty
ISPRS-Vaihingen	IR-R-G	8	6.7	✔	`x`	`x`	++
EuroSDR-Vaihingen	R-G-B	20	6.7	✔	`x`	`x`	++
Toulouse-UMBRA	R-G-B	12.5	2-4	`x`	☑	☑	++++
Toulouse-Métropole	R-G-B	5	8	✔	`x`	`x`	+
Enschede	R-G-B	10	10	`x`	☑	☑	+++
DublinCity	R-G-B	3.4	250-348	`x`	☑	`x`	++

In the table, the origin orientation accuracy influence the data accuracy, in order to improve the quality of the dataset, an ICP based Image-LiDAR is proposed to refine the orientation.

Dataset structure

The training and evaluation dataset is also provided, the structure of the folder is same with the old version.

Because the whole dataset is too large, so only the used in the paper is uploaded. The data will be hosted by Zenodo, because the limitation is 50GB, so the dataset used be in the paper and the pre-trained models are available on Zenodo. (Don't upload data for the community "Photogrammetry - Geological Remote Sensing", the dataset must be reviewed, and it will takes too long time to wait, and I give up.)

ISPRS-Vaihingen

All the training and testing data can be found on Google Drive, this is a newer version compare to the old version, the origin image and LiDAR used are same with old version. To visualize the data in GoogleEarth, we need know the projection system of the data, for Vaihingen data, the system is WGS 84 / UTM zone 32N.


Origin ISPRS-Vaihingen coverage

An example is show here :


Example for ISPRS-Vaihingen

EuroSDR-Vaihingen

This dataset is collected nearly same time with ISPRS-Vaihingen, the difference is that the resolution, the dataset can be found here, because the LiDAR is from ISPRS-Vaihingen, so only a small part of the data is used.


Origin EuroSDR-Vaihingen coverage

An example is show here :


Example for EuroSDR-Vaihingen

Toulouse-UMBRA

This dataset is collected by IGN(French map agency) in 2012, and the camera is produced by IGN, the origin image is 16bit, this dataset is for remote sensing use, to make the training data, we use auto just. And in the experiment, we found that the image is quite different from other dataset.


Origin Toulouse-UMBRA coverage

An example is show here :


Example for Toulouse-UMBRA

Toulouse-Métropole

This dataset is collect by AI4GEO in 2019, the camera is UltraCam Osprey Prime M3, and the LiDAR is ALS70. The origin dataset is too large, only the area same with the Toulous-UMBRA is used in the paper for produce the dataset.


Origin Toulouse-Métropole coverage

An example is show here :


Example for Toulouse-Métropole

Enschede

This dataset is a dataset collected from ITC Faculty Geo-Information Science and Earth Observation in 2011, the LiDAR is AHN2 in 2012. The origin device has 5 cameras, only the nadir camera is used in the experiment.

To visualize the data in GoogleEarth, we need know the projection system of the data, for Enschede data, the system is Amersfoort.


Origin Enschede coverage

An example is show here :


Example for Enschede

DublinCity

DublinCity is an open dataset, the original aerial and LiDAR point cloud can be downloaded, the origin dataset is very large.

You can find the training and testing dataset from another paper. To save the disk, we do not upload this time, more information can be found on Github also.

An example is show here :


Example for DublinCity

Method

In the paper, we evaluate the state of the art methods of deep learning on stereo dense matching before 2020.

MicMac

MicMac can is a open source code, the code can be found from Github. In the experiment, the command line is :

mm3d MM1P left.tif right.tif NONE DefCor=0.2 HasSBG=false HasVeg=true

SGM(GPU)

This method is revised during the experiment, because the origin disparity range is too small, i.e 128, in our experiment, 256 is used. The code can be found in folder.

GraphCuts

This origin code can be found from Github, to make the code run in Linux, a new version can be found here.

CBMV

The code can be found from Github.

MC-CNN

The code can be found from Github.