Skip to content
This repository has been archived by the owner on Jul 2, 2019. It is now read-only.

Commit

Permalink
Merge pull request #10 from SpaceNetChallenge/spacenetV2
Browse files Browse the repository at this point in the history 
Spacenet V2
  • Loading branch information
@dlindenbaum
dlindenbaum authored Mar 21, 2017
2 parents 57b0127 + 78d1661 commit 21dbe6e
Show file tree
Hide file tree
Showing 23 changed files with 3,783 additions and 533 deletions.
3 changes: 3 additions & 0 deletions .idea/dictionaries/dlindenbaum.xml
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions .idea/vcs.xml
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

72 changes: 70 additions & 2 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# SpaceNet Utilities

This repository has two python packages, geoTools and evalTools. The geoTools packages is intended to assist in the preprocessing of [SpaceNet](https://aws.amazon.com/public-data-sets/spacenet/) satellite imagery data corpus to a format that is consumable by machine learning algorithms. The evalTools package is used to evaluate the effectiveness of object detection algorithms using ground truth.

This repository has two python packages, geoTools and evalTools. The geoTools packages is intended to assist in the preprocessing of [SpaceNet](https://spacenetchallenge.github.io/) satellite imagery data corpus hosted on [SpaceNet on AWS](https://aws.amazon.com/public-datasets/spacenet/) to a format that is consumable by machine learning algorithms. The evalTools package is used to evaluate the effectiveness of object detection algorithms using ground truth.
This is version 2.0 and has been updated with more capabilities to
## Download Instructions

Further download instructions for the [SpaceNet Dataset](https://github.com/SpaceNetChallenge/utilities/tree/master/content/download_instructions) can be found [here](https://github.com/SpaceNetChallenge/utilities/tree/master/content/download_instructions)
Expand Down Expand Up @@ -32,6 +32,74 @@ Hints:
* The images provided could contain anywhere from zero to multiple buildings.
* All proposed polygons should be legitimate (they should have an area, they should have points that at least make a triangle instead of a point or a line, etc).
* Use the [metric implementation code](https://github.com/SpaceNetChallenge/utilities/blob/master/python/evaluateScene.py) to self evaluate.
To run the metric you can use the following command.
```
python python/evaluateScene.py /path/to/SpaceNetTruthFile.csv \
/path/to/SpaceNetProposalFile.csv \
--resultsOutputFile /path/to/SpaceNetResults.csv
```

## Data Transformation Code

To make the Spacenet dataset easier to use we have created a tool createDataSpaceNet.py
This tool currently supports the creation of datasets with annotation to support 3 Formats
1. [PASCAL VOC2012](http://host.robots.ox.ac.uk/pascal/VOC/)
2. [Darknet](https://pjreddie.com/darknet/yolo/)
3. [Segmenation Boundaries Dataset (SBD)](http://home.bharathh.info/pubs/codes/SBD/download.html)

It will create the appropriate annotation files and a summary trainval.txt and test.txt in the outputDirectory

### Create an PASCAL VOC2012 Compatiable Dataset
The final product will have image dimensions of 420 pixels
```
python python/createDataSpaceNet.py /path/to/spacenet_sample/AOI_2_Vegas_Train/ \
RGB-PanSharpen \
--outputDirectory /path/to/spacenet_sample/annotations/ \
--annotationType PASCALVOC2012 \
--imgSizePix 400
```
### Changing the raster format
Some GIS Images have 16-bit pixel values which openCV has trouble with. createDataSpaceNet.py can convert the 16bit GeoTiff to an 8bit GeoTiff or 8bit JPEG

To create the 8bit GeoTiff
```
python python/createDataSpaceNet.py /path/to/spacenet_sample/AOI_2_Vegas_Train/ \
RGB-PanSharpen \
--outputDirectory /path/to/spacenet_sample/annotations/ \
--annotationType PASCALVOC2012 \
--convertTo8Bit \
--outputFileType GTiff \
--imgSizePix 400
```

To create the 8bit JPEG
```
python python/createDataSpaceNet.py /path/to/spacenet_sample/AOI_2_Vegas_Train/ \
RGB-PanSharpen \
--outputDirectory /path/to/spacenet_sample/annotations/ \
--annotationType PASCALVOC2012 \
--convertTo8Bit \
--outputFileType JPEG \
--imgSizePix 400
```

For more Features
```
python python/createDataSpaceNet.py -h
```



## Use our Docker Container
We have created two Docker files at /docker/standalone/cpu and /docker/standalone/gpu
These Dockerfiles will build a docker container with all packages neccessary to run the package

More documenation to follow


## Dependencies
All dependencies can be found in [requirements.txt](./python/requirements.txt)
Expand Down
107 changes: 79 additions & 28 deletions content/download_instructions/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,67 +1,118 @@
# SpaceNet Utilities
## Hosting
[SpaceNet](https://aws.amazon.com/public-datasets/spacenet/) is a corpus of commercial satellite imagery and labeled training data to use for machine learning research. The dataset is currently hosted as an [Amazon Web Services (AWS) Public Dataset](https://aws.amazon.com/public-datasets/).

Instruction for the download of [SpaceNet](https://aws.amazon.com/public-data-sets/spacenet/) satellite imagery data corpus to a format that is consumable by machine learning algorithms.


## Catalog
1. Area of Interest 1 (AOI 1) - Location: Rio de Janeiro. 50cm imagery collected from DigitalGlobe’s [WorldView-2 satellite](http://satimagingcorp.s3.amazonaws.com/site/pdf/WorldView-2_datasheet.pdf). The dataset includes building footprints and 8-band multispectral data.
2. Area of Interest 2 (AOI 2) - Location: Vegas. 30cm imagery collected from DigitalGlobe’s [WorldView-3 satellite](https://www.spaceimagingme.com/downloads/sensors/datasheets/DG_WorldView3_DS_2014.pdf). The dataset includes building footprints and 8-band multispectral data.
3. Area of Interest 3 (AOI 3) - Location: Paris. 30cm imagery collected from DigitalGlobe’s [WorldView-3 satellite](https://www.spaceimagingme.com/downloads/sensors/datasheets/DG_WorldView3_DS_2014.pdf). The dataset includes building footprints and 8-band multispectral data.
4. Area of Interest 4 (AOI 4) - Location: Shanghai. 30cm imagery collected from DigitalGlobe’s [WorldView-3 satellite](https://www.spaceimagingme.com/downloads/sensors/datasheets/DG_WorldView3_DS_2014.pdf). The dataset includes building footprints and 8-band multispectral data.
5. Area of Interest 5 (AOI 5) - Location: Khartoum. 30cm imagery collected from DigitalGlobe’s [WorldView-3 satellite](https://www.spaceimagingme.com/downloads/sensors/datasheets/DG_WorldView3_DS_2014.pdf). The dataset includes building footprints and 8-band multispectral data.
6. Point of Interest (POI) Dataset- Location: Rio de Janeiro. The dataset includes POIs.

## Dependencies
Required that AWS CLI is installed and that an active AWS account with credit card is associated with the aws cli
The [AWS Command Line Interface (CLI)](https://aws.amazon.com/cli/) must be installed with an active AWS account. Configure the AWS CLI using 'aws configure'

Configure the AWS CLI using aws configure

SpaceNet AWS Structure
## SpaceNet Simple Storage Service (S3) Directory Structure (AOI 1)
```
s3://spacenet-dataset/
-- AOI_1_Rio
|-- processedData
| -- processedBuildingLabels.tar.gz # Compressed 3band and 8band 200m x 200m tiles with associated building foot print labels
# This dataset is the Training Dataset for the first [Top Coder Competition](https://community.topcoder.com/longcontest/?module=ViewProblemStatement&rd=16835&pm=14439)
| -- processedBuildingLabels.tar.gz # Compressed 3band and 8band 200m x 200m tiles with associated building foot print labels # This dataset is the Training Dataset for the first Top Coder Competition
`-- srcData
|-- rasterData
| |-- 3-Band.tar.gz # 3band (RGB) Raster Mosaic for Rio De Jenairo area (2784 sq KM) collected by [WorldView-2](http://satimagingcorp.s3.amazonaws.com/site/pdf/WorldView-2_datasheet.pdf)
| -- 8-Band.tar.gz # 8band Raster Mosaic for Rio De Jenairo area (2784 sq KM) collected by [WorldView-2](http://satimagingcorp.s3.amazonaws.com/site/pdf/WorldView-2_datasheet.pdf)
| |-- 3-Band.tar.gz # 3band (RGB) Raster Mosaic for Rio De Jenairo area (2784 sq KM) collected by WorldView-2
| -- 8-Band.tar.gz # 8band Raster Mosaic for Rio De Jenairo area (2784 sq KM) collected by WorldView-2
-- vectorData
|-- Rio_BuildingLabels.tar.gz # Source Dataset that contains Building the building foot prints traced from the Mosaic
|-- Rio_HGIS_Metro.gdb.tar.gz # Source Point of Interest Dataset in GeoDatabase Format. Best if Used with ESRI
-- Rio_HGIS_Metro_extract.tar # Source Point of Interest Dataset in GeoJSON with associated .jpg. Easy to Use without ESRI toolset
-- AOI_1_Rio
|-- processedData
| -- processedBuildingLabels.tar.gz # Compressed 3band and 8band 200m x 200m tiles with associated building foot print labels # This dataset is the Training Dataset for the first Top Coder Competition
`-- srcData
|-- rasterData
| |-- 3-Band.tar.gz # 3band (RGB) Raster Mosaic for Rio De Jenairo area (2784 sq KM) collected by WorldView-2
| -- 8-Band.tar.gz # 8band Raster Mosaic for Rio De Jenairo area (2784 sq KM) collected by WorldView-2
-- vectorData
|-- Rio_BuildingLabels.tar.gz # Source Dataset that contains Building the building foot prints traced from the Mosaic
|-- Rio_HGIS_Metro.gdb.tar.gz # Source Point of Interest Dataset in GeoDatabase Format. Best if Used with ESRI
-- Rio_HGIS_Metro_extract.tar # Source Point of Interest Dataset in GeoJSON with associated .jpg. Easy to Use without ESRI toolset
-- AOI_1_Rio
|-- processedData
| -- processedBuildingLabels.tar.gz # Compressed 3band and 8band 200m x 200m tiles with associated building foot print labels # This dataset is the Training Dataset for the first Top Coder Competition
`-- srcData
|-- rasterData
| |-- 3-Band.tar.gz # 3band (RGB) Raster Mosaic for Rio De Jenairo area (2784 sq KM) collected by WorldView-2
| -- 8-Band.tar.gz # 8band Raster Mosaic for Rio De Jenairo area (2784 sq KM) collected by WorldView-2
-- vectorData
|-- Rio_BuildingLabels.tar.gz # Source Dataset that contains Building the building foot prints traced from the Mosaic
|-- Rio_HGIS_Metro.gdb.tar.gz # Source Point of Interest Dataset in GeoDatabase Format. Best if Used with ESRI
-- Rio_HGIS_Metro_extract.tar # Source Point of Interest Dataset in GeoJSON with associated .jpg. Easy to Use without ESRI toolset
```

#To download the Imagery

##To download processed 200mx200m Tiles with associated building foot prints for building foot print extraction tests do the following
## SpaceNet Simple Storage Service (S3) Directory Structure (AOI 2-5)
```
## Warning this file is 3.4 GB
aws s3api get-object --bucket spacenet-dataset --key AOI_1_Rio/processedData/processedBuildingLabels.tar.gz --request-payer requester processedBuildingLabels.tar.gz
├── AOI_[Num]_[City]_Train
│ ├── geojson
│ │ └── buildings # Contains GeoJson labels of buildings for each tile
│ ├── MUL # Contains Tiles of 8-Band Multi-Spectral raster data from WorldView-3
│ ├── MUL-PanSharpen # Contains Tiles of 8-Band Multi-Spectral raster data pansharpened to 0.3m
│ ├── PAN # Contains Tiles of Panchromatic raster data from Worldview-3
│ ├── RGB-PanSharpen # Contains Tiles of RGB raster data from Worldview-3
│ └── summaryData # Contains CSV with pixel based labels for each building in the Tile Set.
```

## Download instructions

##To download the Source Imagery Mosaic
### AOI 1 - Rio de Janeiro
To download processed 200mx200m tiles of AOI 1 (3.4 GB) with associated building footprints do the following:
```
aws s3api get-object --bucket spacenet-dataset --key AOI_1_Rio/processedData/processedBuildingLabels.tar.gz --request-payer requester processedBuildingLabels.tar.gz
```
To download the Source Imagery Mosaic (3-band = 2.3 GB and 8-band = 6.5 GB):
```
## Warning this file is 2.3 GB
aws s3api get-object --bucket spacenet-dataset --key AOI_1_Rio/srcData/rasterData/3-Band.tar.gz --request-payer requester 3-Band.tar.gz
## Warning this file is 6.5 GB
aws s3api get-object --bucket spacenet-dataset --key AOI_1_Rio/srcData/rasterData/8-Band.tar.gz --request-payer requester 8-Band.tar.gz
```

##To download the Source Vector Data for the Building Extraction Challenge
To download the Source Vector Data (0.18 GB):
```
## Warning this file is 0.18 GB
aws s3api get-object --bucket spacenet-dataset --key AOI_1_Rio/srcData/vectorData/Rio_BuildingLabels.tar.gz --request-payer requester Rio_BuildingLabels.tar.gz
```

### AOI 2 - Vegas
To download processed 200mx200m tiles of AOI 2 (23 GB) with associated building footprints do the following:
```
aws s3api get-object --bucket spacenet-dataset --key AOI_2_Vegas/AOI_2_Vegas_Train.tar.gz --request-payer requester AOI_2_Vegas_Train.tar.gz
```

##To download the Rio Point of Interest Dataset in ESRI GeoDatabase Form
### AOI 3 - Paris
To download processed 200mx200m tiles of AOI 3 (5 GB) with associated building footprints do the following:
```
## Warning this file is 5 GB
aws s3api get-object --bucket spacenet-dataset --key AOI_3_Paris/AOI_3_Paris_Train.tar.gz --request-payer requester AOI_3_Paris_Train.tar.gz
```
## Warning this file is 31 GB
aws s3api get-object --bucket spacenet-dataset --key AOI_1_Rio/srcData/vectorData/Rio_HGIS_Metro.gdb.tar.gz --request-payer requester Rio_HGIS_Metro.gdb.tar.gz

### AOI 4 - Shanghai
To download processed 200mx200m tiles of AOI 4 (23 GB) with associated building footprints do the following:
```
aws s3api get-object --bucket spacenet-dataset --key AOI_4_Shanghai/AOI_4_Shanghai_Train.tar.gz --request-payer requester AOI_4_Shanghai_Train.tar.gz
```

##To download the Rio Point of Interest Dataset Extracted into GeoJSONs with associated .jpg
### AOI 5 - Khartoum
To download processed 200mx200m tiles of AOI 5 (4 GB) with associated building footprints do the following:
```
aws s3api get-object --bucket spacenet-dataset --key AOI_5_Khartoum/AOI_5_Khartoum_Train.tar.gz --request-payer requester AOI_5_Khartoum_Train.tar.gz
```

### Point of Interest Dataset in ESRI GeoDatabase Form (31 GB)
```
aws s3api get-object --bucket spacenet-dataset --key AOI_1_Rio/srcData/vectorData/Rio_HGIS_Metro.gdb.tar.gz --request-payer requester Rio_HGIS_Metro.gdb.tar.gz
```
## Warning this file is 29 GB
aws s3api get-object --bucket spacenet-dataset --key AOI_1_Rio/srcData/vectorData/Rio_HGIS_Metro_extract.tar --request-payer requester Rio_HGIS_Metro_extract.tar

### Point of Interest Dataset Extracted into GeoJSONs with associated .jpg (29 GB)
```
aws s3api get-object --bucket spacenet-dataset --key AOI_1_Rio/srcData/vectorData/Rio_HGIS_Metro_extract.tar --request-payer requester Rio_HGIS_Metro_extract.tar
```


Expand Down
93 changes: 93 additions & 0 deletions docker/Dockerfile
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
LABEL maintainer dlindenbaum


## List of Python packages needed
###dataTools
#import numpy as np
#from osgeo import ogr, gdal, osr
#import cv2
###evalTools
###geoTools
#from osgeo import gdal, osr, ogr
# import rtree
#from osgeo import gdal, osr, ogr, gdalnumeric
#from PIL import Image

## Install External Libraries for Pillow
#apt-get install
#libjpeg
#zlib
#libtiff
#libfreetype
#littlecms
#libwebp
#openjpeg
#
### install Python openCV
#libopencv-dev \
#python-opencv \
#python-numpy \
#python-pip \
#python-setuptools \
#gdal-bin \
#python-gdal

## Install General Requirements
RUN apt-get update && apt-get install -y --no-install-recommends \
apt-utils \
build-essential \
cmake \
git \
wget \
vim \
python-dev \
python-pip \
python-setuptools
RUN pip install --upgrade pip

## Install Basics for Python
RUN apt-get update && apt-get install -y --no-install-recommends \
python-numpy \
python-scipy

## Install Essentials for Pillow
RUN apt-get update && apt-get install -y --no-install-recommends \
libjpeg-dev \
zlib1g \
libtiff5-dev \
libfreetype6-dev \
libwebp-dev \
libopenjpeg-dev

RUN pip install Pillow

## Install GDAL Requirments
RUN apt-get update && apt-get install -y --no-install-recommends \
gdal-bin \
python-gdal

## Instal OpenCV Requirements
RUN apt-get update && apt-get install -y --no-install-recommends \
libopencv-dev \
python-opencv

## Install RTRee
RUN apt-get update && apt-get install -y --no-install-recommends \
libspatialindex-dev

RUN pip install rtree

ENV GIT_BASE=/opt/
WORKDIR $GIT_BASE

# Download spaceNetUtilities
# FIXME: use ARG instead of ENV once DockerHub supports this
ENV CLONE_TAG=spacenetV2
RUN git clone -b ${CLONE_TAG} --depth 1 https://github.com/SpaceNetChallenge/utilities.git

ENV PYUTILS_ROOT $GIT_BASE/utilities/python
ENV PYTHONPATH $PYUTILS_ROOT:$PYTHONPATH

WORKDIR /workspace


Loading

0 comments on commit 21dbe6e

Please sign in to comment.