Clean Data Run Statistics Harvey

Train/val/test 2048 chips

TODO: There are fewer chips used than in geojson. Need to clean geojson

• train: 722 (580 train + 142 val)
• test: 153
• folder name: /home/ubuntu/anyan/harvey_data/
  • harvey_train_train_bigtiff_v3
  • harvey_train_val_bigtiff_v3
  • harvey_test_bigtiff_v3

Geojson labels

• Before manual cleaning

• The total number of bboxes for training + test: 126937
• The total number of bboxes for damaged buildings training + test: 12407
• The total number of bboxes for non-damaged buildings training + test: 114530
• The total number of 2048 chips for training + test: 1014

• Geojson cleaning

  • automatic cloud removal: 3163
  • manual removal:3508
  • total (set) : 6671

• After cleaning

• The total number of bboxes for training + test: 120277
• The total number of bboxes for damaged buildings training + test: 12255
• The total number of bboxes for non-damaged buildings training + test: 108022
• The total number of 2048 chips for training + test: 985 (more than actual used chips)

SSD 2class + augmentation

on ssd_xView

• train data (harvey_train_tomnod_2class_aug.record)

• INFO:root:Max chips per resolution: 25972
• INFO:root:Tot Box: 212175
• INFO:root:Chips: 17315
• for class: 1
• augmentation applied: 26035
• num of black small chips removed: 996
• num of small chips containing clouds: 46
• num of original class 1 bboxes: 26490 (original + shifted chips)
• num of original class 2 bboxes: 66170 (original + shifted chips)
• num of class 1 bbox augmented: 71134
• num of class 2 bbox augmented: 48381
• num of class 1 bbox in total: 97624
• num of class 2 bbox in total: 114551
• num of original chips that contain class 1: 5933
• num of original chips that cntain class 2 bboxes: 15007
• num of class 1 chips augmented: 25227
• num of class 2 chips augmented: 16216
• number of chips added by shift: 3283
• num of chips that contain class 1 bbox in total: 31160
• num of chips that contain class 2 bbox in total: 31223
• INFO:root:saved: 43350 train chips

• val data (harvey_val_tomnod_2class.record )

• INFO:root:Max chips per resolution: 4969
• INFO:root:Tot Box: 16134
• INFO:root:Chips: 3313
• num of black small chips removed: 298
• num of small chips containing clouds: 11
• num of original class 1 bboxes: 2368
• num of original class 2 bboxes: 13766
• num of original chips that contain class 1: 727
• num of original chips that cntain class 2 bboxes: 3015
• INFO:root:saved: 3313 test chips

• test data (harvey_test_tomnod_2class.record)

• INFO:root:Max chips per resolution: 4701
• INFO:root:Tot Box: 13611
• INFO:root:Chips: 3134
• num of black small chips removed: 0
• num of small chips containing clouds: 9
• num of original class 1 bboxes: 1819
• num of original class 2 bboxes: 11792
• num of original chips that contain class 1: 682
• num of original chips that cntain class 2 bboxes: 2868
• INFO:root:saved: 3134 test chips

SSD 2class + no augmentation

• on 54.244.179.66 (ssd_tomnod_2class_noaug)

• train data

• INFO:root:Max chips per resolution: 21121
• INFO:root:Tot Box: 67632
• INFO:root:Chips: 14081
• num of black small chips removed: 985
• num of small chips containing clouds: 45
• num of original class 1 bboxes: 7714
• num of original class 2 bboxes: 59918
• num of class 1 bbox augmented: 0
• num of class 2 bbox augmented: 0
• num of class 1 bbox in total: 7714
• num of class 2 bbox in total: 59918
• num of original chips that contain class 1: 2699
• num of original chips that cntain class 2 bboxes: 13023
• num of class 1 chips augmented: 0
• num of class 2 chips augmented: 0
• number of chips added by shift: 0
• num of chips that contain class 1 bbox in total: 2699
• num of chips that contain class 2 bbox in total: 13023
• INFO:root:saved: 14081 train chips

Tomnod 1 class + Augmentation

• 172.31.6.181 (ssd_tomnod_1class_aug)

• train:

• INFO:root:Max chips per resolution: 29620
• INFO:root:Tot Box: 106043
• INFO:root:Chips: 19747
• for class: 1
• augmentation applied: 30882
• num of black small chips removed: 1003
• num of small chips containing clouds: 46
• num of original class 1 bboxes: 27460
• num of original class 2 bboxes: 69124
• num of class 1 bbox augmented: 78583
• num of class 2 bbox augmented: 0
• num of class 1 bbox in total: 106043
• num of class 2 bbox in total: 69124
• num of original chips that contain class 1: 6444
• num of original chips that cntain class 2 bboxes: 17067
• num of class 1 chips augmented: 30882
• num of class 2 chips augmented: 0
• number of chips added by shift: 3410
• num of chips that contain class 1 bbox in total: 37326
• num of chips that contain class 2 bbox in total: 17067
• INFO:root:saved: 50629 train chips

• val data

• INFO:root:Max chips per resolution: 5778
• INFO:root:Tot Box: 2496
• INFO:root:Chips: 3852
• num of black small chips removed: 298
• num of small chips containing clouds: 11
• num of original class 1 bboxes: 2496
• num of original class 2 bboxes: 14331
• num of original chips that contain class 1: 832
• num of original chips that cntain class 2 bboxes: 3442
• INFO:root:saved: 3852 test chips

test dat

• INFO:root:Max chips per resolution: 5475
• INFO:root:Tot Box: 1969
• INFO:root:Chips: 3650
• num of black small chips removed: 0
• num of small chips containing clouds: 9
• num of original class 1 bboxes: 1969
• num of original class 2 bboxes: 12301
• num of original chips that contain class 1: 793
• num of original chips that cntain class 2 bboxes: 3280
• INFO:root:saved: 3650 test chips

Tomnod 1 class no augmentation

• 34.218.75.232 (ssd_tomnod_1class_noaug)

• training data

• INFO:root:Max chips per resolution: 24532
• INFO:root:Tot Box: 8159
• INFO:root:Chips: 16355
• num of black small chips removed: 985
• num of small chips containing clouds: 45
• num of original class 1 bboxes: 8159
• num of original class 2 bboxes: 62460
• num of class 1 bbox augmented: 0
• num of class 2 bbox augmented: 0
• num of class 1 bbox in total: 8159
• num of class 2 bbox in total: 62460
• num of original chips that contain class 1: 3057
• num of original chips that cntain class 2 bboxes: 14940
• num of class 1 chips augmented: 0
• num of class 2 chips augmented: 0
• number of chips added by shift: 0
• num of chips that contain class 1 bbox in total: 3057
• num of chips that contain class 2 bbox in total: 14940
• INFO:root:saved: 16355 train chips

Faster RCNN inception V2 on Tomnod 2class + aug data

• ssd_tomnod_frcnn