Added support for HDF5 dataset and an HDF5 creation tool #1468
Conversation
keras_retinanet/bin/build_hdf5.py
Outdated
|
|
||
| import h5py | ||
| import numpy as np | ||
| from tqdm import tqdm |
There was a problem hiding this comment.
There are other scripts using progressbar2 instead of tqdm. I think we should choose one of them.
There was a problem hiding this comment.
Agreed.
Already changed and pushed changes :)
| return {'labels': self.labels[image_index], | ||
| 'bboxes': self.bboxes[image_index]} | ||
|
|
||
| def compute_input_output(self, group): |
There was a problem hiding this comment.
You override this to remove the filtering, right? Does it have a large computational impact? I'd expect it to be minimal, in which case it would be cleaner to not override this function. Do you have a measurement for this?
There was a problem hiding this comment.
Yes, I removed filtering because filtering happens when creating the hdf5. This process relies in the CSVGenerator class which filters the annotations already, so I considered removing that
|
I haven’t made a lot of testing yet, since I haven’t had the time for it, but I was getting roughly 6x faster epochs. I have to say that my dataset is composed of large images (around 3000x2000) and with smaller images the speedup will be less significant. |
|
I would be interested to see the differences on a more "normal" dataset like COCO. I expect the difference will be much smaller there because AFAIK the most time spent there is from anchor target generation, not data loading. |
Added retinanet-build-hdf5 entry point which allows the creation of datasets in the hdf5 format and a new option 'hdf5' to retinanet-train. This allows the dataset to be loaded in main memory the whole time and drastically reduces training times.