Huge datasets

[abreufilho]

Hello, this network is awesome. I've been using your code on my custom dataset but now i am trying to run it on a huge dataset and this code only loads all the dataset on the RAM memory. I already have 64 GB of RAM but for my usage i would need much more RAM.

So i am submitting this issue to ask if there's any way to load data for each batch directly on the disk, so i can use this huge dataset.

[dilraj1983]

The code posted in the repository loads the entire data in the RAM while defining the ChangeDetectionDataset Class. You can alter it by reading for example only the paths to the images and then reading each batch only while training on the fly by using the Dataloader Class. You can go through the pytorch docs for in depth understanding of the Dataloader Class