Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle abstract loader that doesn't have a dataset member #840

Closed
versatran01 opened this issue Feb 14, 2020 · 0 comments · Fixed by #955
Closed

Handle abstract loader that doesn't have a dataset member #840

versatran01 opened this issue Feb 14, 2020 · 0 comments · Fixed by #955
Assignees
Labels
bug Something isn't working

Comments

@versatran01
Copy link

versatran01 commented Feb 14, 2020

Feature / Bug (?)

I have an abstract loader that chains multiple pytorch loaders when training on video sequences. Each small loader contains one sequence and the chained loader just use itertools.chain to chain them together.

I cannot put all data in a single loader because it does not make sense to read images from two different sequences.
I cannot use an IterableDataset because that would still have the same problem.

https://github.com/PyTorchLightning/pytorch-lightning/blob/06242c200a318a37d1f882c786e60354ec04533f/pytorch_lightning/trainer/data_loading.py#L58

This assumes that a loader must have a dataset field, which I felt is too restrictive. I suggest putting a check to see if the loader has a dataset member before doing this check. My workaround, for now, is just to declare a dataset member in my ChainedLoader to be None.

Fix

Since this kind of abstract loader does not have an explicit handle to a dataset, it should belong to the concept of an IterableDataset, which the user specifies the number of batches rather than a percentage. So a simple fix would be to put a check of hasattr somewhere in the same line.

@versatran01 versatran01 added the bug Something isn't working label Feb 14, 2020
@ethanwharris ethanwharris self-assigned this Feb 17, 2020
@Borda Borda added the need fix label Feb 22, 2020
@ethanwharris ethanwharris mentioned this issue Feb 26, 2020
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
3 participants