New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sublinear speed-up with dataparallel #13

Open

grey-area opened this issue Feb 18, 2019 · 1 comment

grey-area commented Feb 18, 2019

With default arguments apart from cardinality (set to 16), I get:

On 1 1080 ti with minibatch size 20: ~9 minutes per epoch.
Using dataparallel across 4 1080 ti's with minibatch size 128: ~4.5 minutes per epoch.

Perfect linear scaling would give you 2.25 minutes per epoch.
Any idea what's going on here/how to get better scaling?

Owner

prlz77 commented Feb 19, 2019

Hi!

Have you tried with a vanilla resnet? to check whether the problem is in the model or the dataloader?

Pau

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment