Add distributed sampler error

[mvpatel2000]

What does this PR do?

Adds error if dataloader is used without distributed sampler.

What issue(s) does this change relate to?

CO-1066

[mvpatel2000]

I'm very unsure why the grad accum check is failing. Would love fresh eyes

[mvpatel2000]

@eracah @dakinggg @bcui19 would love some help on this failing test.... I have no idea what's going on :(. Would love suggestions

[eracah]

@eracah @dakinggg @bcui19 would love some help on this failing test.... I have no idea what's going on :(. Would love suggestions

Make the tolerance higher?

[mvpatel2000]

@eracah @dakinggg @bcui19 would love some help on this failing test.... I have no idea what's going on :(. Would love suggestions

Make the tolerance higher?

@hanlint thinks there might be a genuine failure. It's not clear to me how the PR changes would affect the values we see... so its weird to see failures starting now. Maybe just randomness change?

[eracah]

@eracah @dakinggg @bcui19 would love some help on this failing test.... I have no idea what's going on :(. Would love suggestions

Make the tolerance higher?

@hanlint thinks there might be a genuine failure. It's not clear to me how the PR changes would affect the values we see... so its weird to see failures starting now. Maybe just randomness change?

what sources of stochasticity do you need to fix? just the rank_zero_seed?

[bcui19]

I checked this last night, I also wasn't sure why it was failing. Talked with @bandish-shah, mixed precision won't always produce the same results, so it could've been the random seed.

to repo it would be possible to re-run w/ the same seed that made the test fail.

Also now tests are passing, so it could've been the rank_zero_seed?

[mvpatel2000]

I checked this last night, I also wasn't sure why it was failing. Talked with @bandish-shah, mixed precision won't always produce the same results, so it could've been the random seed. to repo it would be possible to re-run w/ the same seed that made the test fail.
It seems to always be running the same seed. Each time the tests run, the amount it fails by is the same. So in this case, I don't think its per stochasticity @eracah @bcui19.

Greatest absolute difference: 1.691281795501709e-06 at index (1, 0) (up to 1e-08 allowed)
Greatest relative difference: 2.489731502748192e-05 at index (1, 0) (up to 1e-05 allowed)

It could just be the seed we used before worked, and the new seed doesn't work....

Also now tests are passing, so it could've been the rank_zero_seed?

I think it marked as passed because jenkins failed before it ran the tests.

[mvpatel2000]

on batch 12, we get different gradients because we get different forward pass outputs. The parameters are the same, and the grad_accum 2 value is closer to being correct (value you get with full precision) than the reference. So, I'm going to raise thresholds and call it a day

[bcui19]

Based on offline discussion around the error, LGTM! Thanks for deep dive.