Flaky Dask test

[freddyaboulton]

I've seen the test_automl_immediate_quit dask test flake intermittently on my prs, on a merge to main, and on #2317 link.

Rerunning the tests usually makes the failure go away but that's not ideal.

The expected outcome of this issue is an explanation of why the test is flaking and/or a solution to make the test not flake anymore!

[chukarsten]

I can't seem to recover the specific test runs but this seems to always flake on Python 3.7. I saw it at least 4 times flaking on 3.7.

[chukarsten]

Just to add some more information on this, build_conda_pkg has failed on Python 3.7 on test_submit_training_job_single(), here.

[chukarsten]

Here is an instance of the flake happening on Windows and Python 3.8. First time I've seen it on this version, so I guess the 3.7/dask version disagreement is out.