⚠️ CI failed ⚠️

[github-actions]

Workflow Run URL

[ncclementi]

runtime = 'coiled-0.0.4-py3.9', name = 'test_double_diff', category = 'benchmarks', last_three_duration [s] = (426.233430147171, 606.2415187358856, 397.9905481338501), duration_threshold [s] = 381.5963892449 

This one is a bit tricky to judge because it is quite noisy. Note that the runs that failed (x in the plot) are not used to compute the mean that we use for the analysis. This is running with runtime 0.0.4 (no packages changed, except maybe coiled), which leads me to think that this regression report is a result of noise.

The only thing that makes me doubt is that in upstream about the same date we see an increment in durations, but not so bad yet to call it regression.
Upstream:

@ian-r-rose do you have any thoughts?

[ncclementi]

On upstream, even though not reported as regression since the signal it's very noisy, wall time seems to be better since this was reported, Although the average memory seems to be increasing.

@gjoseph92 any thoughts on this? trying to asses whether I should close this, or keep it open.

[gjoseph92]

I wonder if this is due to

• Withhold root tasks [no co assignment] dask/distributed#6614
• Revert "Fix co-assignment for binary operations" dask/distributed#6985

However, the movement is the opposite of what I'd expect. The first PR should have made runtime go down from baseline, then the follow-up should have made it go back up to baseline. Unless the "fix" really didn't work properly.

I also would expect it to be affected by dask/distributed#6975.

It's kind of unbelievable how much noisiness and variance there is on 0.0.4. I would say that this workload is simply unrunnable on those older versions.

[ncclementi]

It's kind of unbelievable how much noisiness and variance there is on 0.0.4. I would say that this workload is simply unrunnable on those older versions.

Sure, but the pictures on my latest comment are from upstream (see here https://coiled.github.io/coiled-runtime/coiled-upstream-py3.9.html) and I see that wall time went down 09/02 but avg memory went up is this expected

[ncclementi]

Closing, as current upstream looks like