Investigate asyncio.wait_for for possible bug

[mrocklin]

I've encountered situations that I think point to asyncio.wait_for not actually cancelling the underlying coroutine if the timeout is reached. I've had to work around this in a couple of situations:

• Fix test_worker_waits_for_scheduler #6155
• Xfail test_worker_death_timeout #6186

See for a possible root cause: python/cpython#86296

We use asyncio.wait_for commonly throughout the codebase. If it genuinely sometimes misses then we should find a workaround, including possibly writing this ourselves.

cc @graingert do you have any thoughts on the CPython bug? Could it be causing what we're seeing? Do you see an easy solution here?

[graingert]

and this is related too #6064

[graingert]

one option is to wait for a backport of async with asyncio.timeout(delay): from python 3.11

python/cpython#90927

[mrocklin]

Backporting sounds fine to me, as long as it's reliable.

…

On Wed, Apr 27, 2022 at 9:21 AM Thomas Grainger ***@***.***> wrote: one option is to wait for a backport of async with asyncio.timeout(delay): from python 3.11 python/cpython#90927 <python/cpython#90927> — Reply to this email directly, view it on GitHub <#6211 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTB3PVHMNO2XTKIN5KTVHFEOXANCNFSM5UMIZHEQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[fjetter]

We would wait for an official backport. This is likely a feature we can not just copy-paste. Depending on how long it takes for a backport (assuming it actually comes) we can invest more time into #6064 for the trivial situations where we are basically creating and waiting for the task in the same context (e.g. not nested tasks) this is a sufficient fix.

[mrocklin]

we can invest more time into #6064 for the trivial situations where we are basically creating and waiting for the task in the same context (e.g. not nested tasks) this is a sufficient fix

I think that this is almost all cases that we use? I'm not sure what the less trivial case would look like.

To be clear, I would hope that this would be a function in utils.py that we could use throughout the codebase, not just in the case presented in that PR.