Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix test_worker_waits_for_scheduler #6155

Merged
merged 1 commit into from
Apr 19, 2022

Conversation

mrocklin
Copy link
Member

Remove asyncio.wait_for from test.
It seems to be malfunctioning very rarely.
We replace it with a less sophisticated task + sleep + cancel.

Fixes #6153

Remove asyncio.wait_for from test.
It seems to be malfunctioning very rarely.
We replace it with a less sophisticated task + sleep + cancel.

Fixes dask#6153
@mrocklin mrocklin mentioned this pull request Apr 19, 2022
3 tasks
Copy link
Member

@fjetter fjetter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has a lower level root cause since we're using asyncio.wait_for internally in a bunch of places, particularly the core.connect. The way asyncio.wait_for handles cancellation is subject to a race condition.

I'm fine with rewriting the test if that resolves things in this specific case

@mrocklin
Copy link
Member Author

This has a lower level root cause since we're using asyncio.wait_for internally in a bunch of places, particularly the core.connect. The way asyncio.wait_for handles cancellation is subject to a race condition.

Is this python/cpython#86296 ?

@mrocklin
Copy link
Member Author

Or, more broadly, is there an issue somewhere for this? (doing a search on github for wait_for is less helpful than you might expect)

@fjetter
Copy link
Member

fjetter commented Apr 19, 2022

Sorry, I meant to copy the link... #6064

Is this python/cpython#86296 ?

This might be related but I need to read it a bit more thoroughly...

@mrocklin
Copy link
Member Author

Heh, this example is also a good case study in how you and I approach issues. I find a fix for the immedate pain in a few minutes, but ignore the deeper problem. You address the deeper problem, but things take a while. It's probably good that we're both here 🙂

@github-actions
Copy link
Contributor

Unit Test Results

       16 files  ±0         16 suites  ±0   7h 15m 48s ⏱️ - 23m 5s
  2 745 tests ±0    2 665 ✔️ +4       80 💤 ±0  0  - 4 
21 845 runs  ±0  20 810 ✔️ +5  1 035 💤 ±0  0  - 5 

Results for commit 4d4b318. ± Comparison against base commit fc5b460.

@mrocklin mrocklin merged commit 15ca62e into dask:main Apr 19, 2022
@mrocklin mrocklin deleted the fix-test_worker_waits_for_scheduler branch April 19, 2022 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Intermittent failure with test_worker_waits_for_scheduler
2 participants