Log if closing an executor is not possible in thread #6644
Conversation
distributed/worker.py
Outdated
| @@ -1556,6 +1556,11 @@ def _close(): | |||
| try: | |||
| await to_thread(_close) | |||
| except RuntimeError: # Are we shutting down the process? | |||
| logger.error( | |||
There was a problem hiding this comment.
Should we add some information to the log message saying that this error is okay when we are shutting down the process? Otherwise, people might be confused to see an error log.
There was a problem hiding this comment.
I changed the logic. If python is shutting down, we're calling close immediately without trying to dispatch to the asyncio threadpool.
In all other cases, we'll try the dispatch and log this error when calling it explicitly.
This way we
- do not spam warnings for ordinary shutdown
- do not block for threads when the interpreter shuts down
- log an error if something happens we actually do not anticipate
this code is obviously much more verbose but encodes our intentions better.
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 15 files ±0 15 suites ±0 10h 45m 9s ⏱️ + 27m 0s For more details on these failures, see this check. Results for commit 272ee14. ± Comparison against base commit 40c9420. ♻️ This comment has been updated with latest results. |
| await to_thread(_close) | ||
| except RuntimeError: # Are we shutting down the process? | ||
| _close() # Just run it directly | ||
| if is_python_shutting_down(): |
This should only happen if the process is actually tearing down. If we hit this exception for any other reason, I would like to know