Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Text prompting asyncio fix #1305

Merged
merged 5 commits into from
Apr 27, 2023
Merged

Conversation

isabella618033
Copy link
Contributor

@isabella618033 isabella618033 commented Apr 26, 2023

Fixed the error below by avoiding running dendrite_pool forward for both inference and training at the same time that shares the same grpc resourses.
-> Added priority threadpool with max_worker = 1 inside dendrite_pool so that only 1 forward/backward would be processed at a time.

Exception in callback PollerCompletionQueue._handle_events(<_UnixSelecto...e debug=False>)()
handle: <Handle PollerCompletionQueue._handle_events(<_UnixSelecto...e debug=False>)()>
Traceback (most recent call last):
  File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/completion_queue.pyx.pxi", line 147, in grpc._cython.cygrpc.PollerCompletionQueue._handle_events 
blockingIOError: [Errno 11] Resource temporarily unavailable 

@isabella618033 isabella618033 marked this pull request as ready for review April 26, 2023 20:53
@@ -348,7 +348,8 @@ def inference(
roles = roles,
messages = contents,
uids = uids,
timeout = timeout
timeout = timeout,
priority = 2 # the priority of inference is always higher then training
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should inference take priority over training? Wouldnt this create issues where the model is always in inference?

@isabella618033 isabella618033 merged commit ee9b378 into text_prompting Apr 27, 2023
@ifrit98 ifrit98 deleted the text_prompting_asyncio_fix branch May 30, 2023 12:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants