event driven approach #237
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RaviKhandavilli
previously approved these changes
Sep 28, 2022
lib/python/flame/backend/p2p.py
Outdated
| self.p2pbe._set_heart_beat(req.end_id) | ||
|
|
||
| await dck_task | ||
|
|
||
| async def _dummy_context_keeper(self, context): | ||
| """Sleep 1000 sec (an abitrary big value) to keep context alive.""" | ||
| """Sleep 100000 sec (an abitrary big value) to keep context alive.""" |
There was a problem hiding this comment.
check if there is any method , as long sleeps does not look good
There was a problem hiding this comment.
That is a good suggestion. I will address your comment.
The flame library presents some inefficiency. 1) In order to check if there is at least one peer in a channel, it busy-waits by calling empty() function and sleep one second. 2) head-of-line blocking problem: to recv data from peers of a channel, the implementation gets the list of peers, loops through each of peers and calls recv(). If the first peer is a straggler, it causes a head-of-line blocking problem. 3) When model size increases, it takes a while to merge transmitted messages (it is byte array concatenation operation). This process takes place in a coroutine; and it isn't suspended until it is finished. This delays a heart beat transmission as the heart beat function is also a coroutine. The delayed heart beat triggers a timeout at the other side of grpc channel, which breaks a training job. The first two issues are addressed in an event-driven manner. The third problem is addressed by using threading. The byte array concatenation takes place in a separate thread. So, the asyncio coroutine of handling message will release cpu resource quickly. Therefore, the heart beat transmission function works in a timely manner. Also, graceful termination is enforced by ensuring data transfer is complete.
myungjin
force-pushed
the
event_driven_approach
branch
from
17f97b3
to
a3b3c46
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The flame library presents some inefficiency.
In order to check if there is at least one peer in a channel, it busy-waits by calling empty() function and sleep one second.
head-of-line blocking problem: to recv data from peers of a channel, the implementation gets the list of peers, loops through each of peers and calls recv(). If the first peer is a straggler, it causes a head-of-line blocking problem.
When model size increases, it takes a while to merge transmitted messages (it is byte array concatenation operation). This process takes place in a coroutine; and it isn't suspended until it is finished. This delays a heart beat transmission as the heart beat function is also a coroutine. The delayed heart beat triggers a timeout at the other side of grpc channel, which breaks a training job.
The first two issues are addressed in an event-driven manner. The third problem is addressed by using threading. The byte array concatenation takes place in a separate thread. So, the asyncio coroutine of handling message will release cpu resource quickly. Therefore, the heart beat transmission function works in a timely manner.
Also, graceful termination is enforced by ensuring data transfer is complete.