This repository has been archived by the owner on Apr 26, 2024. It is now read-only.
Optimise queueing of inbound replication commands #7861
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When we get behind on replication, we tend to stack up background processes behind a linearizer. Bg processes are heavy (particularly with respect to prometheus metrics) and linearizers aren't terribly efficient once the queue gets long either. A better approach is to maintain a queue of requests to be processed, and nominate a single process to work its way through the queue. Fixes: #7444
|
(this is best reviewed with whitespace changes hidden) |
richvdh
force-pushed
the
rav/optimise_replication_queue
branch
from
1bc4e93
to
1cc6c93
Compare
richvdh
force-pushed
the
rav/optimise_replication_queue
branch
from
1cc6c93
to
a507496
Compare
... because stock `typing` doesn't have it in Python 3.5.
erikjohnston
approved these changes
Jul 16, 2020
auscompgeek
reviewed
Jul 16, 2020
synapse/replication/tcp/handler.py
Outdated
Comment on lines
148
to
150
| stream_name: Deque[ | ||
| Tuple[Union[RdataCommand, PositionCommand], AbstractConnection] | ||
| ]() |
There was a problem hiding this comment.
FWIW, from PEP 484:
It is not recommended to use the subscripted class (e.g.
Node[int]) directly in an expression -- using a type alias (e.g.IntNode = Node[int]) instead is preferred. (First, creating the subscripted class, e.g.Node[int], has a runtime cost. Second, using a type alias is more readable.)
Since this is instantiating the type a few times, and the type is going across 3 lines, probably worth aliasing it?
There was a problem hiding this comment.
great suggestion, thanks.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When we get behind on replication, we tend to stack up background processes behind a linearizer. Bg processes are heavy (particularly with respect to prometheus metrics) and linearizers aren't terribly efficient once the queue gets long either.
A better approach is to maintain a queue of requests to be processed, and nominate a single process to work its way through the queue.
Fixes: #7444