Add scaling docs for V2 #4661
Merged
Add scaling docs for V2 #4661
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sakoush
approved these changes
Feb 13, 2023
| replicas: 3 | ||
| ``` | ||
|
|
||
| The number of replicas will need not to exceed the replicas of the Server the model is scheduled to. |
There was a problem hiding this comment.
Suggested change
| The number of replicas will need not to exceed the replicas of the Server the model is scheduled to. | |
| Currently, the number of replicas will need not to exceed the replicas of the Server the model is scheduled to. |
| serverConfig: mlserver | ||
| ``` | ||
|
|
||
| Models scheduled to a server can only scale up to the server replica count. |
There was a problem hiding this comment.
Suggested change
| Models scheduled to a server can only scale up to the server replica count. | |
| Currently, models scheduled to a server can only scale up to the server replica count. |
|
|
||
| ## Internal Components | ||
|
|
||
| Seldon core v2 runs with several control and dataplane components. The scaling of these resurces is discussed below: |
There was a problem hiding this comment.
Suggested change
| Seldon core v2 runs with several control and dataplane components. The scaling of these resurces is discussed below: | |
| Seldon Core v2 runs with several control and dataplane components. The scaling of these resurces is discussed below: |
| - Model gateway. | ||
| - This component pulls model requests from Kafka and sends them to inference servers. It can be scaled up to the partition factor of your Kafka topics. At present we set a uniform partition factor for all topics in one installation of Seldon Core V2. | ||
| - Dataflow engine. | ||
| - The dataflow engine runs KStream topologies to manage Pipelines. It can run as multiple replicas and the scheduler will balance Pipelines to run across it with a consistent hashing load balancer with each Pipeline managed up to the partition factor of Kafka (presently hardwired to one). |
There was a problem hiding this comment.
Suggested change
| - The dataflow engine runs KStream topologies to manage Pipelines. It can run as multiple replicas and the scheduler will balance Pipelines to run across it with a consistent hashing load balancer with each Pipeline managed up to the partition factor of Kafka (presently hardwired to one). | |
| - The dataflow engine runs KStream topologies to manage Pipelines. It can run as multiple replicas and the scheduler will balance Pipelines to run across it with a consistent hashing load balancer. Each Pipeline is managed up to the partition factor of Kafka (presently hardwired to one). |
| - Scheduler. | ||
| - This manages the control plane operations. It is presently required to be one replica as it maintains internal state within a BadgerDB held on local persistent storage (stateful set in Kubernetes). Performance tests have shown this not to be a bottleneck at present. | ||
| - Kubernetes Controller. | ||
| - The Kubernetes controller manages resources updates on the cluster which it passes on to the Scheduler, It is by default one replica but has the ability to scale. |
There was a problem hiding this comment.
Suggested change
| - The Kubernetes controller manages resources updates on the cluster which it passes on to the Scheduler, It is by default one replica but has the ability to scale. | |
| - The Kubernetes controller manages resources updates on the cluster which it passes on to the Scheduler. It is by default one replica but has the ability to scale. |
|
Should we also add a todo section about any upcoming work that can help with scalability? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds initial scaling docs section for V2.