Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connect shard throughput #5151

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Connect shard throughput #5151

wants to merge 2 commits into from

Conversation

fulmicoton
Copy link
Contributor

@fulmicoton fulmicoton commented Jun 24, 2024

Tested on simian with a pathologic workload:

- chico(rate=40mb)
- chico(rate=10kb)*10

On the left, without any per-shard weighting.
On the right, with per-shard weighting.

indexing throughput
image

cpu usage
image

Copy link

github-actions bot commented Jun 24, 2024

On SSD:

Average search latency is 1.0x that of the reference (lower is better).
Ref run id: 2268, ref commit: 9fddb68
Link

On GCS:

Average search latency is 1.05x that of the reference (lower is better).
Ref run id: 2269, ref commit: 9fddb68
Link

@fulmicoton fulmicoton force-pushed the connect-shard-throughput branch 3 times, most recently from a2686ca to a43c7a7 Compare July 1, 2024 06:03
Copy link
Contributor

@trinity-1686a trinity-1686a left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some lints are firing, otherwise lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants