Skip to content

Commit

Permalink
best-practice: support extra strategy for fine-tuning the performance…
Browse files Browse the repository at this point in the history
… in massive regions (#18221) (#18231)
  • Loading branch information
ti-chi-bot committed Jul 12, 2024
1 parent 81985a5 commit d0a7ae2
Show file tree
Hide file tree
Showing 3 changed files with 21 additions and 1 deletion.
20 changes: 20 additions & 0 deletions best-practices/massive-regions-best-practices.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,18 @@ You can check the following monitoring metrics in Grafana's **TiKV Dashboard**:

![Check Propose wait duration](/media/best-practices/propose-wait-duration.png)

+ `Commit log duration` in the **Raft IO** panel

`Commit log duration` is the time Raftstore takes to commit Raft logs to the majority of members in the respective Region. The possible reasons for a high value of this metric with significant fluctuations include the following:

- The workload on Raftstore is heavy.
- The append log operation is slow.
- Raft logs cannot be committed timely due to network congestion.

Reference value: lower than 200-500 ms.

![Check Commit log duration](/media/best-practices/commit-log-duration.png)

## Performance tuning methods

After finding out the cause of a performance problem, try to solve it from the following two aspects:
Expand Down Expand Up @@ -135,6 +147,14 @@ The default size of a Region is 96 MiB, and you can reduce the number of Regions
> + The query performance, especially for queries that deal with a large range of data, might decrease.
> + The Region scheduling slows down.
### Method 7: Increase the maximum number of connections for Raft communication

By default, the maximum number of connections used for Raft communication between TiKV nodes is 1. Increasing this number can help alleviate blockage issues caused by heavy communication workloads of a large number of Regions. For detailed instructions, see [`grpc-raft-conn-num`](/tikv-configuration-file.md#grpc-raft-conn-num).

> **Note:**
>
> To reduce unnecessary thread switching overhead and mitigate potential negative impacts from batch processing, it is recommended to set the number within the range of `[1, 4]`.
## Other problems and solutions

This section describes some other problems and solutions.
Expand Down
Binary file added media/best-practices/commit-log-duration.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion tikv-configuration-file.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ This document only describes the parameters that are not included in command-lin

### `grpc-raft-conn-num`

+ The maximum number of links among TiKV nodes for Raft communication
+ The maximum number of connections between TiKV nodes for Raft communication
+ Default value: `1`
+ Minimum value: `1`

Expand Down

0 comments on commit d0a7ae2

Please sign in to comment.