-
-
Notifications
You must be signed in to change notification settings - Fork 269
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make tracing an optional feature #713
Comments
While test integrating tonic into TiKV to replace grpc-rs, I found that tonic's performance is not as good as grpc-rs as it consumes more cpu. And from the profile, I saw that a lot of cpu time is cost by the tracing logic in h2. In some workloads, the qps with tonic can be 15% lower than grpc-rs. Here is a cpu profiling snapshot: ![]() After patching tonic with a h2 package with all tracing code removed, tonic's performance is slightly better than gprc-rs in almost all workload without any further optimization. |
There's something about this that surprises me. Namely, should tracing really incur a 15% overhead if the tracing level is low? That seems shocking. @glorv any interest in providing repro steps or raw profiles? |
@ajwerner It's an attempt to replace grpc-rs with tonic for tikv. The workload is sysbench 's oltp_write_only that is a CPU heavy workload with high write concurrency(256 in my benchmark). So any part consumes more CPU will result with a performance regression. Here are 2 profiles with the same workload, the 2nd is patched with all tracing code removed. profile.zip |
Make
tracing
available as a Cargo feature, probably default off.The text was updated successfully, but these errors were encountered: