Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v3rpc: Log expensive request in UnaryInterceptor #12086

Merged
merged 2 commits into from
Jul 1, 2020

Conversation

tangcong
Copy link
Contributor

@tangcong tangcong commented Jun 27, 2020

recently, In the process of troubleshooting the etcd request timeout, I encountered the following problems:

  1. Slow query (large response packet) Although the warn log will be printed when apply takes more than 100ms, but lack of remote peer info, it is not convenient to quickly locate the source of the call.
  2. Like the Authenticate interface does not go through the apply process, there is no available log for its latency(no warn apply log,no trace log). issue Issue: etcd stops answering after ~250 concurent HTTP API clients. #11826 has the same feedback.

Adjusting the log level to debug can solve the above problems, but the production environment needs to be changed
and performance will drop sharply.

This pr will print expensive request information in grpc interceptor to solve the above problems and improve the efficiency of the troubleshooting.

@tangcong tangcong force-pushed the print-expensive-request-v3rpc branch from f2b9865 to c5ea67e Compare June 30, 2020 16:52
@tangcong tangcong force-pushed the print-expensive-request-v3rpc branch from c5ea67e to 5c6d7b4 Compare June 30, 2020 16:56
Copy link
Contributor

@gyuho gyuho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@tangcong tangcong changed the title etcdserver,v3rpc: Log expensive request in UnaryInterceptor v3rpc: Log expensive request in UnaryInterceptor Jun 30, 2020
Copy link
Contributor

@jingyih jingyih left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jingyih jingyih merged commit 15884e9 into etcd-io:master Jul 1, 2020
@tangcong tangcong deleted the print-expensive-request-v3rpc branch February 26, 2021 18:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants