-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Proposal] dump query processing performance metrics from various stages #3324
Comments
Yes please, this would be something we're interested in. See: https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/druid-development/MNHqZl7weLw/3CoM1MrgBAAJ |
👍 |
for some queries , number of segments scanned might be very large and that can blow up the context... so i will probably limit the number of segments reported per historical to something like 10 (...assuming other segments behaved similarly, this much information would be enough) also may be, have separate flags for only broker reporting the performance, and a "detailed" flag to include reports from historicals too. |
This issue has been marked as stale due to 280 days of inactivity. |
This issue has been closed due to lack of activity. If you think that |
While executing a query, Druid Broker and Historicals (and realtime tasks) publish very useful metrics like
at broker -
query/time
query/bytes
query/node/time
query/node/ttfb
at historical
query/time
query/bytes
query/segment/time
query/segment/wait
...
all the metrics contain queryId, host etc in the dimensions. so if Druid metrics were ingested in another Druid cluster, then users can understand where all the time for a query execution was spent. And, we do have a druid cluster (aka metrics-cluser) to debug performance issues.
However,
some users do not have bandwidth to maintain another druid cluster for metrics and push aggregated metrics to monitoring systems like Graphite. With aggregation, it becomes difficult to understand performance issues for specific queryId.
even with having a druid "metrics" cluster, it takes some time for metrics to get ingested to that cluster. sometimes we want to be able to do the debugging interactively, that is be able to send a query and see all the performance metrics in one place.
introduce /druid/v3 query endpoint that gives query responseContext #3319 and WIP: optionally configure DirectDruidClient to use /druid/v3 instead of /druid/v2 #3323 enable the ability to have large responseContext from broker (and same accumulated from all historicals)
This proposal is to enable dumping the query performance metrics in the responseContext if query context contains a flag, "dumpPerformance".
With the flag, end user would see a responseContext like below( which would be very useful to debug query performance problems).....
Depends on #3319 and #3323
The text was updated successfully, but these errors were encountered: