Log query and emit metrics for grpc request and multi-stage leaf stage

[Jackie-Jiang]

• Separate server query log and metric emitting into a separate class: ServerQueryLogger
• Allow logging query and emitting metric for GRPC request and multi-stage leaf stage
• Bugfixes:
  • Add numResizes into prometheus config
  • Change resizeTimeMs to be a meter instead of gauge
• Configuration change:
  • pinot.server.query.log.maxRatePerSecond to replace the deprecated pinot.query.scheduler.query.log.maxRatePerSecond (backward compatible)
  • pinot.server.query.log.droppedReportMaxRatePerSecond to configure the dropped log report frequency

Release Notes

New config added for server:

• pinot.server.query.log.maxRatePerSecond: query log max rate (QPS, default 10K)
• pinot.server.query.log.droppedReportMaxRatePerSecond: dropped query log report max rate (QPS, default 1)

[codecov-commenter]

Codecov Report

Merging #11838 (4b05e5a) into master (86d3c44) will decrease coverage by 0.08%.
Report is 3 commits behind head on master.
The diff coverage is 9.60%.

@@             Coverage Diff              @@
##             master   #11838      +/-   ##
============================================
- Coverage     66.35%   66.28%   -0.08%     
  Complexity      207      207              
============================================
  Files          2350     2351       +1     
  Lines        127281   127310      +29     
  Branches      19603    19606       +3     
============================================
- Hits          84463    84383      -80     
- Misses        36924    37037     +113     
+ Partials       5894     5890       -4     

Flag	Coverage Δ
custom-integration1	<0.01% <0.00%> (ø)
integration	<0.01% <0.00%> (ø)
integration1	<0.01% <0.00%> (ø)
integration2	0.00% <0.00%> (ø)
java-11	66.25% <9.60%> (-0.08%)	⬇️
java-21	49.82% <6.40%> (-16.41%)	⬇️
skip-bytebuffers-false	66.26% <9.60%> (-0.09%)	⬇️
skip-bytebuffers-true	49.80% <6.40%> (+31.99%)	⬆️
temurin	66.28% <9.60%> (-0.08%)	⬇️
unittests	66.28% <9.60%> (-0.08%)	⬇️
unittests1	67.05% <6.66%> (-0.07%)	⬇️
unittests2	17.80% <3.20%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...a/org/apache/pinot/common/metrics/ServerGauge.java	92.85% <ø> (-0.17%)	⬇️
...a/org/apache/pinot/common/metrics/ServerMeter.java	97.70% <100.00%> (+0.02%)	⬆️
...rg/apache/pinot/server/starter/ServerInstance.java	84.02% <100.00%> (+0.11%)	⬆️
...va/org/apache/pinot/spi/utils/CommonConstants.java	28.00% <ø> (ø)
.../java/org/apache/pinot/server/conf/ServerConf.java	93.10% <75.00%> (-2.90%)	⬇️
...che/pinot/core/query/scheduler/QueryScheduler.java	62.50% <50.00%> (-3.96%)	⬇️
...che/pinot/core/transport/grpc/GrpcQueryServer.java	0.00% <0.00%> (ø)
...e/operator/LeafStageTransferableBlockOperator.java	85.00% <20.00%> (-1.29%)	⬇️
...che/pinot/core/query/logger/ServerQueryLogger.java	2.85% <2.85%> (ø)

... and 21 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more