Add record availability lag for Kafka connector

[navina]

Introduces record availability lag for realtime connectors via the rest apis - consumingSegmentsInfo
and /debug/tables/{tableName}

Label: release-notes

/consumingSegmentsInfo and debug/tables/{tableName} API now provides the following per-partition numbers for Kafka data sources:
currentConsumingOffset, latestUpstreamOffset, recordsLag, availabilityLagMs

[codecov-commenter]

Codecov Report

Merging #9621 (b13c20f) into master (d580bbc) will decrease coverage by 0.07%.
The diff coverage is 13.20%.

@@             Coverage Diff              @@
##             master    #9621      +/-   ##
============================================
- Coverage     35.02%   34.95%   -0.08%     
  Complexity      189      189              
============================================
  Files          1943     1943              
  Lines        103817   103841      +24     
  Branches      15743    15744       +1     
============================================
- Hits          36361    36293      -68     
- Misses        64369    64448      +79     
- Partials       3087     3100      +13     

Flag	Coverage Δ
integration1	25.80% <5.66%> (-0.04%)	⬇️
unittests2	15.66% <10.52%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
.../common/restlet/resources/SegmentConsumerInfo.java	0.00% <0.00%> (ø)
...ugin/stream/kafka20/KafkaConsumerPartitionLag.java	0.00% <0.00%> (ø)
...in/stream/kafka20/KafkaStreamMetadataProvider.java	48.38% <0.00%> (-6.16%)	⬇️
...lugin/stream/kafka20/server/KafkaDataProducer.java	32.60% <0.00%> (-0.73%)	⬇️
...ache/pinot/server/api/resources/DebugResource.java	0.00% <0.00%> (ø)
...che/pinot/server/api/resources/TablesResource.java	27.31% <0.00%> (-0.25%)	⬇️
...pache/pinot/spi/stream/ConsumerPartitionState.java	0.00% <0.00%> (ø)
...org/apache/pinot/spi/stream/PartitionLagState.java	0.00% <0.00%> (ø)
...manager/realtime/LLRealtimeSegmentDataManager.java	58.10% <60.00%> (+2.66%)	⬆️
...ot/controller/util/ConsumingSegmentInfoReader.java	89.36% <100.00%> (+0.23%)	⬆️
... and 44 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[navina]

@npawar / @KKcorps Please review!

I seem to have added docs before merging the code :))

[Jackie-Jiang]

Try to keep the json property and variable name the same

[Jackie-Jiang]

We may also pick a shorter name for them to be more concise, but let's keep the name consistent across classes:

• recordsLag: lag in records (offsets)
• availabilityLagMs: availability lag in millis
• timeLagMs: time lag in millis

[Jackie-Jiang]

LGTM