Expose Additional Realtime Ingestion Metrics

[suddendust]

• Expose invalidRealtimeRowsDropped to measure decoding errors in stream.
• Expose incompleteRealtimeRowsConsumed to measure transformation errors when continueOnError has been set to true.
• Expose rowsWithErrors to cover any error while processing a row (catch-all, covers both the cases above).

JConsole output:

Regex test:

public class RegexMatcher {
  public static void main(String[] args) {
    // Define the regex patterns
    String[] regexPatterns = {
        "pinot.server.([^\\.]*?)_(OFFLINE|REALTIME)\\-(.+)\\-(\\w+).(invalidRealtimeRowsDropped|incompleteRealtimeRowsConsumed|rowsWithErrors|realtimeRowsConsumed)"
    };

    // Input strings to test against the regex patterns
    String[] inputStrings = {
        "pinot.server.airlineStats_REALTIME-flights-realtime-8.invalidRealtimeRowsDropped",
        "pinot.server.airlineStats_OFFLINE-flights-realtime-8.invalidRealtimeRowsDropped",
        "pinot.server.githubEvents_REALTIME-githubEvents-0.incompleteRealtimeRowsConsumed",
        "pinot.server.githubEvents_OFFLINE-githubEvents-0.incompleteRealtimeRowsConsumed",
        "pinot.server.meetupRsvp_REALTIME-meetupRSVPEvents-1.rowsWithErrors",
        "pinot.server.meetupRsvp_OFFLINE-meetupRSVPEvents-1.rowsWithErrors",
        "pinot.server.meetupRsvp_REALTIME-meetupRSVPEvents-1.realtimeRowsConsumed",
        "pinot.server.meetupRsvp_OFFLINE-meetupRSVPEvents-1.realtimeRowsConsumed"
    };

    // Iterate over each regex pattern
    for (String regexPattern : regexPatterns) {
      // Compile the regex pattern into a Pattern object
      Pattern pattern = Pattern.compile(regexPattern);

      // Iterate over each input string
      for (String inputString : inputStrings) {
        // Create a Matcher object to match the pattern against the input string
        Matcher matcher = pattern.matcher(inputString);

        // Perform the matching and check if it matches
        if (matcher.matches()) {
          System.out.println("Input string '" + inputString + "' matches the regex pattern: " + regexPattern);
        } else {
          System.out.println("Input string '" + inputString + "' does not match the regex pattern: " + regexPattern);
        }
      }
    }
  }
}

Output:

Input string 'pinot.server.airlineStats_REALTIME-flights-realtime-8.invalidRealtimeRowsDropped' matches the regex pattern: pinot.server.([^\.]*?)_(OFFLINE|REALTIME)\-(.+)\-(\w+).(invalidRealtimeRowsDropped|incompleteRealtimeRowsConsumed|rowsWithErrors|realtimeRowsConsumed)
Input string 'pinot.server.airlineStats_OFFLINE-flights-realtime-8.invalidRealtimeRowsDropped' matches the regex pattern: pinot.server.([^\.]*?)_(OFFLINE|REALTIME)\-(.+)\-(\w+).(invalidRealtimeRowsDropped|incompleteRealtimeRowsConsumed|rowsWithErrors|realtimeRowsConsumed)
Input string 'pinot.server.githubEvents_REALTIME-githubEvents-0.incompleteRealtimeRowsConsumed' matches the regex pattern: pinot.server.([^\.]*?)_(OFFLINE|REALTIME)\-(.+)\-(\w+).(invalidRealtimeRowsDropped|incompleteRealtimeRowsConsumed|rowsWithErrors|realtimeRowsConsumed)
Input string 'pinot.server.githubEvents_OFFLINE-githubEvents-0.incompleteRealtimeRowsConsumed' matches the regex pattern: pinot.server.([^\.]*?)_(OFFLINE|REALTIME)\-(.+)\-(\w+).(invalidRealtimeRowsDropped|incompleteRealtimeRowsConsumed|rowsWithErrors|realtimeRowsConsumed)
Input string 'pinot.server.meetupRsvp_REALTIME-meetupRSVPEvents-1.rowsWithErrors' matches the regex pattern: pinot.server.([^\.]*?)_(OFFLINE|REALTIME)\-(.+)\-(\w+).(invalidRealtimeRowsDropped|incompleteRealtimeRowsConsumed|rowsWithErrors|realtimeRowsConsumed)
Input string 'pinot.server.meetupRsvp_OFFLINE-meetupRSVPEvents-1.rowsWithErrors' matches the regex pattern: pinot.server.([^\.]*?)_(OFFLINE|REALTIME)\-(.+)\-(\w+).(invalidRealtimeRowsDropped|incompleteRealtimeRowsConsumed|rowsWithErrors|realtimeRowsConsumed)
Input string 'pinot.server.meetupRsvp_REALTIME-meetupRSVPEvents-1.realtimeRowsConsumed' matches the regex pattern: pinot.server.([^\.]*?)_(OFFLINE|REALTIME)\-(.+)\-(\w+).(invalidRealtimeRowsDropped|incompleteRealtimeRowsConsumed|rowsWithErrors|realtimeRowsConsumed)
Input string 'pinot.server.meetupRsvp_OFFLINE-meetupRSVPEvents-1.realtimeRowsConsumed' matches the regex pattern: pinot.server.([^\.]*?)_(OFFLINE|REALTIME)\-(.+)\-(\w+).(invalidRealtimeRowsDropped|incompleteRealtimeRowsConsumed|rowsWithErrors|realtimeRowsConsumed)

Process finished with exit code 0

[Jackie-Jiang]

Please refer to the definition for realtimeRowsConsumed, where we can properly extract the topic and partition

Out of the scope of this PR, just realize we have different format for Gauge (table name in the end), not sure if it is intentional. Ideally we should use the same format for Meter, Timer and Gauge.

[Jackie-Jiang]

We may also consider updating the regex to match all meters following this format so that we don't need to keep updating it whenever adding a new meter

[suddendust]

We may also consider updating the regex to match all meters following this format so that we don't need to keep updating it whenever adding a new meter

I can add that but not sure of the regex preference rules. A metric can match multiple regex in that case. Any idea around this?

[suddendust]

Please refer to the definition for realtimeRowsConsumed, where we can properly extract the topic and partition

Done

[KKcorps]

The partition label seems incorrect here. When I run your code I get the following

Input string 'pinot.server.meetupRsvp_OFFLINE-meetupRSVPEvents-1.realtimeRowsConsumed' matches the regex pattern: pinot.server.([^\.]*?)_(OFFLINE|REALTIME)\-(.+)\-(\w+).(invalidRealtimeRowsDropped|incompleteRealtimeRowsConsumed|rowsWithErrors|realtimeRowsConsumed)
metricName: pinot.server.meetupRsvp_OFFLINE-meetupRSVPEvents-1.realtimeRowsConsumed
Group 0: pinot.server.meetupRsvp_OFFLINE-meetupRSVPEvents-1.realtimeRowsConsumed
Group 1: meetupRsvp
Group 2: OFFLINE
Group 3: meetupRSVPEvents
Group 4: 1
Group 5: realtimeRowsConsumed

So it should be $4 and not $5, right?
And metric name should contain $5 instead of $4

[suddendust]

Changed as discussed

[KKcorps]

LGTM!