Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add nop operation to aggregate #135

Closed
ronensc opened this issue Mar 8, 2022 · 2 comments · Fixed by #152
Closed

Add nop operation to aggregate #135

ronensc opened this issue Mar 8, 2022 · 2 comments · Fixed by #152
Assignees

Comments

@ronensc
Copy link
Collaborator

ronensc commented Mar 8, 2022

Currently in the extract/aggregate stage we group entries by some field and apply an aggregate function on the group.
We support sum, min, max, avg and count
But, there are cases where we are interested in grouping entries without applying the aggregate function (nop - no operation).
We want to retain the original value of the entries.

One example for such case is histograms in prometheus. We need the original values so we could put it in the right buckets.
Actually, counters in promethues needs this too because their API accepts only deltas and not the final count/sum.

The current implementation saves the original value of the entries in the recentRawValues field.

"value": fmt.Sprintf("%f", group.value),
"recentRawValues": group.RecentRawValues,

Adding nop will allow us to merge the recentRawValues field into the value field.

@ronensc
Copy link
Collaborator Author

ronensc commented Mar 8, 2022

cc @eranra

@ronensc ronensc self-assigned this Mar 8, 2022
@ronensc
Copy link
Collaborator Author

ronensc commented Mar 8, 2022

To conclude, if the aggregate operation is one of:
max, min, avg, sum, count
the prometheus metric should be Gauge

If the aggregate operation is nop,
the prometheus metric should be Counter or Histogram. The valuekey field in the flowlogs-pipeline.conf.yaml will determine which field will be used.

There is still an edge case where we want to use count functionality with Prometheus Counters (rather than Gauges). In this case we need to add a new field recent_raw_count which is the count of the slice and in the prometheus encoder to handle both cases (slices and counts).
An alternative option is to add a transformer that adds a field with constant value of 1. Then we can use this 1s in the slice.

@ronensc ronensc linked a pull request Apr 26, 2022 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants