Instrument the operator with metrics

[sebgl]

Metrics we're interested in

• rate of reconciliation loops execution, for:
  • stack controller
  • elasticsearchcluster controller
  • kibana controller
• rate of reconciliation loop errors (for each of the 3 controllers), labeled with the error "type"
• reconciliation loop duration/latency (for each of the 3 controllers): average, ideally p95 & p99

ES communication metrics (labeled with the stack name):

• rate of the requests to a stack ES endpoint
• duration/latency of those requests

K8S requests metrics:

• rate of requests to the apiserver, labeled by verbs, route and status code
• rate of req errors to the apiserver, labeled by verbs, route and status code
• req duration: average, percentiles

Optional (to discuss?):

• gauge of the number of stack/es/kb the operator is responsible for

Metrics collector

1. prometheus lib instrumentation <- metricbeat -> Elasticsearch

• rates are usually expressed with counters, which we visualize by applying a rate() function. Counters can be reset when the process restarts. How well does that fit ES/Kibana? We can use a derivative aggregation, does it handle restarts well? Edit: yes, we can tweak this in the TS visual builder: https://www.youtube.com/watch?v=CNR-4kZ6v_E

• latencies are usually histograms, with values falling into buckets that we define:

# TYPE stack_operator_stack_reconciliations_duration_seconds histogram
stack_operator_stack_reconciliations_duration_seconds_bucket{le="0.005"} 1
stack_operator_stack_reconciliations_duration_seconds_bucket{le="0.01"} 1
stack_operator_stack_reconciliations_duration_seconds_bucket{le="0.025"} 1
stack_operator_stack_reconciliations_duration_seconds_bucket{le="0.05"} 1
stack_operator_stack_reconciliations_duration_seconds_bucket{le="0.1"} 1
stack_operator_stack_reconciliations_duration_seconds_bucket{le="0.25"} 1
stack_operator_stack_reconciliations_duration_seconds_bucket{le="0.5"} 1
stack_operator_stack_reconciliations_duration_seconds_bucket{le="1"} 1
stack_operator_stack_reconciliations_duration_seconds_bucket{le="2.5"} 1
stack_operator_stack_reconciliations_duration_seconds_bucket{le="5"} 1
stack_operator_stack_reconciliations_duration_seconds_bucket{le="10"} 1
stack_operator_stack_reconciliations_duration_seconds_bucket{le="+Inf"} 1
stack_operator_stack_reconciliations_duration_seconds_sum 0.000187009
stack_operator_stack_reconciliations_duration_seconds_count 1

Not sure how to visualize that with ES/Kibana.

• gauges represent values that can go up/down (a good example for that is CPU usage).

2. go-metrics <- metricbeats -> elasticsearch

Main benefit: histograms are simpler than the prometheus alternative, since they emit avg and percentiles values directly.

3. logs <- filebeat -> Elasticsearch

If each reconciliation loop execution is logged anyway, it's quite easy to include all metrics in the logs we produce, and build dashboards with that. In other terms: leave the entire aggregation to ES, don't pre-aggregate.

[sebgl]

Basic metrics set up in PR #214.
But not really exposed yet. It should be behind an optional service for that, with maybe the Prometheus annotation set.

[donbowman]

https://github.com/vvanholl/elasticsearch-prometheus-exporter
would be nice

[anyasabo]

@donbowman that exporter is for the actual ES clusters themselves, where this issue is about instrumenting the operator. That said, since that ES exporter installs as a plugin you can install it with an init container as described here:
https://github.com/elastic/cloud-on-k8s/blob/master/docs/snapshots.asciidoc

[charith-elastic]

It would be very useful to have an instrumented Elasticsearch client and gather metrics about the API calls. One of the observations from scale testing (#357) was that the operator seems to spend most of its time on API calls when managing a large number of Elasticsearch clusters. Having the metrics to backup these observations can help us measure the effects of any optimization efforts on that end.

[sebgl]

Relates #1189.

[dulltz]

Usage data about elastic-licensing might be also worth exposing as Prometheus metrics for admins.
This will prevent users from inadvertently violating limits.
https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-licensing.html#k8s_getting_usage_data

[pebrc]

We have controller-runtime metrics as of #214, the Elasticsearch client is instrumented as of #1189. The question about support for Prometheus histograms is answered as of Elasticsearch 7.6 with the new histogram field mapper

I just created #3140 to follow up on the last comment here and am suggesting to close this issue for now. Please reopen if you disagree.