-
Notifications
You must be signed in to change notification settings - Fork 735
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kafka-Minion as alternative to Burrow for consumer lag monitoring #259
Merged
Merged
Changes from all commits
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
3789317
From the helm chart, not working yet
solsson 0e4f82c
Working http://metrics-minion.kafka:8080/metrics
solsson 9913600
Adds by-convention scrape annotations
solsson 4ccf36d
For Prometheus Operator with the kube-prometheus contrib
solsson 9f669f4
Kafka Minion 0.1.2
solsson 31baa04
Reflects the Kafka Minion use case more clearly,
solsson 419c629
Reorganizes our options for monitoring with Prometheus
solsson 5fc33f4
Uses dedicated endpoints, default options for now
solsson dcae04c
Comments on a dependency that I couldn't get apply -k to install first
solsson 54ab8b4
Gets to a point where variants must be manually tailored
solsson f848f56
Enables scrape for #259 through the SD that is no longer recommended
solsson c318eda
Merge branch 'prometheus-operator-kube-prometheus' into metrics-kafka…
solsson 7fcabf3
A heavy kustomization like this one is clearly only an example
solsson 22c60c5
Applying the prometheus exporter sidecar with -k is still TODO
solsson bba77fb
Avoids a Target Down until the sidecar can be applied
solsson 8c1d7ec
Lists the new exporter on the start page
solsson b2eb754
Switches to a Kafka Minion build from current master
solsson File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
apiVersion: v1 | ||
kind: Service | ||
metadata: | ||
name: metrics-minion | ||
namespace: kafka | ||
labels: &labels | ||
app: kafka-minion | ||
type: openmetrics | ||
spec: | ||
selector: *labels | ||
ports: | ||
- name: http | ||
port: 8080 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,49 @@ | ||
apiVersion: apps/v1 | ||
kind: Deployment | ||
metadata: | ||
name: metrics-minion | ||
namespace: kafka | ||
labels: &labels | ||
app: kafka-minion | ||
type: openmetrics | ||
spec: | ||
replicas: 1 | ||
selector: | ||
matchLabels: *labels | ||
template: | ||
metadata: | ||
labels: *labels | ||
annotations: | ||
prometheus.io/scrape: "true" | ||
prometheus.io/port: "8080" | ||
prometheus.io/path: /metrics | ||
spec: | ||
containers: | ||
- name: kafka-minion | ||
image: solsson/kafka-consumers-prometheus@sha256:a005aa02581fa46884b4a4499f7b4d56e889770c64e7e84d0a9ad195935fa59c | ||
env: | ||
- name: TELEMETRY_HOST | ||
value: 0.0.0.0 | ||
- name: TELEMETRY_PORT | ||
value: "8080" | ||
- name: EXPORTER_IGNORE_SYSTEM_TOPICS | ||
value: "true" | ||
- name: EXPORTER_METRICS_PREFIX | ||
value: kafka_minion | ||
- name: LOG_LEVEL | ||
value: info | ||
- name: KAFKA_BROKERS | ||
value: kafka-0.broker:9092, kafka-1.broker:9092, kafka-2.broker:9092 | ||
- name: KAFKA_CONSUMER_OFFSETS_TOPIC_NAME | ||
value: __consumer_offsets | ||
ports: | ||
- name: http | ||
containerPort: 8080 | ||
readinessProbe: | ||
httpGet: | ||
port: http | ||
path: /readycheck | ||
livenessProbe: | ||
httpGet: | ||
port: http | ||
path: /healthcheck |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
resources: | ||
- kafka-minion-service.yaml | ||
- kafka-minion.yaml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,10 +1,27 @@ | ||
# Export metrics to Prometheus | ||
|
||
Kafka uses JMX to expose metrics, as is already [enabled](https://github.com/Yolean/kubernetes-kafka/pull/96) for broker pods. There's many ways to use JMX. For example [Kafka Manager](../yahoo-kafka-manager/) uses it to display current broker traffic. | ||
JMX is already [enabled](https://github.com/Yolean/kubernetes-kafka/pull/96) for broker pods (TODO extract to kustomization). There's many ways to use JMX. For example [Kafka Manager](../yahoo-kafka-manager/) uses it to display current broker traffic. | ||
|
||
At Yolean we use Prometheus. This folder adds a sidecar to the broker pods that exports selected JMX metrics over HTTP in Prometheus format. To add a container to an existing pod we must use the `patch`command: | ||
This folder adds a sidecar to the broker pods that exports selected JMX metrics over HTTP in Prometheus format. To add a container to an existing pod we must use the `patch`command: | ||
|
||
Using kubectl 1.14+ | ||
|
||
``` | ||
kubectl --namespace kafka apply -k prometheus/ | ||
``` | ||
|
||
Using pre-1.14 kubectl: | ||
|
||
``` | ||
kubectl --namespace kafka apply -f prometheus/10-metrics-config.yml | ||
kubectl --namespace kafka patch statefulset kafka --patch "$(cat prometheus/50-kafka-jmx-exporter-patch.yml )" | ||
``` | ||
|
||
## Consumer lag monitoring | ||
|
||
See [Burrow](../linkedin-burrow) | ||
or [Kafka Minion](../consumers-prometheus/) | ||
|
||
## Prometheus Operator | ||
|
||
Use the [prometheus-operator](../variants/prometheus-operator/) kustomization. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
bases: | ||
# This kustomization needs to depend on one with kafka in it, to add the sidecar, | ||
# but it needs to be the kafka from the chosen variant, as ../kafka here would override other kustomizations | ||
#- ../kafka | ||
#- ../variants/scale-1 | ||
resources: | ||
- 10-metrics-config.yml | ||
patchesStrategicMerge: | ||
- 50-kafka-jmx-exporter-patch.yml |
12 changes: 12 additions & 0 deletions
12
variants/prometheus-operator-example/additional-scrape-configs.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
|
||
apiVersion: monitoring.coreos.com/v1 | ||
kind: Prometheus | ||
metadata: | ||
name: k8s | ||
namespace: monitoring | ||
spec: | ||
additionalScrapeConfigs: | ||
# github.com/kubernetes-sigs/kustomize/blob/master/examples/kvSourceGoPlugin.md is clearly WIP | ||
name: additional-scrape-configs-5m4c7m6mc9 | ||
# See https://github.com/prometheus/prometheus/pull/4131, and upon disagreement see https://github.com/prometheus/prometheus/issues/4484 | ||
key: pods-discovery-by-prometheus-io-annotations.yaml |
7 changes: 7 additions & 0 deletions
7
variants/prometheus-operator-example/alertmanager-main-scale-1.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
apiVersion: monitoring.coreos.com/v1 | ||
kind: Alertmanager | ||
metadata: | ||
name: main | ||
namespace: monitoring | ||
spec: | ||
replicas: 1 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
# Allows the "k8s" prometheus from Prometheus Operator contrib to do service discovery iin the kafka namespace | ||
--- | ||
apiVersion: rbac.authorization.k8s.io/v1 | ||
kind: Role | ||
metadata: | ||
name: prometheus-k8s | ||
namespace: kafka | ||
rules: | ||
- apiGroups: | ||
- "" | ||
resources: | ||
- services | ||
- endpoints | ||
- pods | ||
verbs: | ||
- get | ||
- list | ||
- watch | ||
--- | ||
apiVersion: rbac.authorization.k8s.io/v1 | ||
kind: RoleBinding | ||
metadata: | ||
name: prometheus-k8s | ||
namespace: kafka | ||
roleRef: | ||
apiGroup: rbac.authorization.k8s.io | ||
kind: Role | ||
name: prometheus-k8s | ||
subjects: | ||
- kind: ServiceAccount | ||
name: prometheus-k8s | ||
namespace: monitoring |
38 changes: 38 additions & 0 deletions
38
variants/prometheus-operator-example/k8s-kafka-servicemonitor.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
--- | ||
apiVersion: v1 | ||
kind: Service | ||
metadata: | ||
name: broker-monitoring | ||
namespace: kafka | ||
labels: | ||
app: kafka | ||
spec: | ||
publishNotReadyAddresses: true | ||
ports: | ||
- name: fromjmx | ||
port: 5556 | ||
selector: | ||
app: kafka | ||
--- | ||
apiVersion: monitoring.coreos.com/v1 | ||
kind: ServiceMonitor | ||
metadata: | ||
name: kafka | ||
namespace: monitoring | ||
labels: | ||
k8s-app: kafka | ||
spec: | ||
namespaceSelector: | ||
matchNames: | ||
- kafka | ||
selector: | ||
matchLabels: | ||
app: kafka | ||
endpoints: | ||
# https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#endpoint | ||
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token | ||
interval: 120s | ||
scrapeTimeout: 119s | ||
port: fromjmx | ||
scheme: http | ||
path: /metrics |
22 changes: 22 additions & 0 deletions
22
variants/prometheus-operator-example/k8s-minion-servicemonitor.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
apiVersion: monitoring.coreos.com/v1 | ||
kind: ServiceMonitor | ||
metadata: | ||
name: kafka-metrics-minion | ||
namespace: monitoring | ||
labels: | ||
k8s-app: kafka-metrics-minion | ||
spec: | ||
namespaceSelector: | ||
matchNames: | ||
- kafka | ||
selector: | ||
matchLabels: | ||
app: kafka-minion | ||
type: openmetrics | ||
endpoints: | ||
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token | ||
interval: 30s | ||
scrapeTimeout: 30s | ||
port: http | ||
scheme: http | ||
path: /metrics |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
bases: | ||
# With prometheus-operator only you must add your own Prometheus and Alertmanager resources | ||
#- github.com/coreos/prometheus-operator?ref=de9a6e1 | ||
- github.com/coreos/kube-prometheus?ref=3a64636 | ||
- ../../consumers-prometheus | ||
# The ../../prometheus base must be edited to point to the chosen kafka base | ||
# Actually to apply the sidecar with apply -k it has to be included with the kafka variant; can't be its own kustomization because you'll get | ||
# either "failed to find an object with apps_v1_StatefulSet|kafka to apply the patch" or "id 'apps_v1_StatefulSet|kafka|~P|zoo|~S' already used" | ||
#- ../../prometheus | ||
resources: | ||
- k8s-kafka-rbac.yaml | ||
# with base ../../prometheus | ||
#- k8s-kafka-servicemonitor.yaml | ||
# with base ../../consumers-prometheus | ||
- k8s-minion-servicemonitor.yaml | ||
patchesStrategicMerge: | ||
- prometheus-k8s-scale-1.yaml | ||
- prometheus-k8s-2.9.2.yaml | ||
- alertmanager-main-scale-1.yaml | ||
- prometheus-k8s-nodeport.yaml | ||
- additional-scrape-configs.yaml | ||
secretGenerator: | ||
- name: additional-scrape-configs | ||
namespace: monitoring | ||
#kvSources: | ||
#- name: pods-discovery-by-prometheus-io-annotations.yaml | ||
#pluginType: builtin | ||
#args: | ||
#- scrape-configs/pods-discovery-by-prometheus-io-annotations.yaml | ||
files: | ||
- scrape-configs/pods-discovery-by-prometheus-io-annotations.yaml |
8 changes: 8 additions & 0 deletions
8
variants/prometheus-operator-example/prometheus-k8s-2.9.2.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
apiVersion: monitoring.coreos.com/v1 | ||
kind: Prometheus | ||
metadata: | ||
name: k8s | ||
namespace: monitoring | ||
spec: | ||
baseImage: quay.io/prometheus/prometheus | ||
version: v2.7.2 |
10 changes: 10 additions & 0 deletions
10
variants/prometheus-operator-example/prometheus-k8s-nodeport.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
apiVersion: v1 | ||
kind: Service | ||
metadata: | ||
name: prometheus-k8s | ||
namespace: monitoring | ||
spec: | ||
type: NodePort | ||
ports: | ||
- port: 9090 | ||
nodePort: 32490 |
7 changes: 7 additions & 0 deletions
7
variants/prometheus-operator-example/prometheus-k8s-scale-1.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
apiVersion: monitoring.coreos.com/v1 | ||
kind: Prometheus | ||
metadata: | ||
name: k8s | ||
namespace: monitoring | ||
spec: | ||
replicas: 1 |
35 changes: 35 additions & 0 deletions
35
...ometheus-operator-example/scrape-configs/pods-discovery-by-prometheus-io-annotations.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
# Example scrape config for pods | ||
# | ||
# The relabeling allows the actual pod scrape endpoint to be configured via the | ||
# following annotations: | ||
# | ||
# * `prometheus.io/scrape`: Only scrape pods that have a value of `true` | ||
# * `prometheus.io/path`: If the metrics path is not `/metrics` override this. | ||
# * `prometheus.io/port`: Scrape the pod on the indicated port instead of the | ||
# pod's declared ports (default is a port-free target if none are declared). | ||
- job_name: 'kubernetes-pods' | ||
|
||
kubernetes_sd_configs: | ||
- role: pod | ||
|
||
relabel_configs: | ||
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] | ||
action: keep | ||
regex: true | ||
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] | ||
action: replace | ||
target_label: __metrics_path__ | ||
regex: (.+) | ||
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] | ||
action: replace | ||
regex: ([^:]+)(?::\d+)?;(\d+) | ||
replacement: $1:$2 | ||
target_label: __address__ | ||
- action: labelmap | ||
regex: __meta_kubernetes_pod_label_(.+) | ||
- source_labels: [__meta_kubernetes_namespace] | ||
action: replace | ||
target_label: kubernetes_namespace | ||
- source_labels: [__meta_kubernetes_pod_name] | ||
action: replace | ||
target_label: kubernetes_pod_name |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe some additional comments what one may prefer depending on the use case / environment?
In fact they can supplement each other and it may be a valid desire to operate both of them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's lots and lots of research to be done for anyone who wants to set up a Kafka stack and I see this repository as a collection of examples rather than a way to discuss the choices.