Skip to content
This repository has been archived by the owner on Sep 4, 2024. It is now read-only.

Latest commit

 

History

History
65 lines (47 loc) · 2.31 KB

README.md

File metadata and controls

65 lines (47 loc) · 2.31 KB
Raw

kube-custom-monitor

kube-custom-monitor - a set of useful Prometheus metrics for a kubernetes cluster

This project is in a very early state, but it might already be useful

Leader

Kubernetes setup

This monitor outputs two Prometheus metrics leader_monitor_is_leader and leader_monitor_leader_transitions which reflect an active leader for kube-scheduler, kube-controller-manager and other kubernetes controllers which uses the same leader election library.

SLO

SLO monitor compute a series of metrics which reflect time which a POD took to start:

  • slomonitor_pod_e2e_startup_latency_seconds - POD end-to-end startup latency without image pull time
  • slomonitor_pod_image_pull_latency_seconds - POD image pull latency
  • slomonitor_pod_full_startup_latency_seconds - POD's end-to-end startup latency

A startup latency is computed as the time from POD's object creating time to the start time of the last container in a pod minus time it took to download the images and minus time it took to run init-containers.

This module is heavily inspired by slo-monitor. However, they differently understand startup time. For slo-monitor, the startup time includes init-containers and readiness probes. In a production environment, these two values can be modified by cluster's users, and such modifications can heavily skew the cluster's SLO.

POD

One of the disadvantages of kube-state-metrics is that the set of metrics it often reports hard to correlate with the POD's state which kubectl get pods outputs.

$ kubectl get pods
NAME                             READY     STATUS    RESTARTS   AGE
api-f13f3c93-0-b469c776c-x8b25   1/1       Running   8          8d

This monitor reads the API server and computes a single metric kube_pod_status which value matches the status of a POD as how kubectl would show it.

Build and use

$ https://github.com/ikruglov/kube-custom-monitor.git
$ cd kube-custom-monitor
$ go build -v
$ ./kube-custom-monitor -kubeconfig kubeconfig -logtostderr -v 1

# in parallel terminal
$ curl -q http://localhost:8082/metrics

Acknowledgement

  • This module was originally developed for Booking.com.