kubelet-stats.md

kubelet-stats

This monitor pulls cadvisor metrics through a Kubernetes kubelet instance via the /stats/container endpoint.

Monitor Type: kubelet-stats

Monitor Source Code

Accepts Endpoints: No

Multiple Instances Allowed: Yes

Configuration

Config option	Required	Type	Description
kubeletAPI	no	object (see below)	Kubelet client configuration

The nested kubeletAPI config object has the following fields:

Config option	Required	Type	Description
url	no	string	URL of the Kubelet instance. This will default to https://<current node hostname>:10250 if not provided.
authType	no	string	Can be none for no auth, tls for TLS client cert auth, or serviceAccount to use the pod's default service account token to authenticate. (default: none)
skipVerify	no	bool	Whether to skip verification of the Kubelet's TLS cert (default: true)
caCertPath	no	string	Path to the CA cert that has signed the Kubelet's TLS cert, unnecessary if skipVerify is set to false.
clientCertPath	no	string	Path to the client TLS cert to use if authType is set to tls
clientKeyPath	no	string	Path to the client TLS key to use if authType is set to tls
logResponses	no	bool	Whether to log the raw cadvisor response at the debug level for debugging purposes. (default: false)

Metrics

The following table lists the metrics available for this monitor. Metrics that are marked as Included are standard metrics and are monitored by default.

Name	Type	Included	Description
container_cpu_cfs_periods	counter		Total number of elapsed CFS enforcement intervals
container_cpu_cfs_throttled_periods	counter		Total number of times tasks in the cgroup have been throttled
container_cpu_cfs_throttled_time	counter		Total time duration, in nanoseconds, for which tasks in the cgroup have been throttled
container_cpu_percent	counter		Cumulative cpu utilization as a percentage of the total host CPU available. This metric is equivalent to container_cpu_utilization / <# of CPUs/cores on host>.
container_cpu_system_seconds_total	counter		Cumulative system cpu time consumed in nanoseconds
container_cpu_usage_seconds_total	counter		Cumulative cpu time consumed per cpu in nanoseconds
container_cpu_user_seconds_total	counter		Cumulative user cpu time consumed in nanoseconds
container_cpu_utilization	counter	✔	Cumulative cpu utilization in percentages. This is equivalent to "centicores", or hundreths of CPU cores consumed. This metric is NOT normalized by the total # of cores on the system.
container_fs_io_current	gauge		Number of I/Os currently in progress
container_fs_io_time_seconds_total	counter		Cumulative count of seconds spent doing I/Os
container_fs_io_time_weighted_seconds_total	counter		Cumulative weighted I/O time in seconds
container_fs_limit_bytes	gauge		Number of bytes that the container may occupy on this filesystem
container_fs_read_seconds_total	counter		Cumulative count of seconds spent reading
container_fs_reads_merged_total	counter		Cumulative count of reads merged
container_fs_reads_total	counter		Cumulative count of reads completed
container_fs_sector_reads_total	counter		Cumulative count of sector reads completed
container_fs_sector_writes_total	counter		Cumulative count of sector writes completed
container_fs_usage_bytes	gauge		Number of bytes that are consumed by the container on this filesystem
container_fs_write_seconds_total	counter		Cumulative count of seconds spent writing
container_fs_writes_merged_total	counter		Cumulative count of writes merged
container_fs_writes_total	counter		Cumulative count of writes completed
container_last_seen	gauge		Last time a container was seen by the exporter
container_memory_failcnt	counter		Number of memory usage hits limits
container_memory_failures_total	counter	✔	Cumulative count of memory allocation failures
container_memory_usage_bytes	gauge	✔	Current memory usage in bytes
container_memory_working_set_bytes	gauge		Current working set in bytes
container_spec_cpu_period	gauge	✔	The number of microseconds that the CFS scheduler uses as a window when limiting container processes
container_spec_cpu_quota	gauge	✔	In CPU quota for the CFS process scheduler. In K8s this is equal to the containers's CPU limit as a fraction of 1 core and multiplied by the container_spec_cpu_period. So if the CPU limit is 500m (500 millicores) for a container and the container_spec_cpu_period is set to 100,000, this value will be 50,000.
container_spec_cpu_shares	gauge		CPU share of the container
container_spec_memory_limit_bytes	gauge	✔	Memory limit for the container.
container_spec_memory_swap_limit_bytes	gauge		Memory swap limit for the container.
container_start_time_seconds	gauge		Start time of the container since unix epoch in seconds.
container_tasks_state	gauge		Number of tasks in given state
machine_cpu_cores	gauge	✔	Number of CPU cores on the node.
machine_cpu_frequency_khz	gauge		Node's CPU frequency.
machine_memory_bytes	gauge	✔	Amount of memory installed on the node.
pod_network_receive_bytes_total	counter	✔	Cumulative count of bytes received
pod_network_receive_errors_total	counter	✔	Cumulative count of errors encountered while receiving
pod_network_receive_packets_dropped_total	counter		Cumulative count of packets dropped while receiving
pod_network_receive_packets_total	counter		Cumulative count of packets received
pod_network_transmit_bytes_total	counter	✔	Cumulative count of bytes transmitted
pod_network_transmit_errors_total	counter	✔	Cumulative count of errors encountered while transmitting
pod_network_transmit_packets_dropped_total	counter		Cumulative count of packets dropped while transmitting
pod_network_transmit_packets_total	counter		Cumulative count of packets transmitted

To specify custom metrics you want to monitor, add a metricsToInclude filter to the agent configuration, as shown in the code snippet below. The snippet lists all available custom metrics. You can copy and paste the snippet into your configuration file, then delete any custom metrics that you do not want sent.

Note that some of the custom metrics require you to set a flag as well as add them to the list. Check the monitor configuration file to see if a flag is required for gathering additional metrics.

metricsToInclude:
  - metricNames:
    - container_cpu_cfs_periods
    - container_cpu_cfs_throttled_periods
    - container_cpu_cfs_throttled_time
    - container_cpu_percent
    - container_cpu_system_seconds_total
    - container_cpu_usage_seconds_total
    - container_cpu_user_seconds_total
    - container_fs_io_current
    - container_fs_io_time_seconds_total
    - container_fs_io_time_weighted_seconds_total
    - container_fs_limit_bytes
    - container_fs_read_seconds_total
    - container_fs_reads_merged_total
    - container_fs_reads_total
    - container_fs_sector_reads_total
    - container_fs_sector_writes_total
    - container_fs_usage_bytes
    - container_fs_write_seconds_total
    - container_fs_writes_merged_total
    - container_fs_writes_total
    - container_last_seen
    - container_memory_failcnt
    - container_memory_working_set_bytes
    - container_spec_cpu_shares
    - container_spec_memory_swap_limit_bytes
    - container_start_time_seconds
    - container_tasks_state
    - machine_cpu_frequency_khz
    - pod_network_receive_packets_dropped_total
    - pod_network_receive_packets_total
    - pod_network_transmit_packets_dropped_total
    - pod_network_transmit_packets_total
    monitorType: kubelet-stats

Dimensions

The following dimensions may occur on metrics emitted by this monitor. Some dimensions may be specific to certain metrics.

Name	Description
container_id	The ID of the running container
container_image	The container image name
container_name	The container's name as it appears in the pod spec, the same as container_spec_name but retained for backwards compatibility.
container_spec_name	The container's name as it appears in the pod spec
kubernetes_namespace	The K8s namespace the container is part of
kubernetes_pod_name	The pod instance under which this container runs
kubernetes_pod_uid	The UID of the pod instance under which this container runs