Question about the meaning of container.cpu.utilization from kubeletstats receiver

[chenlujjj]

Component(s)

receiver/kubeletstats

Describe the issue you're reporting

Hi Team,
We are trying to use the kubeletstats receiver, but don't understand the container.cpu.utilization metric very well.

1. From the documentation, its type is gauge and unit is 1, but the plot of the metric indicates it is more like a counter and has a unit of seconds.
   

Note: we are using signalfx to store and visualize metrics.

2. I apply a rate function to the raw metric and scale it with a factor of 0.01, then the metric value is consistent with another metric container/cpu/core_usage_time from GCP (you can find the metric description here). I'm confused why we need the 0.01 scale factor, does it mean the metric is a percent in fact?

[github-actions]

Pinging code owners:

• receiver/kubeletstats: @dmitryax @TylerHelmuth

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[TylerHelmuth]

@chenlujjj you're correct that the name isn't quite right. We are discussing this as part of #24905. I am going to close this issue since I believe it is being handled by #24905, please ping me if you disagree.

[chenlujjj]

@TylerHelmuth Thanks. How about the 2nd question ? Do we need to scale it with 0.01 to get the accurate cpu usage ?

[TylerHelmuth]

The cpu usage being reported is in "CPU core-nanoseconds per second". Scaling it with 0.01 changes it from nanoseconds to milliseconds I think.

[chenlujjj]

@TylerHelmuth I think it is normalized to second unit here. I'm wondering if you have some real-life examples using this metric, and could you please share it out ?

[TylerHelmuth]

@chenlujjj you're right, we do normalize it to seconds. Maybe @dmitryax from Splunk can answer the question.

[jinja2]

@chenlujjj are you by any chance also sending in data from the signalfx-agent's kubelet-metrics monitor? Specifically the metric container_cpu_utilization

[dmitryax]

Hi @chenlujjj, there are some query time mappings on the Splunk Observability backend that causing this confusion. We are going to resolve it. Can you please submit a Splunk support ticket and we will engage there?

[chenlujjj]

@jinja2 Not sure how to use signalfx-agent, it has reached End of Support. We are using splunk-otel-collector to collect metrics/logs/traces.

[chenlujjj]

@dmitryax ok, I'll raise a ticket there.