Filter terminated pods from node request metrics. #104
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
chadpatel
approved these changes
Sep 29, 2023
sky333999
approved these changes
Oct 2, 2023
adam-mateen
approved these changes
Oct 2, 2023
jefchien
merged commit 1957d95
into
amazon-contributing:aws-cwa-dev
86 of 87 checks passed
lisguo
pushed a commit
to lisguo/opentelemetry-collector-contrib
that referenced
this pull request
Oct 20, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description: The
node_<cpu|memory>_requestmetrics and metrics derived from them (node_<cpu|memory>_reserved_capacity) differ from the output ofkubectl describe node <node_name>. This is because kubectl filters out terminated pods. See linked issue for more details.Adds a filter for terminated (succeeded/failed state) pods.
Link to tracking Issue: open-telemetry#27262
Testing: Added unit test to validate pod state filtering. Built and deployed changes to cluster. Deployed
cpu-testpod.The gap is when the change was deployed. The metric can be see spiking up while the
cpu-testpod is running and then returns to the previous request size after it has terminated.