Add Resiliency Metrics

[Reham77]

Description:
The HyperPod team will tag each node on the Kubernetes level with a label which describes it’s health status, this PR goal is to add a feature to extract these labels value and emit a metrics to CW.

Testing: Deployed a custom agent on a testing cluster

Documentation: N/A

[movence]

so this will be always true and not contorlled by a flag or something?

[movence]

also add unit test for this?

[Reham77]

This should be empty if the node is not hyperpod (instance type doesn't start with ml)

[Reham77]

I don't have direct access to captureOnlyNodeLabelInfo from leaderElection. also i don't see the CaptureNodeLevelInfo getting tested. any suggestions?

[spanaik]

so this will be always true and not contorlled by a flag or something?

Yes this will run only on the leader agent to collect the labels for all nodes for now, we might end up collecting other labels for the node local agent as well, so we will need it for that as well.

[sky333999]

It looks like this map will have a max of one label for the health status and not have any other labels.
Does this even have to be a map in that case? Cant this just be a string called sageMakerNodeHealthStatus as part of NodeInfo and itll be empty for non hyperpod nodes. Callers can check for empty string or ret pointer and check for nil.

You can probably remove the captureOnlyNodeLabelInfo as well at that point?

[sky333999]

Calling it something generic like NodeToLabelsMap gives readers an expectation that all the labels set on the node are accessible via this map.

[spanaik]

The map are part of an allowlist that uses an ENUM instead of strings to store the label keys/values, any additional labels to be stored will have to be made into an ENUM, thus we will continue to use this pattern for now.