Add metric to track Unhealthy Stale machine removal count

[jguipi]

What this PR does / why we need it:
There are two kinds of stale machine which machineSet controller collects

• stale due to unhealthiness
• stale due to creation timeout

In this metric , we are interested to expose the count of stale due to unhealthiness machine which get terminated between two scrapings by prometheus.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

I have tested the code for different scenarios like:

• deleting a machine obj directly doesn't increase count
• health timeout machines in different machineDeployments don't cause race and update the counter properly
• restarting mcm resets counter back to 0
• stale machines due to creation timeout collection doesn't increase the counter

Release note:

Added a new metric that will allow to get the number of stale (due to unhealthiness) machines  that are getting terminated

[jguipi]

/area monitoring
/kind enhancement

Here's a capture of the exported metric visibility :

cc.: @dmahmalatsap @istvanballok @rickardsjp @himanshu-kun @rishabh-11

[himanshu-kun]

/assign

[himanshu-kun]

there are many worker go-routines which reconcile the machineSets , and the current metric is per machineSet
So I don't feel this is a thread safe way of keeping count , I'll have to discuss internally and research and only then I can comment on that.
otherwise I have requested some changes.

[elankath]

nice work.

[dmahmalatsap]

Thanks for the refactor and for keeping us in mind in the process !

[rishabh-11]

Looks good. thanks for the changes.

/lgtm

[himanshu-kun]

/lgtm
approving my own contribution to the PR :')

[himanshu-kun]

Note: Since this metric is expose by MCM core, so it doesn't require to be vendored in mcm-providers to start exposing the metrics and will be available in next MCM release (v0.50.0)