Support setting timestamp on metric

[tesharp]

As noted in one of the issues the actual timestamp of metrics received from Azure API is slightly behind scrape time. This change has an opt in feature to set the timestamp on metrics so it is the correct timestamp.

At the moment this does not deal with caching, so only works if caching is disabled. It cannot guarantee behavior of prometheus server if it will reject old samples etc. Left the old code intact so this is an opt in feature.

Example usage:

.\azure-metrics-exporter.exe --development.webui --metrics.resourceid.lowercase --metrics.set-timestamp

[mblaschke]

doesn't that create more and more load on prometheus side if you always publish the timestamp as label?
And how big is your latency? How do you scrape the metrics?

[tesharp]

It doesnt add timestamp as label. prometheus supports adding timestamp to the end, like this:

azure_metric_psqlf_iops_average_count{***} 0 1646861760000

Regarding latency; from my observations I generally get metrics 2 min delayed if scaping every minute. I'd rather miss last 2 min of data, but have the data at correct timestamp in grafana.

[tesharp]

Added a temporary fix for missing some metrics.. The collector fails if it tries to add same metric many times (same set of labels), GetMetricList would have to only return one metric value (latest value) for each unique combination of labels for that to properly work.

Right now it works if there is only one value, ie. if timespan and interval is the same. If timespan is set to PT5M and interval PT1M azure will return 5 metric values for same metric and it will fail as it tries to add 5 values for same label set. Works if both are PT1M for instance, then it only returns one value

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

No Coverage information
0.0% Duplication

[tesharp]

Had to fix the problem with getting unique label sets in the end. Requesting interval = PT1M, timespan = PT1M didn't always return all metrics and could leave gaps if there was a delay in metric being available in Azure API. So, now its possible to still set interval = PT1M but increase timespan (=PT10M for instance) and it will always use the latest value available.

[mblaschke]

What about a solution for caching?
Update the timestamp when cache is valid?

[mblaschke]

can you fix the conflicts?

[ethirolle]

We are seeing this exact issue now: data points collected via azure-metrics-exporter appear to be about 3 minutes ahead of those queried directly from Grafana's Azure Monitor datasource.

I know it has been a couple of years, but is there any chance of getting this PR merged, so that azure-metrics-exporter has the option to include the correct timestamps along with the data points?

Thank you!!