Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix metrics wiring. #4691

Merged
merged 1 commit into from
Nov 3, 2020
Merged

fix metrics wiring. #4691

merged 1 commit into from
Nov 3, 2020

Conversation

raulk
Copy link
Member

@raulk raulk commented Nov 2, 2020

Problem

Some components like go-ds-measure, go-ipfs-blockstore and go-bitswap expose metrics via ipfs/go-metrics-interface, but Lotus never injects the Prometheus exporter (ipfs/go-metrics-prometheus). Therefore, those metrics never surface in instrumentation.

Instead, Lotus uses OpenCensus directly.

Changes

This commit injects the Prometheus exporter for go-metrics-interface, and instructs the OpenCensus Prometheus exporter to use the DefaultRegistry.

This has the effect of exposing blending the metrics of both metrics libraries.

With this patch, the datastore, cache utilisation, and bitswap metrics are now exported via the /debug/metrics endpoint.

This commit also fixes an issue where the metrics scope was empty, making go-metrics-interface default to "". Angle brackets are inadmissible characters for Prometheus, so it was refusing to export the affected metrics. (These were the ARC cache metrics.)

Some components like go-ds-measure, go-ipfs-blockstore and go-bitswap
expose metrics via ipfs/go-metrics-interface, but Lotus never injects
the Prometheus exporter (ipfs/go-metrics-prometheus). Therefore, those
metrics never surface in instrumentation.

Instead, Lotus uses OpenCensus directly.

This commit injects the Prometheus exporter for go-metrics-interface, and
instructs the OpenCensus Prometheus exporter to use the DefaultRegistry.

This has the effect of exposing blending the metrics of both metrics
libraries.

With this patch, the datastore, cache utilisation, and bitswap metrics
are now exported via the /debug/metrics endpoint.

This commit also fixes an issue where the metrics scope was empty, making
go-metrics-interface default to "<no-scope>". Angle brackets are inadmissible
characters for Prometheus, so it was refusing to export the affected metrics.
(These were the ARC cache metrics.)
@raulk
Copy link
Member Author

raulk commented Nov 2, 2020

Some new metrics

image

image

image

Old metrics are still there, of course

image

image

@lanzafame
Copy link
Contributor

@raulk This looks good. Is there any way to have the fsrepo_* metrics be prefixed with lotus_? There is a chance for this to be exposed by IPFS on the same host, this problem can be solved on the querying side of things by creating graphs that sort by the job label which prometheus adds automatically when scraping but it would be the only one of all the lotus metrics so there will be confusion.

@raulk
Copy link
Member Author

raulk commented Nov 3, 2020

@lanzafame I investigated and unfortunately came up dry. Due to the way that measures are being registered, and the way go-metrics-interface is built, we'd need to change go-ds-metrics to take a context, which allows us to inject a custom namespace.

I suggest we merge this as is, and if it becomes a problem, we can look at doing that.

@lanzafame
Copy link
Contributor

I figured the lack of context in that interface was the reason but I wanted to hope otherwise for a little bit

@magik6k magik6k merged commit 1df02d5 into master Nov 3, 2020
@magik6k magik6k deleted the fix/metrics branch November 3, 2020 10:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants