[Prometheus compatibility] Do not require trimming suffixes by default

[dashpole]

Part of open-telemetry/wg-prometheus#72

Changes

We currently require Prometheus -> OpenTelemetry translation to remove type and unit suffixes in metric names. For example, the Prometheus metric http_request_duration_seconds_total would become http_request_duration in the resulting OTel metric. When this change was made, it received a large amount of user pushback, as it broke many existing metric names.

This PR makes trimming suffixes from names disabled by default, and encourages implementations to provide a config knob to enable the suffix trimming.

The original motivation for trimming suffixes by default was so that metrics could be round-tripped between an OTel SDK and the collector through the prometheus endpoint. For example, the OTel metric process.network.io becomes process_network_io_bytes_total with type and unit suffixes. When converting that back to OpenTelemetry, trimming suffixes would result in process_network_io, which is close to the original OpenTelemetry metric name, but isn't the same as the original.

We would like to make the removal of suffixes disabled by default because:

• It eliminates a breaking change for users
• It would preserve the name for existing prometheus metrics (e.g. not from OTel instrumentation) which many users are familiar with.
• Removing suffixes doesn't/didn't achieve the objective of round-tripping the metric name because . are substituted for _.

cc @open-telemetry/wg-prometheus @jmacd @gouthamve

[pirgeo]

Maybe I am missing something, but what is the use case for round-tripping through a Prometheus endpoint when I am already using an OTel SDK and an OTel collector? Maybe I am just misunderstanding the meaning of "round-trip" here.

[dashpole]

what is the use case for round-tripping through a Prometheus endpoint when I am already using an OTel SDK and an OTel collector?

Good question. Some users may prefer prometheus' pull model for ease of local development, for the up metric, or because they have already standardized on Prometheus endpoints within their team/org. They would be able to adopt OTel instrumentation and collector without changing the protocol if they wanted to. Being able to round-trip is also generally a reasonable definition of "compatibility" for us to work towards. But that use-case isn't compelling enough (IMO) to justify the breaking changes that we would have to make to get there, and trimming suffixes isn't sufficient to be able to round-trip metric names.

[pirgeo]

Gotcha, so this would allow folks to basically use the collector in an all-prometheus deployment.
This is an example of how I understood what you are saying (please correct me if this is not what you meant):

• an app has a prom endpoint that the (local) collector scrapes
• (optional) The collector might add some host metrics or similar
• The "local" collector sends it to a different ("remote") collector (e.g. via OTLP)
  • that remote collector combines data from multiple apps or instances
• The remote collector offers a Prometheus endpoint which can again be scraped by the backend.

The idea is that the data produced in the app and scraped by the local collector is the same as the data showing up in the backend, even if the data format was OTLP in between, is that right?

[dashpole]

Your example demonstrates the Prometheus -> OTLP -> Prometheus round-trip. The simpler example of this is a single collector with a Prometheus receiver and a Prometheus exporter, since the internal representation in the collector is OTLP. That round-trip works well today, and is not impacted by this PR.

The other round-trip, which is impacted by this PR, is the OTLP -> Prometheus -> OTLP round trip. This would occur when someone uses an OTel SDK (which has an OTel representation of metric data) with a prometheus exporter that is scraped by an OTel collector's prometheus receiver, and then pushed via OTLP somewhere.

[pirgeo]

I see, thanks for clarifying that.