System metrics - Define transition plan for breaking changes in attributes

[joaopgrassi]

In open-telemetry/opentelemetry-collector-contrib#89 the attributes of all existing metrics were changed based on the decision in open-telemetry/opentelemetry-collector-contrib#51. An overview of the changes can be found in this comment.

From @jsuereth on the PR:

We should do one of two things with this PR:

• Open a (blocking) bug about providing guidance on how to deal with the breaking changes in system metrics.
• Advise instrumentation to continue using the older System metrics specification until the system metrics working group has had a chance to walk through their issues and declare this stable.

I'm creating this issue to discuss what we should do next:

• Should we merge Perf test: TestTrace10kSPS/OpenCensus CPU consumption is 40.3%, max expected is 35% opentelemetry-collector-contrib#89 first and then define whatever guidelines/plan?
• Can we know if these metrics are wildly adopted/used?
• If we need a plan, what everyone thinks doing something similar to HTTP: Defining transition period for pre-stability HTTP semconv breaking changes opentelemetry-specification#3362

CC @jsuereth @ChrsMark @dmitryax @mx-psi

[mx-psi]

Do we have an idea as to what OpenTelemetry components implement system metrics? I am aware and have worked on the OpenTelemetry Collector hostmetrics receiver, but I don't know what language libraries have support for it and I think the specific implementers may change our plans (we have different tooling available in the Collector vs. libraries e.g. feature gates)

[trask]

Do we have an idea as to what OpenTelemetry components implement system metrics?

I think you should be able to find out by searching the github org for the different system.* namespaces, e.g.

• https://github.com/search?q=org%3Aopen-telemetry+system.cpu&type=code
• https://github.com/search?q=org%3Aopen-telemetry+system.memory&type=code

Also note this prior sentiment that non-collector components should not emit system.* metrics: open-telemetry/opentelemetry-specification#2388 (comment)

[dmitryax]

I think it should be similar to open-telemetry/opentelemetry-specification#3362 with a long transition period because the hostmetrics receiver is pretty popular and must be widely used in the field.

Speaking about the transition mechanics, open-telemetry/opentelemetry-collector#10903 (I'm working on it) will introduce a concept of default and optional attributes that users can override along with the ability to assign deprecation warnings for them in metadata.yaml files. Similar functionality is currently available for metrics https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/cmd/mdatagen/metadata-schema.yaml#L60. Once it's in place, we can utilize that mechanism for this transition.

[mx-psi]

Thanks @trask! Based on the searches from #246 (comment) as well as searches for all other prefixes from the system namespace we have the following implementations of at least some system metrics:

• go.opentelemetry.io/contrib/instrumentation/host (Go Contrib)
• @opentelemetry/host-metrics (JS contrib)
• opentelemetry-instrumentation-system-metrics (Python contrib)
• hostmetrics receiver (core Collector)

[mx-psi]

On the August 30th System Semantic Convention WG meeting we decided that:

• We want to continue merging changes to the spec up until we declare the system metrics stable
• We will not make the transition on any implementations until we have declared it stable, we will use the experience of the HTTP transition to understand and adjust the transition plan for language libraries

[joaopgrassi]

Discussed about the plan on September 11th in the Semconv SIG meeting:

• Collector uses feature gates to manage breaking changes - the plan should accommodate for that
• As an initial "warning" we should add a similar section as done for HTTP, indicating instrumentations to continue producing the old version of the conventions
• Leave the actual transition plan blank, until we know more which SIGs are affected and if/how we will handle it. System semconv SIG should be filling this gap there with the proper plan

[dmitryax]

Collector uses feature gates to manage breaking changes - the plan should accommodate for that

I shared a preferred way for the collector in the comment above #246 (comment). I think we should use that one instead of feature gates. It's more flexible, e.g. it allows users to send both attributes during the transition if they want to.