Initial Entity and Resource proposal.

[jsuereth]

This is a proposal to address Resource and Entity data model interactions, including a path forward to address immediate friction and issues in the current resource specification.

The proposal includes all links and context needed to justify it, but duplicating a snapshot here:

Motivation

This proposal attempts to focus on the following problems within OpenTelemetry to unblock multiple working groups:

• Allowing mutating attributes to participate in Resource (OTEP 208).
• Allow Resource to handle entities whose lifetimes don't match the SDK's lifetime (OTEP 208).
• Provide support for async resource lookup (spec#952).
• Fix current Resource merge rules in the specification, which most implementations violate (oteps#208, spec#3382, spec#3710).
• Allow semantic convention resource modeling to progress (spec#605, spec#559, etc).

[tigrannajaryan]

Thanks for the draft @jsuereth !

This is an important next step for entities. I left some comments, but they don't prevent us from making progress.

[austinlparker]

A question for @jsuereth (and perhaps this has been addressed in WG meetings, if so, feel free to point me to those discussions) but my read of this proposal is that there is some future state where resource attributes are all contained inside entities, rather than flat, thus changing the 'logical' top-level resource attributes into complex attributes rather than flat ones. This is not a pattern that existing tools really use; Is the intention that these attributes should be flattened on ingest in order to allow traditional group-by/analysis functions and the entity definition ingested as some sort of hash (or other platform-appropriate representation)?

[bboreham]

Drive-by comment: I expected the top-level description to reference https://github.com/open-telemetry/oteps/blob/main/text/entities/0256-entities-data-model.md

[jack-berg]

Is there any prototype implementation of this? It sounds reasonable to me but is pretty abstract.

[jsuereth]

A question for @jsuereth (and perhaps this has been addressed in WG meetings, if so, feel free to point me to those discussions) but my read of this proposal is that there is some future state where resource attributes are all contained inside entities, rather than flat, thus changing the 'logical' top-level resource attributes into complex attributes rather than flat ones. This is not a pattern that existing tools really use; Is the intention that these attributes should be flattened on ingest in order to allow traditional group-by/analysis functions and the entity definition ingested as some sort of hash (or other platform-appropriate representation)?

The goal of this is that tooling can continue to engage with the flattened resource attribtue model we provide today. optionally tooling can engage with the new bundled entity definitions (and OTEL itself, e.g. the collector, would do so).

This gives flexibility for users to engage or not engage with entities, depending on their need, and opens up further exploration of entities.

[tigrannajaryan]

Is there any prototype implementation of this? It sounds reasonable to me but is pretty abstract.

@jack-berg There are early prototypes in Go and in Collector (core and contrib). The prototypes somewhat deviate from this proposal (e.g. the prototype only allows one associated EntityRef in the Resource, it uses key/value pairs, not key arrays, etc). Despite some deviation I think the prototypes demonstrate the idea (and we can update the prototypes to align with this proposal).

[aknuds1]

As proposed - we can create new entity-specific info metrics with descriptive attributes. I'd love to see @aknuds1 confirm, the last time I could check, these info metrics would work with the proposed info PromQL support exactly the way we want them to.

@jsuereth Sorry I'm late in reviewing this, but reading the PR now.

I should perhaps clarify the proposed MVP version of the info PromQL function only works with target_info (by default) or other info metrics (can be specified through a __name__ label matcher) that are identified through the instance and job labels. For info to be more intelligent about which info metrics exist, and which are their identifying labels, I think native metadata support first has to be added to Prometheus (that's something @codesome and I are trying to design).

[jsuereth]

Edit: adding CNCF thread on this: https://cloud-native.slack.com/archives/C01LSCJBXDZ/p1726584544475279

@aknuds1 - We should discuss more. My assumption on OTLP<->Prometheus here is that we can use identifying attributes to synthesize a job/instance label rather than forcing prometheus to understand all of these other labels.

Then, you can use info to interact with target_info (by default, has all the resource attributes) and/or a possible per-entity info metric.

We should continue this discussion somewhere higher bandwidth, cncf slack otel-prometheus channel?

[aknuds1]

Please see question regarding Prometheus compatibility.

[zalegrala]

Does the opt-in nature here conflict with the goals of unblocking the pieces you mention at the start? I'm wondering specifically about some components in a pipeline using the entities while others are only using resource. In such a case, I would expect that two entities could appear in the pipeline because now we have two areas of reference for the same concept.

If entiies becomes a new scope, should this eventually seek to replace resource?

[jsuereth]

I think entities will eventually replace existing resource usage. The reason we're layering is to not break the entire ecosystem and effectively create a V2 of the protocol. Instead, we see this as an evolution. If you want to engage with Entities, you would NEED to ensure all components are entity aware. As soon as one component is not, you're back to today's behavior. So users who want to use entities MUST ensure all components leverage entities.

However - there's a higher level question. Do I need to understand entities to interact with Metrics/Logs/Spans/etc. from OpenTelemetry? The answer to that is no. You can blindly leverage Resource.attributes as important slices of information for the data, and you will always be able to continue doing that. That's addressing consumers / storage solutions.

[ArthurSens]

Sorry for the delay reviewing this! It's a great read and I think that a clear separation of identifying/descriptive attributes is a big step forward for better compatibility with Prometheus.

I'd like to propose something different though :)

[ocelotl]

What about the resource attributes? Are they also dependent on the entities set E?

[jsuereth]

Yes.

You construct a Resource using this entity set.

• The Resource.attributes field contains alL identifying and descriptive attributes from the entity set.
• The Resource.schema_url field is set to the schema_url shared by all entities (if they all have the same one) or unset if they do not have the same schema_url.
• The Resource.entities field would contain the EntityRef proto, which just contains key names and the entity type for the entity set.

[carlosalberto]

Overall LGTM, although I'd like to know about details going forward, to set expectations on how the SDKs will be updated:

• Will Resource stay immutable (with the addition of the new entities field), and will be internally updated by the SDK with the latest instance tracked? If not, will attributes stay as read-only at least, in a best-effort attempt?
• The merge algorithm mentions schema_url as a comparison point between entities of the same type, and then one of the examples mentions having the same schema_url with different schema version. Is this something to be tuned/clarified later on, regarding on how to merge values in such case?

[aknuds1]

LGTM, thanks (see suggested change)!

[ArthurSens]

@aknuds1 since the proposal is being referenced in other places, like this OTEP, I wonder if we should update the proposal with what is being implemented here. 🤔

[ArthurSens]

• (desired) Conversion from OTLP -> prometheus can be reversed such that OTLP -> Prometheus -> OTLP is non-lossy.

I'm bringing that up to the Prometheus team. We've recently decided that Prometheus won't support exporting OTLP directly, but we're working on Prometheus Remote Write Receiver in OTel collector and that's our plan to transform Prometheus metrics into OTLP format, would that be ok?

[jsuereth]

Yes - to me it's the same domain modelling issue

[ArthurSens]

Happy to see this idea moving forward. With clear distinction of identifying attributes I believe we can design a better UX for Prometheus as a OTel metrics backend :)

[jsuereth]

Thanks @carlosalberto !

Will Resource stay immutable (with the addition of the new entities field), and will be internally updated by the SDK with the latest instance tracked? If not, will attributes stay as read-only at least, in a best-effort attempt?

I think a follow on OTEP will define the needs and design for "mutable" resource. This OTEP calls out that users want descriptive/mutable attributes on Resource. I expect the ResourceProvider concept to be expanded on to provide that (cc @tedsuo who I think is working on that proposal, or updating the existing one).

For now - this proposal has resource remain read-only and defines a merge algorithm for a ResourceProvider, but with identifying "descriptive" (or changing) attributes as a first step. This does not solve the needs of browser until the follow on OTEP is complete and approved.

The merge algorithm mentions schema_url as a comparison point between entities of the same type, and then one of the examples mentions having the same schema_url with different schema version. Is this something to be tuned/clarified later on, regarding on how to merge values in such case?

We clarify schema url versions in the specificaiton already - https://github.com/open-telemetry/opentelemetry-specification/tree/main/specification/schemas#schema-url

So we already have a way to understand version and whether or not they match. We do not have any notion of "compatibility" encoded in version but we can check for equality and order them (to know which is latest).

[annabokhan]

Nothing blocking, follow ups noted

[jsuereth]

Thanks @annabokhan !