Metadata fields to include in transaction metric documents

[AlexanderWert]

With the APM UI increasingly relying on metrics for data retrieval and visualization, we need to consider scenarios with transaction sampling rates of zero. In these cases, only information available about transactions are the transaction metrics. However, as of today there is a big gap between metadata fields available on transaction documents and the fields available on metric documents.

This thread is about starting a discussion on additional fields to be mirrored on transaction metrics documents.

Below is a first summary of the metadata fields and their availability on metric documents:

Fields already available with transaction metrics

• cloud.provider
• cloud.region
• agent.name
• agent.version
• service.name
• service.node.name
• service.environment
• host.hostname
• host.name
• container.id
• kubernetes.pod.name

Additional field candidates to include

field	already available in metrics	potentially relevant for metrics	usage examples
cloud.account.id	❌	✅	---
cloud.account.name	❌	✅	---
cloud.instance.id	❌	❔	---
cloud.instance.name	❌	❔	---
cloud.machine.type	❌	✅	---
cloud.project.id	❌	✅	---
cloud.project.name	❌	✅	---
cloud.service.name	❌	✅	To show the cloud service in the metadata fly-out.
network.connection.type	❔	❔	---
service.framework.name	❌	❔	---
service.framework.version	❌	❔	---
process.title	❌	❔	---
service.language.name	❌	✅	---
service.language.version	❌	✅	---
service.runtime.name	❌	✅	For the UI to determine whether a service is running in a serverless (AWS lambda) context. Or, determine that a Ruby service is executed on jRuby (different service icon).
service.runtime.version	❌	✅	---
host.architecture	❌	❔	---
host.os.platform	❌	✅	---
kubernetes.namespace	❔	❔	---
kubernetes.node.name	❔	❔	---
faas.coldstart	❌	✅	differentiate metrics between cold and warm starts of a serverless service
faas.trigger.type	❌	✅	filter serverless transaction metrics by faas trigger type

Fields not to include

• process.argv
• process.pid
• process.ppid
• service.id
• kubernetes.pod.uid

[alex-fedotyev]

I am 100% supportive as this would be helpful in several areas:

• Performance improvements
• Better support for aggressive sampling configuration
• Features like version comparisons or exploratory UI (going forward)

[cyrille-leclerc]

FYI AIOps faces a very similar problem for Transaction Correlations which requires to have access to many fields, including manually defined labels.

[felixbarny]

I think that by default, all metadata fields that are available for traces that don't significantly increase the cardinality should be available on metrics, too. A good check for that is whether the cardinality would be higher than a unique ephemeral ID that re-generates when an application restarts. While there can be exceptions, such as process.argv, the exclusion should be explicit and manual whereas the inclusion should be implicit and automatic.

access to many fields, including manually defined labels.

I think labels, with the exception of global labels, don't lend themselves well to be added to the dimensions of metrics. Falling back to sampled transaction documents and interpolating throughput based on the sample_rate seems like a better option for transaction-specific custom labels.

[cyrille-leclerc]

Good catch @felixbarny . We said with @sophiec20 and @stevedodson that it's more important for transaction correlations to ensure that we have a representative set of "transaction data", whatever these data are traces or metrics; it's more important for than benefiting of faster response time leveraging the density of "traces summarized as metrics".
With this rationale, we will postpone the effort to adopt "traces summarized as metrics" in the transaction correlation algorithm.

[felixbarny]

That makes sense to me. For error/latency correlation, we should include as many dimensions as feasible. Ideally even the ones with higher cardinality, such as url, user name, and custom labels. Adding these dimensions to metrics is not feasible, however.
So the correlations either have to solely work on trace documents or use both metrics and traces. For example, if trace correlations are slow, we could first do correlations on metrics and refresh the table once the trace correlations have been computed. As that would be rather complex and potentially confusing to the user, I wouldn't focus on that in the beginning.

[cyrille-leclerc]

you are right. @sophiec20 reminded us this opportunity of optimizing the response time first showing the results produced by "traces summarized as metrics" while waiting for the more accurate result of "traces as documents" but we have not proven that we have a to display the result causes dissatisfaction.

[dgieselaar]

I think that by default, all metadata fields that are available for traces that don't significantly increase the cardinality should be available on metrics, too.

Does that imply still using an allow list rather than exception list? I would be in favor of that, IMHO it would be too easy for users to add a custom field that blows up the cardinality of transaction metrics.

[axw]

@dgieselaar what I had in mind is that the dimensions would continue to be based on known fields only. We wouldn't, for example, consider dynamic user-defined fields like global labels.

If a new field comes along, then we should have a test in apm-server which fails if we don't explicitly identify it as a dimension or not. In this way it's not really an allow list or an exception list, but rather an explicit partitioning of the entire set of known fields.