Link visualizations concerns: too many incoming links, showing transitive links

[lmolkova]

When working on representing connections as spans (e.g. open-telemetry/semantic-conventions#756 or open-telemetry/semantic-conventions#1192), we'd like to be able to

• record connection (setup and/or lifetime) as a span
• record link from HTTP request to the connection

As a result, it could be possible to detect connection pool misconfiguration, leaked connection streams, see when request is slow because it's waiting for connection to be initiated or becomes available to it, etc.

Problem

There are a couple visualization concerns here:

1. The single connection serves 100-10000+ requests. When I open a visualization for a trace for specific request, I don't really want to see all other requests that were served on this connection.
2. Visualizing connection span (with a lot of incoming links) is useful when there are a few, for example, in dev/test environment, but overwhelming in common case. Having the data to analyze is still useful.
3. Connection initiation and request could be minutes apart, spans become tiny on a scale that shows time between operations.

Note:
This does not affect backends that don't show incoming links (such as Jaeger).

Questions:

How do backends visualize links? Are they ready to show tons of incoming links and potentially huge graphs?

• Do they show incoming links? Azure Monitor does, are there others?
• Do they show the whole graph? E.g. if A and C are both linked to B, and I open trace A, should I see A->B or all three? Is there a depth limit?
• What if linked operations are minutes apart? That's common in messaging scenarios. Do backends show it nicely? (e.g. add a break in the timeline)

And a couple of fundamental questions:

• Is links the right tool for the job? It seems the answer is yes - we want to record relationships between spans in different traces.
• If not links then what? Events + artificial connection-id to correlate on and give up on visualization?

[danielo515]

I have a very similar problem. At first, I thought the solution was to create custom traces out of our correlation IDs and force them on every span handling a message. But given how hard OTEL sdl fights back to prevent us from setting our own trace IDs I star to think it is not the right solution. Then someone suggested to use links, and I have the same exact concerns: will I be able to see connected traces? That is the whole reason I want to use open telemetry, so if the answer is no, I will be quite disappointed