Question: span.kind & peer.service

[CodeBlanch]

I have an instrumented service calling a third party service. If I set span.kind = client & peer.service = service_name on the outgoing span should Jaeger UI treat that span as a dependency with its own color?

It seems to treat it as internal, parent and child have the same service color.

It works if I send 2 batches with the process set on the second one to the peer.service value:

Notice on the second version the HTTP GET span to google is treated as an outgoing/dependency span with a different color than the parent.

No problem to send multiple batches, but it requires more processing to do so because I have to take all the spans ready for export and split them up into groups by service name.

[jpkrohling]

The color is defined based on the process' service name, not on tag values for a specific tag key.

It works if I send 2 batches with the process set on the second one to the peer.service value:

Could you share a code that does that? The service name is a "process" value, set when you set the tracer. All batches created by the same tracer instance should have the same service name, so, I would expect the second batch to have the same service (and color) as the first one.

[CodeBlanch]

@jpkrohling Sorry I think I put this issue in the wrong spot. It's probably more of a Jaeger UI thing? The client breaking up spans into batches by service name, in this case, is the OpenTelemetry .NET JaegerExporter: https://github.com/open-telemetry/opentelemetry-dotnet/blob/master/src/OpenTelemetry.Exporter.Jaeger/JaegerExporter.cs#L148-L184

[yurishkuro]

I think JaegerExporter.cs is doing the wrong thing (filed open-telemetry/opentelemetry-dotnet#1388):

var spanServiceName = jaegerSpan.PeerServiceName ?? this.Process.ServiceName;

Peer service has nothing to do with Process.ServiceName, the former is what YOUR service talks to, the latter is YOUR service.

If I set span.kind = client & peer.service = service_name on the outgoing span should Jaeger UI treat that span as a dependency with its own color?

No, it should not (at least it's how it is currently designed), because your service is still generating the client span. If the receiving server generates its own span, then it will be shown in a different color.

[CodeBlanch]

No, it should not (at least it's how it is currently designed), because your service is still generating the client span. If the receiving server generates its own span, then it will be shown in a different color.

I called Google, they aren't willing to send spans to my Jaeger 🤣

Sorry about the sensational example, but that is the issue I'm getting at. It is impossible to guarantee the receiving service will report. When you need to call out to some third party, or some service lacking instrumentation, or a product like SQL that doesn't report, the UI treats those spans as internal. The only way to make it work currently (that I've found) is to send a batch/process for each service. Agree 100% with you that is not how the API is designed to work, but I also don't think it was a design goal to represent external spans as internal.

I was hoping I was just missing something in the API to make it work (like remoteEndpoint in Zipkin), but it sounds like a gap?

[yurishkuro]

Yes, you could call it a gap. We also discussed that it would be useful to treat leaf client spans with peer.service tags as proxies for uninstrumented services, for example in the service dependency graphs.

In terms of the impact on the Gantt chart view, it has pros and cons. Using a different color for spans emitted by the same service runs counter to the current design of the UI, where service-color association is present in many screens.

[CodeBlanch]

One more reply and then I'll get out of your hair! Thanks for the dialog.

What you are saying makes total sense. But I think it might be a case of us being too close to the problem to really see it.

How I got here is I was demoing OpenTelemetry integration to a team at my company. Trying to get them to instrument their codebase so our traces flow better throughout the enterprise. I showed them the trace visualization using Jaeger UI. What did they say? Almost immediately: "It's broken, some of the outgoing calls aren't colored."

A user looking at it wants to see all dependencies colored. That is the killer feature. Sometimes both sides don't report, that's totally expected, and, IMO, we should have a solution for that. If there's anything I can do to get the ball rolling let me know.

Anyway, I'm going to close this because the question is basically answered: working as designed/not currently supported.

[yurishkuro]

The fix is coming in jaegertracing/jaeger-ui#659