What's INTERNAL span kind and when it should be used

[lmolkova]

See #4178 for the context.

Internal kind is defined as

opentelemetry-specification/specification/trace/api.md

Lines 801 to 803 in 5381b55

	* `INTERNAL` Default value. Indicates that the span represents an
	internal operation within an application, as opposed to an
	operations with remote parents or children.

Presumably it means that span cannot have remote parents and remote children.

Based on discissions referenced in #4178, #110, #526:

• instrumentations don't know if underlying layer is instrumented. Underlying layer may be instrumented, not enabled, or conditionally suppressed (still propagating current context)
• there are multiple nested layers of physical operations (e.g. HTTP -> DNS) and lower layers don't usually support context propagation
• there are protocols that don't support context propagation (yet)
• there are multiple layers that each support context propagation (e.g. messaging + CloudEvents over HTTP could carry 3 different contexts at the same time)

I.e. there is no reliably way to know whether something will have remote children (or any descendants). It depends on application, enabled instrumentations, potentially interchangeable protocols.

INTERNAL vs SERVER is also controversial. Imagine a cron-job (e.g. FaaS time trigger) - it never has a parent, should it be Internal?

The existing status-quo:

• the boundary between INTERNAL and CLIENT or SERVER in the spec is unclear.
• semantic conventions that describe logical operations (DB, messaging, LLM) document the kind

[lmolkova]

Proposal:

keep the status quo - clarify INTERNAL description:

• it should be used for strictly local operations (that are not expected to involve network calls)
• it could be used in other cases when other kinds don't apply and should not be used when another kind applies
• semantic conventions should document span kind

Alternatives:

• ?

[austinlparker]

INTERNAL vs SERVER is also controversial. Imagine a cron-job (e.g. FaaS time trigger) - it never has a parent, should it be Internal?

Well, depends on where you're modeling from. The trigger would be the client, and the function would be the server.