Add new Sampler and SpanProcessor to allow for generating metrics from 100% of spans without impacting sampling

[thpierce]

Is your feature request related to a problem? Please describe.
Yes. We wish to generate key metrics from OTEL auto-instrumented spans. I had previously engaged the community to discuss this, and after some conversations, decided that the best approach would be to implement a Sampler and a Span Processor to the contrib repositories.

The major requirements for our project are as follows:

• Metrics should be produced from 100% of Client/Server/Producer/Consumer spans produced by OTEL Auto-Instrumentation, regardless of sampling decision
  • Span sampling decisions should still be respected w.r.t. exporting spans to the collector and propagating sampling decisions to child spans
• Metric generation must be done when the span is ended, as metric attributes are derived from some span attributes that are not always available when the span is started (e.g. http.route).
• Metric attributes need to be added to the span attributes before export to Collector, to be used to correlate metrics and spans in a downstream process.

I am raising this issue to garner attention for this change before I raise a PR, and perhaps run conversations in parallel with implementation, which is ongoing. Please note there may be some differences in the final implementation, but the broad strokes should remain the same.

Describe the solution you'd like

All contributions would go to opentelemetry-java-contrib/aws-xray/src/main/java/io/opentelemetry/contrib/awsxray.

• First, a new Sampler, tentatively called AlwaysRecordSampler.
  • This is an aggregate Sampler that is initialized with a root Sampler (akin to ParentBasedSampler).
  • shouldSample() will call the root sampler and, if the SampleDecision is RECORD_AND_SAMPLE or RECORD_ONLY, it simply returns the result. However, if the SampleDecision is DROP, it will instead create a new result with SampleDecision set to RECORD_ONLY (all other result behaviour unchanged).
  • This Sampler will ensure that 100% of spans get send to the SpanProcessor, without changing the actual Sampling rate/behaviour.
• Second, a new SpanProcessor, tentatively called AwsSpanMetricsProcessor.
  • This SpanProcessor is initialized with a MeterProvider and a downstream SpanProcessor (akin to the MultiSpanProcessor)
  • All APIs but onEnd() and isEndRequired() will passthrough to the downstream SpanProcessor, as needed.
  • isEndRequired will return true, and onEnd() will:
    • Determine attributes for metrics
    • Emit metrics with attributes
    • If the span is marked for Sampling,
      • Replace span with a new ReadableSpan identical to the original span, but with metric attributes embedded into span attributes.
    • Call downstreamSpanProcessor.onEnd() with span, if required.
  • This SpanProcessor will create desired metrics and embed attributes into processed spans before sending them to the downstream SpanProcessor (for additional processing/export).

Describe alternatives you've considered

We have evaluated a couple different options here:

• Always exporting spans, regardless of sample decision, implementing a collector SpanProcessor to emit metrics, then drop anything not marked as Sampled. We felt this did not align well with the OTEL specification & would result in heavy traffic to the collector, as well as introduce backwards incompatibility risks for the customer
• Implement the SpanProcessor as a SpanExporter. The trouble here is that SpanExporters are not supposed to receive RECORD_ONLY spans, and no default SpanProcessor offers that functionality, so we would end up having to write a SpanProcessor anyways. No major value was seen in this approach.

Additional context

• Originally discussed here: Introduce means of producing metrics from all Spans regardless of sampling decision. opentelemetry-specification#3145
• Reviewed approach with @willarmiros, component owner for aws-xray.

[willarmiros]

+1, I support this component's implementation in the X-ray contrib.

[thpierce]

One thing we are evaluating is: where exactly should these components go? We originally thought they would live in aws-xray, but AlwaysRecordSampler is fairly generic - is there a better package that they could fall under? Happy to take guidance here.

[trask]

@thpierce is this issue closed by #802?

[thpierce]

Yes! Resolving now, ty for help getting these merged.