End-to-end demonstration of DELTA and CUMULATIVE metric export pipeline

[jmacd]

In the draft Metrics API spec update for 0.4, I've written a section on the fact that choice of cumulative- vs. delta-oriented export is significant. Any time a conversion has to be made from cumulative to delta, or from delta to cumulative, there is a memory requirement. The converter has to track the last value that was current in either case.

The use of these terms was a stumbling block for OTEP 98. Filing this issue to resolve issues there. The resolution is to avoid the terms cumulative and delta in the API documentation. The API terms will be "additive synchronous" (Counter and UpDownCounter) or "additive asynchronous" (SumObserver and UpDownSumObserver).

There is still validation to be done, to answer #143. Another way this question was phrased recently in gitter (thanks @jkwatson), was this:

Do we think that anyone in this SIG is in a position to write a full end-to-end description of how metrics would flow from API through to export, including OLTP, statsd, and prometheus, with the views API thrown into the mix? I'm less concerned with the SDK internals, and more about how the data inputs map to data outputs given the various instrument types and exporter flavors?

To close this issue, we should demonstrate an end-to-end metrics export pipeline that includes a memory-less client, OTLP transport, and collector-based exporters for statsd and prometheus that remember last-values in order to convert between delta- and cumulative- values during the export pipeline. Statsd exporters have to remember last-values in order to export deltas. Prometheus exporters have to remember last-values in order to export cumulatives.

[jmacd]

On this topic of memory requirements, here's the draft from a WIP API spec update for the 0.4 metrics API:

### Memory requirements

The API is designed not to impose long-lived memory requirements on
the user, although for some exporters it cannot be avoided.  The
potential for unbounded memory growth comes from "momentary-use"
labels, labels which are used briefly and then are not used again.
If the SDK is required to retain memory of every combination of
instrument and label set it has ever seen, long-lived memory can
become a problem.

Nevertheless, exporters may be forced to allocate long-lived memory to
perform their function, particularly with respect to the additive
instruments.  Metric exposition formats commonly have two ways of
exposing additive instrument data: in terms of the change in a sum, or
in terms of the sum itself.  (The terms "delta" and "cumulative" are
used to describe these two approaches in an export pipeline, we avoid
their use to describe the additive instruments.)

There are two cases where an exporter must retain memory in order to
operate correctly.

1. An exporter that exports sums but receives changes in the sum
2. An exporter that exports changes in a sum but receives the sum.

In both of these cases, the exporter must maintain memory of the last
value of the sum, to convert between the two representations of
additive data when both forms of input are present.

An exporter can avoid this memory requirement by supporting exposition
of additive metric data in both ways, as with OTLP.  A client
configured with OTLP for metrics export does not have a long-lived
memory requirement, because that protocol supports both forms of
additive data.  Although, the memory requirement then arises in the
downstream system for any exporter that does not support both forms.

It is tempting to consider not supporting one of these forms of
additive metric data, as a way to avoid exporter memory requirements.
However, legacy protocols exist with both forms, so this would not
necessarily help.  Besides, the fact that synchronous additive
instruments accept one, while asynchronous additive instruments
accepts the other, is ultimately meant as a user convenience.  Had
OpenTelemetry not specified that asynchronous additive instruments
accept sums directly, this burden would fall to the user (e.g., since
operating systems report heap usage as a sum, the user would be forced
into a memory requirement in order to report changes in the sum).