add support for explicitly disabling sending to APM server

[trentm]

Use case: Use the Node.js APM agent for async context tracking (via its transactions), but explicitly not have an APM server setup/configured to which to send trace data. Some things that the user can get out of this (even though transactions/spans/metrics/errors are not being sent to APM server) are:

• handle distributed tracing headers (traceparent, tracestate, eventually perhaps baggage),
• handle adding trace.id to logging records to correlate logs within the service and across services for distributed traces,
• provide the async context tracking to allow storing context from an incoming request and retrieving that context to use in outgoing requests.

The latter is a possible use case for Kibana (from elastic/kibana#97934) to be able to track the source of a Kibana action (called "execution context" on that issue; examples are Kibana front-end calls to the Kibana back-end API, other non-Kibana calls to the API, internal Kibana background tasks) through to an Elasticsearch slowlogs entry.

This would be a third run mode for the agent, separate from:

• active: false - which means the agent is not running at all, packages have not been instrumented, async context tracking is not being done
• recording: false - which means the agent is minimally running, packages have been instrumented, async context tracking is not being done, central config is being polled to allow recording to be turned back on. Note that the Node.js APM agent currently doesn't yet support the recording mode. (Java agent docs, Python agent docs)

Since v3.14.0, an active agent will work without an APM server. However, not nicely:

• It will log.error every 30s or so (depends on events being generated and a backoff time strategy) that it cannot contact APM server.
• It will serialize and buffer up to maxQueueSize events in memory.

The goal of this mode is to make the state explicit (no log.erroring) and limit overhead to a minimum.

[trentm]

Ideas for the configuration var for this:

1. serverUrl = "none" Given other APM agents are unlikely to implement functionality like this, having special handling for a shared config var seems unhelpful. Also, in general special inline sentinel values can be surprising and error-prone.
2. sending = true|false Feels clearer and somewhat fits with the existing enabled and recording, but I don't love that grepping for "sending" gets lots of false positives. Also it doesn't make clear that using APM server for central config is also out.
3. useServer = true|false Groups well with the other /server/i config vars.
4. ... something else?

Ideas welcome.

[trentm]

In Python, we could easily write a processor that just drops all
spans. That's what I would tell a user to do if they asked for this
feature. -- Colton

As a quick PoC hack one could configure the Node.js APM agent to do the same via apm.addFilter().

[trentm]

Eyal and Felix pointed out that

Java, Python, and Ruby have the disable_send config option: https://www.elastic.co/guide/en/apm/agent/java/current/config-reporter.html#config-disable-send
But we have never fully aligned or specified the behavior cross-agent.

This sounds like (almost?) exactly what we want here. If so, then we should use the equivalent config name.

[trentm]

Some related comments at elastic/kibana#101587 (comment)