Add failure handling to ingest pipelines.

[simitt]

Avoid unexpected infinite retries on pipeline failures due to internal callback handling, by defining failure handling on pipelines.

For the geoip pipeline the client.ip field will be removed on failure. The field has an ip mapping in Elasticsearch. Therefore ingesting the document in ES would fail, even if the pipeline error is ignored in case of a malformed IP address. Dropping the malformed field, ensures the document can be indexed. Information in client.ip is always duplicated from some other field, therefore no original information is lost. While there is some pre-processing in place to avoid malformed IP addresses, this is the only case that comes to my mind that could make the pipeline fail.

For the user_agent pipeline failures are simply ignored. The user_agent.original field is indexed as a keyword and therefore less susceptible to ingestion failure on malformed user_agent information, and the failure handling only ensures edge cases are covered and unexpected ingestion issues avoided.

closes #2880

[codecov-io]

Codecov Report

Merging #2896 into master will decrease coverage by 0.46%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2896      +/-   ##
==========================================
- Coverage   79.73%   79.26%   -0.47%     
==========================================
  Files          89       83       -6     
  Lines        4830     4355     -475     
==========================================
- Hits         3851     3452     -399     
+ Misses        979      903      -76

Impacted Files	Coverage Δ
beater/beater.go	69.66% <0%> (-8.35%)	⬇️
beater/config/config.go	76.66% <0%> (-5.1%)	⬇️
sourcemap/mapper.go	87.23% <0%> (-3.4%)	⬇️
model/stacktrace_frame.go	98.36% <0%> (-1.64%)	⬇️
beater/api/mux.go	71.96% <0%> (-0.01%)	⬇️
sourcemap/store.go
beater/config/rum.go
beater/config/mode.go
elasticsearch/client.go
sourcemap/es_store.go
... and 3 more