Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ensure pipeline errors do not cause infinite ingest retries #2880

Closed
simitt opened this issue Oct 31, 2019 · 2 comments · Fixed by #2896
Closed

Ensure pipeline errors do not cause infinite ingest retries #2880

simitt opened this issue Oct 31, 2019 · 2 comments · Fixed by #2896
Assignees
@simitt
Milestone
7.5

Comments

@simitt
Copy link
Contributor

simitt commented Oct 31, 2019 •
edited
Loading

The APM Server currently uses a pipeline with two processors, where no on_failure or ignore_failure is defined. In case a processor fails on an event, ES will return a status code 500 leading to retries of indexing the failed events in the APM Server. See libbeat implementation.
Since the APM Server uses GuaranteedSend, the retry results in an infinite loop, see retry implementation.

Ensure to user proper error handling on pipelines to avoid an infinite retry loop.
@urso urso mentioned this issue Nov 1, 2019
Closed
@simitt
Copy link
Contributor Author

simitt commented Nov 1, 2019

related elastic/elasticsearch#48803

@graphaelli
Copy link
Member

Let's go ahead with the ignore_failure patch for master & 7.5 and track the linked elaticsearch issue to see if it makes the next minor, and revert going forward at that point.

@graphaelli graphaelli added this to the 7.5 milestone Nov 12, 2019
@simitt simitt self-assigned this Nov 12, 2019
simitt added a commit to simitt/apm-server that referenced this issue Nov 13, 2019
@simitt
Add failure handling to ingest pipelines.
b5f6bb6 
Avoid unexpected infinite retries on pipeline failures due to internal
callback handling, by defining failure handling on pipelines.

closes elastic#2880
@simitt simitt mentioned this issue Nov 13, 2019
Merged
simitt added a commit that referenced this issue Nov 13, 2019
@simitt
Add failure handling to ingest pipelines. (#2896)
c21d3f8 
Avoid unexpected infinite retries on pipeline failures due to internal
callback handling, by defining failure handling on pipelines.

closes #2880
simitt added a commit to simitt/apm-server that referenced this issue Nov 13, 2019
@simitt
Add failure handling to ingest pipelines. (elastic#2896)
b00e6c4 
Avoid unexpected infinite retries on pipeline failures due to internal
callback handling, by defining failure handling on pipelines.

closes elastic#2880
simitt added a commit to simitt/apm-server that referenced this issue Nov 13, 2019
@simitt
Add failure handling to ingest pipelines. (elastic#2896)
daed5d5 
Avoid unexpected infinite retries on pipeline failures due to internal
callback handling, by defining failure handling on pipelines.

closes elastic#2880
simitt added a commit that referenced this issue Nov 13, 2019
@simitt
Add failure handling to ingest pipelines. (#2896) (#2900)
5fde973 
Avoid unexpected infinite retries on pipeline failures due to internal
callback handling, by defining failure handling on pipelines.

closes #2880
simitt added a commit that referenced this issue Nov 13, 2019
@simitt
Add failure handling to ingest pipelines. (#2896) (#2901)
67192f9 
Avoid unexpected infinite retries on pipeline failures due to internal
callback handling, by defining failure handling on pipelines.

closes #2880
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@simitt simitt

Labels
None yet
Projects
None yet
Milestone
7.5
Development

Successfully merging a pull request may close this issue.

Add failure handling to ingest pipelines. simitt/apm-server
2 participants
@graphaelli @simitt