Reindex search resiliency prototype #43187
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit is just the (first) integTest for validating that reindex can survive data node restarts.
henningandersen
changed the title
This cements the SPI to be the individual operations and the capability to convert the results and failures to common format.
ywelsch
reviewed
Jun 18, 2019
|
|
||
| ensureGreen("test"); | ||
| String reindexNode = internalCluster().startCoordinatingOnlyNode(Settings.EMPTY); | ||
| NodeClient reindexNodeClient = internalCluster().getInstance(NodeClient.class, reindexNode); |
There was a problem hiding this comment.
internalCluster().client(reindexNode) should be simpler
There was a problem hiding this comment.
The reason this is done like this is to get the NodeClient since it has executeLocally, which returns a Task object.
|
|
||
| private void assertSameDocs(int numberOfDocuments, String... indices) { | ||
| refresh(indices); | ||
| SearchSourceBuilder sourceBuilder = SearchSourceBuilder.searchSource().size(0) |
There was a problem hiding this comment.
perhaps just do a count or use a cardinality aggregation on the "data" field? No need for scripting I think.
| } else if (retryFromRequest.source().query() == null) { | ||
| retryFromRequest.source().query(rangeQueryBuilder); | ||
| } else { | ||
| throw new UnsupportedOperationException("not yet implemented"); |
There was a problem hiding this comment.
perhaps wrap original query using BoolQueryBuilder where the rangeQueryBuilder is added as filter and the original query as must.
and add seq_no range as filter.
…rch_resiliency_poc
henningandersen
added a commit
to henningandersen/elasticsearch
that referenced
this pull request
Jul 2, 2019
Refactor ScrollableHitSource to pump data out and have a simplified interface (callers should no longer call startNextScroll, instead they simply mark that they are done with the previous result, triggering a new batch of data). This eases making reindex resilient, since we will sometimes need to rerun search during retries. Relates elastic#43187 and elastic#42612
henningandersen
added a commit
that referenced
this pull request
Jul 9, 2019
Refactor ScrollableHitSource to pump data out and have a simplified interface (callers should no longer call startNextScroll, instead they simply mark that they are done with the previous result, triggering a new batch of data). This eases making reindex resilient, since we will sometimes need to rerun search during retries. Relates #43187 and #42612
henningandersen
added a commit
that referenced
this pull request
Jul 9, 2019
Refactor ScrollableHitSource to pump data out and have a simplified interface (callers should no longer call startNextScroll, instead they simply mark that they are done with the previous result, triggering a new batch of data). This eases making reindex resilient, since we will sometimes need to rerun search during retries. Relates #43187 and #42612
henningandersen
added a commit
to henningandersen/elasticsearch
that referenced
this pull request
Aug 13, 2019
Local reindex can now survive loosing data nodes that contain source data. The original query will be restarted with a filter for `_seq_no >= last_seq_no` when a failure is detected. Part of elastic#42612 and split out from elastic#43187
henningandersen
added a commit
that referenced
this pull request
Aug 19, 2019
Local reindex can now survive loosing data nodes that contain source data. The original query will be restarted with a filter for `_seq_no >= last_seq_no` when a failure is detected. The original/first search request is not retried/restarted since this is not what we used to do and it leads to long wait times to get the info back that a search request is bad. Part of #42612 and split out from #43187
|
This is all covered in split out PRs, mainly #45497 so closing this. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains a prototype for ensuring that reindex searching can survive data node restarts and/or related network issues.
Primarily created this PR to collaborate internally.
Most tests only modified to compile.
ReindexResilientSearchITchecks that resiliency works.A large part of the change is changing the retry handling to now be handled by
ScrollableHitSourceinstead of the two subclasses. This ensures it is handled the same way and made it easier to restart the query from scratch. The subclasses main responsibility is not to execute one search or scroll request and convert the result to common format.Notable questions:
SearchRequest_seq_no >= retryFromValuecondition if there is an existing condition.ThreadContexthandling may not be complete, though I would expect transport and threadPool to handle this automatically?PRs to split out:
ScrollableHitSource