[ILM] Add unfollow action #36970

martijnvg · 2018-12-23T21:08:45Z

This change adds the unfollow action for CCR follower indices.

This is needed for the shrink action in case an index is a follower index.
This will give the follower index the opportunity to fully catch up with
the leader index, pause index following and unfollow the leader index.
After this the shrink action can safely perform the ilm shrink.

The unfollow action needs to be added to the hot phase and acts as
barrier for going to the next phase (warm or delete phases), so that
follower indices are being unfollowed properly before indices are expected
to go in read-only mode. This allows the force merge action to execute
its steps safely.

The unfollow action has three steps:

wait-for-indexing-complete step: waits for the index in question
to get the index.lifecycle.indexing_complete setting be set to true
wait-for-follow-shard-tasks step: waits for all the shard follow tasks
for the index being handled to report that the leader shard global checkpoint
is equal to the follower shard global checkpoint.
unfollow-index step: actually performs the unfollow. This consists
out of multiple operations being executed on the index being handled:
pause index following, close index, unfollow and open index. (a follower
index can only be unfollowed when it is closed, because the underlying
engine is changed)

In the case of the last two steps, if the index in being handled is
a regular index then the steps acts as a no-op.

Relates to #34648

This change adds the unfollow action for CCR follower indices. This is needed for the shrink action in case an index is a follower index. This will give the follower index the opportunity to fully catch up with the leader index, pause index following and unfollow the leader index. After this the shrink action can safely perform the ilm shrink. The unfollow action needs to be added to the hot phase and acts as barrier for going to the next phase (warm or delete phases), so that follower indices are being unfollowed properly before indices are expected to go in read-only mode. This allows the force merge action to execute its steps safely. The unfollow action has three steps: * `wait-for-indexing-complete` step: waits for the index in question to get the `index.lifecycle.indexing_complete` setting be set to `true` * `wait-for-follow-shard-tasks` step: waits for all the shard follow tasks for the index being handled to report that the leader shard global checkpoint is equal to the follower shard global checkpoint. * `unfollow-index` step: actually performs the unfollow. This consists out of multiple operations being executed on the index being handled: pause index following, close index, unfollow and open index. (a follower index can only be unfollowed when it is closed, because the underlying engine is changed) In the case of the last two steps, if the index in being handled is a regular index then the steps acts as a no-op. Relates to elastic#34648

elasticmachine · 2018-12-23T21:08:47Z

Pinging @elastic/es-core-features

changed WaitForIndexingComplete step to always ignore non follower indices.

gwbrown

Made some comments - some are questions, some are just to help track what still needs to be done with the understanding that this is still a WIP.

@martijnvg Don't worry about this until you get back - I'll work on this a bit like we discussed. I don't think this needs major changes, just a few tweaks/another integration test/etc.

...e/src/main/java/org/elasticsearch/xpack/core/indexlifecycle/WaitForFollowShardTasksStep.java

.../core/src/main/java/org/elasticsearch/xpack/core/indexlifecycle/WaitForIndexingComplete.java

.../src/test/java/org/elasticsearch/xpack/core/indexlifecycle/UnfollowFollowIndexStepTests.java

.../core/src/main/java/org/elasticsearch/xpack/core/indexlifecycle/UnfollowFollowIndexStep.java

gwbrown · 2018-12-27T20:47:04Z

.../test/java/org/elasticsearch/xpack/core/indexlifecycle/WaitForFollowShardTasksStepTests.java

+        Client client = Mockito.mock(Client.class);
+        List<FollowStatsAction.StatsResponse> statsResponses = Arrays.asList(
+            new FollowStatsAction.StatsResponse(createShardFollowTaskStatus(0, 9, 9)),
+            new FollowStatsAction.StatsResponse(createShardFollowTaskStatus(1, 3, 3))


These checkpoint values seems like a good candidate for randomization.

.../multi-cluster/src/test/java/org/elasticsearch/xpack/indexlifecycle/CCRIndexLifecycleIT.java

.../core/src/main/java/org/elasticsearch/xpack/core/indexlifecycle/UnfollowFollowIndexStep.java

.../src/test/java/org/elasticsearch/xpack/core/indexlifecycle/TimeseriesLifecycleTypeTests.java

This aligns WaitForIndexingCompleteStep with naming conventions

.../multi-cluster/src/test/java/org/elasticsearch/xpack/indexlifecycle/CCRIndexLifecycleIT.java

The shard follow tasks continuesly poll changes from leader index, in case no new writes arrive in the leader shards, then the leader shards wait up to the defined read_poll_timeout before returning an empty result. The result also includes information about whether the index settings need to be synced. Because in this test no data was indexed it took a full minute for ccr to realize that the followe index settings needed to be updated and caused the test to fail due to an assert busy statement to time out.

dakrone

I left some comments, thanks for doing this Martijn!

...ck/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifecycle/UnfollowAction.java

.../core/src/main/java/org/elasticsearch/xpack/core/indexlifecycle/UnfollowFollowIndexStep.java

...e/src/main/java/org/elasticsearch/xpack/core/indexlifecycle/WaitForIndexingCompleteStep.java

gwbrown · 2019-01-15T00:09:17Z

I spent a bit of time trying to track down the test failures, I saw a couple variations of failure but the one that reliably reproduces is that the test times out waiting for the IndexRoutingTable to become non-null, but I wasn't able to figure out why it never gets there. I may be able to spend some more time on this later tonight, but wanted to write this up in case I can't.

martijnvg · 2019-01-15T10:00:42Z

The reason the qa test failed, was because of a silly bug: OpenFollowerIndexStep was extending from AbstractUnfollowIndexStep instead of just extending AsyncActionStep. Because of this, the open index api call was never made (the unfollow api had removed the custom ccr metadata).

because after the unfollow step it is no longer a follower index.

gwbrown · 2019-01-15T15:58:42Z

As requested by @bmcconaghy, we're going to add the status of the indexing_complete setting to the StepInfo while waiting for it to be set as a boolean field, so that the UI can display it more easily.

...e/src/main/java/org/elasticsearch/xpack/core/indexlifecycle/WaitForIndexingCompleteStep.java

martijnvg · 2019-01-16T13:22:45Z

@elasticmachine run gradle build tests 2

gwbrown · 2019-01-16T21:59:37Z

@elasticmachine run gradle build tests 2

martijnvg · 2019-01-17T13:02:16Z

@elasticmachine run gradle build tests 1

martijnvg · 2019-01-17T13:02:22Z

@elasticmachine run gradle build tests 2

This is necessary, at least in the Hot phase, so that the priority will be set before waiting for the index to unfollow, which is much more likely what the user expects.

gwbrown · 2019-01-18T05:26:37Z

@elasticmachine run gradle build tests 2 please

gwbrown · 2019-01-18T16:53:46Z

@elasticmachine run gradle build tests 2 please

* elastic/master: (104 commits) Permission for restricted indices (elastic#37577) Remove Watcher Account "unsecure" settings (elastic#36736) Add cache cleaning task for ML snapshot (elastic#37505) Update jdk used by the docker builds (elastic#37621) Remove an unused constant in PutMappingRequest. Update get users to allow unknown fields (elastic#37593) Do not add index event listener if CCR disabled (elastic#37432) Add local session timeouts to leader node (elastic#37438) Add some deprecation optimizations (elastic#37597) refactor inner geogrid classes to own class files (elastic#37596) Remove obsolete deprecation checks (elastic#37510) ML: Add support for single bucket aggs in Datafeeds (elastic#37544) ML: creating ML State write alias and pointing writes there (elastic#37483) Deprecate types in the put mapping API. (elastic#37280) [ILM] Add unfollow action (elastic#36970) Packaging: Update marker used to allow ELASTIC_PASSWORD (elastic#37243) Fix setting openldap realm ssl config Document the need for JAVA11_HOME (elastic#37589) SQL: fix object extraction from sources (elastic#37502) Nit in settings.gradle for Eclipse ...

This change adds the unfollow action for CCR follower indices. This is needed for the shrink action in case an index is a follower index. This will give the follower index the opportunity to fully catch up with the leader index, pause index following and unfollow the leader index. After this the shrink action can safely perform the ilm shrink. The unfollow action needs to be added to the hot phase and acts as barrier for going to the next phase (warm or delete phases), so that follower indices are being unfollowed properly before indices are expected to go in read-only mode. This allows the force merge action to execute its steps safely. The unfollow action has three steps: * `wait-for-indexing-complete` step: waits for the index in question to get the `index.lifecycle.indexing_complete` setting be set to `true` * `wait-for-follow-shard-tasks` step: waits for all the shard follow tasks for the index being handled to report that the leader shard global checkpoint is equal to the follower shard global checkpoint. * `pause-follower-index` step: Pauses index following, necessary to unfollow * `close-follower-index` step: Closes the index, necessary to unfollow * `unfollow-follower-index` step: Actually unfollows the index using the CCR Unfollow API * `open-follower-index` step: Reopens the index now that it is a normal index * `wait-for-yellow` step: Waits for primary shards to be allocated after reopening the index to ensure the index is ready for the next step In the case of the last two steps, if the index in being handled is a regular index then the steps acts as a no-op. Relates to #34648 Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com> Co-authored-by: Gordon Brown <gordon.brown@elastic.co>

martijnvg added >feature WIP :Data Management/ILM+SLM Index and Snapshot lifecycle management labels Dec 23, 2018

martijnvg added 3 commits December 24, 2018 11:11

Replaced ccr / ilm integ test with multi cluster integ test and

f9f530e

changed WaitForIndexingComplete step to always ignore non follower indices.

Fixed tests

80bb2ef

Added more tests

a265c2a

gwbrown reviewed Dec 27, 2018

View reviewed changes

gwbrown assigned martijnvg and gwbrown Dec 27, 2018

gwbrown reviewed Dec 27, 2018

View reviewed changes

.../core/src/main/java/org/elasticsearch/xpack/core/indexlifecycle/UnfollowFollowIndexStep.java Outdated Show resolved Hide resolved

colings86 mentioned this pull request Dec 28, 2018

[ILM] Allow ILM and CCR to work well together #34648

Closed

3 tasks

gwbrown reviewed Jan 2, 2019

View reviewed changes

.../src/test/java/org/elasticsearch/xpack/core/indexlifecycle/TimeseriesLifecycleTypeTests.java Show resolved Hide resolved

gwbrown added 8 commits January 3, 2019 11:10

Rename class to WaitForIndexingCompleteStep

e281b03

This aligns WaitForIndexingCompleteStep with naming conventions

Factor "ccr" out into constant

07f0136

Add note about leader index to error message

509718c

Factor out mocking into methods

974bb8b

Merge branch 'master' into pr/36970

97df36b

Fix error message in test

37bd0ef

[WIP] Add rollover-based test case

94dc951

Merge branch 'master' into pr/36970

e996f83

gwbrown reviewed Jan 5, 2019

View reviewed changes

.../multi-cluster/src/test/java/org/elasticsearch/xpack/indexlifecycle/CCRIndexLifecycleIT.java Outdated Show resolved Hide resolved

martijnvg added 2 commits January 7, 2019 13:29

Adjusted testGetNextActionName() test

0ef7002

dakrone requested changes Jan 8, 2019

View reviewed changes

gwbrown added 4 commits January 8, 2019 13:35

Merge branch 'master' into pr/36970

6bb4055

More informative message when following is paused

2d368ab

Remove index settings from step info

7485773

Javadoc for UnfollowAction

d96ebbb

Merge remote-tracking branch 'es/master' into ccr_ilm_follower_indices

7f521e8

OpenFollowerIndexStep cannot extend from AbstractUnfollowIndexStep,

5d3d661

because after the unfollow step it is no longer a follower index.

gwbrown added 2 commits January 15, 2019 12:27

Add indexing_complete setting to Wait step info

f7e74d4

Added Unfollow to the Actions docs ToC

7c1749b

dakrone reviewed Jan 15, 2019

View reviewed changes

...e/src/main/java/org/elasticsearch/xpack/core/indexlifecycle/WaitForIndexingCompleteStep.java Outdated Show resolved Hide resolved

dakrone reviewed Jan 15, 2019

View reviewed changes

...e/src/main/java/org/elasticsearch/xpack/core/indexlifecycle/WaitForIndexingCompleteStep.java Outdated Show resolved Hide resolved

gwbrown and others added 2 commits January 15, 2019 16:15

Switch Info to hardcoded false

fcf34e6

fixed checkstyle violation

c02f8cb

Merge remote-tracking branch 'es/master' into ccr_ilm_follower_indices

a9b2663

gwbrown added 4 commits January 17, 2019 11:28

Merge branch 'master' into pr/36970

bb67cdb

Fix a couple tests

3ee6298

Run SetPriority before Unfollow

f1c05bd

This is necessary, at least in the Hot phase, so that the priority will be set before waiting for the index to unfollow, which is much more likely what the user expects.

Merge branch 'master' into pr/36970

ca0d2cc

gwbrown merged commit a3030c5 into elastic:master Jan 18, 2019

gwbrown added the backport pending label Jan 18, 2019

gwbrown mentioned this pull request Jan 19, 2019

Inject Unfollow before Rollover and Shrink #37625

Merged

gwbrown removed the backport pending label Jan 22, 2019

jimczi added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ILM] Add unfollow action #36970

[ILM] Add unfollow action #36970

martijnvg commented Dec 23, 2018 •

edited

Loading

elasticmachine commented Dec 23, 2018

gwbrown left a comment •

edited

Loading

gwbrown Dec 27, 2018

dakrone left a comment

gwbrown commented Jan 15, 2019

martijnvg commented Jan 15, 2019

gwbrown commented Jan 15, 2019

martijnvg commented Jan 16, 2019

gwbrown commented Jan 16, 2019

martijnvg commented Jan 17, 2019

martijnvg commented Jan 17, 2019

gwbrown commented Jan 18, 2019

gwbrown commented Jan 18, 2019

[ILM] Add unfollow action #36970

[ILM] Add unfollow action #36970

Conversation

martijnvg commented Dec 23, 2018 • edited Loading

elasticmachine commented Dec 23, 2018

gwbrown left a comment • edited Loading

Choose a reason for hiding this comment

gwbrown Dec 27, 2018

Choose a reason for hiding this comment

dakrone left a comment

Choose a reason for hiding this comment

gwbrown commented Jan 15, 2019

martijnvg commented Jan 15, 2019

gwbrown commented Jan 15, 2019

martijnvg commented Jan 16, 2019

gwbrown commented Jan 16, 2019

martijnvg commented Jan 17, 2019

martijnvg commented Jan 17, 2019

gwbrown commented Jan 18, 2019

gwbrown commented Jan 18, 2019

martijnvg commented Dec 23, 2018 •

edited

Loading

gwbrown left a comment •

edited

Loading