Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ILM] Add unfollow action #36970

Merged
merged 46 commits into from
Jan 18, 2019
Merged

Commits on Dec 23, 2018

  1. [ILM] Add unfollow action

    This change adds the unfollow action for CCR follower indices.
    
    This is needed for the shrink action in case an index is a follower index.
    This will give the follower index the opportunity to fully catch up with
    the leader index, pause index following and unfollow the leader index.
    After this the shrink action can safely perform the ilm shrink.
    
    The unfollow action needs to be added to the hot phase and acts as
    barrier for going to the next phase (warm or delete phases), so that
    follower indices are being unfollowed properly before indices are expected
    to go in read-only mode. This allows the force merge action to execute
    its steps safely.
    
    The unfollow action has three steps:
    * `wait-for-indexing-complete` step: waits for the index in question
      to get the `index.lifecycle.indexing_complete` setting be set to `true`
    * `wait-for-follow-shard-tasks` step: waits for all the shard follow tasks
      for the index being handled to report that the leader shard global checkpoint
      is equal to the follower shard global checkpoint.
    * `unfollow-index` step: actually performs the unfollow. This consists
      out of multiple operations being executed on the index being handled:
      pause index following, close index, unfollow and open index. (a follower
      index can only be unfollowed when it is closed, because the underlying
      engine is changed)
    
    In the case of the last two steps, if the index in being handled is
    a regular index then the steps acts as a no-op.
    
    Relates to elastic#34648
    martijnvg committed Dec 23, 2018
    Configuration menu
    Copy the full SHA
    c0fc17e View commit details
    Browse the repository at this point in the history

Commits on Dec 24, 2018

  1. Replaced ccr / ilm integ test with multi cluster integ test and

    changed WaitForIndexingComplete step to always ignore non follower indices.
    martijnvg committed Dec 24, 2018
    Configuration menu
    Copy the full SHA
    f9f530e View commit details
    Browse the repository at this point in the history
  2. Fixed tests

    martijnvg committed Dec 24, 2018
    Configuration menu
    Copy the full SHA
    80bb2ef View commit details
    Browse the repository at this point in the history
  3. Added more tests

    martijnvg committed Dec 24, 2018
    Configuration menu
    Copy the full SHA
    a265c2a View commit details
    Browse the repository at this point in the history

Commits on Jan 3, 2019

  1. Rename class to WaitForIndexingCompleteStep

    This aligns WaitForIndexingCompleteStep with naming conventions
    gwbrown committed Jan 3, 2019
    Configuration menu
    Copy the full SHA
    e281b03 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    07f0136 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    509718c View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    974bb8b View commit details
    Browse the repository at this point in the history

Commits on Jan 4, 2019

  1. Configuration menu
    Copy the full SHA
    97df36b View commit details
    Browse the repository at this point in the history
  2. Fix error message in test

    gwbrown committed Jan 4, 2019
    Configuration menu
    Copy the full SHA
    37bd0ef View commit details
    Browse the repository at this point in the history

Commits on Jan 5, 2019

  1. Configuration menu
    Copy the full SHA
    94dc951 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e996f83 View commit details
    Browse the repository at this point in the history

Commits on Jan 7, 2019

  1. Decreased read_poll_timeout from 60 seconds to 1 second.

    The shard follow tasks continuesly poll changes from leader index,
    in case no new writes arrive in the leader shards, then the leader
    shards wait up to the defined read_poll_timeout before returning an
    empty result. The result also includes information about whether
    the index settings need to be synced.
    
    Because in this test no data was indexed it took a full minute for ccr
    to realize that the followe index settings needed to be updated and
    caused the test to fail due to an assert busy statement to time out.
    martijnvg committed Jan 7, 2019
    Configuration menu
    Copy the full SHA
    3949445 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0ef7002 View commit details
    Browse the repository at this point in the history

Commits on Jan 8, 2019

  1. Configuration menu
    Copy the full SHA
    6bb4055 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2d368ab View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    7485773 View commit details
    Browse the repository at this point in the history
  4. Javadoc for UnfollowAction

    gwbrown committed Jan 8, 2019
    Configuration menu
    Copy the full SHA
    d96ebbb View commit details
    Browse the repository at this point in the history
  5. UnfollowAction hashcode

    gwbrown committed Jan 8, 2019
    Configuration menu
    Copy the full SHA
    464c256 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    5d63a5a View commit details
    Browse the repository at this point in the history

Commits on Jan 9, 2019

  1. added assertion messages

    martijnvg committed Jan 9, 2019
    Configuration menu
    Copy the full SHA
    f63e73d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    deac47a View commit details
    Browse the repository at this point in the history
  3. Moved pause follower index, close follower index and

    open follower index operation from UnfollowFollowerIndexStep
    to the following steps respectively:
    * PauseFollowerIndexStep
    * CloseFollowerIndexStep
    * OpenFollowerIndexStep
    
    The UnfollowAction now exists of the following steps:
    * WaitForIndexingCompleteStep
    * WaitForFollowShardStep
    * PauseFollowerIndexStep
    * CloseFollowerIndexStep
    * UnfollowFollowerIndexStep
    * OpenFollowerIndexStep
    martijnvg committed Jan 9, 2019
    Configuration menu
    Copy the full SHA
    886a3f5 View commit details
    Browse the repository at this point in the history
  4. Remove unused imports

    gwbrown committed Jan 9, 2019
    Configuration menu
    Copy the full SHA
    2b60e54 View commit details
    Browse the repository at this point in the history

Commits on Jan 10, 2019

  1. Configuration menu
    Copy the full SHA
    6d0e236 View commit details
    Browse the repository at this point in the history

Commits on Jan 11, 2019

  1. Configuration menu
    Copy the full SHA
    533bc4f View commit details
    Browse the repository at this point in the history
  2. Add Unfollow action to HLRC

    gwbrown committed Jan 11, 2019
    Configuration menu
    Copy the full SHA
    731d8d2 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    67f970e View commit details
    Browse the repository at this point in the history
  4. added wait for yellow step. That will run after open index step,

    so that after unfollow is done then the unfollowed index is ready
    to handle requests.
    martijnvg committed Jan 11, 2019
    Configuration menu
    Copy the full SHA
    272b8d6 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    ea93eda View commit details
    Browse the repository at this point in the history

Commits on Jan 14, 2019

  1. Configuration menu
    Copy the full SHA
    e9b252e View commit details
    Browse the repository at this point in the history
  2. docs iter

    martijnvg committed Jan 14, 2019
    Configuration menu
    Copy the full SHA
    0416490 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    443fc4a View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    5abf557 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    caa13ce View commit details
    Browse the repository at this point in the history

Commits on Jan 15, 2019

  1. Configuration menu
    Copy the full SHA
    7f521e8 View commit details
    Browse the repository at this point in the history
  2. OpenFollowerIndexStep cannot extend from AbstractUnfollowIndexStep,

    because after the unfollow step it is no longer a follower index.
    martijnvg committed Jan 15, 2019
    Configuration menu
    Copy the full SHA
    5d3d661 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    f7e74d4 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    7c1749b View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    fcf34e6 View commit details
    Browse the repository at this point in the history

Commits on Jan 16, 2019

  1. fixed checkstyle violation

    martijnvg committed Jan 16, 2019
    Configuration menu
    Copy the full SHA
    c02f8cb View commit details
    Browse the repository at this point in the history

Commits on Jan 17, 2019

  1. Configuration menu
    Copy the full SHA
    a9b2663 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    bb67cdb View commit details
    Browse the repository at this point in the history
  3. Fix a couple tests

    gwbrown committed Jan 17, 2019
    Configuration menu
    Copy the full SHA
    3ee6298 View commit details
    Browse the repository at this point in the history

Commits on Jan 18, 2019

  1. Run SetPriority before Unfollow

    This is necessary, at least in the Hot phase, so that the priority will
    be set before waiting for the index to unfollow, which is much more
    likely what the user expects.
    gwbrown committed Jan 18, 2019
    Configuration menu
    Copy the full SHA
    f1c05bd View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ca0d2cc View commit details
    Browse the repository at this point in the history