-
Notifications
You must be signed in to change notification settings - Fork 24.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Snapshot Lifecycle Management #43934
Commits on Mar 12, 2019
-
Add SnapshotLifecycleService and related CRUD APIs
This commit adds `SnapshotLifecycleService` as a new service under the ilm plugin. This service handles snapshot lifecycle policies by scheduling based on the policies defined schedule. This also includes the get, put, and delete APIs for these policies Relates to #38461
Configuration menu - View commit details
-
Copy full SHA for 960041e - Browse repository at this point
Copy the full SHA 960041eView commit details -
Configuration menu - View commit details
-
Copy full SHA for acb8135 - Browse repository at this point
Copy the full SHA acb8135View commit details -
Configuration menu - View commit details
-
Copy full SHA for d7b41e0 - Browse repository at this point
Copy the full SHA d7b41e0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0043de4 - Browse repository at this point
Copy the full SHA 0043de4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 54ac17c - Browse repository at this point
Copy the full SHA 54ac17cView commit details -
Configuration menu - View commit details
-
Copy full SHA for c5f7738 - Browse repository at this point
Copy the full SHA c5f7738View commit details
Commits on Mar 13, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 06a6a13 - Browse repository at this point
Copy the full SHA 06a6a13View commit details
Commits on Mar 14, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 25ebdf3 - Browse repository at this point
Copy the full SHA 25ebdf3View commit details
Commits on Mar 15, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 6d0aad5 - Browse repository at this point
Copy the full SHA 6d0aad5View commit details
Commits on Mar 18, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 4ba731a - Browse repository at this point
Copy the full SHA 4ba731aView commit details -
Handle snapshot lifecycle policy updates and deletions (#40062)
(Note this is a PR against the `snapshot-lifecycle-management` feature branch) This adds logic to `SnapshotLifecycleService` to handle updates and deletes for snapshot policies. Policies with incremented versions have the old policy cancelled and the new one scheduled. Deleted policies have their schedules cancelled when they are no longer present in the cluster state metadata. Relates to #38461
Configuration menu - View commit details
-
Copy full SHA for 1738b9b - Browse repository at this point
Copy the full SHA 1738b9bView commit details
Commits on Mar 21, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 6837354 - Browse repository at this point
Copy the full SHA 6837354View commit details
Commits on Mar 22, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 817dd41 - Browse repository at this point
Copy the full SHA 817dd41View commit details
Commits on Mar 26, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 8095a32 - Browse repository at this point
Copy the full SHA 8095a32View commit details -
Take a snapshot for the policy when the SLM policy is triggered (#40383)
(This is a PR for the `snapshot-lifecycle-management` branch) This commit fills in `SnapshotLifecycleTask` to actually perform the snapshotting when the policy is triggered. Currently there is no handling of the results (other than logging) as that will be added in subsequent work. This also adds unit tests and an integration test that schedules a policy and ensures that a snapshot is correctly taken. Relates to #38461
Configuration menu - View commit details
-
Copy full SHA for 5b97fff - Browse repository at this point
Copy the full SHA 5b97fffView commit details
Commits on Mar 28, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 1f5811b - Browse repository at this point
Copy the full SHA 1f5811bView commit details
Commits on Apr 1, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 4321b24 - Browse repository at this point
Copy the full SHA 4321b24View commit details
Commits on Apr 2, 2019
-
Configuration menu - View commit details
-
Copy full SHA for c0b2a70 - Browse repository at this point
Copy the full SHA c0b2a70View commit details -
Record most recent snapshot policy success/failure (#40619)
Keeping a record of the results of the successes and failures will aid troubleshooting of policies and make users more confident that their snapshots are being taken as expected. This is the first step toward writing history in a more permanent fashion.
Configuration menu - View commit details
-
Copy full SHA for 51d7f0e - Browse repository at this point
Copy the full SHA 51d7f0eView commit details
Commits on Apr 3, 2019
-
Validate snapshot lifecycle policies (#40654)
(This is a PR against the `snapshot-lifecycle-management` branch) With the commit, we now validate the content of snapshot lifecycle policies when the policy is being created or updated. This checks for the validity of the id, name, schedule, and repository. Additionally, cluster state is checked to ensure that the repository exists prior to the lifecycle being added to the cluster state. Part of #38461
Configuration menu - View commit details
-
Copy full SHA for a53da7d - Browse repository at this point
Copy the full SHA a53da7dView commit details
Commits on Apr 4, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 3a19a30 - Browse repository at this point
Copy the full SHA 3a19a30View commit details
Commits on Apr 8, 2019
-
Hook SLM into ILM's start and stop APIs (#40871)
(This pull request is for the `snapshot-lifecycle-management` branch) This change allows the existing `/_ilm/stop` and `/_ilm/start` APIs to also manage snapshot lifecycle scheduling. When ILM is stopped all scheduled jobs are cancelled. Relates to #38461
Configuration menu - View commit details
-
Copy full SHA for 68b4387 - Browse repository at this point
Copy the full SHA 68b4387View commit details -
Add tests for SnapshotLifecyclePolicyItem (#40912)
Adds serialization tests for SnapshotLifecyclePolicyItem.
Configuration menu - View commit details
-
Copy full SHA for 9074c08 - Browse repository at this point
Copy the full SHA 9074c08View commit details -
Configuration menu - View commit details
-
Copy full SHA for 06e616a - Browse repository at this point
Copy the full SHA 06e616aView commit details
Commits on Apr 9, 2019
-
Configuration menu - View commit details
-
Copy full SHA for e8d40d8 - Browse repository at this point
Copy the full SHA e8d40d8View commit details -
Configuration menu - View commit details
-
Copy full SHA for bddbd00 - Browse repository at this point
Copy the full SHA bddbd00View commit details
Commits on Apr 10, 2019
-
Configuration menu - View commit details
-
Copy full SHA for d4fab06 - Browse repository at this point
Copy the full SHA d4fab06View commit details
Commits on Apr 11, 2019
-
Add human readable version of modified date for snapshot lifecycle po…
…licy (#41035) * Add human readable version of modified date for snapshot lifecycle policy This small change changes it from: ``` ... "modified_date": 1554843903242, ... ``` To ``` ... "modified_date" : "2019-04-09T21:05:03.242Z", "modified_date_millis" : 1554843903242, ... ``` Including the `"modified_date"` field when the `?human` field is used. Relates to #38461 * Fix test
Configuration menu - View commit details
-
Copy full SHA for e564b48 - Browse repository at this point
Copy the full SHA e564b48View commit details
Commits on Apr 12, 2019
-
Add API to execute SLM policy on demand (#41038)
This commit adds the ability to perform a snapshot on demand for a policy. This can be useful to take a snapshot immediately prior to performing some sort of maintenance. ```json PUT /_ilm/snapshot/<policy>/_execute ``` And it returns the response with the generated snapshot name: ```json { "snapshot_name" : "production-snap-2019.04.09-rfyv3j9qreixkdbnfuw0ug" } ``` Note that this does not allow waiting for the snapshot, and the snapshot could still fail. It *does* record this information into the cluster state similar to a regularly trigged SLM job. Relates to #38461
Configuration menu - View commit details
-
Copy full SHA for 1b4574a - Browse repository at this point
Copy the full SHA 1b4574aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5797aaa - Browse repository at this point
Copy the full SHA 5797aaaView commit details
Commits on Apr 15, 2019
-
Configuration menu - View commit details
-
Copy full SHA for ead7146 - Browse repository at this point
Copy the full SHA ead7146View commit details
Commits on Apr 16, 2019
-
Add next_execution to SLM policy metadata (#41221)
* Add next_execution to SLM policy metadata This adds the next time a snapshot lifecycle policy will be executed when retriving a policy's metadata, for example: ```json GET /_ilm/snapshot?human { "production" : { "version" : 1, "modified_date" : "2019-04-15T21:16:21.865Z", "modified_date_millis" : 1555362981865, "policy" : { "name" : "<production-snap-{now/d}>", "schedule" : "*/30 * * * * ?", "repository" : "repo", "config" : { "indices" : [ "foo-*", "important" ], "ignore_unavailable" : true, "include_global_state" : false } }, "next_execution" : "2019-04-15T21:16:30.000Z", "next_execution_millis" : 1555362990000 }, "other" : { "version" : 1, "modified_date" : "2019-04-15T21:12:19.959Z", "modified_date_millis" : 1555362739959, "policy" : { "name" : "<other-snap-{now/d}>", "schedule" : "0 30 2 * * ?", "repository" : "repo", "config" : { "indices" : [ "other" ], "ignore_unavailable" : false, "include_global_state" : true } }, "next_execution" : "2019-04-16T02:30:00.000Z", "next_execution_millis" : 1555381800000 } } ``` Relates to #38461 * Fix and enhance tests * Figured out how to Cron
Configuration menu - View commit details
-
Copy full SHA for 7ea37fe - Browse repository at this point
Copy the full SHA 7ea37feView commit details
Commits on Apr 22, 2019
-
Change SLM endpoint from /_ilm/* to /_slm/* (#41320)
This commit changes the endpoint for snapshot lifecycle management from: ``` GET /_ilm/snapshot/<policy> ``` to: ``` GET /_slm/policy/<policy> ``` It mimics the ILM path only using `slm` instead of `ilm`. Relates to #38461
Configuration menu - View commit details
-
Copy full SHA for d6de587 - Browse repository at this point
Copy the full SHA d6de587View commit details -
Configuration menu - View commit details
-
Copy full SHA for a7d18da - Browse repository at this point
Copy the full SHA a7d18daView commit details
Commits on Apr 24, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 5736be6 - Browse repository at this point
Copy the full SHA 5736be6View commit details
Commits on Apr 26, 2019
-
Configuration menu - View commit details
-
Copy full SHA for e13ebd1 - Browse repository at this point
Copy the full SHA e13ebd1View commit details
Commits on Apr 29, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 47b5655 - Browse repository at this point
Copy the full SHA 47b5655View commit details -
Add initial documentation for SLM (#41510)
* Add initial documentation for SLM This adds the initial documentation for snapshot lifecycle management. It also includes the REST spec API json files since they're sort of documentation. Relates to #38461
Configuration menu - View commit details
-
Copy full SHA for d63d521 - Browse repository at this point
Copy the full SHA d63d521View commit details -
Add
manage_slm
andread_slm
roles (#41607)* Add `manage_slm` and `read_slm` roles This adds two more built in roles - `manage_slm` which has permission to perform any of the SLM actions, as well as stopping, starting, and retrieving the operation status of ILM. `read_slm` which has permission to retrieve snapshot lifecycle policies as well as retrieving the operation status of ILM. Relates to #38461 * Add execute to the test * Fix ilm -> slm typo in test
Configuration menu - View commit details
-
Copy full SHA for acc3749 - Browse repository at this point
Copy the full SHA acc3749View commit details
Commits on May 2, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 1adcd6d - Browse repository at this point
Copy the full SHA 1adcd6dView commit details
Commits on May 8, 2019
-
Configuration menu - View commit details
-
Copy full SHA for f316925 - Browse repository at this point
Copy the full SHA f316925View commit details
Commits on May 9, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 1684aaa - Browse repository at this point
Copy the full SHA 1684aaaView commit details
Commits on May 10, 2019
-
Record SLM history into an index (#41707)
It is useful to have a record of the actions that Snapshot Lifecycle Management takes, especially for the purposes of alerting when a snapshot fails or has not been taken successfully for a certain amount of time. This adds the infrastructure to record SLM actions into an index that can be queried at leisure, along with a lifecycle policy so that this history does not grow without bound. Additionally, SLM automatically setting up an index + lifecycle policy leads to `index_lifecycle` custom metadata in the cluster state, which some of the ML tests don't know how to deal with due to setting up custom `NamedXContentRegistry`s. Watcher would cause the same problem, but it is already disabled (for the same reason).
Configuration menu - View commit details
-
Copy full SHA for d0973fc - Browse repository at this point
Copy the full SHA d0973fcView commit details
Commits on May 20, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 8216a3c - Browse repository at this point
Copy the full SHA 8216a3cView commit details
Commits on May 21, 2019
-
Configuration menu - View commit details
-
Copy full SHA for ce945a8 - Browse repository at this point
Copy the full SHA ce945a8View commit details
Commits on May 28, 2019
-
Configuration menu - View commit details
-
Copy full SHA for dc973a5 - Browse repository at this point
Copy the full SHA dc973a5View commit details -
High Level Rest Client support for SLM (#41767)
* High Level Rest Client support for SLM This commit add HLRC support for SLM. Relates to #38461 * Fill out documentation tests with tags * Add more callouts and asciidoc for HLRC * Update javadoc links to real locations
Configuration menu - View commit details
-
Copy full SHA for e6ce229 - Browse repository at this point
Copy the full SHA e6ce229View commit details
Commits on May 29, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 7427724 - Browse repository at this point
Copy the full SHA 7427724View commit details
Commits on May 31, 2019
-
Configuration menu - View commit details
-
Copy full SHA for ce649f4 - Browse repository at this point
Copy the full SHA ce649f4View commit details
Commits on Jun 3, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 7fe4d05 - Browse repository at this point
Copy the full SHA 7fe4d05View commit details
Commits on Jun 4, 2019
-
Add security test testing SLM cluster privileges (#42678)
* Add security test testing SLM cluster privileges This adds a test to `PermissionsIT` that uses the `manage_slm` and `read_slm` cluster privileges. Relates to #38461 * Don't redefine vars
Configuration menu - View commit details
-
Copy full SHA for 0adf14d - Browse repository at this point
Copy the full SHA 0adf14dView commit details
Commits on Jun 5, 2019
-
Add Getting Started Guide for SLM (#42878)
This commit adds a basic Getting Started Guide for SLM.
Configuration menu - View commit details
-
Copy full SHA for f452ecd - Browse repository at this point
Copy the full SHA f452ecdView commit details
Commits on Jun 6, 2019
-
Configuration menu - View commit details
-
Copy full SHA for c89a5bc - Browse repository at this point
Copy the full SHA c89a5bcView commit details
Commits on Jun 10, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 27173fa - Browse repository at this point
Copy the full SHA 27173faView commit details
Commits on Jun 13, 2019
-
Include SLM policy name in Snapshot metadata (#43132)
Keep track of which SLM policy in the metadata field of the Snapshots taken by SLM. This allows users to more easily understand where the snapshot came from, and will enable future SLM features such as retention policies.
Configuration menu - View commit details
-
Copy full SHA for c7669b3 - Browse repository at this point
Copy the full SHA c7669b3View commit details
Commits on Jun 14, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 6baaa39 - Browse repository at this point
Copy the full SHA 6baaa39View commit details
Commits on Jun 20, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 9bd7de5 - Browse repository at this point
Copy the full SHA 9bd7de5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1232e05 - Browse repository at this point
Copy the full SHA 1232e05View commit details -
[TEST] Move exception wrapping for devious exception throwing
Fixes an issue where an exception was created from one line and thrown in another.
Configuration menu - View commit details
-
Copy full SHA for 0138f52 - Browse repository at this point
Copy the full SHA 0138f52View commit details
Commits on Jun 25, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 404be63 - Browse repository at this point
Copy the full SHA 404be63View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9a3b0db - Browse repository at this point
Copy the full SHA 9a3b0dbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9e87a4e - Browse repository at this point
Copy the full SHA 9e87a4eView commit details
Commits on Jun 26, 2019
-
Configuration menu - View commit details
-
Copy full SHA for a26e302 - Browse repository at this point
Copy the full SHA a26e302View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6393ac8 - Browse repository at this point
Copy the full SHA 6393ac8View commit details
Commits on Jul 1, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 88d915e - Browse repository at this point
Copy the full SHA 88d915eView commit details
Commits on Jul 2, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 1a9988e - Browse repository at this point
Copy the full SHA 1a9988eView commit details
Commits on Jul 3, 2019
-
Add a note mentioning the privileges needed for SLM (#43708)
* Add a note mentioning the privileges needed for SLM This adds a note to the top of the "getting started with SLM" documentation mentioning that there are two built-in privileges to assist with creating roles for SLM users and administrators. Relates to #38461 * Mention that you can create snapshots for indices you can't read
Configuration menu - View commit details
-
Copy full SHA for 91fdcb5 - Browse repository at this point
Copy the full SHA 91fdcb5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1055836 - Browse repository at this point
Copy the full SHA 1055836View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1af1f8d - Browse repository at this point
Copy the full SHA 1af1f8dView commit details -
Configuration menu - View commit details
-
Copy full SHA for dc3a284 - Browse repository at this point
Copy the full SHA dc3a284View commit details
Commits on Jul 5, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 8f25b3c - Browse repository at this point
Copy the full SHA 8f25b3cView commit details
Commits on Jul 10, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 8806125 - Browse repository at this point
Copy the full SHA 8806125View commit details -
Configuration menu - View commit details
-
Copy full SHA for 74b4e75 - Browse repository at this point
Copy the full SHA 74b4e75View commit details
Commits on Jul 15, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 6aa24fa - Browse repository at this point
Copy the full SHA 6aa24faView commit details -
Configuration menu - View commit details
-
Copy full SHA for aacd841 - Browse repository at this point
Copy the full SHA aacd841View commit details