diff --git a/docs/management/index-lifecycle-policies/example-index-lifecycle-policy.asciidoc b/docs/management/index-lifecycle-policies/example-index-lifecycle-policy.asciidoc deleted file mode 100644 index 0097bf8c648f07..00000000000000 --- a/docs/management/index-lifecycle-policies/example-index-lifecycle-policy.asciidoc +++ /dev/null @@ -1,179 +0,0 @@ -[role="xpack"] - -[[example-using-index-lifecycle-policy]] -=== Tutorial: Use {ilm-init} to manage {filebeat} time-based indices - -With {ilm} ({ilm-init}), you can create policies that perform actions automatically -on indices as they age and grow. {ilm-init} policies help you to manage -performance, resilience, and retention of your data during its lifecycle. This tutorial shows -you how to use {kib}’s *Index Lifecycle Policies* to modify and create {ilm-init} -policies. You can learn more about all of the actions, benefits, and lifecycle -phases in the {ref}/overview-index-lifecycle-management.html[{ilm-init} overview]. - - -[discrete] -[[example-using-index-lifecycle-policy-scenario]] -==== Scenario - -You’re tasked with sending syslog files to an {es} cluster. This -log data has the following data retention guidelines: - -* Keep logs on hot data nodes for 30 days -* Roll over to a new index if the size reaches 50GB -* After 30 days: -** Move the logs to warm data nodes -** Set {ref}/glossary.html#glossary-replica-shard[replica shards] to 1 -** {ref}/indices-forcemerge.html[Force merge] multiple index segments to free up the space used by deleted documents -* Delete logs after 90 days - - -[discrete] -[[example-using-index-lifecycle-policy-prerequisites]] -==== Prerequisites - -To complete this tutorial, you'll need: - -* An {es} cluster with hot and warm nodes configured for shard allocation -awareness. If you’re using {cloud}/ec-getting-started-templates-hot-warm.html[{ess}], -choose the hot-warm architecture deployment template. - -+ -For a self-managed cluster, add node attributes as described for {ref}/shard-allocation-filtering.html[shard allocation filtering] -to label data nodes as hot or warm. This step is required to migrate shards between -nodes configured with specific hardware for the hot or warm phases. -+ -For example, you can set this in your `elasticsearch.yml` for each data node: -+ -[source,yaml] --------------------------------------------------------------------------------- -node.attr.data: "warm" --------------------------------------------------------------------------------- - -* A server with {filebeat} installed and configured to send logs to the `elasticsearch` -output as described in {filebeat-ref}/filebeat-getting-started.html[Getting Started with {filebeat}]. - -[discrete] -[[example-using-index-lifecycle-policy-view-fb-ilm-policy]] -==== View the {filebeat} {ilm-init} policy - -{filebeat} includes a default {ilm-init} policy that enables rollover. {ilm-init} -is enabled automatically if you’re using the default `filebeat.yml` and index template. - -To view the default policy in {kib}, open the menu, go to *Stack Management > Data > Index Lifecycle Policies*, -search for _filebeat_, and choose the _filebeat-version_ policy. - -This policy initiates the rollover action when the index size reaches 50GB or -becomes 30 days old. - -[role="screenshot"] -image::images/tutorial-ilm-hotphaserollover-default.png["Default policy"] - - -[float] -==== Modify the policy - -The default policy is enough to prevent the creation of many tiny daily indices. -You can modify the policy to meet more complex requirements. - -. Activate the warm phase. - -+ -. Set either of the following options to control when the index moves to the warm phase: - -** Provide a value for *Timing for warm phase*. Setting this to *15* keeps the -indices on hot nodes for a range of 15-45 days, depending on when the initial -rollover occurred. - -** Enable *Move to warm phase on rollover*. The index might move to the warm phase -more quickly than intended if it reaches the *Maximum index size* before the -the *Maximum age*. - -. In the *Select a node attribute to control shard allocation* dropdown, select -*data:warm(2)* to migrate shards to warm data nodes. - -. Change *Number of replicas* to *1*. - -. Enable *Force merge data* and set *Number of segments* to *1*. -+ -NOTE: When rollover is enabled in the hot phase, action timing in the other phases -is based on the rollover date. - -+ -[role="screenshot"] -image::images/tutorial-ilm-modify-default-warm-phase-rollover.png["Modify to add warm phase"] - -. Activate the delete phase and set *Timing for delete phase* to *90* days. -+ -[role="screenshot"] -image::images/tutorial-ilm-delete-rollover.png["Add a delete phase"] - -[float] -==== Create a custom policy - -If meeting a specific retention time period is most important, you can create a -custom policy. For this option, you will use {filebeat} daily indices without -rollover. - -. To create a custom policy, open the menu, go to *Stack Management > Data > Index Lifecycle Policies*, then click -*Create policy*. - -. Activate the warm phase and configure it as follows: -+ -|=== -|*Setting* |*Value* - -|Timing for warm phase -|30 days from index creation - -|Node attribute -|`data:warm` - -|Number of replicas -|1 - -|Force merge data -|enable - -|Number of segments -|1 -|=== - -+ -[role="screenshot"] -image::images/tutorial-ilm-custom-policy.png["Modify the custom policy to add a warm phase"] - - -+ -. Activate the delete phase and set the timing. -+ -|=== -|*Setting* |*Value* -|Timing for delete phase -|90 -|=== - -+ -[role="screenshot"] -image::images/tutorial-ilm-delete-phase-creation.png["Delete phase"] - -. To configure the index to use the new policy, open the menu, then go to *Stack Management > Data > Index Lifecycle -Policies*. - -.. Find your {ilm-init} policy. -.. Click the *Actions* link next to your policy name. -.. Choose *Add policy to index template*. -.. Select your {filebeat} index template name from the *Index template* list. For example, `filebeat-7.5.x`. -.. Click *Add Policy* to save the changes. - -+ -NOTE: If you initially used the default {filebeat} {ilm-init} policy, you will -see a notice that the template already has a policy associated with it. Confirm -that you want to overwrite that configuration. - -+ -+ -TIP: When you change the policy associated with the index template, the active -index will continue to use the policy it was associated with at index creation -unless you manually update it. The next new index will use the updated policy. -For more reasons that your {ilm-init} policy changes might be delayed, see -{ref}/update-lifecycle-policy.html#update-lifecycle-policy[Update Lifecycle Policy]. diff --git a/docs/user/management.asciidoc b/docs/user/management.asciidoc index 1704a80847652c..bc96463f6efbad 100644 --- a/docs/user/management.asciidoc +++ b/docs/user/management.asciidoc @@ -190,8 +190,6 @@ include::{kib-repo-dir}/management/index-lifecycle-policies/manage-policy.asciid include::{kib-repo-dir}/management/index-lifecycle-policies/add-policy-to-index.asciidoc[] -include::{kib-repo-dir}/management/index-lifecycle-policies/example-index-lifecycle-policy.asciidoc[] - include::{kib-repo-dir}/management/managing-indices.asciidoc[] include::{kib-repo-dir}/management/ingest-pipelines/ingest-pipelines.asciidoc[]