Skip to content

Commit

Permalink
Merge remote-tracking branch 'upstream/main' into migrate-toolbar-button
Browse files Browse the repository at this point in the history
  • Loading branch information
rshen91 committed Feb 9, 2022
2 parents 1e5da62 + c78a4f3 commit e8e2e40
Show file tree
Hide file tree
Showing 4 changed files with 58 additions and 62 deletions.
2 changes: 1 addition & 1 deletion .bazeliskversion
Original file line number Diff line number Diff line change
@@ -1 +1 @@
1.10.1
1.11.0
23 changes: 10 additions & 13 deletions docs/setup/upgrade.asciidoc
Original file line number Diff line number Diff line change
@@ -1,33 +1,30 @@
[[upgrade]]
== Upgrade {kib}

To upgrade from 7.16 or earlier to {version},
**you must first upgrade to {prev-major-last}**.
This enables you to use the Upgrade Assistant to
{stack-ref}/upgrading-elastic-stack.html#prepare-to-upgrade[prepare to upgrade].
You must resolve all critical issues identified by the Upgrade Assistant
before proceeding with the upgrade.
To upgrade from 7.16.0 or earlier to {version},
**you must first upgrade to {prev-major-last}**, which enables you to use the *Upgrade Assistant* to
{stack-ref}/upgrading-elastic-stack.html#prepare-to-upgrade[prepare for the upgrade].
Before you upgrade, you must resolve all critical issues identified by the *Upgrade Assistant*.

{kib} does not support rolling upgrades.
You must shut down all {kib} instances, install the new software, and restart {kib}.
Rolling upgrades are unsupported in {kib}. To upgrade,
you must shut down all {kib} instances, install the new software, and restart {kib}.
Upgrading while older {kib} instances are running can cause data loss or upgrade failures.

[WARNING]
====
{kib} automatically runs <<saved-object-migrations, saved object migrations>>
when required.
When required, {kib} automatically migrates <<saved-object-migrations, saved objects>>.
In case of an upgrade failure, you can roll back to an
earlier version of {kib}. To roll back, you **must** have a
{ref}/snapshot-restore.html[backup snapshot] that includes the `kibana` feature
state. Snapshots include this feature state by default.
state. By default, snapshots include the `kibana` feature state.
====

For more information about upgrading,
refer to {stack-ref}/upgrading-elastic-stack.html[Upgrading to Elastic {version}.]

IMPORTANT: You can upgrade to pre-release versions for testing,
but upgrading from a pre-release to the General Available version is not supported.
Pre-releases should only be used for testing in a temporary environment.
but upgrading from a pre-release to the General Available version is unsupported.
You should use pre-release versions only for testing in a temporary environment.

include::upgrade/upgrade-migrations.asciidoc[leveloffset=-1]

Expand Down
2 changes: 1 addition & 1 deletion docs/setup/upgrade/logging-configuration-changes.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
[[logging-config-changes]]
=== Logging configuration changes

WARNING: {kib} 8.0 and later uses a new logging system. Be sure to read the documentation for your version of {kib} before proceeding.
WARNING: {kib} 8.0.0 and later uses a new logging system. Before you upgrade, read the documentation for your {kib} version.

[[logging-pattern-format-old-and-new-example]]
[options="header"]
Expand Down
93 changes: 46 additions & 47 deletions docs/setup/upgrade/upgrade-migrations.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,31 +2,31 @@
[[saved-object-migrations]]
=== Saved object migrations

Every time {kib} is upgraded it will perform an upgrade migration to ensure that all <<managing-saved-objects,saved objects>> are compatible with the new version.
Each time you upgrade {kib}, an upgrade migration is performed to ensure that all <<managing-saved-objects,saved objects>> are compatible with the new version.

NOTE: 6.7 includes an https://www.elastic.co/guide/en/kibana/6.7/upgrade-assistant.html[Upgrade Assistant]
to help you prepare for your upgrade to 7.0. To access the assistant, go to *Management > 7.0 Upgrade Assistant*.
NOTE: To help you prepare for the upgrade to 7.0.0, 6.7.0 includes an https://www.elastic.co/guide/en/kibana/6.7/upgrade-assistant.html[*Upgrade Assistant*].
To access the assistant, go to *Management > 7.0 Upgrade Assistant*.

WARNING: {kib} 7.12.0 and later uses a new migration process and index naming scheme. Be sure to read the documentation for your version of {kib} before proceeding.
WARNING: {kib} 7.12.0 and later uses a new migration process and index naming scheme. Before you upgrade, read the documentation for your version of {kib}.

WARNING: The following instructions assumes {kib} is using the default index names. If the `kibana.index` or `xpack.tasks.index` configuration settings were changed these instructions will have to be adapted accordingly.
WARNING: The following instructions assumes {kib} is using the default index names. If the `kibana.index` or `xpack.tasks.index` configuration settings are different from the default, adapt the instructions accordingly.

[float]
[[upgrade-migrations-process]]
==== Background

Saved objects are stored in two indices:

* `.kibana_{kibana_version}_001`, e.g. for Kibana v7.12.0 `.kibana_7.12.0_001`.
* `.kibana_task_manager_{kibana_version}_001`, e.g. for Kibana v7.12.0 `.kibana_task_manager_7.12.0_001`.
* `.kibana_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_7.12.0_001`.
* `.kibana_task_manager_{kibana_version}_001`, e.g. for {kib} 7.12.0 `.kibana_task_manager_7.12.0_001`.

The index aliases `.kibana` and `.kibana_task_manager` will always point to
The index aliases `.kibana` and `.kibana_task_manager` always point to
the most up-to-date saved object indices.

When you start a new {kib} installation, an upgrade migration is performed before starting plugins or serving HTTP traffic.
Before you upgrade, shut down old nodes to prevent losing acknowledged writes.
To reduce the likelihood of old nodes losing acknowledged writes, {kib} 7.12.0 and later
adds a write block to the outdated index. Table 1 lists the saved objects indices used by previous versions of {kib}.
adds a write block to the outdated index. Table 1 lists the saved objects indices used by previous {kib} versions.

.Saved object indices and aliases per {kib} version
[options="header"]
Expand All @@ -46,8 +46,8 @@ adds a write block to the outdated index. Table 1 lists the saved objects indice
When upgrading several {kib} instances connected to the same {es} cluster,
ensure that all outdated instances are shut down before starting the upgrade.

{kib} does not support rolling upgrades. However, once outdated instances are shut down,
all upgraded instances can be started in parallel, in which case all instances will participate in the upgrade migration in parallel.
Rolling upgrades are unsupported in {kib}. However, when outdated instances are shut down, you can start all upgraded instances in parallel,
which allows all instances to participate in the upgrade migration in parallel.

For large deployments with more than 10 {kib} instances, and more than 10,000 saved objects,
you can reduce the upgrade downtime by bringing up a single {kib} instance and waiting for it to
Expand All @@ -56,7 +56,7 @@ complete the upgrade migration before bringing up the remaining instances.
[float]
[[preventing-migration-failures]]
==== Preventing migration failures
This section highlights common causes of {kib} upgrade failures and how to prevent them.
Review the common causes of {kib} upgrade failures and how to prevent them.

[float]
===== timeout_exception or receive_timeout_transport_exception
Expand All @@ -71,18 +71,18 @@ Error: Unable to complete saved object migrations for the [.kibana] index. Pleas
Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Error: [timeout_exception]: Timed out waiting for completion of [org.elasticsearch.index.reindex.BulkByScrollTask@6a74c54]
--------------------------------------------

Instructions to work around this issue are in https://github.com/elastic/kibana/issues/95321[this GitHub issue].
For instructions on how to mitigate the known issue, refer to https://github.com/elastic/kibana/issues/95321[the GitHub issue].

[float]
===== Corrupt saved objects
We highly recommend testing your {kib} upgrade in a development cluster to find and remedy problems
caused by corrupt documents, especially when there are custom integrations creating saved objects in your environment.
To find and remedy problems caused by corrupt documents, we highly recommend testing your {kib} upgrade in a development cluster,
especially when there are custom integrations that create saved objects in your environment.

Saved objects that were corrupted through manual editing or integrations will cause migration
failures with a log message like `Unable to migrate the corrupt Saved Object document ...`.
Saved objects that are corrupted through manual editing or integrations cause migration
failures with a log message, such as `Unable to migrate the corrupt Saved Object document ...`.
For a successful upgrade migration, you must fix or delete corrupt documents.

For example, given the following error message:
For example, you receive the following error message:

[source,sh]
--------------------------------------------
Expand Down Expand Up @@ -112,18 +112,18 @@ DELETE .kibana_7.12.0_001/_doc/marketing_space:dashboard:e3c5fc71-ac71-4805-bcab

. Restart {kib}.
+
In this example, the dashboard with ID `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` that belongs to the space `marketing_space` **is no longer available**.
The dashboard with the `e3c5fc71-ac71-4805-bcab-2bcc9cc93275` ID that belongs to the `marketing_space` space **is no longer available**.

Be sure you have a snapshot before you delete the corrupt document. If restoring from a snapshot is not an option, it is recommended to also delete the `temp` and `target` indices the migration created before restarting {kib} and retrying.
Be sure you have a snapshot before you delete the corrupt document. If you are unable to restore from a snapshot, it is recommended to also delete the `temp` and `target` indices the migration creates before you restart {kib} and retry the snapshot restore.

[float]
===== User defined index templates that causes new `.kibana*` indices to have incompatible settings or mappings
===== User defined index templates that cause new `.kibana*` indices to have incompatible settings or mappings
Matching index templates that specify `settings.refresh_interval` or `mappings` are known to interfere with {kib} upgrades.

Prevention: Narrow down the {data-sources} of any user-defined index templates to ensure that these won't apply to new `.kibana*` indices.
To make sure the index templates won't apply to new `.kibana*` indices, narrow down the {data-sources} of any user-defined index templates.

NOTE: {kib} < 6.5 creates it's own index template called `kibana_index_template:.kibana`
and uses an index pattern of `.kibana`. This index template will not interfere and does not need to be changed or removed.
NOTE: In {kib} 6.5.0 and earlier, {kib} creates a `kibana_index_template:.kibana` index template
and uses a `.kibana` index pattern. You do not need to change or remove the index template.

[float]
===== An unhealthy {es} cluster
Expand All @@ -140,7 +140,7 @@ Ensure that all {kib} instances are running the same version, configuration, and

[float]
===== Incompatible `xpack.tasks.index` configuration setting
For {kib} 7.5.0 and earlier, when the task manager index is set to `.tasks` with the configuration setting `xpack.tasks.index: ".tasks"`,
In {kib} 7.5.0 and earlier, when the task manager index is set to `.tasks` with the configuration setting `xpack.tasks.index: ".tasks"`,
upgrade migrations fail. In {kib} 7.5.1 and later, the incompatible configuration setting prevents upgrade migrations from starting.

[float]
Expand All @@ -149,56 +149,55 @@ upgrade migrations fail. In {kib} 7.5.1 and later, the incompatible configuratio

If {kib} unexpectedly terminates while migrating a saved object index, {kib} automatically attempts to
perform the migration again when the process restarts. Do not delete any saved objects indices to
attempt to fix a failed migration. Unlike previous versions, {kib} 7.12.0 and
later does not require deleting any indices to release a failed migration lock.
to fix a failed migration. Unlike previous versions, {kib} 7.12.0 and
later does not require deleting indices to release a failed migration lock.

If upgrade migrations fail repeatedly, follow the advice in
If upgrade migrations fail repeatedly, refer to
<<preventing-migration-failures, preventing migration failures>>.
Once the root cause for the migration failure has been addressed,
{kib} will automatically retry the migration without any further intervention.
If you're unable to resolve a failed migration following these steps, please contact support.
When you address the root cause for the migration failure,
{kib} automatically retries the migration.
If you're unable to resolve a failed migration, contact Support.

[float]
[[upgrade-migrations-rolling-back]]
==== Rolling back to a previous version of {kib}

If you've followed the advice in <<preventing-migration-failures, preventing migration failures>>
and <<resolve-migrations-failures, resolving migration failures>> and
If {kib} is still unable to upgrade successfully, rollback {kib} until
If you've followed <<preventing-migration-failures, preventing migration failures>>
and <<resolve-migrations-failures, resolving migration failures>>, and
{kib} is still unable to successfully upgrade, rollback {kib} until
you're able to identify and fix the root cause.

WARNING: Before rolling back {kib}, ensure that the version you want to rollback to is compatible with
your {es} cluster. If the version you're rolling back to is not compatible, you must also rollback {es}.
Any changes made after an upgrade are lost when rolling back to a previous version.
WARNING: Before you roll back {kib}, ensure that the version you want to roll back to is compatible with
your {es} cluster. If the version you want to roll back to is not compatible, you must also rollback {es}.
Any changes made after an upgrade are lost when you roll back to a previous version.

To rollback after a failed upgrade migration, the saved object indices have to be
rolled back to be compatible with the previous {kib} version.
To roll back after a failed upgrade migration, you must also rollback the saved object indices to be compatible with the previous {kib} version.

[float]
===== Rollback by restoring a backup snapshot
===== Roll back by restoring a backup snapshot

. Before proceeding, {ref}/snapshots-take-snapshot.html[take a snapshot] that contains the `kibana` feature state.
Snapshots include this feature state by default.
By default, snapshots include the `kibana` feature state.
. To make sure no {kib} instances are performing an upgrade migration, shut down all {kib} instances.
. Delete all saved object indices with `DELETE /.kibana*`.
. To delete all saved object indices, use `DELETE /.kibana*`.
. {ref}/snapshots-restore-snapshot.html[Restore] the `kibana` feature state from the snapshot.
. Start all {kib} instances on the older version you want to rollback to.

[float]
===== (Not recommended) Rollback without a backup snapshot
===== (Not recommended) Roll back without a backup snapshot

. To make sure no {kib} instances are performing an upgrade migration, shut down all {kib} instances.
. {ref}/snapshots-take-snapshot.html[Take a snapshot] that includes the `kibana` feature state. By default, snapshots include the feature state.
. {ref}/snapshots-take-snapshot.html[Take a snapshot] that includes the `kibana` feature state. By default, snapshots include the `kibana` feature state.
. Delete the version-specific indices created by the failed upgrade migration.
+
For example, to rollback from a failed upgrade
to v7.12.0: `DELETE /.kibana_7.12.0_*,.kibana_task_manager_7.12.0_*`
to v7.12.0, use `DELETE /.kibana_7.12.0_*,.kibana_task_manager_7.12.0_*`.
. Inspect the output of `GET /_cat/aliases`.
+
If the `.kibana` or `.kibana_task_manager` aliases are missing, you must create them manually.
Find the latest index from the output of `GET /_cat/indices` and create the missing alias to point to the latest index.
For example, if the `.kibana` alias is missing, and the latest index is `.kibana_3`, create a new alias with `POST /.kibana_3/_aliases/.kibana`.
. To remove the write block from the rollback indices:
For example, if the `.kibana` alias is missing, and the latest index is `.kibana_3`, create a new alias using `POST /.kibana_3/_aliases/.kibana`.
. To remove the write block from the roll back indices, use
`PUT /.kibana,.kibana_task_manager/_settings {"index.blocks.write": false}`
. Start {kib} on the older version you want to rollback to.

Expand Down

0 comments on commit e8e2e40

Please sign in to comment.