Delete deprecated index settings "archived.*"

[faabsen]

Is your feature request related to a problem? Please describe.
When migrating from ElasticSearch to OpenSearch (via the snapshot restore path), indexes can/will be restored with deprecated index settings, e.g. archived.index.lifecycle.name. However, when applying an index policy and switching its state, the ISM fails with the following info: "unknown setting [archived.index.lifecycle.name] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"

Describe the solution you'd like
It would be wonderful, to either let the user delete deprecated "archived" settings or fix the problem for the ISM (the ISM state change should work independently from existing deprecated "archived" settings.

Describe alternatives you've considered
Currently, only a complete reindex will fix these ISM failures. However, reindexing is time and CPU intensive.

Additional context
OpenSearch 1.2.3
Migration from ElasticSearch 7.10.2

[dbbaughe]

archived.index.lifecycle.name isn't an ISM setting, it's an ILM setting from Elastic's x-pack feature (Index Lifecycle Management).

This error: "unknown setting [archived.index.lifecycle.name] please check that any required plugins are installed, or check the breaking changes documentation for removed settings" isn't related much to ISM.. it's the core not knowing what to do with the archived setting that was imported from the snapshot (in this case the x-pack ILM setting, but could be any setting that's in Elasticsearch and not in OpenSearch).

@anasalkouz This isn't really an ISM issue... this is a core issue relating to imported settings that OpenSearch does not support and them being archived and causing the cluster to stop specific API requests (I think it'll block all setting API calls, I forget exactly though). I'd probably move it back to core repo.

[mch2]

Issue moved to opensearch-project/OpenSearch #2800 via ZenHub

[mch2]

This looks like a bwc concern for the OpenSearch repo. @anasalkouz can you pls transfer back? Looks like my transfer permissions/options are different than yours.

[AssafMentzer]

Another alternative is to ignore these ilm settings in the _restore
POST _snapshot/my-repository/my-snapshot/_restore { "ignore_index_settings": ["index.lifecycle.*"] }

the indices will be restored without the deprecated index settings (e.g. archived.index.lifecycle.name, archived.index.lifecycle.rollover_alias)

[indrajohn7]

Similar issue found on one of the cluster while removing the index block write.

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "unknown setting [archived.index.lifecycle.indexing_complete] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "unknown setting [archived.index.lifecycle.indexing_complete] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
    "suppressed" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "unknown setting [archived.index.lifecycle.name] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
      },
      {
        "type" : "illegal_argument_exception",
        "reason" : "unknown setting [archived.index.lifecycle.rollover_alias] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
      }
    ]
  },

[ankitkala]

Restoring snapshot without index setting should help in mitigation in this particular case but doesn't solve one major issue.The issue is that if we move any index setting to archived, there is no way to clear that setting via update setting.

By adding archived. as prefix, the index level settings index.lifecycle.indexing_complete, in this case, becomes archived.index.lifecycle.indexing_complete.

There is a bug(link1 link2) in the code due to which we can't clear these settings. During update settings call, we expect all index settings to start with index prefix and have the logic to prepend index if its not already there. This won't work for archived settings as those start with archived.index.. and would fail with something like this.

{
    "error": {
        "reason": "unknown setting [index.archived.index.lifecycle.indexing_complete] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
        "root_cause": [
            {
                "reason": "unknown setting [index.archived.index.lifecycle.indexing_complete] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
                "type": "illegal_argument_exception"
            }
        ],
        "type": "illegal_argument_exception"
    },
    "status": 400
}

We need to fix the logic to skip adding the index prefix in such cases.

[ankitkala]

Options for Remediation:

1. Update the index settings to remove the archived ones(doesn't work due to the bug mentioned above).
2. Clear the settings from cluster state using NodeToolCli (Works for cluster settings but not index level settings. feature gap??)
3. Reindex the data into a new index without the settings(can be done, but messy specially if you need to do this for all indices in a cluster)
4. Restore all the indices from snapshot with ignore_index_settings to skip the archived settings.