[BUG] Telemetry docs migration issue

[kavilla]

Describe the bug

There is a problem with indices that came from a legacy version with telemetry added and migrating to OpenSearch Dashboards that prevents the clean up of the previous system index and prevents it from starting automatically.

To Reproduce
Steps to reproduce the behavior:

1. Have a doc like the following:

 {
    "_index" : ".kibana_1",
    "_type" : "_doc",
    "_id" : "ui-metric:console:GET_cluster.state",
    "_score" : 1.0,
    "_source" : {
      "ui-metric" : {
        "count" : 1
      },
      "type" : "ui-metric",
      "references" : [ ],
      "updated_at" : "2021-03-02T14:08:50.823Z"
    }
  },

2. Start OpenSearch Dashboards
3. See if failure in migration
4. Check _cat/aliases to see if the same alias for a system index points to two system indices.

Expected behavior
No exception thrown

OpenSearch Version
1.0+

Dashboards Version
1.0+

Plugins
Security

Screenshots
n/a

Additional context
A temporary workaround can be to manually cleanup the previous alias with
curl -X DELETE localhost:9200/.kibana_1/_alias/.kibana if you have a .kibana_2. To prevent future issues until this is patched you can delete those telemetry docs. Using the app Dev Tools with DELETE /.kibana/_doc/{ID of TELEMETRY OBJECT. [REF]

[AMoo-Miki]

A similar problem was also observed with Detected mapping change in "properties.originId" on OSD 1.3.2.

[bandinib-amzn]

Is there any plan to fix this?

[bandinib-amzn]

Is this issue only present in 1.x version? What about 2.x?

[wbeckler]

I propose that we skip this type during the migration. Thoughts? I know this will lead to data loss, but it's better than completely dying.

[wbeckler]

Also, what about a startup flag that includes "ignore this thing" which can be enabled optionally, so that some users can have a hard fail that's more transparent?

[kavilla]

Looking into there is a migrations.skip that can be configured. However, the logic in how it's being used seems to still then migrate if plugins aren't not initialized at the time of startup. Which seems to be always, then in goes into the plugin migration part. Still looking into.