Allow system indices when domain is index create block

[Arpit-Bandejiya]

Is your feature request related to a problem? Please describe.
During the cluster write block, sometimes we have seen the opensearch-dasboards is not able to work due to kibana index being red.

For background, kibana tries to creates “.kibana*” indices in two cases, while boot up or index migration. Index migration is done during version upgrade. For ex: if current index is “.kibana_1” so after version upgrades new index is created with name “.kibana_2” and so on.

However if there is cluster create-index block, opensearch-dashboards index creation fails.

Below are logs found in kibana.log

{
    "type": "log",
    "@timestamp": "...",
    "tags": ["info", "savedobjects-service"],
    "pid": ***,
    "message": "Creating index .kibana_-190122424_cognitouswest2qt8dmhfaysnuser_2."
}
{
    "type": "error",
    "@timestamp": "...",
    "tags": ["error", "migration"],
    "pid": ***,
    "level": "error",
    "error": {
        "message": "[index_create_block_exception] blocked by: [FORBIDDEN/10/cluster create-index blocked (api)];",
        "name": "Error",
        "stack": "Error: [index_create_block_exception] blocked by: [FORBIDDEN/10/cluster create-index blocked (api)];\n ......"
    },
    "message": "[index_create_block_exception] blocked by: [FORBIDDEN/10/cluster create-index blocked (api)];"
}

Describe the solution you'd like
The idea is to create a dynamic setting which can used to specify the system indices to be skipped when the cluster is write blocked. The setting will be set to false by default which would mean no indices will be allowed to skip the index create block. The setting will be like this when applied:

cluster.blocks.create_index.allowed_sys_indices_pattern: [".kibana"]

In the index create block we will check if the index is of the following pattern and we will skip the index create block for these cases.

Describe alternatives you've considered

We can also create a new setting for allowing all the sys indices to skip the index creation block. The setting will be like this:

cluster.blocks.create_index.allow_sys_indices: True

The major disadvantage of this is that this can cause the domain to go down if the size of the system index is too big since we are not filtering the system index here.

Additional context
Add any other context or screenshots about the feature request here.

[ankitkala]

Ideal solution would be to migrate from hidden index to system index and have this support added for system indices(and not hidden indices): #3942

[ankitkala]

There is already an effort going on to identify the system generated indices using system index plugin: #9239

After this is done, we should be able to distinguish between user & system indices which can be used for write block validation logic.