This commit adds a new NoOpEngine implementation based on the current 
ReadOnlyEngine. This new implementation uses an empty DirectoryReader 
with no segments readers and will always returns 0 docs. The NoOpEngine 
is the default Engine created for IndexShards of closed indices. It expects 
an empty translog when it is instantiated.

Relates to #33888

Changes were made in #34357 and #36467

When a NoOpEngine is instanciated, the current implementation verifies 
that the translog contains no operations and that it contains the same 
UUID as the last Lucene commit data.We can relax those two constraints 
because the Close Index API now ensure that all translog operations are 
flushed before closing a shard. The detection of coherence between translog 
UUID / Lucene commit data is not specific to NoOpEngine, and is already 
done by IndexShard.innerOpenEngineAndTranslog().

Related to #33888

…8024)

This commit allows shards of indices in CLOSE state to be replicated as normal shards. 
It changes the MetaDataIndexStateService so that index routing tables of closed indices 
are kept in cluster state when the index is closed. Index routing tables are modified so 
that shard routings are reinitialized with the INDEX_CLOSED unassigned information. 
The IndicesClusterStateService is modified to remove IndexService instances of closed 
or reopened indices. In combination with the ShardRouting being in INITIALIZING state 
the shards are recreated on the data nodes to reflect the new state. If the index state is 
closed, the IndexShard instances will be created using the NoOpEngine as the engine
implementation.

This commit also mutes two tests that rely on the fact that shard locks are released when 
an index is closed, which is not the case anymore with replicated closed indices (actually 
the locks are released but reacquired once the shard is reinitialized after being closed). 
These tests will be adapted in follow up PRs.

Finally, many things will require to be adapted or improved in follow up PRs (see #33888)
 but this is the first big step towards replicated closed indices.

Relates to #33888

Relates to #33888

…38327)

Relates to #33888 and  #38024

…dices (#38329)

Replicated closed indices do not need to be refreshed, neither they need 
their translogs or global checkpoint to be fsync. This pull request changes 
how `BaseAsyncTask` tasks are rescheduled in `IndexService` instances 
so that the tasks are rescheduled only when the index is opened.

Relates to #33888

…rs (#38955)

This pull request removes the legacy way of closing indices (aka "direct close") 
in mixed versions clusters, since this backward compatibility logic is not
 required anymore on master/8.0.0.

It also changes the closing logic so that routing tables of closed indices are
 removed when the cluster contains a node in version < 8.0.

Relates #33888

…8631)

This pull request modifies the `ClusterAllocationExplainIT` test suite so 
that it always runs the tests with opened and closed indices. The only test 
that was not adapted for closed indices is `testAllocationFilteringOnIndexCreation` 
because we don't allow to directly create indices in the closed state.

Relates to #33888

This commit adds a simple test which verifies that a replica can be promoted 
as a primary when the index is closed.

Relates to #33888

* Adapt more tests suites to closed indices

Similarly to #38631, this pull request modifies multiple test suites so
that they runs the tests with opened or closed indices.

The suites are testing:
- shard allocation filtering
- shard allocation awereness
- Reroute API

Relates to #33888

This commit adapts the Recovery API to make it work with 
shards of replicated closed indices.

Relates #33888

Closing an index is a process that can be broken down into several steps:
1. first, the state of the cluster is updated to add a write block on the index 
to be closed
2. then, a transport replication action is executed on all shards of the index. 
This action checks that the maximum sequence number and the global 
checkpoint have identical values, indicating that all in flight writing operations 
have been completed on the shard.
3. finally, and if the previous steps were successful, the cluster state is 
updated again to change the state of the index from `OPEN`to `CLOSE`.

During the last step, the master node retrieves the minimum node version 
among all the nodes that compose the cluster:
* If a node is in pre 8.0 version, the index is closed and the index routing 
table is removed from the cluster state. This is the "old" way of closing 
indices and closed indices with no routing table are not replicated.
* If all nodes are in version 8.0 or higher, the index is closed and its routing 
table is reinitialized in cluster state. This is the new way of closing indices 
and such closed indices will be replicated in the cluster.

But routing tables are not persisted in the cluster state, so after a full cluster 
restart there is no way to make the distinction between an index closed in 
7.x and an index closed and replicated on 8.0.

This commit introduces a new private index settings named 
`index.verified_before_close` that is added to closed indices that are replicated 
at closing time. This setting serves as a marker to indicate that the index has 
been closed using the new Close Index API on a cluster that supports
 replication of closed indices.

This way, after a full cluster restart, the Gateway service can automatically 
recovers those closed indices as if they were opened indices. Closed indices 
that don't have this setting (because they were closed on a pre-8.0 cluster, 
or a cluster in mixed version) won't be recovered and will need to be reopened 
and closed again on a 8.0 cluster.

Note that reopening the index removes the private setting. 

Relates to #33888

Now the test `CloseFollowerIndexIT` has been added in #38702, it needs to 
be adapted for replicated closed indices.

The test closes the follower index which is lagging behind the leader index. 
When it's closed, no sanity checks are executed because it's a follower index 
(this is a consequence of #38702). But with replicated closed indices, the index
 is reinitialized as a closed index with a `NoOpEngine` and such engines make 
strong assertions on the values of the maximum sequence number and the 
global checkpoint. Since the values do not match, the shards cannot be created 
and fail and the cluster health turns RED.

This commit adapts the `CloseFollowerIndexIT` test so that it wraps the 
default `UncaughtExceptionHandler` with a handler that tolerates any exception 
thrown by `ReadOnlyEngine.assertMaxSeqNoEqualsToGlobalCheckpoint()`. 
Replacing the default uncaught exception handler requires specific permissions,
 and instead of creating another gradle project it duplicates the 
`internalClusterTest` task to make it work without security manager for this 
specific test only.

Relates to #33888

This commit adapts the Cluster Health API to support replicated closed indices. 
In order to do that, it removes the hard coded indices options from the 
`ClusterHealthRequest` and replaces it with a new `IndicesOptions.lenientExpand()` 
option. This option will be used by the master node (once it is upgraded to 8.0) 
to compute the global cluster health using both opened and closed indices 
information by default. The `expand_wildcards` REST parameter is also 
documented and tests where added to ensure that a specific expansion type 
can be used to monitoring the health of a only opened or only closed indices.

Since the Cat Indices relies on the Cluster Health API, it has been adapted 
to report information about closed indices too. Note that the health and 
number of shards/replicas is only printed out for closed indices that have an 
index routing table. Closed indices without routing table have the same 
output as before.

Related to #33888

This commit changes the Close Index API to add a `wait_for_active_shards`
 parameter that allows to wait for shards of closed indices to be active before 
returning a response.

Relates #33888