Closed index replica allocation

[henningandersen]

When an index is closed, we expect primary and replicas to be identical.
This commit improves the gateway replica shard allocator to consider
shards with identical sequence numbers sync'ed for closed indices. This
ensures that we will pick a fast recovery regardless of whether synced
flush was performed prior to closing an index.

Fixed InternalTestCluster to allow doing operations inside onStopped()
when using restartXXXNode().

Relates #41400 and #33888

Please notice the todo on the explain API.

[elasticmachine]

Pinging @elastic/es-distributed

[henningandersen]

ci/1 failed with unrelated failure, reported here: #41794
@elasticmachine run elasticsearch-ci/1

[dnhatn]

@henningandersen I have merged #41400.

[henningandersen]

@elasticmachine run elasticsearch-ci/1

[ywelsch]

It looks like TransportNodesListShardStoreMetaData is loading the last commit, not the safe commit. For the peer recovery, it’s the local checkpoint of the safe commit, however, which counts. This means that some of these replica allocation decisions by the master might be non-optimal (and require full file-based recoveries). We can of course argue that the likelihood of that is very small. The other issue is that this might work well for closed indices now (given the specialiation for closed indices), but the allocation code will not take frozen indices into account, which share many of the properties with closed replicated indices when it comes to recovery.

I’m mostly wondering if we should generalize the logic a bit more, and not rely on the max seq no / local checkpoint of the last commit, but explicitly enrich the TransportNodesListShardStoreMetaData response to contain additional info:

• minimum sequence number from which this shard copy can offer operation-based recoveries (= local checkpoint of safe commit for now) (minProvRecoverySeq)
• minimum sequence number of range which this shard copy requires for an operation-based recovery (= local checkpoint of safe commit + 1 for now) (minReqRecoverySeq)
• maximum sequence number of the shard copy (maxSeq)

If we then have the condition that
primary.minProvRecoverySeq <= replica.minReqRecoverySeq == primary.maxSeq + 1 && primary.maxSeq == replica.maxSeq
we can assume that a recovery will be operation-based and not require sending any ops and therefore instanteneous (similar to synced flush). This condition won't require any qualification whether it is for closed / frozen or regular indices.

[ywelsch]

^^

[dnhatn]

@henningandersen Should we close this PR?

[henningandersen]

Thanks @dnhatn , yes this can be closed now.