Adapt CloseFollowerIndexIT for replicated closed indices

[tlrx]

Note: this pull request targets the replicated-closed-indices feature branch

Now the test CloseFollowerIndexIT has been added in #38702, it needs to be adapted for replicated closed indices.

The test closes the follower index which is lagging behind the leader index. When it's closed, no sanity checks are executed because it's a follower index (this is a consequence of #38702). But with replicated closed indices, the index is reinitialized as a closed index with a NoOpEngine and such engines make strong assertions on the values of the maximum sequence number and the global checkpoint. Since the values do not match, the shards cannot be created and fail and the cluster health turns RED.

This is the expected behavior but the test now requires to be adapted to catch the following uncaught exception:

WARNING: Uncaught exception in thread: Thread[elasticsearch[follower1][generic][T#3],5,TGRP-CloseFollowerIndexIT]
java.lang.AssertionError: max seq. no. [-1] does not match [31]
	at __randomizedtesting.SeedInfo.seed([A19FDEFBFCF2B7B1]:0)
	at org.elasticsearch.index.engine.ReadOnlyEngine.assertMaxSeqNoEqualsToGlobalCheckpoint(ReadOnlyEngine.java:141)
	at org.elasticsearch.index.engine.ReadOnlyEngine.<init>(ReadOnlyEngine.java:115)
	at org.elasticsearch.index.engine.NoOpEngine.<init>(NoOpEngine.java:40)
	at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1438)
	at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1391)
	at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:424)
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95)
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:302)
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93)
	at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1685)
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2267)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

This pull request adapt the CloseFollowerIndexIT test so that it wraps the default UncaughtExceptionHandler with a handler that tolerates any exception thrown by ReadOnlyEngine.assertMaxSeqNoEqualsToGlobalCheckpoint(). Replacing the default uncaught exception handler requires specific permissions, and instead of creating another gradle project I chose to duplicate the internalClusterTest task to make it work without security manager for this specific test only.

I tried to come up with other ways to adapt this test (disabling allocation for closed shards, wraps ReadOnlyEngine in a factory that swallows the assertion error, dedicated gradle project etc) but this one is the only solution I found that works correctly and checks the real assertion error.

[elasticmachine]

Pinging @elastic/es-distributed

[tlrx]

Unrelate test UnicastZenPingTests.testSimplePings failed, let's try again:
@elasticmachine run elasticsearch-ci/1

[tlrx]

@elasticmachine run elasticsearch-ci/1

[martijnvg]

LGTM - assuming that changing the default UncaughtExceptionHandler is the cleanest way.

[ywelsch]

We need to think how to integrate closed replicated indices with closed follower indices. In the current form, closing follower indices will result in them not being allocated, making the cluster go red. Perhaps instead of requiring max sequence number to be equal to global checkpoint for these shards, which is not guaranteed by the CCR following logic, it would be sufficient to check that max seq number and global checkpoint agree with the primary and require a sync id marker (i.e. perform a synced flush during the VerifyShardBeforeCloseAction) so that peer recovery will be instantaneous.

[tlrx]

Thanks @martijnvg and @ywelsch .

Yannick and I talked via another channel and we decided to merge this change so that the CloseFollowerIndexIT test will still run after the feature branch will be merged.

We still agree that we need to think how to integrate closed replicated indices with closed follower indices and this is tracked as part of #33888.