CI: CcrRepositoryIT.testDocsAreRecovered and testIndividualActionsTimeout fails

[alpar-t]

Example build failure

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+internalClusterTest/161/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+internalClusterTest/202/console

Reproduction line

does not reproduce locally

./gradlew :x-pack:plugin:ccr:internalClusterTest -Dtests.seed=1FA1BA206A169A3C -Dtests.class=org.elasticsearch.xpack.ccr.CcrRepositoryIT -Dtests.method="testDocsAreRecovered" -Dtests.security.manager=true -Dtests.locale=ro-RO -Dtests.timezone=Etc/Universal -Dcompiler.java=11 -Druntime.java=8

Example relevant log:

21:51:51 FAILURE 1.60s J3 | CcrRepositoryIT.testIndividualActionsTimeout <<< FAILURES!
21:51:51    > Throwable #1: java.lang.AssertionError: expected:<0> but was:<2>
21:51:51    > 	at __randomizedtesting.SeedInfo.seed([1FA1BA206A169A3C:7BBC56FCD964A80F]:0)
21:51:51    > 	at org.elasticsearch.xpack.ccr.CcrRepositoryIT.testIndividualActionsTimeout(CcrRepositoryIT.java:370)
21:51:51    > 	at java.lang.Thread.run(Thread.java:748)

Frequency

14 times yesterday, 3 times Today

[elasticmachine]

Pinging @elastic/es-distributed

[Tim-Brooks]

I'm pretty sure that it was CcrRepositoryIT.testIndividualActionsTimeout that failed here?

[alpar-t]

Both failed actually and I mixed them up when writing the ticket.
I think testIndividualActionsTimeout failed most often but I saw multiple test failures for this class.

[Tim-Brooks]

My theory is that this test was failing in conjunction with testIndividualActionsTimeout because testIndividualActionsTimeout left bad state around when it failed. testDocsAreRecovered has been unmuted.