[CI] SharedClusterSnapshotRestoreIT.testBasicWorkFlow failure

[davidkyle]

Build Url: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-darwin-compatibility/99/console

This does not reproduce

gradle :core:integTest \
  -Dtests.seed=2538815628A1B83B \
  -Dtests.class=org.elasticsearch.snapshots.SharedClusterSnapshotRestoreIT \
  -Dtests.method="testBasicWorkFlow" \
  -Dtests.security.manager=true \
  -Dtests.locale=es-PY \
  -Dtests.timezone=Etc/GMT-10

Failed due to an assertion error:

08:32:33   1> [2017-08-30T18:32:05,409][INFO ][o.e.s.SharedClusterSnapshotRestoreIT] [testBasicWorkFlow]: after test
08:32:33 FAILURE 3.07s J0 | SharedClusterSnapshotRestoreIT.testBasicWorkFlow <<< FAILURES!
08:32:33    > Throwable #1: java.lang.AssertionError: Count is 99 but 100 was expected.  Total shards: 4 Successful shards: 4 & 0 shard failures:
08:32:33    > 	at __randomizedtesting.SeedInfo.seed([2538815628A1B83B:84E5F7F39CF8EF7B]:0)
08:32:33    > 	at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertHitCount(ElasticsearchAssertions.java:263)
08:32:33    > 	at org.elasticsearch.snapshots.SharedClusterSnapshotRestoreIT.testBasicWorkFlow(SharedClusterSnapshotRestoreIT.java:209)```

[imotov]

@jasontedor and I discussed this issue, and it seems that this issues might be related to #26544. The test basically creates a few indices, makes snapshots, deletes some records from some of the indices, closes the indices and restores them from the snapshot. The test fails because one record is missing in one of the indices. It is possible that some lingering deletes in replicas might have messed up the count. The failure doesn't reproduce. It didn't fail again in the last 20 days and if our theory is correct it shouldn't happen again after #26694 is merged. So, I am going to close this issue for now assuming that it was fixed. Please, reopen the issue if it happens again.