Restore remote index shards with ExistingStoreRecoverySource after restore from remote state

[linuxpi]

Description

After Quorum loss recovery, remote index shards will be restore in cluster metadata as EXISTING_STORE as RecoverySource. This is same as non remote shards.

Related Issues

Resolves #10658

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change 91b3ea2

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/28150/
• CommitID: 4650fbb
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[linuxpi]

@linuxpi can this be simplified further where indices are restored in cluster state after metadata recovery from remote as "CLUSTER_RECOVERED" (which is default behavior ), now if there is a latest copy of a shard available on any index, it will be assigned else shard will turn red. Essentially, let it behave like shards are recovering from local disk after cluster restart. So when recovering metadata, if you don't add any entry for shard RoutingTable then ClusterStateUpdaters will add a recovery entry own it own

OpenSearch/server/src/main/java/org/opensearch/gateway/ClusterStateUpdaters.java

Line 137 in 7936f94

routingTableBuilder.addAsRecovery(cursor);

(lets remove the code we added here earlier for special handling for remote indices)

and when user trigger the restore API, it will set the recovery source and everything properly.

So if i got it correctly, in case of remote state auto restore, we want to initialize shards with ExistingStore recovery. If any of the shards find data in local disk on data node it will comeup and others will fail?

so thinking from the perspective of split primaries issue, if the shard comes up in the network partition with cluster manager in it, the isolated primary in other partition would try to do primary term validation with this new shard that just came up and it wont be able to?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/28519/
• CommitID: f853be1
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/28529/
• CommitID: 1a2e529
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/28534/
• CommitID: d08ae22
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/28546/
• CommitID: 71ea786
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[linuxpi]

Flaky Tests
#5329
#10542

org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation
org.opensearch.indices.replication.SegmentReplicationIT.testSendCorruptBytesToReplica
org.opensearch.indices.replication.SegmentReplicationIT.testSendCorruptBytesToReplica
org.opensearch.indices.replication.SegmentReplicationIT.testSendCorruptBytesToReplica
org.opensearch.indices.replication.SegmentReplicationIT.testSendCorruptBytesToReplica

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/28561/
• CommitID: 91b3ea2
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[linuxpi]

Flaky Test - #9464

org.opensearch.search.query.QueryPhaseTests.testQueryTimeoutChecker {p0=0 p1=org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher@42b43a6}