[Segment Replication] Add Segment Replication Specific Integration Tests #11773

Rishikesh1159 · 2024-01-05T17:25:54Z

Description

This PR adds new segment replication specific integ tests. As discussed on issue here, this PR provides implementation of Steps 1,2,3.

For Step 2: We have chosen Approach 2 (modifying indexing helper methods). Please feel free to comment on this PR if anyone has strong opinions against this approach. In this PR we added 3 new segrep specific integtests, each one representing different way to support Approach 2:
-> SegmentReplicationIndicesRequestCacheIT - IndicesRequestCacheIT - indexRandom()
-> SegmentReplicationSimpleClusterStateIT - SimpleClusterStateIT - index()
-> SegmentReplicationIndexStatsIT - IndexStatsIT - client().prepareIndex()

For Step 3: We have chosen Approach 2 (new segrep test class for every test). Again please feel free to comment on this PR if anyone has strong opinions against this approach.

This PR has small initial set of tests running with segment replication. But later there will follow up PR's that will add bunch of new segrep specific integ tests similar to tests in this PR.

Related Issues

Resolves #6761

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

github-actions · 2024-01-05T17:45:47Z

❌ Gradle check result for 56b1d57: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-01-05T17:47:48Z

Compatibility status:

Checks if related components are compatible with change e6d3703

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git]

github-actions · 2024-01-05T18:42:18Z

❌ Gradle check result for 56b1d57: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-02-07T17:03:25Z

❌ Gradle check result for 19d99d8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-02-07T17:05:23Z

❌ Gradle check result for 718ecb3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-02-07T17:12:51Z

❌ Gradle check result for 4649c42: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/internalClusterTest/java/org/opensearch/recovery/RecoveryWhileUnderLoadIT.java

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

…sh1159/OpenSearch into segrep-enabled-integTest

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

github-actions · 2024-02-08T01:13:01Z

❌ Gradle check result for bcc0ca0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-02-08T01:19:29Z

❕ Gradle check result for d7e9027: UNSTABLE

TEST FAILURES:

      1 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testOverriddenBufferInterval

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2024-02-08T01:20:05Z

❕ Gradle check result for de8e7e2: UNSTABLE

TEST FAILURES:

      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2024-02-08T01:45:31Z

❕ Gradle check result for 2c305cb: UNSTABLE

TEST FAILURES:

      2 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing
      1 org.opensearch.search.SearchWeightedRoutingIT.testShardRoutingWithNetworkDisruption_FailOpenEnabled
      1 org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringFetchPhase

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

github-actions · 2024-02-08T20:21:28Z

❌ Gradle check result for e6d3703: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Rishikesh1159 · 2024-02-08T20:41:23Z

❌ Gradle check result for e6d3703: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

#12197

github-actions · 2024-02-08T21:28:21Z

✅ Gradle check result for e6d3703: SUCCESS

@ignore

…sts (#11773) * Run few tests with Segment Replication enabled. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Update reason for ignoring test. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * remove @ignore to resolve :server:forbiddenApisInternalClusterTest check. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * fix spotlessCheck. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * add conditional logic of force refresh. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Address comments on PR. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * fix failing errors. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Use parameterization for running segment replication tests. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Fix failing tests. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Fix failing test. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * add new waitForReplication() and refactor. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Address comments on PR and revert back changes made to SegmentReplication Tests. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * revert changes made to Segrep tests. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Refactor and address comments. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * fix failure of using forbidden api new Random(). Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Add comments to debug. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Remove non-critical tests from running with segrep. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Fix test to run with segrep. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * separate out refresh and waitForReplication into different methods. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * refactor with usage of waitForReplication(). Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * fix parameters passed in factory for IndexStatsIT. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Update IndexstatsIT to run with segrep Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> --------- Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> (cherry picked from commit 30c4210) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

@ignore

…sts (#11773) (#12270) * Run few tests with Segment Replication enabled. * Update reason for ignoring test. * remove @ignore to resolve :server:forbiddenApisInternalClusterTest check. * fix spotlessCheck. * add conditional logic of force refresh. * Address comments on PR. * fix failing errors. * Use parameterization for running segment replication tests. * Fix failing tests. * Fix failing test. * add new waitForReplication() and refactor. * Address comments on PR and revert back changes made to SegmentReplication Tests. * revert changes made to Segrep tests. * Refactor and address comments. * fix failure of using forbidden api new Random(). * Add comments to debug. * Remove non-critical tests from running with segrep. * Fix test to run with segrep. * separate out refresh and waitForReplication into different methods. * refactor with usage of waitForReplication(). * fix parameters passed in factory for IndexStatsIT. * Update IndexstatsIT to run with segrep --------- (cherry picked from commit 30c4210) Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

@ignore

…sts (opensearch-project#11773) * Run few tests with Segment Replication enabled. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Update reason for ignoring test. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * remove @ignore to resolve :server:forbiddenApisInternalClusterTest check. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * fix spotlessCheck. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * add conditional logic of force refresh. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Address comments on PR. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * fix failing errors. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Use parameterization for running segment replication tests. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Fix failing tests. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Fix failing test. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * add new waitForReplication() and refactor. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Address comments on PR and revert back changes made to SegmentReplication Tests. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * revert changes made to Segrep tests. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Refactor and address comments. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * fix failure of using forbidden api new Random(). Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Add comments to debug. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Remove non-critical tests from running with segrep. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Fix test to run with segrep. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * separate out refresh and waitForReplication into different methods. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * refactor with usage of waitForReplication(). Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * fix parameters passed in factory for IndexStatsIT. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Update IndexstatsIT to run with segrep Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> --------- Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

@ignore

…sts (opensearch-project#11773) * Run few tests with Segment Replication enabled. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Update reason for ignoring test. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * remove @ignore to resolve :server:forbiddenApisInternalClusterTest check. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * fix spotlessCheck. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * add conditional logic of force refresh. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Address comments on PR. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * fix failing errors. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Use parameterization for running segment replication tests. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Fix failing tests. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Fix failing test. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * add new waitForReplication() and refactor. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Address comments on PR and revert back changes made to SegmentReplication Tests. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * revert changes made to Segrep tests. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Refactor and address comments. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * fix failure of using forbidden api new Random(). Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Add comments to debug. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Remove non-critical tests from running with segrep. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Fix test to run with segrep. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * separate out refresh and waitForReplication into different methods. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * refactor with usage of waitForReplication(). Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * fix parameters passed in factory for IndexStatsIT. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Update IndexstatsIT to run with segrep Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> --------- Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

@ignore

…sts (opensearch-project#11773) * Run few tests with Segment Replication enabled. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Update reason for ignoring test. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * remove @ignore to resolve :server:forbiddenApisInternalClusterTest check. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * fix spotlessCheck. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * add conditional logic of force refresh. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Address comments on PR. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * fix failing errors. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Use parameterization for running segment replication tests. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Fix failing tests. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Fix failing test. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * add new waitForReplication() and refactor. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Address comments on PR and revert back changes made to SegmentReplication Tests. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * revert changes made to Segrep tests. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Refactor and address comments. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * fix failure of using forbidden api new Random(). Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Add comments to debug. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Remove non-critical tests from running with segrep. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Fix test to run with segrep. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * separate out refresh and waitForReplication into different methods. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * refactor with usage of waitForReplication(). Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * fix parameters passed in factory for IndexStatsIT. Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> * Update IndexstatsIT to run with segrep Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> --------- Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

Rishikesh1159 added 2 commits January 4, 2024 17:37

Run few tests with Segment Replication enabled.

1e47a6e

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

Update reason for ignoring test.

56b1d57

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

mch2 reviewed Feb 7, 2024

View reviewed changes

server/src/internalClusterTest/java/org/opensearch/recovery/RecoveryWhileUnderLoadIT.java Outdated Show resolved Hide resolved

Rishikesh1159 and others added 5 commits February 7, 2024 16:19

Merge branch 'opensearch-project:main' into segrep-enabled-integTest

bcc0ca0

separate out refresh and waitForReplication into different methods.

d2a73b6

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

Merge branch 'segrep-enabled-integTest' of https://github.com/Rishike…

d7e9027

…sh1159/OpenSearch into segrep-enabled-integTest

refactor with usage of waitForReplication().

de8e7e2

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

fix parameters passed in factory for IndexStatsIT.

2c305cb

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

Update IndexstatsIT to run with segrep

e6d3703

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

andrross approved these changes Feb 9, 2024

View reviewed changes

andrross added backport 2.x Backport to 2.x branch and removed v2.11.0 Issues and PRs related to version 2.11.0 labels Feb 9, 2024

Rishikesh1159 merged commit 30c4210 into opensearch-project:main Feb 9, 2024
34 checks passed

opensearch-trigger-bot bot mentioned this pull request Feb 9, 2024

[Backport 2.x] [Segment Replication] Add Segment Replication Specific Integration Tests #12270

Merged

Rishikesh1159 mentioned this pull request Feb 14, 2024

[Segment Replication] Add random replication strategy #12297

Merged

8 tasks

This was referenced Feb 28, 2024

[Remote Store] Add support for Running Parameterized Remote Store Enabled Integration Tests #12487

Merged

[Rmote Store] Run integ tests with segrep and remote store enabled #12458

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Segment Replication] Add Segment Replication Specific Integration Tests #11773

[Segment Replication] Add Segment Replication Specific Integration Tests #11773

Rishikesh1159 commented Jan 5, 2024 •

edited by andrross

Loading

github-actions bot commented Jan 5, 2024

github-actions bot commented Jan 5, 2024 •

edited

Loading

github-actions bot commented Jan 5, 2024

github-actions bot commented Feb 7, 2024

github-actions bot commented Feb 7, 2024

github-actions bot commented Feb 7, 2024

github-actions bot commented Feb 8, 2024

github-actions bot commented Feb 8, 2024

github-actions bot commented Feb 8, 2024

github-actions bot commented Feb 8, 2024

github-actions bot commented Feb 8, 2024

Rishikesh1159 commented Feb 8, 2024

github-actions bot commented Feb 8, 2024

[Segment Replication] Add Segment Replication Specific Integration Tests #11773

[Segment Replication] Add Segment Replication Specific Integration Tests #11773

Conversation

Rishikesh1159 commented Jan 5, 2024 • edited by andrross Loading

Description

Related Issues

Check List

github-actions bot commented Jan 5, 2024

github-actions bot commented Jan 5, 2024 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Jan 5, 2024

github-actions bot commented Feb 7, 2024

github-actions bot commented Feb 7, 2024

github-actions bot commented Feb 7, 2024

github-actions bot commented Feb 8, 2024

github-actions bot commented Feb 8, 2024

github-actions bot commented Feb 8, 2024

github-actions bot commented Feb 8, 2024

github-actions bot commented Feb 8, 2024

Rishikesh1159 commented Feb 8, 2024

github-actions bot commented Feb 8, 2024

Rishikesh1159 commented Jan 5, 2024 •

edited by andrross

Loading

github-actions bot commented Jan 5, 2024 •

edited

Loading