[Segment Replication] Introduce cluster level setting `cluster.index.restrict.replication.type` to prevent replication type setting override during index creations #11583

dreamer-89 · 2023-12-11T23:48:41Z

Description

Introduces node level cluster setting cluster.index.restrict.replication.type to restrict setting replication type setting at index level

Related Issues

Resolves ##11457

Documentation Issue

opensearch-project/documentation-website#5806

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-12-11T23:57:34Z

❌ Gradle check result for 54c4bae: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2023-12-12T00:08:07Z

❌ Gradle check result for 722f064: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2023-12-12T00:10:46Z

Compatibility status:

Checks if related components are compatible with change 948ce3a

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/sql.git]

github-actions · 2023-12-12T21:14:02Z

❌ Gradle check result for 2333244: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Suraj Singh <surajrider@gmail.com>

dreamer-89 · 2023-12-20T02:43:33Z

@kotwanikunal @andrross : Thanks for the feedback. Addressed the review comments. Please have a look.

github-actions · 2023-12-20T03:28:25Z

❕ Gradle check result for 39d47be: UNSTABLE

TEST FAILURES:

      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

...lClusterTest/java/org/opensearch/indices/replication/SegmentReplicationClusterSettingIT.java

Signed-off-by: Suraj Singh <surajrider@gmail.com>

dreamer-89 · 2023-12-20T19:27:11Z

@andrross @kotwanikunal : Addressed review comments. Please have a look.

github-actions · 2023-12-20T20:09:10Z

❕ Gradle check result for 81d63ba: UNSTABLE

TEST FAILURES:

      1 org.opensearch.remotestore.RemoteStoreStatsIT.testDownloadStatsCorrectnessSinglePrimaryMultipleReplicaShards
      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-12-20T20:22:19Z

❌ Gradle check result for 46c530d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Suraj Singh <surajrider@gmail.com>

github-actions · 2023-12-20T22:23:47Z

❌ Gradle check result for 46c530d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2023-12-20T23:13:41Z

❕ Gradle check result for 948ce3a: UNSTABLE

TEST FAILURES:

      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

andrross · 2023-12-21T18:15:22Z

Looks good to me. @kotwanikunal what do you think?

kotwanikunal · 2023-12-21T19:22:51Z

LGTM.

opensearch-trigger-bot · 2023-12-21T20:18:40Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-11583-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 6a6ab32e532d16de222645ccfae376c778c68991
# Push it to GitHub
git push --set-upstream origin backport/backport-11583-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-11583-to-2.x.

…restrict.replication.type` to prevent replication type setting override during index creations (opensearch-project#11583) * Add new cluster level setting that prevents index level replication type setting overrides Signed-off-by: Suraj Singh <surajrider@gmail.com> * Block restore on mis-matching replication type setting Signed-off-by: Suraj Singh <surajrider@gmail.com> * Address review comments and rebase Signed-off-by: Suraj Singh <surajrider@gmail.com> * Address review comments Signed-off-by: Suraj Singh <surajrider@gmail.com> * Use appropriate variable names Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix failing integ test Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Suraj Singh <surajrider@gmail.com> (cherry picked from commit 6a6ab32)

…estrict.replication.type` to prevent replication type setting override during index creations (#11583) (#11707) * [Segment Replication] Introduce cluster level setting `cluster.index.restrict.replication.type` to prevent replication type setting override during index creations (#11583) * Add new cluster level setting that prevents index level replication type setting overrides Signed-off-by: Suraj Singh <surajrider@gmail.com> * Block restore on mis-matching replication type setting Signed-off-by: Suraj Singh <surajrider@gmail.com> * Address review comments and rebase Signed-off-by: Suraj Singh <surajrider@gmail.com> * Address review comments Signed-off-by: Suraj Singh <surajrider@gmail.com> * Use appropriate variable names Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix failing integ test Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Suraj Singh <surajrider@gmail.com> (cherry picked from commit 6a6ab32) * Remove duplicate IndicesService.CLUSTER_REMOTE_INDEX_RESTRICT_ASYNC_DURABILITY_SETTING setting introduce from conflict resolution Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Suraj Singh <surajrider@gmail.com>

…restrict.replication.type` to prevent replication type setting override during index creations (opensearch-project#11583) * Add new cluster level setting that prevents index level replication type setting overrides Signed-off-by: Suraj Singh <surajrider@gmail.com> * Block restore on mis-matching replication type setting Signed-off-by: Suraj Singh <surajrider@gmail.com> * Address review comments and rebase Signed-off-by: Suraj Singh <surajrider@gmail.com> * Address review comments Signed-off-by: Suraj Singh <surajrider@gmail.com> * Use appropriate variable names Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix failing integ test Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Suraj Singh <surajrider@gmail.com>

…restrict.replication.type` to prevent replication type setting override during index creations (opensearch-project#11583) * Add new cluster level setting that prevents index level replication type setting overrides Signed-off-by: Suraj Singh <surajrider@gmail.com> * Block restore on mis-matching replication type setting Signed-off-by: Suraj Singh <surajrider@gmail.com> * Address review comments and rebase Signed-off-by: Suraj Singh <surajrider@gmail.com> * Address review comments Signed-off-by: Suraj Singh <surajrider@gmail.com> * Use appropriate variable names Signed-off-by: Suraj Singh <surajrider@gmail.com> * Fix failing integ test Signed-off-by: Suraj Singh <surajrider@gmail.com> --------- Signed-off-by: Suraj Singh <surajrider@gmail.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

dreamer-89 changed the title ~~[Segment Replication] Introduce cluster level setting to restrict setting replication type setting at index level~~ [Segment Replication] Introduce cluster level setting to force prevent index level replication type overrides Dec 20, 2023

Address review comments and rebase

39d47be

Signed-off-by: Suraj Singh <surajrider@gmail.com>

dreamer-89 force-pushed the cluster_level_setting branch from db7f2d9 to 39d47be Compare December 20, 2023 02:36

andrross reviewed Dec 20, 2023

View reviewed changes

dreamer-89 added 2 commits December 20, 2023 11:21

Address review comments

81d63ba

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Use appropriate variable names

46c530d

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Fix failing integ test

948ce3a

Signed-off-by: Suraj Singh <surajrider@gmail.com>

andrross approved these changes Dec 21, 2023

View reviewed changes

kotwanikunal approved these changes Dec 21, 2023

View reviewed changes

kotwanikunal merged commit 6a6ab32 into opensearch-project:main Dec 21, 2023
31 checks passed

andrross added the backport 2.x Backport to 2.x branch label Dec 21, 2023

opensearch-trigger-bot bot added the backport-failed label Dec 21, 2023

dreamer-89 mentioned this pull request Jan 2, 2024

[Segment Replication] Introduce cluster level setting cluster.indexrestrict.replication.type to prevent replication type setting override during index creations (#11583) #11707

Merged

BrewTestBot mentioned this pull request Feb 21, 2024

opensearch 2.12.0 Homebrew/homebrew-core#163463

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Segment Replication] Introduce cluster level setting `cluster.index.restrict.replication.type` to prevent replication type setting override during index creations #11583

[Segment Replication] Introduce cluster level setting `cluster.index.restrict.replication.type` to prevent replication type setting override during index creations #11583

dreamer-89 commented Dec 11, 2023 •

edited

Loading

github-actions bot commented Dec 11, 2023

github-actions bot commented Dec 12, 2023

github-actions bot commented Dec 12, 2023 •

edited

Loading

github-actions bot commented Dec 12, 2023

dreamer-89 commented Dec 20, 2023

github-actions bot commented Dec 20, 2023

dreamer-89 commented Dec 20, 2023

github-actions bot commented Dec 20, 2023

github-actions bot commented Dec 20, 2023

github-actions bot commented Dec 20, 2023

github-actions bot commented Dec 20, 2023

andrross commented Dec 21, 2023

kotwanikunal commented Dec 21, 2023

opensearch-trigger-bot bot commented Dec 21, 2023

[Segment Replication] Introduce cluster level setting cluster.index.restrict.replication.type to prevent replication type setting override during index creations #11583

[Segment Replication] Introduce cluster level setting cluster.index.restrict.replication.type to prevent replication type setting override during index creations #11583

Conversation

dreamer-89 commented Dec 11, 2023 • edited Loading

Description

Related Issues

Documentation Issue

Check List

github-actions bot commented Dec 11, 2023

github-actions bot commented Dec 12, 2023

github-actions bot commented Dec 12, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Dec 12, 2023

dreamer-89 commented Dec 20, 2023

github-actions bot commented Dec 20, 2023

dreamer-89 commented Dec 20, 2023

github-actions bot commented Dec 20, 2023

github-actions bot commented Dec 20, 2023

github-actions bot commented Dec 20, 2023

github-actions bot commented Dec 20, 2023

andrross commented Dec 21, 2023

kotwanikunal commented Dec 21, 2023

opensearch-trigger-bot bot commented Dec 21, 2023

[Segment Replication] Introduce cluster level setting `cluster.index.restrict.replication.type` to prevent replication type setting override during index creations #11583

[Segment Replication] Introduce cluster level setting `cluster.index.restrict.replication.type` to prevent replication type setting override during index creations #11583

dreamer-89 commented Dec 11, 2023 •

edited

Loading

github-actions bot commented Dec 12, 2023 •

edited

Loading