Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Backport 2.x] Adding Remote translog upload, download and failover changes #5757

Merged
merged 5 commits into from
Jan 9, 2023

Conversation

gbbafna
Copy link
Collaborator

@gbbafna gbbafna commented Jan 9, 2023

Description

#5755
#5638
#5392
#5649
#5699

Issues Resolved

[List any issues this PR will resolve]

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Bukhtawar and others added 4 commits January 9, 2023 13:44
…ch-project#4172)

* Introduce TranslogFactory for Local/Remote Translog support

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
…opensearch-project#5392)

* Introduce remote translog with upload functionality 

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
Co-authored-by: Bukhtawar Khan <bukhtawa@amazon.com>
…5638)

* Enable creation of indices using Remote Translog behind a setting and feature flag
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
…log (opensearch-project#5649)

* Add support to download translog from remote store during recovery

Signed-off-by: Sachin Kale <kalsac@amazon.com>
@github-actions
Copy link
Contributor

github-actions bot commented Jan 9, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Jan 9, 2023

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testWriteLargeBlob

CHANGELOG.md Outdated
@@ -43,6 +44,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Change http code for DecommissioningFailedException from 500 to 400 ([#5283](https://github.com/opensearch-project/OpenSearch/pull/5283))
- Pre conditions check before updating weighted routing metadata ([#4955](https://github.com/opensearch-project/OpenSearch/pull/4955))
- Support remote translog transfer for request level durability([#4480](https://github.com/opensearch-project/OpenSearch/pull/4480))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets fix the change log to include what has been added

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added #5392 for Remote Translog Functionality

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add support for refresh level durability is added under Added, we can remove it from here. Also, we can remove segment store integration in failover flow entry.

* Integrate remote translog download on failover

Signed-off-by: Ashish Singh <ssashish@amazon.com>
@github-actions
Copy link
Contributor

github-actions bot commented Jan 9, 2023

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      2 org.opensearch.cluster.service.MasterServiceTests.classMethod
      1 org.opensearch.cluster.service.MasterServiceTests.testThrottlingForMultipleTaskTypes

@github-actions
Copy link
Contributor

github-actions bot commented Jan 9, 2023

Gradle Check (Jenkins) Run Completed with:

@Bukhtawar Bukhtawar changed the title [Backport 2.x] Adding Remote Translog , upload , download and failover changes [Backport 2.x] Adding Remote translog, upload, download and failover changes Jan 9, 2023
@Bukhtawar Bukhtawar changed the title [Backport 2.x] Adding Remote translog, upload, download and failover changes [Backport 2.x] Adding Remote translog upload, download and failover changes Jan 9, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Jan 9, 2023

Gradle Check (Jenkins) Run Completed with:

@gbbafna gbbafna merged commit d694685 into opensearch-project:2.x Jan 9, 2023
@gbbafna gbbafna deleted the backport-download branch January 9, 2023 16:10
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.5 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.5 2.5
# Navigate to the new working tree
pushd ../.worktrees/backport-2.5
# Create a new branch
git switch --create backport/backport-5757-to-2.5
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 d694685c109b189443fde2340599720116c24c4c
# Push it to GitHub
git push --set-upstream origin backport/backport-5757-to-2.5
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.5

Then, create a pull request where the base branch is 2.5 and the compare/head branch is backport/backport-5757-to-2.5.

sachinpkale pushed a commit to sachinpkale/OpenSearch that referenced this pull request Jan 10, 2023
…hanges (opensearch-project#5757)

* Introduce TranslogFactory for Local/Remote Translog support (opensearch-project#4172)

* Introduce TranslogFactory for Local/Remote Translog support

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

* [Remote Translog] Introduce remote translog with upload functionality (opensearch-project#5392)

* Introduce remote translog with upload functionality 

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
Co-authored-by: Bukhtawar Khan <bukhtawa@amazon.com>

* Enable creation of indices using Remote Translog    (opensearch-project#5638)

* Enable creation of indices using Remote Translog behind a setting and feature flag
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

* [Remote Translog] Add support for downloading files from remote translog (opensearch-project#5649)

* Add support to download translog from remote store during recovery

Signed-off-by: Sachin Kale <kalsac@amazon.com>

* Integrate remote translog download on failover (opensearch-project#5699)

* Integrate remote translog download on failover

Signed-off-by: Ashish Singh <ssashish@amazon.com>

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Signed-off-by: Ashish Singh <ssashish@amazon.com>
gbbafna added a commit that referenced this pull request Jan 10, 2023
* Introduce TranslogFactory for Local/Remote Translog support (#4172)

* Introduce TranslogFactory for Local/Remote Translog support

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

* [Remote Translog] Introduce remote translog with upload functionality (#5392)

* Introduce remote translog with upload functionality 

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
Co-authored-by: Bukhtawar Khan <bukhtawa@amazon.com>

* Enable creation of indices using Remote Translog    (#5638)

* Enable creation of indices using Remote Translog behind a setting and feature flag
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

* [Remote Translog] Add support for downloading files from remote translog (#5649)

* Add support to download translog from remote store during recovery

Signed-off-by: Sachin Kale <kalsac@amazon.com>

* Integrate remote translog download on failover (#5699)

* Integrate remote translog download on failover

Signed-off-by: Ashish Singh <ssashish@amazon.com>

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Signed-off-by: Ashish Singh <ssashish@amazon.com>

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Signed-off-by: Ashish Singh <ssashish@amazon.com>
Co-authored-by: Gaurav Bafna <85113518+gbbafna@users.noreply.github.com>
kotwanikunal pushed a commit that referenced this pull request Jan 25, 2023
…hanges (#5757)

* Introduce TranslogFactory for Local/Remote Translog support (#4172)

* Introduce TranslogFactory for Local/Remote Translog support

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>

* [Remote Translog] Introduce remote translog with upload functionality (#5392)

* Introduce remote translog with upload functionality 

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
Co-authored-by: Bukhtawar Khan <bukhtawa@amazon.com>

* Enable creation of indices using Remote Translog    (#5638)

* Enable creation of indices using Remote Translog behind a setting and feature flag
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

* [Remote Translog] Add support for downloading files from remote translog (#5649)

* Add support to download translog from remote store during recovery

Signed-off-by: Sachin Kale <kalsac@amazon.com>

* Integrate remote translog download on failover (#5699)

* Integrate remote translog download on failover

Signed-off-by: Ashish Singh <ssashish@amazon.com>

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Signed-off-by: Ashish Singh <ssashish@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants