Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Backport 2.x] Add experimental support for zstd compression. #6982

Merged
merged 1 commit into from
Apr 5, 2023
Merged

[Backport 2.x] Add experimental support for zstd compression. #6982

merged 1 commit into from
Apr 5, 2023

Conversation

opensearch-trigger-bot[bot]
Copy link
Contributor

Backport f071c9b from #3577.

@github-actions @dblock
Add experimental support for zstd compression. (#3577)
0730657 
* Add experimental support for zstd and lz4 (native) compression.

Add experimental support for zstd (with and without dictionary support) and lz4 (native) compressions as discussed in: #3354.

Users would be able to set the index.codec setting with the values "lz4_native" (for lz4 native), "zstd" (for zstd with dictionary), and "zstd_no_dict" (for zstd without a dictionary).

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Fix license issues, add tests for zstd, zstd_no_dict, and lz4_native compressions.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Fix DCO and and issues with CodecTests.java.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Fix forbidden api violation error for lz4-java.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Fix license headers. Remove and fix unnecessary fields.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Fix magic numbers. Use more restrictive access modifiers.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Use protected access modifier for Zstd and LZ4 compression mode classes.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Allow negative compression levels for zstd. Use more restrictive access modifiers.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Use a more restrictive permission for loading zstd-jni and lz4-java libraries.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Rename a file (follow a consistent version naming convention).

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Refactor and create a new custom-codecs sandbox module.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Remove blank lines.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Restore Lucene92CustomCodec to extend from FilterCodec.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Make use of the compressionLevel argument.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Make Lucene92CustomCodec abstract and use a package-private access modifier.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Fix missing JavaDoc issues. Remove unused field in PerFieldMappingPostingFormatCodec.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Fix lint errors.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Fix the description for the custom-codecs plugin.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Fix wildcard import and improve documentation.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Access control exception fixed. Removed lz4-java support for now.

- PRs were made to zstd-jni and lz4-java to use AccessController.doPrivileged.
- The zstd-jni PR is merged since version 1.5.4-1.
- The lz4-java support temporarily removed until the PR gets merged.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Upgrade plugin to use Lucene95Codec. Rename files accordingly.

- Upgrade plugin to use Lucene95Codec. Rename files accordingly.
- Fix lint issue with plugin-security.
- Remove thridPartyAudit that was there for supporting lz4-java.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Add test cases for compression/decompression. Other minor changes.

  - add test cases for compression/decompression.
  - rename package.
  - add a CHANGELOG entry.
  - add more checks for signed integer arithmetic.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Remove ES grant in plugin-security.policy. Fix minor javadoc issue.

  - Remove ES grant in plugin-security.policy file.
  - Replace @link and @see to fix javadoc error.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Upgrade jettison version to 1.5.4.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

* Update SHA for jettison 1.5.4.

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>

---------

Signed-off-by: Mulugeta Mammo <mulugeta.mammo@intel.com>
Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>
(cherry picked from commit f071c9b)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@dblock
Copy link
Member

dblock commented Apr 4, 2023

So this will ship with 2.7.0, @mulugetam WDYT about writing a blog post or some kind of reference about the advantages of ZSTD and how to try the experimental feature? we can start/should be documenting it in https://github.com/opensearch-project/documentation-website too.

@dblock dblock added the documentation pending Tracks issues which have PRs merged but documentation changes pending label Apr 4, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Apr 4, 2023

Gradle Check (Jenkins) Run Completed with:

@dblock
Copy link
Member

dblock commented Apr 4, 2023

docker pull rate limit ... will need to retry later ... grrr

@github-actions
Copy link
Contributor

github-actions bot commented Apr 4, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Apr 4, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Apr 5, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Apr 5, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Apr 5, 2023

Gradle Check (Jenkins) Run Completed with:

@reta reta merged commit 3accec8 into 2.x Apr 5, 2023
@github-actions github-actions bot deleted the backport/backport-3577-to-2.x branch April 5, 2023 12:42
@mulugetam
Copy link
Contributor

So this will ship with 2.7.0, @mulugetam WDYT about writing a blog post or some kind of reference about the advantages of ZSTD and how to try the experimental feature? we can start/should be documenting it in https://github.com/opensearch-project/documentation-website too.

@dblock Sure, will do that and share.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@reta reta reta approved these changes

@dblock dblock dblock approved these changes

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna is a code owner

@setiah setiah Awaiting requested review from setiah

@kartg kartg Awaiting requested review from kartg

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 is a code owner

@nknize nknize Awaiting requested review from nknize nknize is a code owner

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

@ryanbogan ryanbogan Awaiting requested review from ryanbogan

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

@dreamer-89 dreamer-89 Awaiting requested review from dreamer-89

@tlfeng tlfeng Awaiting requested review from tlfeng

@VachaShah VachaShah Awaiting requested review from VachaShah VachaShah is a code owner

@xuezhou25 xuezhou25 Awaiting requested review from xuezhou25

Assignees
No one assigned
Labels
documentation pending Tracks issues which have PRs merged but documentation changes pending
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dblock @mulugetam @reta