Support different compression types for snapshoting

[willyborankin]

Is your feature request related to a problem? Please describe.

For now Opensearch supports only deflate compression for snapshoting (compress setting for all Cloud plugins).
It would be great to extend compression to use different compression algorithms like:

• ZSTD
• LZ4
  or even more.

Describe the solution you'd like
To support different compression types by
extending the repository settings for compression using such JSON:

{
  ...
   "settings": {
      "compression": {
         "type": "zstd", // could be `deflate`, `xz4` etc.
         "compression_settings": {
              // different compression settings like level and so on
         }
      }
   }
}

In this case it is difficult to keep backward compatibility

Describe alternatives you've considered
Compression type could be set when the repository is created/updated using such JSON:

{
  ...
   "settings": {
      "compress": true,
      "compression_type": "zstd", // `deflate`, `lz4`
   }
}

and add compression settings in opensearch.yml something like:

compression.<propery>: <some_property_value>

In this case it is possible to keep backward compatibility between versions. e.g. deflate is a default type for compression

[CEHENKLE]

@willyborankin Hey! Thanks for this.

Is this something you're thinking about contributing, or something you'd like to see (regardless of who contributes it)?

Thanks!
/C

[willyborankin]

@willyborankin Hey! Thanks for this.

Is this something you're thinking about contributing, or something you'd like to see (regardless of who contributes it)?

Thanks! /C

@CEHENKLE Hey!,
I would say both, to see and implement :-). As a first step ZSTD compression, and after that LZ4, what do you think?
Thanks!

[dblock]

@willyborankin makes sense to me!

[CEHENKLE]

@willyborankin Cool beans! I look forward to seeing the PRs :)