Index Level Encryption plugin

[asonje]

Description

This pull request adds index level encryption features to OpenSearch based on the issue #3469. Each OpenSearch index is individually encrypted based on user provided encryption keys. A new cryptofs store type index.store.type is introduced which instantiates a CryptoDirectory that encrypts and decrypts files as they are written and read respectively

Related Issues

Resolves #[Issue number to be closed when this PR is merged]
#3469

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change 2da3eca

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git]

[github-actions]

❌ Gradle check result for e5e930b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 2da3eca: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[Bukhtawar]

As a part of moving security closer to the core should we consider moving CryptoDirectory to core or even better module and crypto providers as plugins instead

[varun-lodaya]

@asonje thanks for putting the proposal and PR on this. This is definitely something we should get into OpenSearch.

I am yet to see the PR but couple of high level questions -

• Have you done any performance benchmarking to understand the impact? I see your mention of some initial testing and impact in the Proposal but not quantification - On The Fly Encryption Feature Proposal #3469?
• +1 to Bukhtawar's comment. Will let @peternied add on this, but generally speaking, does this need a new plugin? Can we build this functionality in core?

[andrross]

As a part of moving security closer to the core should we consider moving CryptoDirectory to core or even better module and crypto providers as plugins instead

From a software architecture perspective I think it make sense to build on the existing abstractions (e.g. the directory factory/IndexStorePlugin). As to whether it is a module in core, a plugin in the core repo, or a plugin in a separate repo are good questions. However, we shouldn't take any decision here lightly as we'll need to support whatever we release here for multiple major versions since it is essentially writing a new data format. My initial suggestion is that the sandbox in the core repo is probably a good fit for this feature while it incubates and is proven out.

[varun-lodaya]

From a software architecture perspective I think it make sense to build on the existing abstractions (e.g. the directory factory/IndexStorePlugin). As to whether it is a module in core, a plugin in the core repo, or a plugin in a separate repo are good questions. However, we shouldn't take any decision here lightly as we'll need to support whatever we release here for multiple major versions since it is essentially writing a new data format. My initial suggestion is that the sandbox in the core repo is probably a good fit for this feature while it incubates and is proven out.

Sounds fair to me.

[Bukhtawar]

The idea is to decouple abstractions and implementation. The abstraction can be closer to the goal, while the implementations or crypto providers can be plugins.

[asonje]

• Have you done any performance benchmarking to understand the impact?

Till now I have focused more on correctness and don't have any performance numbers to share at the moment. There will be a noticeable impact to performance though relative to the HybridDirectory; there is no crypto equivalent yet for MMapDirectory. That would be a good addition for performance.

[asonje]

If there are no objections or further comments I will move this to sandbox/plugins per @andrross' suggestion and submit a new PR. Thanks

[asonje]

updated PR: #12902