Set target routing shard by partition key #316

acarbonetto · 2023-07-26T23:43:32Z

Description

Allows user to set routing shard via the relation partition keys. For example:

select _id, _index, _routing, int0 from calcs_routing PARTITION(KEY1, KEY2) where _routing = 'KEY1' or _routing = 'KEY2'

Note that multiple keys can be send to OpenSearch storage by using comma-delimited list.

IT tests

New IT tests running against a multi-node cluster have been added to the build. These can be run using ./gradlew :integ-test:multiClusterSearch and will be automatically run against CI.

TODO

Remove quotes/backticks from around the routingId before sending
Rebase
Propose PPL syntax

Issues Resolved

resolves Allow individual shards to be targeted during query execution [FEATURE] opensearch-project/sql#1478 for SQL

Note (out of scope): PPL syntax is still TBD

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov · 2023-07-26T23:57:18Z

Codecov Report

Merging #316 (087a5b3) into integ-metafields-set-routing-shard (31b2b28) will decrease coverage by 0.01%.
The diff coverage is 97.43%.

@@                           Coverage Diff                            @@
##             integ-metafields-set-routing-shard     #316      +/-   ##
========================================================================
- Coverage                                 97.42%   97.41%   -0.01%     
- Complexity                                 4647     4651       +4     
========================================================================
  Files                                       408      408              
  Lines                                     11526    11549      +23     
  Branches                                    839      845       +6     
========================================================================
+ Hits                                      11229    11251      +22     
  Misses                                      290      290              
- Partials                                      7        8       +1

Flag	Coverage Δ
sql-engine	`97.41% <97.43%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...java/org/opensearch/sql/storage/StorageEngine.java	`100.00% <ø> (ø)`
...ql/prometheus/storage/PrometheusStorageEngine.java	`100.00% <ø> (ø)`
...ensearch/sql/spark/storage/SparkStorageEngine.java	`100.00% <ø> (ø)`
...org/opensearch/sql/sql/domain/SQLQueryRequest.java	`100.00% <ø> (ø)`
...ql/opensearch/request/OpenSearchScrollRequest.java	`98.24% <88.88%> (-1.76%)`	⬇️
...ain/java/org/opensearch/sql/analysis/Analyzer.java	`100.00% <100.00%> (ø)`
...sql/opensearch/request/OpenSearchQueryRequest.java	`100.00% <100.00%> (ø)`
...l/opensearch/request/OpenSearchRequestBuilder.java	`100.00% <100.00%> (ø)`
...ch/sql/opensearch/response/OpenSearchResponse.java	`100.00% <100.00%> (ø)`
...search/sql/opensearch/storage/OpenSearchIndex.java	`100.00% <100.00%> (ø)`
... and 2 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

MaxKsyunz

@acarbonetto do you think we can pass routingId from query to OpenSearchRequest without having to modify

legacy/src/main/java/org/opensearch/sql/legacy/request/SqlRequestFactory.java

MaxKsyunz · 2023-07-28T01:35:27Z

legacy/src/test/java/org/opensearch/sql/legacy/plugin/RestSQLQueryActionCursorFallbackTest.java

@@ -98,6 +99,7 @@ private static SQLQueryRequest createSqlQueryRequest(String query, Optional<Stri
    builder.endObject();
    JSONObject jsonContent = new JSONObject(Strings.toString(builder));

+    // TODO pass through


MaxKsyunz · 2023-07-28T01:45:22Z

docs/user/dql/basics.rst

@@ -363,6 +363,17 @@ SQL query::
 	{
 	  "query" : "SELECT account_number FROM accounts/account"
 	}
+Example 4: Selecting From Index using Partition Shard


What happens when PARTITION is provided for another data source, like Prometheus or Spark?

They will need to implement a sharing or partition option separately. Most datasources would have such an option.

MaxKsyunz · 2023-07-28T01:53:52Z

docs/user/dql/basics.rst

+
+	POST /_plugins/_sql
+	{
+	  "query" : "SELECT account_number FROM account PARTITION(shard1, shard2)"


It may be confusing to use PARTITION an existing operation in SQL for this.

What about introducing a general notion of data source options, keeping them as key-value pairs in Relation that get analyzed by each StorageEngine.

This would allow other data sources to benefit from this and avoid question about features that only apply to OpenSearch.

This is exactly what the PARTITION in SQL is designed for. Also, the proposal in the issue suggested using this syntax.

I DO like your idea, but I'd rather stick with well-understood SQL syntax than introduce something new.

Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

* Fix core refactor: StreamIO from common to core.common Signed-off-by: acarbonetto <andrewc@bitquilltech.com> * Fix core refactor: StreamIO from common to core.common Signed-off-by: acarbonetto <andrewc@bitquilltech.com> --------- Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

…-set-routing-shard Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

…routing-shard Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

acarbonetto requested review from MaxKsyunz, Yury-Fridlyand, forestmvey and GumpacG as code owners July 26, 2023 23:43

acarbonetto changed the title ~~Dev metafields set routing shard~~ Set target routing shard by partition key Jul 27, 2023

MaxKsyunz reviewed Jul 28, 2023

View reviewed changes

acarbonetto added 11 commits July 28, 2023 08:40

Update documentation for _routing

f6e0e21

Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

Proof of Concept: request routing shard through SQL partition

21318f1

Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

Add IT tests for multi-cluster

382e955

Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

Fix cross-cluster tests

6ca8fec

Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

Revert query parameter 'routing'

8edcc54

Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

Fix unit tests with partition add

97379bb

Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

clean up checkstyle and add test coverage

5de0897

Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

Fix IT tests

d543849

Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

Add multi-cluster search tests to build

283428c

Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

Updates for comments

b1928ae

Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

acarbonetto force-pushed the dev-metafields-set-routing-shard branch 2 times, most recently from b1928ae to 430d7a9 Compare July 28, 2023 16:06

acarbonetto added 4 commits July 28, 2023 09:47

Update comment

ef9332a

Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

Merge branch 'integ-metafields-set-routing-shard' into dev-metafields…

5d18908

…-set-routing-shard Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

Merge remote-tracking branch 'upstream/main' into dev-metafields-set-…

f1371f7

…routing-shard Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

Fix doctests

087a5b3

Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

acarbonetto merged commit f6643f7 into integ-metafields-set-routing-shard Aug 23, 2023
5 of 12 checks passed

acarbonetto deleted the dev-metafields-set-routing-shard branch August 23, 2023 20:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set target routing shard by partition key #316

Set target routing shard by partition key #316

acarbonetto commented Jul 26, 2023 •

edited

Loading

codecov bot commented Jul 26, 2023 •

edited

Loading

MaxKsyunz left a comment

MaxKsyunz Jul 28, 2023

MaxKsyunz Jul 28, 2023

acarbonetto Jul 28, 2023

MaxKsyunz Jul 28, 2023

acarbonetto Jul 28, 2023

Set target routing shard by partition key #316

Set target routing shard by partition key #316

Conversation

acarbonetto commented Jul 26, 2023 • edited Loading

Description

IT tests

TODO

Issues Resolved

Check List

codecov bot commented Jul 26, 2023 • edited Loading

Codecov Report

MaxKsyunz left a comment

Choose a reason for hiding this comment

MaxKsyunz Jul 28, 2023

Choose a reason for hiding this comment

MaxKsyunz Jul 28, 2023

Choose a reason for hiding this comment

acarbonetto Jul 28, 2023

Choose a reason for hiding this comment

MaxKsyunz Jul 28, 2023

Choose a reason for hiding this comment

acarbonetto Jul 28, 2023

Choose a reason for hiding this comment

acarbonetto commented Jul 26, 2023 •

edited

Loading

codecov bot commented Jul 26, 2023 •

edited

Loading