Fixes Numeric exact match queries to use range queries internally #11209

harshavamsi · 2023-11-15T02:31:43Z

Description

Updates term and terms queries for numeric fields to use IndexOrDocValues where appropriate.

Related Issues

Resolves #11097

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-11-15T02:41:08Z

❌ Gradle check result for 6a324a4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2023-11-15T02:53:17Z

Compatibility status:

Checks if related components are compatible with change 339a681

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/performance-analyzer.git]

github-actions · 2023-11-15T17:59:47Z

❌ Gradle check result for 443e205: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2023-11-15T22:47:35Z

❌ Gradle check result for 36708f8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2023-11-15T23:29:41Z

❌ Gradle check result for d97865e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2023-11-16T01:21:41Z

❌ Gradle check result for bbd4e4e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2023-11-16T02:08:03Z

❌ Gradle check result for 1289b3e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2023-11-16T17:55:56Z

❌ Gradle check result for 27f4e47: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2023-11-16T19:33:47Z

❕ Gradle check result for c5acd4c: UNSTABLE

TEST FAILURES:

      1 org.opensearch.index.shard.RemoteIndexShardTests.testRepicaCleansUpOldCommitsWhenReceivingNew

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-11-16T19:34:39Z

Codecov Report

Attention: 166 lines in your changes are missing coverage. Please review.

Comparison is base (8440468) 71.44% compared to head (339a681) 70.90%.
Report is 3 commits behind head on main.

Files	Patch %	Lines
...org/opensearch/index/mapper/NumberFieldMapper.java	40.26%	68 Missing and 21 partials ⚠️
.../document/SortedUnsignedLongDocValuesSetQuery.java	15.09%	43 Missing and 2 partials ⚠️
.../main/java/org/apache/lucene/util/LongHashSet.java	39.62%	21 Missing and 11 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #11209      +/-   ##
============================================
- Coverage     71.44%   70.90%   -0.54%     
+ Complexity    59267    58864     -403     
============================================
  Files          4909     4911       +2     
  Lines        278427   278651     +224     
  Branches      40460    40524      +64     
============================================
- Hits         198910   197585    -1325     
- Misses        63004    64661    +1657     
+ Partials      16513    16405     -108

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

harshavamsi · 2023-12-16T05:54:11Z

Gradle failure is flaky test -- #5329

github-actions · 2024-01-03T19:03:39Z

❌ Gradle check result for 430310a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

github-actions · 2024-01-03T19:35:59Z

❌ Gradle check result for b54a539: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

reta · 2024-01-03T19:51:31Z

@harshavamsi please rebase against latest main, thank you

github-actions · 2024-01-03T20:34:08Z

❕ Gradle check result for ebba45f: UNSTABLE

TEST FAILURES:

      2 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing
      1 org.opensearch.index.shard.RemoteIndexShardTests.testNRTReplicaWithRemoteStorePromotedAsPrimaryCommitRefresh

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

reta · 2024-01-03T20:39:18Z

@msfroh could you please take a look? (waiting for your signoff) thank you

CHANGELOG.md

msfroh · 2024-01-03T21:06:28Z

Just a residual merge conflict on the changelog, where you picked up a couple of other changes. Otherwise, it looks good to me 👍

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

harshavamsi · 2024-01-03T21:22:02Z

Just a residual merge conflict on the changelog, where you picked up a couple of other changes. Otherwise, it looks good to me 👍

Thanks @msfroh, fixed the changelog.

github-actions · 2024-01-03T22:30:52Z

❌ Gradle check result for 339a681: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-01-04T18:00:38Z

❕ Gradle check result for 339a681: UNSTABLE

TEST FAILURES:

      2 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.classMethod
      1 org.opensearch.search.SearchTimeoutIT.testSimpleTimeout {p0={"search.concurrent_segment_search.enabled":"true"}}
      1 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testOverriddenBufferInterval
      1 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshAfterCommit

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

…1209) * Updating numeric term and terms queries to use IODVQ Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Addressing comments Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Fix formatting Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Fix changelog Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Addressing more comments + adding tests Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * renaming yaml test Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Adding skip for bwc Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Adding new SortedUnsignedLongDocValuesSetQuery to allow for BitInteger Terms query Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Fixing some tests Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Remove duplicate skip Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Remove unused points declaration Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Change unsigned exact query to be consistent Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Use slowExactQuery from Unsigned Set Query Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Merging different yaml tests into a single test Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Updating test case for main Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Fix changelog Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> --------- Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> (cherry picked from commit 178a7a0) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…1209) (#11747) * Updating numeric term and terms queries to use IODVQ * Addressing comments * Fix formatting * Fix changelog * Addressing more comments + adding tests * renaming yaml test * Adding skip for bwc * Adding new SortedUnsignedLongDocValuesSetQuery to allow for BitInteger Terms query * Fixing some tests * Remove duplicate skip * Remove unused points declaration * Change unsigned exact query to be consistent * Use slowExactQuery from Unsigned Set Query * Merging different yaml tests into a single test * Updating test case for main * Fix changelog --------- (cherry picked from commit 178a7a0) Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ensearch-project#11209) * Updating numeric term and terms queries to use IODVQ Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Addressing comments Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Fix formatting Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Fix changelog Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Addressing more comments + adding tests Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * renaming yaml test Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Adding skip for bwc Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Adding new SortedUnsignedLongDocValuesSetQuery to allow for BitInteger Terms query Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Fixing some tests Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Remove duplicate skip Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Remove unused points declaration Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Change unsigned exact query to be consistent Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Use slowExactQuery from Unsigned Set Query Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Merging different yaml tests into a single test Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Updating test case for main Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Fix changelog Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> --------- Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

…ensearch-project#11209) * Updating numeric term and terms queries to use IODVQ Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Addressing comments Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Fix formatting Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Fix changelog Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Addressing more comments + adding tests Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * renaming yaml test Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Adding skip for bwc Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Adding new SortedUnsignedLongDocValuesSetQuery to allow for BitInteger Terms query Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Fixing some tests Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Remove duplicate skip Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Remove unused points declaration Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Change unsigned exact query to be consistent Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Use slowExactQuery from Unsigned Set Query Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Merging different yaml tests into a single test Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Updating test case for main Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Fix changelog Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> --------- Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

github-actions bot added bug Something isn't working Search:Performance v2.12.0 Issues and PRs related to version 2.12.0 labels Nov 15, 2023

harshavamsi force-pushed the numeric_exact_match branch from 36708f8 to d97865e Compare November 15, 2023 22:57

harshavamsi force-pushed the numeric_exact_match branch from bbd4e4e to 1289b3e Compare November 16, 2023 01:21

harshavamsi marked this pull request as ready for review November 16, 2023 21:40

harshavamsi requested review from abbashus, adnapibar, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89, gbbafna, kartg, kotwanikunal, mch2 and msfroh as code owners November 16, 2023 21:40

Updating test case for main

b54a539

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

harshavamsi force-pushed the numeric_exact_match branch from 430310a to b54a539 Compare January 3, 2024 19:08

Merge branch 'main' into numeric_exact_match

ebba45f

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

reta self-requested a review January 3, 2024 20:37

reta approved these changes Jan 3, 2024

View reviewed changes

msfroh reviewed Jan 3, 2024

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

Fix changelog

339a681

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

msfroh approved these changes Jan 4, 2024

View reviewed changes

reta added the v3.0.0 Issues and PRs related to version 3.0.0 label Jan 4, 2024

reta merged commit 178a7a0 into opensearch-project:main Jan 4, 2024
32 of 33 checks passed

reta added the backport 2.x Backport to 2.x branch label Jan 4, 2024

opensearch-trigger-bot bot mentioned this pull request Jan 4, 2024

[Backport 2.x] Fixes Numeric exact match queries to use range queries internally #11747

Merged

harshavamsi mentioned this pull request Jan 5, 2024

Update skip version to match branches #11776

Merged

8 tasks

msfroh mentioned this pull request Feb 15, 2024

[Feature Request] terms query's term lookup should be able to efficiently handle 100k+ (or 1M+) terms #12341

Closed

BrewTestBot mentioned this pull request Feb 21, 2024

opensearch 2.12.0 Homebrew/homebrew-core#163463

Merged

harshavamsi mentioned this pull request Feb 29, 2024

Fix issue with float field terms query #12499

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes Numeric exact match queries to use range queries internally #11209

Fixes Numeric exact match queries to use range queries internally #11209

harshavamsi commented Nov 15, 2023 •

edited

Loading

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023 •

edited

Loading

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 16, 2023

github-actions bot commented Nov 16, 2023

github-actions bot commented Nov 16, 2023

github-actions bot commented Nov 16, 2023

codecov bot commented Nov 16, 2023 •

edited

Loading

harshavamsi commented Dec 16, 2023

github-actions bot commented Jan 3, 2024

github-actions bot commented Jan 3, 2024

reta commented Jan 3, 2024

github-actions bot commented Jan 3, 2024

reta commented Jan 3, 2024

msfroh commented Jan 3, 2024

harshavamsi commented Jan 3, 2024

github-actions bot commented Jan 3, 2024

github-actions bot commented Jan 4, 2024

Fixes Numeric exact match queries to use range queries internally #11209

Fixes Numeric exact match queries to use range queries internally #11209

Conversation

harshavamsi commented Nov 15, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 16, 2023

github-actions bot commented Nov 16, 2023

github-actions bot commented Nov 16, 2023

github-actions bot commented Nov 16, 2023

codecov bot commented Nov 16, 2023 • edited Loading

Codecov Report

harshavamsi commented Dec 16, 2023

github-actions bot commented Jan 3, 2024

github-actions bot commented Jan 3, 2024

reta commented Jan 3, 2024

github-actions bot commented Jan 3, 2024

reta commented Jan 3, 2024

msfroh commented Jan 3, 2024

harshavamsi commented Jan 3, 2024

github-actions bot commented Jan 3, 2024

github-actions bot commented Jan 4, 2024

harshavamsi commented Nov 15, 2023 •

edited

Loading

github-actions bot commented Nov 15, 2023 •

edited

Loading

codecov bot commented Nov 16, 2023 •

edited

Loading