Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Draft] Search Query Categorizor initial skeleton using QueryBuilderVisitor #10255

Closed

Conversation

deshsidd
Copy link
Contributor

@deshsidd deshsidd commented Sep 27, 2023

  • Initial skeleton changes
  • Using QueryBuilderVisitor to visit all the nodes of the querybuilder tree
  • The tree referred to here is the tree of querybuilders for the incoming search query with the topLevelQueryBuilder as the root.
  • Integration with Metric Framework : Add Telemetry metrics framework #10241
  • Integration with QueryShapeVisitor to log the shape of the queries : https://gist.github.com/msfroh/74aa3fee52f4074c5e7b8d85f76e88ab
  • Need to also figure out all the types of queries we want to increment counters for

@deshsidd deshsidd changed the title Search Query Categorizor initial skeleton using QueryBuilderVisitor [Draft] Search Query Categorizor initial skeleton using QueryBuilderVisitor Sep 27, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Sep 27, 2023

Compatibility status:

Checks if related components are compatible with change 3338d4f

Incompatible components

Skipped components

Compatible components

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@msfroh
Copy link
Collaborator

msfroh commented Oct 2, 2023

The sample implementation doesn't tell us whether given query clause is required or not, or if it's involved in scoring. These have major implications for performance.

Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com>
@deshsidd deshsidd force-pushed the sid/search-query-categorizor branch from d1a1f2c to fadba4c Compare October 4, 2023 05:57
mch2 and others added 27 commits October 9, 2023 01:10
…ch-project#10347)

* Update multipart download path to write to temp files.

This change updates ReadContextListener to first write parts to a temp location
until all parts have been received.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Suppress forbidden IOUtils.fsync

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Remove unnecessary logging format

Signed-off-by: Marc Handalian <handalm@amazon.com>

* sync directory after file rename

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Remove flaky threadpool terminate test

Signed-off-by: Marc Handalian <handalm@amazon.com>

---------

Signed-off-by: Marc Handalian <handalm@amazon.com>
…nsearch-project#9569)

---------

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>
…oject#10326)

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>
…ject#10205)

Bumps org.apache.commons:commons-compress from 1.23.0 to 1.24.0.

---
updated-dependencies:
- dependency-name: org.apache.commons:commons-compress
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Gagan Juneja <gjjuneja@amazon.com>
…h-project#10016)

* fix stale remote cluster uuid state not purged from remote

Signed-off-by: bansvaru <bansvaru@amazon.com>

* fix tests

Signed-off-by: bansvaru <bansvaru@amazon.com>

* use new limit parameter

Signed-off-by: bansvaru <bansvaru@amazon.com>

* minor refactoring

Signed-off-by: bansvaru <bansvaru@amazon.com>

* delete index metadata files before manifest file

Signed-off-by: bansvaru <bansvaru@amazon.com>

* add basic UT

Signed-off-by: bansvaru <bansvaru@amazon.com>

* delete all data related to a cluster uuid in a single call

Signed-off-by: bansvaru <bansvaru@amazon.com>

* fix git diff

Signed-off-by: bansvaru <bansvaru@amazon.com>

* remove unreferenced code

Signed-off-by: bansvaru <bansvaru@amazon.com>

* fix spa

Signed-off-by: bansvaru <bansvaru@amazon.com>

---------

Signed-off-by: bansvaru <bansvaru@amazon.com>
…n/tools/plugin-cli (opensearch-project#10297)

* Bump org.bouncycastle:bc-fips in /distribution/tools/plugin-cli

Bumps org.bouncycastle:bc-fips from 1.0.2.3 to 1.0.2.4.

---
updated-dependencies:
- dependency-name: org.bouncycastle:bc-fips
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

* Update changelog

Signed-off-by: dependabot[bot] <support@github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Signed-off-by: Bhumika Saini <sabhumik@amazon.com>
* Race condition fix for datetime optimization

Signed-off-by: Prabhat Sharma <ptsharma@amazon.com>

* Changed JavaDateTimeFormatter caching of parser from MRU(most recently used) to a simple last used formatter

Signed-off-by: Prabhat Sharma <ptsharma@amazon.com>

---------

Signed-off-by: Prabhat Sharma <ptsharma@amazon.com>
Co-authored-by: Prabhat Sharma <ptsharma@amazon.com>
…ata fields in ingest simulate API (opensearch-project#10101)

* Fix class_cast_exception when passing int to _version and other metadata fields in ingest simulate API

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* modify change log

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Add more tests

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

---------

Signed-off-by: Gao Binlong <gbinlong@amazon.com>
Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>
Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>
…ases (opensearch-project#10370)

* Segment Replication - Fix ShardLockObtained error during corruption cases

This change fixes a bug where shards could not be recreated locally after corruption.
This occured because the store was not decref'd to 0 if the commit on close would fail
with a corruption exception.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Remove exra logs

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Remove flaky assertion on store refcount

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Remove flaky test.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* PR Feedback.

Remove hacky handling of corruption when fetching metadata.  This will now check for store corruption
when replication has failed and fail the shard accordingly.

This commit also fixes logging in NRTReplicationEngine.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix unit test.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix test failure testSegRepSucceedsOnPreviousCopiedFiles.

This test broke because we invoked target.indexShard on a closed replicationTarget.
In these cases we can assume the store is not corrupt.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* spotless

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Revert flaky IT

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix flakiness failure by expecting RTE when check index fails.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* reintroduce ITs and use recoveries API instead of waiting on shard state.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix edge case where flush failures would not get reported as corruption.

Signed-off-by: Marc Handalian <handalm@amazon.com>

---------

Signed-off-by: Marc Handalian <handalm@amazon.com>
Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>
…arch-project#10402)

* Fix stats reporting for multistream downloads.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* rename tracker to fileTransferTracker.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

---------

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
… allowed in base64UUID url charset (opensearch-project#10379)

* Refactor Remote Store Metadata Lock Manager Utils

Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com>

* Address PR Comments

Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com>

* Address PR Comments

Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com>

* Update Changelog entry

Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com>

* Update Changelog entry

Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com>

* Unmute testDeleteShallowCopySnapshot test

Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com>

---------

Signed-off-by: Harish Bhakuni <hbhakuni@amazon.com>
Co-authored-by: Harish Bhakuni <hbhakuni@amazon.com>
…eft on device' (opensearch-project#10409)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
…ettings during s3 plugin reload (opensearch-project#10452)

Signed-off-by: vikasvb90 <vikasvb@amazon.com>
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
…25.1 in /plugins/repository-gcs (opensearch-project#10298)

* Bump com.google.api.grpc:proto-google-common-protos

Bumps [com.google.api.grpc:proto-google-common-protos](https://github.com/googleapis/sdk-platform-java) from 2.25.0 to 2.25.1.
- [Release notes](https://github.com/googleapis/sdk-platform-java/releases)
- [Changelog](https://github.com/googleapis/sdk-platform-java/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/sdk-platform-java/commits)

---
updated-dependencies:
- dependency-name: com.google.api.grpc:proto-google-common-protos
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

* Update changelog

Signed-off-by: dependabot[bot] <support@github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…project#10437)

* DiversifiedSamplerIT.testNestedSamples
* QueryProfilePhaseTests.testMaxScore
* QueryProfilePhaseTests.testCollapseQuerySearchResults
* HighlighterSearchIT.testHighlightQueryRewriteDatesWithNow
* FieldCapabilitiesIT.testWithIndexFilter
* QueryProfilePhaseTests.testDisableTopScoreCollection

Signed-off-by: Jay Deng <jayd0104@gmail.com>
Signed-off-by: Jay Deng <dengjay@amazon.com>
…25.1 in /plugins/repository-gcs (opensearch-project#10298) (opensearch-project#10461)

* Bump com.google.api.grpc:proto-google-common-protos

Bumps [com.google.api.grpc:proto-google-common-protos](https://github.com/googleapis/sdk-platform-java) from 2.25.0 to 2.25.1.
- [Release notes](https://github.com/googleapis/sdk-platform-java/releases)
- [Changelog](https://github.com/googleapis/sdk-platform-java/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/sdk-platform-java/commits)

---
updated-dependencies:
- dependency-name: com.google.api.grpc:proto-google-common-protos
  dependency-type: direct:production
  update-type: version-update:semver-patch
...



* Updating SHAs



* Update changelog



---------

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…ad & store recovery (opensearch-project#10449)

---------

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com>
Signed-off-by: Siddhant Deshmukh <deshsid@amazon.com>
@github-actions
Copy link
Contributor

github-actions bot commented Oct 9, 2023

Gradle Check (Jenkins) Run Completed with:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.