Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backport/backport 7648 to 2.x #8342

Closed

Conversation

suranjay
Copy link
Contributor

Description

This change adds request tracing framework in OpenSearch. It also integrates Opentelemetry with OpenSearch as it uses the framework for implementing tracing capabilities in OpenSearch.

The tracing framework automatically handles context propagation between different threads, tasks and nodes.

Note: The integrations tests will be covered in the subsequent PR to limit the changes in this PR.

Related Issues

Resolves #7543, #7544

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

nknize and others added 30 commits March 21, 2022 23:48
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
…rch-project#2574)

Change InboundDecoder ensureVersionCompatibility check for onOrAfter
V_2_0_0 instead of explicit version check. This way bug fix and minor
versions will correctly handshake in a mixed 1.x Cluster. Also fixes 
version filtering and warning handling for bwc testing.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
(cherry picked from commit 511ac88)
…oject#2566)

Upgrades to the official 9.1 release

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
…pts (opensearch-project#2564) (opensearch-project#2567)

Signed-off-by: Suraj Singh <surajrider@gmail.com>
signed-off-by: Nicholas Walter Knize <nknize@apache.org>
(cherry picked from commit c1d5491)
…arch-project#2584)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
(cherry picked from commit b6ca0d1)
opensearch-project#2593)

Changed the log message to cluster-manager from master

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>
(cherry picked from commit cc0e66b)
…search-project#2596) (opensearch-project#2605)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
(cherry picked from commit d4ce87b)

Co-authored-by: Andriy Redko <andriy.redko@aiven.io>
…-project#2535) (opensearch-project#2600)

Enables merge on refresh and merge on commit in OpenSearch by
way of two new index options:
index.merge_on_flush.max_full_flush_merge_wait_time and
index.merge_on_flush.enabled. Default merge_on_flush is disabled and
wait time is 10s.

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
(cherry picked from commit 908682d)
…ct#2604) (opensearch-project#2621)

Signed-off-by: Vacha Shah <vachshah@amazon.com>
(cherry picked from commit 5dd75bb)

Co-authored-by: Vacha Shah <vachshah@amazon.com>
…ch-project#2611) (opensearch-project#2634)

Bumps [forbiddenapis](https://github.com/policeman-tools/forbidden-apis) from 3.2 to 3.3.
- [Release notes](https://github.com/policeman-tools/forbidden-apis/releases)
- [Commits](policeman-tools/forbidden-apis@3.2...3.3)

---
updated-dependencies:
- dependency-name: de.thetaphi:forbiddenapis
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit 8f4aec1)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ns/repository-hdfs (opensearch-project#2618) (opensearch-project#2630)

* Bump htrace-core4 in /plugins/repository-hdfs

Bumps htrace-core4 from 4.1.0-incubating to 4.2.0-incubating.

---
updated-dependencies:
- dependency-name: org.apache.htrace:htrace-core4
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
(cherry picked from commit 2425f64)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…#2636) (opensearch-project#2648)

Signed-off-by: Suraj Singh <surajrider@gmail.com>
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

(cherry picked from commit bcaa06b)

Co-authored-by: Suraj Singh <79435743+dreamer-89@users.noreply.github.com>
…2645)

* Update Gradle to 7.4.1

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Address code review comments, added @PathSensitive(PathSensitivity.RELATIVE) where applicable

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
(cherry picked from commit 223efe6)

Co-authored-by: Andriy Redko <andriy.redko@aiven.io>
Add retry plugin support for Test implementations

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>
(cherry picked from commit 65cc56e)
…ct#2166) (opensearch-project#2628)

* Added jenkinsfile for gradle check

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

* Added jenkinsfile to run gradle check

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

* PR comment

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>
(cherry picked from commit f1d35d0)

Co-authored-by: Owais Kazi <owaiskazi19@gmail.com>
…t#2615) (opensearch-project#2666)

Bumps [jboss-annotations-api_1.2_spec](https://github.com/jboss/jboss-annotations-api_spec) from 1.0.0.Final to 1.0.2.Final.
- [Release notes](https://github.com/jboss/jboss-annotations-api_spec/releases)
- [Commits](jboss/jboss-annotations-api_spec@jboss-annotations-api_1.1_spec-1.0.0.Final...jboss-annotations-api_1.2_spec-1.0.2.Final)

---
updated-dependencies:
- dependency-name: org.jboss.spec.javax.annotation:jboss-annotations-api_1.2_spec
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit b5b0cd1)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ject#2669)

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>
(cherry picked from commit 8ea246e)

Co-authored-by: Owais Kazi <owaiskazi19@gmail.com>
…-gcs (opensearch-project#2616) (opensearch-project#2626)

* Bump google-oauth-client in /plugins/repository-gcs

Bumps [google-oauth-client](https://github.com/googleapis/google-oauth-java-client) from 1.31.0 to 1.33.1.
- [Release notes](https://github.com/googleapis/google-oauth-java-client/releases)
- [Changelog](https://github.com/googleapis/google-oauth-java-client/blob/main/CHANGELOG.md)
- [Commits](googleapis/google-oauth-java-client@v1.31.0...v1.33.1)

---
updated-dependencies:
- dependency-name: com.google.oauth-client:google-oauth-client
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
(cherry picked from commit 0216ab2)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…pensearch-project#2614) (opensearch-project#2624)

* Bump jettison from 1.1 to 1.4.1 in /plugins/discovery-azure-classic

Bumps [jettison](https://github.com/jettison-json/jettison) from 1.1 to 1.4.1.
- [Release notes](https://github.com/jettison-json/jettison/releases)
- [Commits](jettison-json/jettison@jettison-1.1...jettison-1.4.1)

---
updated-dependencies:
- dependency-name: org.codehaus.jettison:jettison
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
(cherry picked from commit e44706e)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Add 1.3.2 to main causing gradle check failures (opensearch-project#2679)

Signed-off-by: Suraj Singh <surajrider@gmail.com>
(cherry picked from commit e051a42)

* Add 1.3.2 to main causing gradle check failures (opensearch-project#2679)

Signed-off-by: Suraj Singh <surajrider@gmail.com>
(cherry picked from commit e051a42)

* Remove 1.4.0 bwc version

Signed-off-by: Suraj Singh <surajrider@gmail.com>
…2692)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
(cherry picked from commit ff7805e)

Co-authored-by: Andriy Redko <andriy.redko@aiven.io>
…ct#2681) (opensearch-project#2713)

* Add Shadow jar publication to lang-painless module.

This change creates a shadow jar for asm dependencies so
that they do not conflict with direct asm dependencies from log4j AL2 patch.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Remove security.manager systemProperty that is not required.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Add explicit task dependency for publishing to maven local required by gradle.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Move asm dependencies back to api scope.

Signed-off-by: Marc Handalian <handalm@amazon.com>
(cherry picked from commit d7d4108)

Co-authored-by: Marc Handalian <handalm@amazon.com>
…emoval - in CAT Nodes API (opensearch-project#2670) (opensearch-project#2695)

The request parameter `master_timeout` for CAT Nodes API is deprecated and alternative parameter `cluster_manager_timeout` is added in commit opensearch-project@a87c9d4 / PR opensearch-project#2435.
This PR refactors a previous code commit, and it's suggested by opensearch-project#2658 (comment).

Change:
This PR put the temporary code related to the 'master_timeout' deprecation in centralized places, so that it will be easier to remove when removing the deprecated parameter `master_timeout` in the future.
- Move the method `parseDeprecatedMasterTimeoutParameter()` into abstract base class `BaseRestHandler`, so that every REST API handler can call it.
- Put the unit tests related to the 'master_timeout' deprecation in one class.

Another notable change: Prohibit using two paramters `master_timeout` and `cluster_manager_timeout` together.

Reason:
There are other 60 REST APIs have got request parameter `master_timeout`, and the parameter pending to be deprecated. (See issue opensearch-project#2511 for the full list). 
- Adding new codes (creating deprecation warning, validating the parameter value and unit tests) in every class for the 60 APIs will cause large duplication.
- Removing the duplicate deprecated codes in the future is also a trouble.

Signed-off-by: Tianli Feng <ftianli@amazon.com>
… alternative for 'master_timeout', and deprecate 'master_timeout' - in CAT APIs (opensearch-project#2716)

Apply the change of CAT Nodes API in PR opensearch-project#2435 to other applicable CAT APIs.

- Deprecate the request parameter `master_timeout` that used in many CAT APIs.
- Add alternative new request parameter `cluster_manager_timeout`.
- Add unit tests.

Signed-off-by: Tianli Feng <ftianli@amazon.com>
(cherry picked from commit 78465b4)
…, during deprecation of request parameter 'master_timeout' (opensearch-project#2702) (opensearch-project#2740)

Temporarily build rest client with setStrictDeprecationMode(false) to allow
deprecation warning in HLRC tests while master_timeout parameters is
being refactored.

Signed-off-by: Tianli Feng <ftianli@amazon.com>
(cherry picked from commit 6a2a33d)
…pensearch-project#2742)

Removes duplicate Settings

Signed-off-by: ruanwenjun <wenjun@apache.org>
Signed-off-by: Nicholas Walter knize <nknize@apache.org>
(cherry picked from commit d848c63)

Co-authored-by: Wenjun Ruan <wenjun@apache.org>
…opensearch-project#2697)

Removes types from rest-api-spec endpoints

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
(cherry picked from commit 86eb24d)

Co-authored-by: Nick Knize <nknize@apache.org>
…roject#2617) (opensearch-project#2632)

* Bump asm-tree from 7.2 to 9.2 in /modules/lang-painless

Bumps asm-tree from 7.2 to 9.2.

---
updated-dependencies:
- dependency-name: org.ow2.asm:asm-tree
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
(cherry picked from commit 932cab6)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
opensearch-trigger-bot bot and others added 14 commits April 4, 2022 12:15
…sitory-gcs (opensearch-project#2738) (opensearch-project#2743)

* Bump proto-google-common-protos in /plugins/repository-gcs

Bumps [proto-google-common-protos](https://github.com/googleapis/java-iam) from 1.16.0 to 2.8.0.
- [Release notes](https://github.com/googleapis/java-iam/releases)
- [Changelog](https://github.com/googleapis/java-iam/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/java-iam/commits)

---
updated-dependencies:
- dependency-name: com.google.api.grpc:proto-google-common-protos
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
(cherry picked from commit f9ca90e)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
… (opensearch-project#2750)

Changes version comparison logic to only translate major version when
comparing with legacy 7x versions. This is needed beginning in 2.0 so
that when running 2.0+ versions in bwc mode for 3.0+ upgrades, node
versions no longer have to translate major version or spoof to legacy
versions.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
(cherry picked from commit 2d716ad)
…ct#2655)

Adds resource stats to task framework

Signed-off-by: sruti1312 <srutiparthiban@gmail.com>
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
(cherry picked from commit 8b997c1)

Co-authored-by: Sruti Parthiban <srutiparthiban@gmail.com>
Co-authored-by: Nicholas Walter Knize <nknize@apache.org>
opensearch-project#2760) (opensearch-project#2764)

This change swaps the order of the task to first check for no pointless exclusions.
If not caught first these will be thrown inside of the bogusExcludesCount block that logs a
useless error message.

Signed-off-by: Marc Handalian <handalm@amazon.com>
(cherry picked from commit 21f9950)
…_timeout' - in Cluster APIs (opensearch-project#2658) (opensearch-project#2754)

- Deprecate the request parameter `master_timeout` that used in Cluster APIs which have got the parameter.
- Add alternative new request parameter `cluster_manager_timeout`.
- Add unit tests.

Signed-off-by: Tianli Feng <ftianli@amazon.com>
…_timeout' - in Index APIs except index template APIs (opensearch-project#2660) (opensearch-project#2770)

- Deprecate the request parameter `master_timeout` that used in Index APIs which have got the parameter.
(except index template APIs, which is addressed in PR opensearch-project#2678)
- Add alternative new request parameter `cluster_manager_timeout`.
- Add unit tests.

Signed-off-by: Tianli Feng <ftianli@amazon.com>
(cherry picked from commit 7aa496f)
…ch-project#2752) (opensearch-project#2780)

- Replace `blacklist` with `denylist` in all `tests.rest.blacklist` and `REST_TESTS_BLACKLIST`

Signed-off-by: Tianli Feng <ftianli@amazon.com>
(cherry picked from commit ed040e9)
…earch-project#2777)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
(cherry picked from commit 1dbeda0)

Co-authored-by: Andriy Redko <andriy.redko@aiven.io>
…arch-project#2610) (opensearch-project#2622)

Bumps [json-schema-validator](https://github.com/networknt/json-schema-validator) from 1.0.67 to 1.0.68.
- [Release notes](https://github.com/networknt/json-schema-validator/releases)
- [Changelog](https://github.com/networknt/json-schema-validator/blob/master/CHANGELOG.md)
- [Commits](networknt/json-schema-validator@1.0.67...1.0.68)

---
updated-dependencies:
- dependency-name: com.networknt:json-schema-validator
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit dd79352)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tianli Feng <ftianli@amazon.com>
…ot identified in node bootstrap check (opensearch-project#2779) (opensearch-project#2793)

* Fix issue that deprecated setting 'cluster.initial_master_nodes' is not identified during node bootstrap

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Restore a variable name

Signed-off-by: Tianli Feng <ftianli@amazon.com>
(cherry picked from commit dd24e17)

Co-authored-by: Tianli Feng <ftianli@amazon.com>
…-project#2646) (opensearch-project#2789)

* Bump geoip2 from 2.16.1 to 3.0.1 in /modules/ingest-geoip

Bumps [geoip2](https://github.com/maxmind/GeoIP2-java) from 2.16.1 to 3.0.1.
- [Release notes](https://github.com/maxmind/GeoIP2-java/releases)
- [Changelog](https://github.com/maxmind/GeoIP2-java/blob/main/CHANGELOG.md)
- [Commits](maxmind/GeoIP2-java@v2.16.1...v3.0.1)

---
updated-dependencies:
- dependency-name: com.maxmind.geoip2:geoip2
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

* Fix breaking change with geoip2 version 3.0.1.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix precommit failures caused by ignoreMissingClasses check.

Precommit is failing because forbiddenApis was configured to ignore missing classes
that are present.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Change asn in GeoIpProcessorTests from int to long.

Signed-off-by: Marc Handalian <handalm@amazon.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marc Handalian <handalm@amazon.com>
(cherry picked from commit ce5c55d)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…project#2795) (opensearch-project#2798)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
(cherry picked from commit 0244b2a)
…ethod names (opensearch-project#2784) (opensearch-project#2812)

This is the last commit for replacing non-inclusive terminology `blacklist`. 
Most of the internal usages of non-inclusive terminology `blacklist/whitelist` was replaced by PR opensearch-project#2178, while there are a few remaining.
- Replace `blacklist` with `denylist` in internal class and method names.


Signed-off-by: Tianli Feng <ftianli@amazon.com>
(cherry picked from commit 47a22bb)
Signed-off-by: suranjay <surajkumar.tu@gmail.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@kotwanikunal
Copy link
Member

@suranjay It seems like you have raised the PR against a wrong branch. Please recreate the PR against 2.x if you want to backport to 2.x

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Tracing Framework]Tracing framework implementation
5 participants