Added changes for AdmissionControl Interceptor and AdmissionControlService for RateLimiting #9286

ajaymovva · 2023-08-12T18:50:21Z

Description

This PR consists of below functionalities

Adding core transport interceptors in network module to intercept the requests at the transport layer.
AdmissionControlInterceptor to add the new AdmissionControlHandler, and that handler will be triggered for every transport request.
-The initial implementation of AdmissionControlService, which is currently used to register the admission controllers and trigger the registered admission controllers for every transport request if it is enabled.
AdmissionControlSettings -> Settings related to AdmissionController Feature.
Basic Implementation of the CPU-Based Admission Controller, which currently only tracks the number of transport calls if the admission controller is enabled.
CPUBasedAdmissionControllerSettings, which have the settings specific to the CPU-based admission controller.

Note:

This PR only have initial basic implementation of many classes. As this is the first PR for this feature we are making sure to give basic idea how admission controller is tracing the transport requests and eventually how we will reject the requests if the threshold breaches based on each admission controllers

Related Issues

Ref: #8910

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-08-12T19:00:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22581/
CommitID: d9d0686
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-08-12T19:14:49Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 24m 6s

...les/transport-netty4/src/main/java/org/opensearch/http/netty4/Netty4HttpServerTransport.java

modules/transport-netty4/src/main/java/org/opensearch/transport/Netty4ModulePlugin.java

...les/throttling/src/main/java/org/opensearch/throttling/OpenSearchThrottlingModulePlugin.java

github-actions · 2023-08-15T17:32:49Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22778/
CommitID: babe211
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-15T17:41:31Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22779/
CommitID: efbd221
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-08-15T17:50:57Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/cross-cluster-replication.git]
Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

BUILD SUCCESSFUL in 32m 6s

github-actions · 2023-08-22T20:35:10Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23235/
CommitID: 06a4ff4
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-30T07:36:09Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23800/
CommitID: 9a65c7c
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-30T07:36:55Z

Compatibility status:

Checks if related components are compatible with change fa39f9a

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-08-30T07:38:52Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23801/
CommitID: be089cf
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-30T07:39:13Z

Compatibility status:

Checks if related components are compatible with change be089cf

Incompatible components

Skipped components

Compatible components

github-actions · 2023-09-03T19:14:26Z

Compatibility status:

Checks if related components are compatible with change a1e6134

Incompatible components

Skipped components

Compatible components

github-actions · 2023-09-03T19:18:22Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24284/
CommitID: a1e6134
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-03T21:25:46Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24290/
CommitID: f60c6b5
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-03T21:46:30Z

Compatibility status:

Checks if related components are compatible with change f60c6b5

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-09-04T11:27:53Z

Compatibility status:

Checks if related components are compatible with change de4ea96

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-09-04T11:39:39Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24329/
CommitID: de4ea96
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-04T12:36:11Z

Compatibility status:

Checks if related components are compatible with change 25c16b6

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com>

…ed AdmissionController Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com>

Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com>

github-actions · 2023-10-20T13:17:11Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/28579/
CommitID: fa39f9a

…rvice for RateLimiting (opensearch-project#9286) * Changes for AdmissionControl Interceptor and AdmissionControlService for RateLimiting (opensearch-project#9286) Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com>

opensearch-trigger-bot · 2024-01-22T10:53:28Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-9286-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 7c5a806d5bbee77c0c4a184a500bf5522a8d8cd7
# Push it to GitHub
git push --set-upstream origin backport/backport-9286-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-9286-to-2.x.

…rvice for RateLimiting (opensearch-project#9286) * Changes for AdmissionControl Interceptor and AdmissionControlService for RateLimiting (opensearch-project#9286) Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com>

…ntroller (#12111) * Added changes for AdmissionControl Interceptor and AdmissionControlService for RateLimiting (#9286) * Changes for AdmissionControl Interceptor and AdmissionControlService for RateLimiting (#9286) Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com> * Integrate CPU AC with ResourceUsageCollector and add CPU AC stats to nodes/stats (#10887) * Added changes to integrade cpu AC to ResourceUsageCollector and Emit Stats Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com> Co-authored-by: Bharathwaj G <bharath78910@gmail.com> * Updating Version to 2.12 for the AC Stats Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com> --------- Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com> Co-authored-by: Bharathwaj G <bharath78910@gmail.com> Co-authored-by: Ajay Kumar Movva <movvaam@amazon.com>

…rvice for RateLimiting (opensearch-project#9286) * Changes for AdmissionControl Interceptor and AdmissionControlService for RateLimiting (opensearch-project#9286) Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

bharath-techie reviewed Aug 13, 2023

View reviewed changes

...les/transport-netty4/src/main/java/org/opensearch/http/netty4/Netty4HttpServerTransport.java Outdated Show resolved Hide resolved

bharath-techie reviewed Aug 13, 2023

View reviewed changes

modules/transport-netty4/src/main/java/org/opensearch/transport/Netty4ModulePlugin.java Outdated Show resolved Hide resolved

bharath-techie reviewed Aug 13, 2023

View reviewed changes

...les/throttling/src/main/java/org/opensearch/throttling/OpenSearchThrottlingModulePlugin.java Outdated Show resolved Hide resolved

ajaymovva force-pushed the feature/admission-controller branch 2 times, most recently from babe211 to efbd221 Compare August 15, 2023 17:32

ajaymovva force-pushed the feature/admission-controller branch from efbd221 to 06a4ff4 Compare August 22, 2023 20:26

ajaymovva changed the title ~~Admission Controller Module Rest and Transport Interceptor Commit~~ Admission Controller Module Transport Interceptor Initial Commit Aug 22, 2023

ajaymovva force-pushed the feature/admission-controller branch 2 times, most recently from 9a65c7c to be089cf Compare August 30, 2023 07:34

ajaymovva force-pushed the feature/admission-controller branch from be089cf to a1e6134 Compare September 3, 2023 19:09

ajaymovva force-pushed the feature/admission-controller branch from a1e6134 to f60c6b5 Compare September 3, 2023 21:16

ajaymovva force-pushed the feature/admission-controller branch from f60c6b5 to de4ea96 Compare September 4, 2023 10:55

ajaymovva force-pushed the feature/admission-controller branch from de4ea96 to 25c16b6 Compare September 4, 2023 11:58

Ajay Kumar Movva added 6 commits October 20, 2023 18:03

Admission Controller Module Transport Interceptor Initial Commit

6547a4e

Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com>

Added AdmissionController Interceptor in Core and implemented CPU Bas…

eb53cd8

…ed AdmissionController Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com>

Addressing Comments

3a8e16a

Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com>

Addressing Comments

bdf365b

Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com>

Addressing Comments

7848e7f

Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com>

Addressing Comments

fa39f9a

Signed-off-by: Ajay Kumar Movva <movvaam@amazon.com>

ajaymovva force-pushed the feature/admission-controller branch from b8c2c63 to fa39f9a Compare October 20, 2023 12:34

Bukhtawar merged commit 7c5a806 into opensearch-project:main Oct 21, 2023
16 checks passed

ajaymovva mentioned this pull request Oct 26, 2023

[META] Admission Controller Framework #9504

Open

14 tasks

rramachand21 added the v2.12.0 Issues and PRs related to version 2.12.0 label Oct 31, 2023

ajaymovva self-assigned this Nov 2, 2023

ajaymovva added the backport 2.x Backport to 2.x branch label Jan 22, 2024

opensearch-trigger-bot bot added the backport-failed label Jan 22, 2024

ajaymovva mentioned this pull request Jan 22, 2024

Added changes for AdmissionControl Interceptor and AdmissionControlSe… #11965

Closed

8 tasks

ajaymovva mentioned this pull request Feb 1, 2024

[Backport 2.x] AdmissionControl Framework with CPU Based Admission Controller #12111

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added changes for AdmissionControl Interceptor and AdmissionControlService for RateLimiting #9286

Added changes for AdmissionControl Interceptor and AdmissionControlService for RateLimiting #9286

ajaymovva commented Aug 12, 2023 •

edited

Loading

github-actions bot commented Aug 12, 2023

opensearch-trigger-bot bot commented Aug 12, 2023

github-actions bot commented Aug 15, 2023

github-actions bot commented Aug 15, 2023

opensearch-trigger-bot bot commented Aug 15, 2023

github-actions bot commented Aug 22, 2023

github-actions bot commented Aug 30, 2023

github-actions bot commented Aug 30, 2023 •

edited

Loading

github-actions bot commented Aug 30, 2023

github-actions bot commented Aug 30, 2023

github-actions bot commented Sep 3, 2023

github-actions bot commented Sep 3, 2023

github-actions bot commented Sep 3, 2023

github-actions bot commented Sep 3, 2023

github-actions bot commented Sep 4, 2023

github-actions bot commented Sep 4, 2023

github-actions bot commented Sep 4, 2023

github-actions bot commented Oct 20, 2023

opensearch-trigger-bot bot commented Jan 22, 2024

Added changes for AdmissionControl Interceptor and AdmissionControlService for RateLimiting #9286

Added changes for AdmissionControl Interceptor and AdmissionControlService for RateLimiting #9286

Conversation

ajaymovva commented Aug 12, 2023 • edited Loading

Description

Note:

Related Issues

Check List

github-actions bot commented Aug 12, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 12, 2023

github-actions bot commented Aug 15, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 15, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 15, 2023

github-actions bot commented Aug 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 30, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 30, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 30, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 30, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 3, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 3, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 3, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 3, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 4, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 4, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 4, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Oct 20, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Jan 22, 2024

ajaymovva commented Aug 12, 2023 •

edited

Loading

github-actions bot commented Aug 30, 2023 •

edited

Loading