[Backport 2.x]Support task resource tracking in OpenSearch (#3982)

[ketanv3]

[Backport 2.x] Support task resource tracking in OpenSearch (#3982)

• Support task resource tracking in OpenSearch

• Reopens changes from Support task resource tracking in OpenSearch #2639 (reverted in Revert "Support task resource tracking in OpenSearch" #3046) to add a framework for task resource tracking. Currently, SearchTask and SearchShardTask support resource tracking but it can be extended to any other task.

• Fixed a race-condition when Task is unregistered before its threads are stopped

• Improved error handling and simplified task resource tracking completion listener

• Avoid registering listeners on already completed tasks

Signed-off-by: Ketan Verma ketan9495@gmail.com

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/1254/
• CommitID: 77a5124f900d7bd240768f8f44941c8a7a434a20

[ketanv3]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/1254/
• CommitID: 77a5124

Failed in :qa:wildfly:composeUp due to a transient network error when calling Docker registry. Re-triggering the workflows.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: null ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/1255/
• CommitID: 930ba7cc146ea62a23bd2c6e6687cb9c0d589846

[ketanv3]

Gradle Check (Jenkins) Run Completed with:

• RESULT: null ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/1255/
• CommitID: 930ba7c

The Jenkins' build succeeded, but the GitHub action while trying to get the job status failed.
https://build.ci.opensearch.org/job/gradle-check/1255/

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/1257/
• CommitID: 991632b

[codecov-commenter]

Codecov Report

Merging #4087 (991632b) into 2.x (517b815) will increase coverage by 0.00%.
The diff coverage is 78.13%.

@@            Coverage Diff             @@
##                2.x    #4087    +/-   ##
==========================================
  Coverage     70.61%   70.62%            
- Complexity    56628    56687    +59     
==========================================
  Files          4543     4545     +2     
  Lines        272332   272525   +193     
  Branches      40017    40037    +20     
==========================================
+ Hits         192307   192461   +154     
+ Misses        63902    63865    -37     
- Partials      16123    16199    +76     

Impacted Files	Coverage Δ
...rg/opensearch/common/settings/ClusterSettings.java	91.89% <ø> (ø)
...org/opensearch/action/support/TransportAction.java	55.42% <50.00%> (-0.28%)	⬇️
...rc/main/java/org/opensearch/tasks/TaskManager.java	67.74% <62.50%> (+0.05%)	⬆️
...ster/node/tasks/list/TransportListTasksAction.java	47.61% <66.66%> (+3.17%)	⬆️
...a/org/opensearch/threadpool/TaskAwareRunnable.java	69.56% <69.56%> (ø)
...erver/src/main/java/org/opensearch/tasks/Task.java	79.43% <75.00%> (+1.66%)	⬆️
.../opensearch/tasks/TaskResourceTrackingService.java	79.76% <79.76%> (ø)
...ensearch/common/util/concurrent/ThreadContext.java	92.96% <80.00%> (+0.76%)	⬆️
...ch/common/util/concurrent/OpenSearchExecutors.java	81.72% <83.33%> (+0.11%)	⬆️
.../org/opensearch/action/search/SearchShardTask.java	75.00% <100.00%> (+8.33%)	⬆️
... and 516 more

Help us with your feedback. Take ten seconds to tell us how you rate us.