Monitoring & Tracing capability for multiple Thread pools of Che Server

[skabashnyuk]

What does this PR do?

Add monitoring & Tracing capability for multiple Thread pools of Che Server.

ExecutorServiceWrapper

One of the trickiest thing that was preventing us to add this monitoring earlier was a

• Fact that monitoring and tracing can be enabled/disabled separately.
• Quite a lot of classes constructing ExecutorService in the constructor that is hard to override.
• To be able to handle the timing of the submitted task executor has to be wrapped, it's not enough to have just a reference.
• The sequence of wrapping is important. For example, if we wrap with tracing first we may lose the notion that ExecitorService was actually ThreadPoolExecutor.

To help to handle this kind of situation I've added such interface

public interface ExecutorServiceWrapper {

  ExecutorService wrap(ExecutorService executor, String name, String... tags);

  ScheduledExecutorService wrap(ScheduledExecutorService executor, String name, String... tags);

  CronExecutorService wrap(CronExecutorService executor, String name, String... tags);
}

implementation of this class has to be injected in all classes where ExecutorService is used to wrap the instance of executor.

Minor enhancement.

• JsonRPC. Reduced the number of classes used to configure JSONRpc endpoint.
• ExecutorServiceBuilder - helper class to build new ExecutorService
• CountedRejectedExecutionHandler - class that helps to count task execution rejections
• CountedThreadFactory - the class that helps to count thread creating/running/termination
• Grafana dashboard with traces grouped by execution level. Traces that happen on the same level grouped on the same panels

Grafana

To help to consume values from monitored executor services I've added some changes on Grafana dashboards.

Executors

• Threads running - the number of threads that are not terminated aka alive. May include threads that are in a waiting or blocked state.
• Threads terminated - the number of threads that was finished its execution.
• Threads created - number of threads created by thread factory for given executor service.
• Created thread/minute - Speed of thread creating for the given executor service.

• Executor threads active - number of threads that actively execute tasks.
• Executor pool size - number of threads that actively execute tasks.
• Queued task - the approximate number of tasks that are queued for execution
• Queued occupancy - the percent of the queue used by the tasks that is waining for execution.

• Rejected task - the number of tasks that were rejected from execution.
• Rejected task/minute - the speed of task rejections
• Completed tasks - the number of completed tasks
• Completed tasks/minute - the speed of task execution

• Task execution seconds max - 5min moving maximum of task execution
• Tasks execution seconds avg - 1h moving average of task execution
• Executor idle seconds max - 5min moving maximum of executor idle state.
• Executor idle seconds avg - 1h moving average of executor idle state.

• Scheduled repetitively - number of times how scheduleAtFixedRate or scheduleWithFixedDelay are called.
• Executor scheduled once - number of times schedule methods are called.
• Executor scheduled cron - number of times schedule with CronExpression methods is called.

Traces

• Workspace start Max - maximum workspace start time
• Workspace start Avg - 1h moving average of the workspace start time components
• Workspace stop Max - maximum of workspace stop time
• Workspace stop Avg - 1h moving average of the workspace stop time components

• OpenShiftInternalRuntime#start Max - maximum time of OpenShiftInternalRuntime#start operation
• OpenShiftInternalRuntime#start avg - 1h moving average time of OpenShiftInternalRuntime#start operation
• Plugin Brokering Execution Max - maximum time of PluginBrokerManager#getTooling operation
• Plugin Brokering Execution Avg - 1 of PluginBroker moving average of erManager#getTooling operation

• OpenShiftEnvironmentProvisioner#provision Max - maximum time of OpenShiftEnvironmentProvisioner#provision operation
• OpenShiftEnvironmentProvisioner#provision avg -1h moving average of OpenShiftEnvironmentProvisioner#provision operation
• Plugin Brokering Execution Max - maximum time of PluginBrokerManager#getTooling components execution time
• Plugin Brokering Execution Avg - 1h moving average of time of PluginBrokerManager#getTooling components execution time

• WaitMachinesStart Max - maximim time of WaitMachinesStart operations
• WaitMachinesStart Avg - 1h moving average time of WaitMachinesStart operations
• OpenShiftInternalRuntime#startMachines Max - maximim time of OpenShiftInternalRuntime#startMachines operations
• OpenShiftInternalRuntime#startMachines Avg - 1h moving average of the time of OpenShiftInternalRuntime#startMachines operations

What issues does this PR fix or reference?

#14601

Release Notes

n/a

Docs PR

eclipse-che/che-docs#870

[che-bot]

E2E tests of Eclipse Che Multiuser on OCP has been successful:

• Build details
• Test report
• docker image: eclipseche/che-server:14727

[skabashnyuk]

Wrapping/unwrapping looks fragile, but I've couldn't think of better solution :/ What if something fails (most probably at the reflection)? Does it kill whole server or just logs proper warning and rest keeps working?

I prefer to crash the whole JVM on guice startup phase. Because logs can be configured in the wrong way or it may be skipped or missed. But JVM crash unlikely can't be ignored

I'm also wondering if we will be able to utilize JavaFlightRecorder for monitoring like this of insides of jvm (once we'll switch to java 11+).

Java Flight Recorder requires a commercial license for use in production. To learn more about commercial features and how to enable them please visit http://www.oracle.com/technetwork/java/javaseproducts/.

I think because of that we unlikely can reuse it in Eclipse Che.

[skabashnyuk]

ci-build

[che-bot]

E2E Happy path tests of Eclipse Che Single User on K8S (minikube v1.1.1) has been successful:

• build details
• logs and configs
• "che-server" docker image: maxura/che-server:14727

[che-bot]

E2E Happy path tests of Eclipse Che Single User on K8S (minikube v1.1.1) has failed:

• build details
• test report
• logs and configs
• "che-server" docker image: maxura/che-server:14727
• https://github.com/orgs/eclipse/teams/eclipse-che-qa please check this report.
  Use command "crw-ci-test" to rerun the test.

[che-bot]

E2E tests of Eclipse Che Multiuser on OCP has been successful:

• Build details
• Test report
• docker image: eclipseche/che-server:14727

[skabashnyuk]

ci-build

[sparkoo]

I prefer to crash the whole JVM on guice startup phase. Because logs can be configured in the wrong way or it may be skipped or missed. But JVM crash unlikely can't be ignored

Yes, fail fast and visibly is good. I'm just afraid of incompatibilities with different java versions and compatibility with java modules (if it will ever be a concern for us :)

Java Flight Recorder requires a commercial license for use in production. To learn more about commercial features and how to enable them please visit http://www.oracle.com/technetwork/java/javaseproducts/.

I think because of that we unlikely can reuse it in Eclipse Che.

JFR was opensourced in java 11 (https://openjdk.java.net/jeps/328). We (Red Hat) even have people working on it.

[che-bot]

E2E Happy path tests of Eclipse Che Single User on K8S (minikube v1.1.1) has been successful:

• build details
• logs and configs
• "che-server" docker image: maxura/che-server:14727

[che-bot]

E2E tests of Eclipse Che Multiuser on OCP has been successful:

• Build details
• Test report
• docker image: eclipseche/che-server:14727

[che-bot]

E2E Happy path tests of Eclipse Che Single User on K8S (minikube v1.1.1) has been successful:

• build details
• logs and configs
• "che-server" docker image: maxura/che-server:14727

[che-bot]

E2E Happy path tests of Eclipse Che Single User on K8S (minikube v1.1.1) has been successful:

• build details
• logs and configs
• "che-server" docker image: maxura/che-server:14727

[che-bot]

E2E tests of Eclipse Che Multiuser on OCP has failed:

• Build details
• Test report
• docker image: eclipseche/che-server:14727
  https://github.com/orgs/eclipse/teams/eclipse-che-qa please check this report.
  Use command "ci-test" to rerun the test.

[che-bot]

E2E Happy path tests of Eclipse Che Single User on K8S (minikube v1.1.1) has been successful:

• build details
• logs and configs
• "che-server" docker image: maxura/che-server:14727

[che-bot]

E2E Happy path tests of Eclipse Che Single User on K8S (minikube v1.1.1) has failed:

• build details
• test report
• logs and configs
• "che-server" docker image: maxura/che-server:14727
• https://github.com/orgs/eclipse/teams/eclipse-che-qa please check this report.
  Use command "crw-ci-test" to rerun the test.

[skabashnyuk]

crw-ci-test

[che-bot]

E2E Happy path tests of Eclipse Che Single User on K8S (minikube v1.1.1) has been successful:

• build details
• logs and configs
• "che-server" docker image: maxura/che-server:14727

[che-bot]

E2E tests of Eclipse Che Multiuser on OCP has been successful:

• Build details
• Test report
• docker image: eclipseche/che-server:14727