[META] Automate Integration testing

[setiah]

This is a meta issue for tracking all sub-tasks for full integration testing on OpenSearch distribution.

P0

• Add integration test job to the orchestration pipeline. #178
• Add capability to run IntegTests on built artifacts from builder pipeline #182
• Update IntegTest job to accept parameters based on manifest and kick off multiple integ tests in parallel. #183
• Support integTests for a plugin #327
• Add UT coverage for integTest module #382
• Jenkins job for integration testing #395
• [BLOCKER] Replace remote min artifact with local release artifact for integration testing #319

P1

• support matrix configuration for integTests, which defines a list of integTests that should run against a type of artifact #220
• Integtests support for multiple plugins  #473
• Create a results.yml file after test run #1113
• Run integTests against remote cluster #214
• #326
• Dockerize integration testing #202
• Parallelize integration testing for plugins

Depends on #122

• Automate integration tests for OpenSearch Dashboards #704
• [META] Run integ tests with every build of OpenSearch and OpenSearch Dashboards and publish test results #818
• [IntegTest] Onboard all plugins into automated integration tests for the distribution #58

[setiah]

Describing integration test job in its current form

1. Clone opensearch (1.0), opensearch-build and plugin (based on ref provided) to test. Then clone the dependencies - common-utils, job-scheduler, alerting as notifications.
2. Deploy single node cluster based on params VERSION, TYPE (snapshot | release), ARCHITECTURE (x64 | amd64), SECURITY (w/ | w/o). This downloads artifacts from https://artifacts.opensearch.org, adds settings, disables security (if required), and starts OpenSearch process. It validates the process is running by doing a curl on the ports.
3. Next it pulls integ test dependencies using integtest_dependencies_opensearch.sh into maven-local, based on VERSION (e.g. 1.0.0) and QUALIFIER (e.g. rc1, ga) for each of the repos pulled in step 1 (opensearch, common-utils, job-scheduler, alerting). It uses ./gradlew publishToMavenLocal to publish dependencies to maven local. These dependencies should now be available in ~/.m2/repository/org/opensearch
4. Next it moves the plugin repo to opensearch/plugins folder and executes the integtest.sh from that plugin repo against provided CLUSTER_PORT

[setiah]

Inputs required:

1. Artifact download path
2. git-reference (commit id, etc.)
3. Architecture
4. Security
5. Log publish path (based on BuildId)

What the [modified] integ test job needs to do

1. Add parallelization logic to run tests in parallel on plugins.
2. Spin up multiple single node clusters - one for each parallel test run
3. Publish logs for all runs under common build id, with unique test id.

[peternied]

This item is redundant with #818 closing this one in favor of that one

[dblock]

Going to reopen this until we run integration tests with every build.

[dblock]

Closing in favor of #818