Run Dashboards integ test against static endpoint in Jenkins job

[tianleh]

Is your feature request related to a problem? Please describe

#604

Describe the solution you'd like

Currently there is only OpenSearch integ test in Jenkins

opensearch-build/jenkins/test/testsuite/Jenkinsfile

Line 63 in 86f9cbe

sh "./test.sh ${JOB_NAME} manifests/${opensearch_version}/opensearch-${opensearch_version}-test.yml ${basePath} --test-run-id ${env.BUILD_NUMBER}"

We want to add such for Dashboards as well. The challenge is how we know the public urls for OS and OSD to feed into test.sh

Describe alternatives you've considered

No response

Additional context

No response

[tianleh]

First need to understand how current OpenSearch integ test runs from Jenkins

[tianleh]

[tianleh]

A run needs opensearch_version, build_id, architecture, platform.

Follow up question: is this automatically run? If yes, how the parameters are passed? On the other hand, if manual run, then parameters can be passed manually.

[tianleh]

One option is once distribution-build-opensearch-dashboards finishes, it will trigger another job dashboards-integ test and pass the OSD information to it including version, build_id, platform, architecture (the information to concatenate the CI public URL). Then this new job needs to figure out where to get the OS information, especially what is the latest build. This can be done from multiple ways.

1. use the information from OSD run (version, platform, architecture) to look for the S3 bucket of OS and then get latest build_id
2. query the latest/recent successful run of OS jobs and find the one for (version, platform, architecture)

[tianleh]

Another option is have a main job to coordinate the build/assemble and then integ test, potentially extending the existing check-for-build. If both OS and OSD are triggered, then we can get build_id for both. However, if one of them doesn't trigger, then we need to figure out the latest run with (version, platform, architecture)

[tianleh]

DeclarativeJobPropertyTrackerAction.getParameters can only get the input name "INPUT_MANIFEST" but not the actual content of the value.

[tianleh]

To summarize current status, I am able to get the latest OpenSearch and OpenSearchDashboards build number.
http://jenki-jenki-fpgmrv2ryxko-1366042710.us-east-1.elb.amazonaws.com/job/distribution-build-opensearch-dashboards-tianleh/

However, I cannot move further to get the build details. Why is this import? E.g when a OpenSearchDashboards build/assemble is done, it may need to know which build from distribution-build-opensearch matches the [version, arch, platform].

[tianleh]

@dblock @seraphjiang

[tianleh]

To help explain the POC work, I first try to get the latest build number from distribution-build-opensearch-dashboards and distribution-build-opensearch

This part is working.

Next, assume OpenSearchDashboards is the main driver. We want to

1. get input manifest to understand the version, platform, architecture
2. list OpenSearch recent builds to match the tuple above. (minor release can make the matching logic more complicated)

Currently I am still researching how to get the build details (INPUT_MANIFEST) from the build number.

[dblock]

I think using the current check-for-build workflow (https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/check-for-build.jenkinsfile) to trigger a new coordinator workflow that runs build and test workflows might be a lot easier. Trying to build something independent that tries to fetch the last good build feels unnecessarily complicated.

[tianleh]

After talking with @ohltyler about the PR #1429, we figure out the following result. Tyler's work is for running OpenSearch integ test while mine current task is for such in OpenSearch Dashboards.

The OpenSearch Dashboards test job will need to know both OpenSearch and OpenSearch Dashboards public endpoints which is very different from OpenSearch tests which just need to know the OpenSearch public endpoint. To further clarify the difference, since currently OpenSearch and OpenSearch Dashboards distribution build jobs are separate, Tyler's work can be done purely inside the OpenSearch build job while my work could potentially require cross-job information.

I am currently experimenting the "a new coordinator workflow" from #1322 (comment) which may need to build and test together for both OpenSearch and OpenSearch Dashboards.

To summarize, structure wise there won't be reused work from Tyler's PR towards my work. I would vision "the new coordinator workflow" would trigger the OpenSearch integ test logic by Tyler and also my OpenSearch Dashboards logic in this PR #1368
(At code level, there may be some references to the manipulation of the artifact rool url.)

CC @ohltyler @dblock @seraphjiang @kavilla @Tengda-He for visibility

[dblock]

I think it may be simpler if Dashboards were treated as a separate standalone product, and just use the latest available OpenSearch build, as needed. Do we really need to coordinate the two?

[seraphjiang]

From the issue description, we need public endpoint of both OpenSearch and OpenSearch Dashboards.
I'm curious how Jenkins code get public endpoint for OpenSearch, it is in environment variable of Jenkins function return? What's difficult to do same for OpenSearch Dashbaords?

[dblock]

I see this code that passes endpoint information on.

[tianleh]

We will run the test against the release endpoint instead of every newly bundled artifact. Found a pre-quest task #1482

[tianleh]

Closed in favor of #704