[Meta] Automate Run backward compatibility tests for the distribution (OpenSearch + plugins)

[zelinh]

Background

Currently, each plugin has their own backward compatibility tests running in their GitHub actions. It sets up their own test cluster and test each individual subsequent tasks.
Here is our proposal for BWC/Upgrade testing for OpenSearch project.

All BWC tests include:

1. Rolling upgrade
2. Mixed Cluster search
3. Snapshot upgrade
4. Restart upgrade

In plugin level, the current bwc tests tasks are running against the test cluster framework from the core team, which set up the test cluster with designated OpenSearch min artifacts. Within the test cluster on plugin teams end, it only installs their own plugin and other required dependencies plugins.

Proposal

We would like to see CI/CD that shows each of the plugin would be tested with the latest bundle build of OpenSearch. We like to run backward compatibility tests once the distribution is ready. In another word, running the plugin BWC against the test clusters with all plugins present along with the OpenSearch min once we generate a new bundle distribution.

Currently plugin teams are using the test cluster set up from TestClustersPlugin, within this framework, plugin teams can set up the number of nodes, the version of OpenSearch min artifacts...

I think we could enhance with another test cluster set up option that we could instead install just the core but with the distribution that we generated from the CI distribution build. When we started the new test cluster

testClusters {
    "mycluster" {
        versions = ["1.3.2", "2.4.0"]
        numberOfNodes = 4
    }
}

In the example above, each of those 4 nodes are set up to be started OS version 1.3.6 with all plugins running and the second is the one set up to be the current project version. In this framework, each node will initially be downloaded with DistributionDownloadPlugin and installed with the respective bundle verison. From plugin perspectives, if they want to run BWC tests against the bundle test cluster, they could add a new argument to start the tests.

Works breakdown

• Milestone 1: Update the distributionDownload framework to download bundle distribution from CI build.
• Milestone 2: Onboard the updated gradle BWC tests to plugin repos/ Adapt current BWC gradle tasks to the new test clusters with bundle installed in one plugin
• Milestone 3: Onboard the BWC tests to CI workflow running associated with CI build
• Milestone 4: Report failures for any CI tests failures to plugins teams

[zelinh]

Idea here is to download the bundle from CI bucket with latest index.
e.g.

v1.3.6 Bundle URL: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.6/latest/linux/x64/tar/dist/opensearch/opensearch-1.3.6-linux-x64.tar.gz
v2.4.0 Bundle URL: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.4.0/latest/linux/x64/tar/dist/opensearch/opensearch-2.4.0-linux-x64.tar.gz

The URLs used before for test cluster setup are following this pattern.
https://artifacts.opensearch.org/releases/core/opensearch/1.3.6/opensearch-min-1.3.6-linux-x64.tar.gz for released version
https://artifacts.opensearch.org/snapshots/core/opensearch/2.4.0-SNAPSHOT/opensearch-min-2.4.0-SNAPSHOT-linux-x64-latest.tar.gz for unreleased version.

The approach is to add a new URL pattern to the distributionDownloadPlugin framework.

[zelinh]

Currently I already have the onboard PR ready for the bwc tests on the component repo. It would need previously linked PR against OpenSearch on test cluster to be merged to run. I would raise the PR against plugin repo once the above PR is clear and merged.

[bbarani]

@zelinh Can you please provide the current status? Lets fast track the on-boarding process so that we can use 2.5.0 release as dry run to close the gaps.

[zelinh]

Backport of the PR listed above to 2.x branch is merged. I finally have the backport PR to 2.4 branch in OpenSearch repo pass the gradle checks.
Will trigger the build on 2.4.2 and 2.5 to generate the new build-tools with these changes.

[zelinh]

Since the test cluster setup PRs were merged, I am drafting onboarding PR to component repos for the options to run BWC tests on distribution level soon. Plan to start with AD plugin first.

[zelinh]

For the BWC version (old version) of bundle, I decided to use v1.3.2 to start with. It was the first version that we started to support our current CI bundle URL pattern which follows as this example
https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.2/latest/linux/x64/tar/dist/opensearch/opensearch-1.3.2-linux-x64.tar.gz

[zelinh]

I have onboarded the BWC tests running on distribution level to AD, notification, sql, observability, alerting, index-management plugins.
There are also some other plugins that haven’t been onboarded with BWC tests on distribution level.
- One reason is that these repos’ currently BWC tests on plugin level are failing or unstable. We would need to work with them to fix the test structures or assertions before onboarding with new tests (e.g. cross-cluster-validation, reporting).
- Another reason is that most of our un-onboarded plugins are pretty new and don’t have enough versions/sources for running BWC tests. We would sequentially work on onboarding once they start to have BWC tests. (e.g. ml-commons, geospatial, neural-search)

[bbarani]

@zelinh Whats pending for BWC tests? Can we add automated GitHub issue creation in this workflow as well @rishabh6788

[zelinh]

@bbarani So far I don't see any things pending with the scope of this issue on our side. Once those components repo add stable BWC plugin level tests, we would onboard distribution level tests to them.

[zelinh]

Closing this issue.

[prudhvigodithi]

Hey @zelinh quick question, does this updated gradle BWC tests part of every plugin repo or just the few? Also when you get a chance can you please update the README of backwards-compatibility-tests section part of the test workflow. It would be very helpful for this OpenSearch Release Process documentation and I dont need to add all the details again to the release process doc whereas I can just refer to section part of the test workflow.
Thank you

[zelinh]

@prudhvigodithi Right now, the BWC tests on distribution bundle is only run on few plugins that were onboarded with BWC tests. Yes, I add update the README in our repo regarding of this. I have updated in the plugin repo and could have those docs added here as well.

[prudhvigodithi]

Thanks @zelinh