[Bug]: Opensearch fails to start and deploy for 1.2.4

[abhinavGupta16]

Describe the bug

Opensearch fails to deploy with a missing lib error. The error is consistent with tar and docker deployment.

To reproduce

For docker, run -

docker pull opensearchstaging/opensearch:1.2.4 && docker run -it -p 9200:9200 -e "discovery.type=single-node" -e "DISABLE_SECURITY_PLUGIN=true" opensearchstaging/opensearch:1.2.4

For tar -

$ wget https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.4/638/linux/x64/dist/opensearch/opensearch-1.2.4-linux-x64.tar.gz
$ tar -zxf opensearch-1.2.4-linux-x64.tar.gz
$ cd opensearch-1.2.4
$ ./opensearch-tar-install.sh

Expected behavior

No response

Screenshots

No response

Host / Environment

No response

Additional context

No response

Relevant log output

java.lang.NoClassDefFoundError: org/apache/commons/lang3/tuple/Pair
	at org.opensearch.ad.rest.RestSearchAnomalyDetectorAction.<init>(RestSearchAnomalyDetectorAction.java:35) ~[?:?]
	at org.opensearch.ad.AnomalyDetectorPlugin.getRestHandlers(AnomalyDetectorPlugin.java:282) ~[?:?]
	at org.opensearch.action.ActionModule.initRestHandlers(ActionModule.java:848) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.node.Node.<init>(Node.java:960) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-1.2.4.jar:1.2.4]
	at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-1.2.4.jar:1.2.4]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101) ~[opensearch-1.2.4.jar:1.2.4]
Caused by: java.lang.ClassNotFoundException: org.apache.commons.lang3.tuple.Pair
	at java.net.URLClassLoader.findClass(URLClassLoader.java:435) ~[?:?]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:589) ~[?:?]
	at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:855) ~[?:?]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]
	... 15 more
fatal error in thread [main], exiting
java.lang.NoClassDefFoundError: org/apache/commons/lang3/tuple/Pair
	at org.opensearch.ad.rest.RestSearchAnomalyDetectorAction.<init>(RestSearchAnomalyDetectorAction.java:35)
	at org.opensearch.ad.AnomalyDetectorPlugin.getRestHandlers(AnomalyDetectorPlugin.java:282)
	at org.opensearch.action.ActionModule.initRestHandlers(ActionModule.java:848)
	at org.opensearch.node.Node.<init>(Node.java:960)
	at org.opensearch.node.Node.<init>(Node.java:319)
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412)
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178)
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169)
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.Command.main(Command.java:101)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.lang3.tuple.Pair
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:435)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
	at java.base/java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:855)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	... 15 more

[gaiksaya]

@amitgalitz @Zhangxunmt, can you guys take a look. Looks like AD plugin is causing the issue.
Thanks!

[abhinavGupta16]

Issue for this error that was opened earlier - opensearch-project/anomaly-detection#359

[amitgalitz]

@amitgalitz @Zhangxunmt, can you guys take a look. Looks like AD plugin is causing the issue. Thanks!

Taking a look

[amitgalitz]

I think I found the issue, our dependency is being compiled fine and running fine locally and on all github workflows. But after inspecting the pom file made at build, the lang3 dependency is not being bundled with our JAR which makes it so our artifact itself doesn’t work. Basically our runtime classpath contains a different version of this dependency then when we compile. Actively fixing it now and testing it out.

[abhinavGupta16]

Thanks @amitgalitz for actively looking into this.

[Zhangxunmt]

opensearch-project/anomaly-detection#363. This will fix the issue. After them merge, please work out a build release again for test. @abhinavGupta16

[abhinavGupta16]

opensearch-project/anomaly-detection#363. This will fix the issue. After them merge, please work out a build release again for test. @abhinavGupta16

@Zhangxunmt After the fix is merged, a build will trigger automatically and you should see a successful build notification on the channel for 1.2.4 (it may take some time). You can then use the docker or tar file to test out of the fix worked.

[ylwu-amzn]

Thanks for reporting this issue. Why AD can pass CI:

1. Job scheduler team upgraded cron-utils in this PR Bumps cron-utils version to 9.1.6 job-scheduler#118, which will not include commons-lang3
2. Then AD team added commons-lang3 dependency explicitly. But we used old job-scheduler 1.2.4 zip which still has commons-lang3 packed. So it was ok to use compileOnly in AD and CI can pass.
3. In the release tarball/dockerImage, the job-scheduler is built from latest code which doesn't have commons-lang3 packed any more, then AD can't find commons-lang3.

We plan to pull job scheduler from maven or build locally with latest code to avoid such issue in future.

[abhinavGupta16]

opensearch now builds and starts successfully. Please feel free to reopen this issue if the issue persists
@amitgalitz @ylwu-amzn @Zhangxunmt