[CI] unix packaging tests on rhel-9-packaging OS failing for DockerTests.test010Install

[n1v0lg]

CI Link

https://gradle-enterprise.elastic.co/s/byzfqq7clinbc

Repro line

None

Does it reproduce?

Didn't try

Applicable branches

main, 8.4, 8.3, 7.17

Failure history

Failures for other branches:

8.4 - https://gradle-enterprise.elastic.co/s/ukyv7kx53bbvk
8.3 - https://gradle-enterprise.elastic.co/s/krfseqryq6nbs
7.17 - https://gradle-enterprise.elastic.co/s/m4ik63k3jzxwe

Failure excerpt

It seems that only rhel-9 is affected.

The container fails to start, with below root cause:

org.elasticsearch.packaging.test.DockerTests > test010Install FAILED  
    java.lang.AssertionError: Elasticsearch container did not start successfully.

...

 [2022-08-10T11:27:53,936][WARN ][o.e.p.u.d.Docker         ] [TEST-DockerTests.test010Install-seed#[3C98EAD1D6120B35]-workerest_thread_info] Caught exception while waiting for ES to start	
 org.elasticsearch.packaging.util.Shell$ShellException: Command was not successful: [bash -c docker exec --tty bb6e776314affd4dc171364a2f5e3816113d5ee0d1242cfd2e512241bbcc370c bash -c 'COLUMNS=2000 ps ax']	
    result: exitCode = [126] stdout = [OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: unknown] stderr = []	
  at org.elasticsearch.packaging.util.Shell.runScript(Shell.java:143) ~[test/:?]	
  at org.elasticsearch.packaging.util.Shell.run(Shell.java:73) ~[test/:?]	
  at org.elasticsearch.packaging.util.docker.DockerShell.run(DockerShell.java:49) ~[test/:?]	
  at org.elasticsearch.packaging.util.docker.Docker.waitForElasticsearchToStart(Docker.java:188) ~[test/:?]	
  at org.elasticsearch.packaging.util.docker.Docker.runContainer(Docker.java:121) ~[test/:?]	
  at org.elasticsearch.packaging.test.DockerTests.setupTest(DockerTests.java:118) ~[test/:?]	
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]	
  at java.lang.reflect.Method.invoke(Method.java:577) ~[?:?]	
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) ~[randomizedtesting-runner-2.7.7.jar:?]	
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980) ~[randomizedtesting-runner-2.7.7.jar:?]	
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) ~[randomizedtesting-runner-2.7.7.jar:?]	
  at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) ~[junit-4.12.jar:4.12]	
  at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) ~[junit-4.12.jar:4.12]	
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.7.jar:?]	
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375) ~[randomizedtesting-runner-2.7.7.jar:?]	
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831) ~[randomizedtesting-runner-2.7.7.jar:?]	
  at java.lang.Thread.run(Thread.java:833) [?:?]

[elasticsearchmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[mark-vieira]

I think I've found the root cause here which looks to be opencontainers/runc#3551.

I've reproduced the error as well as confirmed that the tests successfully pass when downgrading containerd to 1.6.6. I'm looking to see if we can just hack our packaging test script to downgrade for us for now as getting the CI images updated with pinned versions is kind of a pain.