[AUTOCUT] Integration Test failed for cross-cluster-replication: 2.12.0 tar distribution #1317
Comments
opensearch-ci-bot
added
autocut
integ-test-failure
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
was able to reproduce the 1> [2024-02-16T00:13:02,727][INFO ][o.o.r.BasicReplicationIT ] [test empty index replication] after test
2> REPRODUCE WITH: ./gradlew ':integTestRemote' --tests "org.opensearch.replication.BasicReplicationIT.test empty index replication" -Dtests.seed=87E473EC64D720CF -Dtests.security.manager=true -Dtests.locale=en-MT -Dtests.timezone=America/Shiprock -Druntime.java=19
2> org.opensearch.client.ResponseException: method [PUT], host [https://localhost:9201/], URI [/_plugins/_replication/lsvvxibneh/_start?wait_for_restore=true], status line [HTTP/1.1 500 Internal Server Error]
{"error":{"root_cause":[{"type":"connect_transport_exception","reason":"[][127.0.0.1:9200] connect_exception"}],"type":"connect_transport_exception","reason":"[][127.0.0.1:9200] connect_exception","caused_by":{"type":"annotated_connect_exception","reason":"Connection refused: localhost/127.0.0.1:9200","caused_by":{"type":"connect_exception","reason":"Connection refused"}}},"status":500}
at __randomizedtesting.SeedInfo.seed([87E473EC64D720CF:E4930300BB801772]:0)
at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:376)
at app//org.opensearch.client.RestClient.performRequest(RestClient.java:346)
at app//org.opensearch.client.RestClient.performRequest(RestClient.java:321)
at app//org.opensearch.replication.ReplicationHelpersKt.startReplication(ReplicationHelpers.kt:96)
at app//org.opensearch.replication.ReplicationHelpersKt.startReplication$default(ReplicationHelpers.kt:63)
at app//org.opensearch.replication.BasicReplicationIT.test empty index replication(BasicReplicationIT.kt:54)this is the root cause: leader-node1-1 | [2024-02-16T07:13:02,609][ERROR][o.o.r.a.s.TransportStopIndexReplicationAction] [leader-node1] Stop replication failed for index[jtdarwgufp] with error java.lang.IllegalArgumentException: No replication in progress for index:jtdarwgufp
leader-node1-1 | at org.opensearch.replication.action.stop.TransportStopIndexReplicationAction.validateReplicationStateOfIndex(TransportStopIndexReplicationAction.kt:196)This isn't related to admin credentials change. Can maintainers take a look and resolve this? |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
I tried to run the integTestRemote from my local on 2 docker images and test are passing fine. Connection Refused error is coming when we create admin user different credentials other than @monusingh-1 Can you check on this |
|
@nisgoel-amazon this is the command I ran after spinning up two single-node clusters locally: ./gradlew integTestRemote --tests "org.opensearch.replication.BasicReplicationIT.test empty index replication" -Dleader.http_host=localhost:9200 -Dfollower.http_host=localhost:9201 -Dleader.transport_host=localhost:9200 -Dfollower.transport_host=localhost:9301 -Dsecurity_enabled=true -Duser=admin -Dpassword='myStrongPassword123!' -PnumNodes=1 --console=plain |
|
Seems like there was an issue in port-forwarding for clusters I setup. With the new setup I was able to execute test successfully. Expand to see docker-compose file used:version: '3'
services:
opensearch-leader:
image: opensearchstaging/opensearch:2.12.0
container_name: opensearch-leader
environment:
- cluster.name=opensearch-leader
- node.name=opensearch-leader1
- discovery.seed_hosts=opensearch-leader1
- cluster.initial_master_nodes=opensearch-leader1
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
hard: 65536
volumes:
- opensearch-data1:/usr/share/opensearch/data
ports:
- 9200:9200
- 9600:9600 # required for Performance Analyzer
networks:
- opensearch-net
opensearch-follower:
image: opensearchstaging/opensearch:2.12.0
container_name: opensearch-follower
environment:
- cluster.name=opensearch-follower
- node.name=opensearch-follower1
- discovery.seed_hosts=opensearch-follower1
- cluster.initial_master_nodes=opensearch-follower1
- bootstrap.memory_lock=true
- OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
volumes:
- opensearch-data2:/usr/share/opensearch/data
ports:
- 9201:9200
- 9601:9600 # required for Performance Analyzer
networks:
- opensearch-net
volumes:
opensearch-data1:
opensearch-data2:
networks:
opensearch-net:Get private ips for both clusters by calling ./gradlew integTestRemote -Dleader.http_host=localhost:9200 -Dleader.transport_host=localhost:9300 -Dfollower.http_host=localhost:9201 -Dfollower.transport_host=localhost:9300 -Dsecurity_enabled=true -Duser=admin -Dpassword=myStrongPassword123! -PnumNodes=1 |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
The integration test failed at distribution level for component cross-cluster-replication |
|
Using this issue for tracking IT failures: #1356 |
The integration test failed at distribution level for component cross-cluster-replication
Version: 2.12.0
Distribution: tar
Architecture: x64
Platform: linux
Please check the logs: https://build.ci.opensearch.org/job/integ-test/7495/display/redirect
* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.12.0/9314/linux/x64/tar/test-results/7495/integ-test/test-report.yml
Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.
The text was updated successfully, but these errors were encountered: