Mgmtvrf test fails with 'Host Unreachable' error

[noaOrMlnx]

Description

test_mgmtvrf.py fails on setup_mvrf() function when tries to execute:
localhost.wait_for(host=var['dut_ip'], port=22, state='started', search_regex=SONIC_SSH_REGEX, timeout=90)

NOTE: mgmt vrf becomes enabled after this function execution.

When ignoring the wait_for() function on setup_mvrf() function, the test fails on test_service_acl() test case - also on wait_for() function.

Steps to reproduce the issue:

1. Execute test_mgmtvrf.py
2. Test fails on wait_for() function saying 'Host Unreachable'.

Describe the results you received:
'Host Unreachable'.

Describe the results you expected:
Test should pass.

[yxieca]

@noaOrMlnx looks like the wait_for is waiting for dut to reboot. This is a known instability with using wait_for for this purpose. Please use work-around similar to reboot test to wait for DUT to come back.

[wangxin]

Checked the test script, it's not related with reboot. @noaOrMlnx Could you please check if manual SSH to DUT works after duthost.command('sudo config vrf add mgmt')?

[vsenchyshyn]

The console where sudo config vrf add mgmt is executed manually hangs and I guess this is why following wait_for() call in the test fails with timeout. It is possible to connect to DUT from the separate console window.

[wangxin]

@vsenchyshyn If running sudo config vrf add mgmt hangs, it's another issue of the product and need further investigation.

I tried from my side on master image and got some findings:

• Initiate SSH connection to DUT from sonic-mgmt docker
• Run "sudo config vrf add mgmt" on DUT. The SSH connection was not broken.
• Run "sudo config vrf del mgmt" on DUT. The SSH connection was not broken.

When the SSH connection is not broken, the localhost.wait_for module will always fail with "Host Unreachable". Probably a limitation of the module.

So, is the SSH connection to DUT supposed to be broken or not when mgmt VRF is configured/removed on DUT? @chitra-raghavan

[chitra-raghavan]

@vsenchyshyn , When the tests were written , the command hangs device for long time and ssh connection got broken. Now in latest images, if ansible ssh doesnt get timed out , wait_for can be removed

[lguohan]

@chitra-raghavan , can you open an pr to resolve this issue?