Skip to content

Commit

Permalink
Merge Azure/master to local fork (#8)
Browse files Browse the repository at this point in the history
* Convert dip_sip ansible test to pytest (sonic-net#1488)

* Convert dip_sip ansible test to pytest

* Fix Review Comments:
- Add ansible wrapper to dip_sip.yml
- Delete unneccessary files

Signed-off-by: Noa Or <noaor@mellanox.com>

* Fix review comments

Check if topology is lag group by minigraph facts.

* Update test_dip_sip.py

* [pmon daemon check] refactoring pmon daemon state check code (sonic-net#1537)

* [daemon utils] move platform/check_daemon_status.py to common/platform/daemon_utils.py

Signed-off-by: Ying Xie <ying.xie@microsoft.com>

* [pmon daemon] refactoring pmon daemon state checking

- Refactor get_pmon_daemon_list to get_pmon_daemon_states:
  returning list of daemon states.
  not returning daemons that are known short lived.
  not returning daemons disabled by configuration.
- Refactor daemon state check to check the returned states.

Signed-off-by: Ying Xie <ying.xie@microsoft.com>

* Fix test_techsupport.py so it could run with T1 topology (sonic-net#1538)

* Fix test_techsupport.py so it could run with T1 topology

* [lgtm] Configure LGTM to analyze test code (sonic-net#1544)

Signed-off-by: Danny Allen <daall@microsoft.com>

* [tests] Fix LGTM errors in tests folder (sonic-net#1545)

Signed-off-by: Danny Allen <daall@microsoft.com>

* Convert vxlan-decap testing to pytest (sonic-net#1518)

Signed-off-by: Xin Wang <xinw@mellanox.com>

Co-authored-by: Xin Wang <xinw@mellanox.com>

* Add the APC PSU support for SNMP PSU controller (sonic-net#1461)

Add APC SNMP MIB for PDU control

PORT_NAME_BASE_OID = ".1.3.6.1.4.1.318.1.1.4.4.2.1.4"
PORT_STATUS_BASE_OID = ".1.3.6.1.4.1.318.1.1.12.3.5.1.1.4"
PORT_CONTROL_BASE_OID = ".1.3.6.1.4.1.318.1.1.12.3.3.1.1.4"

How did you verify/test it?
{username}@{sonic-mgmt-docker}:/var/sonic-mgmt-int/tests/common/plugins/psu_controller$ python
Python 2.7.12 (default, Oct 8 2019, 14:14:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.

from snmp_psu_controllers import *
psucntl = get_psu_controller('<controller_ip>',"<device_name>")
psucntl.get_psu_status()
[{'psu_id': 0, 'psu_on': True}]
psucntl.turn_off_psu('0')
True
psucntl.get_psu_status()
[{'psu_id': 0, 'psu_on': False}]
psucntl.turn_on_psu('0')
True
psucntl.get_psu_status()
[{'psu_id': 0, 'psu_on': True}]
print psucntl.psuType
APC

* [Mellanox] Fix issues for thermal control test cases (sonic-net#1524)

* [thermal fix] 1. should not mock PSU fan speed less than 100; 2. wait 5 seconds after turning on PSU
* Add some check for test case test_thermal_control_psu_absence

* [pytest/hash] add hash key ingress-port test (sonic-net#1509)

1. Iterate once for every port when hash key is ingress port.
2. In loose mode, do not need to enter the check_balancing function.

* [advanced reboot] Add Paramiko module for device connection (sonic-net#1542)

* [advanced reboot] Add Paramiko module for device connection

Parmiko module provides fallback mechanism to using username/password
This is required if we are rebooting into new image using advanced
reboot test fixture.

signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>

* [pytest] Port advanced reboot based test cases (sad path) (sonic-net#1543)

* [pytest] Port advanced reboot based test cases (sad path)

This patch ports remaining sad path advanced reboot test cases
to pytest infra.

signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>

* Update the importing of conn_graph_facts after platform_fixtures.py is removed (sonic-net#1547)

PR sonic-net#1492 copied the implementation of conn_graph_facts in tests/platform/platform_fixtures.py
to tests/common/fixtures/conn_graph_facts.py.

Then PR sonic-net#1503 removed file tests/platform/platform_fixtures.py. But not all related importings
were updated.

This change is to update all the platform_fixtures related importings.

Signed-off-by: Xin Wang <xinw@mellanox.com>

Co-authored-by: Xin Wang <xinw@mellanox.com>

* Create README.testbed.cEOS.md (sonic-net#1549)

* Configure and enable core uploader (sonic-net#1522)

* If core-storage secret key is available, add to /etc/sonic/core_analyzer.rc.json and enable & start core_uploader service
If https_proxy is provided, update /etc/sonic/core_analyzer.rc.json.

* Check the entire dict path before de-referencing.

* Improved regex per comments.

* Fixed syntax error.

* Add a sample file for newly introduced ansible facts.wq

* Removed a redundant empty line.

Co-authored-by: Ubuntu <remanava@remanava-kube-1.hblknyhzkmnujibhxvn3dmavjb.xx.internal.cloudapp.net>

* [tests/common/devices] Add FanoutHost (sonic-net#1532)

* [tests/common/devices] Add FanoutHost

- Add FanoutHost which aggregate different fanout switch host
- Add fanouthosts fixture shortcut to get fanouthost easily
- Support select host instance by os type
- Support get fanout host by both mgmtip and hostname

Co-authored-by: Jing Kan<jika@microsoft.com>

* [tacacs]: replace shell command with service module (sonic-net#1550)

Signed-off-by: Guohan Lu <gulv@microsoft.com>

* [tests/lag]: Convert lag_fallback.yml to pytest (sonic-net#1526)

* [tests/lag]: Convert lag_fallback.yml to pytest

- Convert lag_fallback.yml to pytest
- Reuse int shutdown mehod to shut a member of lag
- Use EOSHost class method shutdown/no_shutdown
- Add post check for EOSHost shutdown/no_shutdown
- Fix typos

Co-authored-by: Jing Kan<jika@microsoft.com>

* [pytest] Fix get_asic_type method and logging (sonic-net#1557)

* [pytest] Fix get_asic_type method and logging

The get_asic_type method dereferences dut which is not a member of the the class.
the dut is an instance of SonicHost class and so should reference self.

signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>

* Added iptables rules to make sure BGP don't ack back to BGP peer syn messages (sonic-net#1561)

* Added iptables rules to make sure BGP don't
ack to peer messages. Otherwise this TX packets
can make egress buffer accounting  off

Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>

* Review Comments Address

* [pytest/hash] Generate available send packet ports instead of hard coding src_ports in hash_test.py. (sonic-net#1511)

Generate available send packet ports instead of hard coding src_ports in hash_test.py.

* [fanout switch] build fanout switch list from device_conn map (sonic-net#1566)

* [fanout switch] build fanout switch list from device_conn map

- Build fanout switch from connection map.
- Build fanout switch DUT port map for tests.
- Build credential dict according to the DUt's inventory.
- Set fanout switch password with right variable name.
- Return command outpus.

Signed-off-by: Ying Xie <ying.xie@microsoft.com>

* Fix an indentation

* add comment for the os type

* Stop arp_update before crm test to prevent DUT learning mac address unexpectly, which caused the fdb test fail. (sonic-net#1559)

* Stop arp_update before crm test to prevent DUT learning mac address unexpectly, which caused the fdb test fail.

* [pytest] Add replace fast-reboot script test option (sonic-net#1563)

Adding an option to replace fast reboot script on dut as part of
testbed preparation.

signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>

* [pytest/creds]: load groups vars into creds for the dut (sonic-net#1575)

Signed-off-by: Guohan Lu <gulv@microsoft.com>

* [pytest/common/devices] Remove immediately checking of shut/no_shut (sonic-net#1576)

- The check can be put in a wait_unitl if user want

Co-authored-by: Blueve <jika@microsoft.com>

* SPyTest documentation initial version (sonic-net#1565)

Co-authored-by: Rama Sasthri, Kristipati <rama.kristipati@broadcom.com>

* [tests/conftest] Read os_type from ansible inventory vars to construct FanoutHost (sonic-net#1577)

- Read os var from ansible inventory manager

Co-authored-by: Jing Kan<jika@microsoft.com>

* [link flap] add link flap pytest (sonic-net#1573)

[link flap] add link flap pytest

* Use show interface status to obtain operational status of interfaces.
* Filter out operational down interfaces from test list.
* Address show_interface.py issue with single interface output.

Signed-off-by: Ying Xie <ying.xie@microsoft.com>

* Fix sensor data for 3700 (sonic-net#1568)

* [Mellanox] Fix issue: remove non-exist sensor sysfs files (sonic-net#1567)

* [pytest/snmp_lldp]: exclude mgmt interface from active interface list (sonic-net#1564)

align the criteria for checking lldpRemManAddrTable, mgmt interface 'eth0' is ruled out from the lldp neighbor list.

* [pytest/features] Add test for show features command (sonic-net#1546)

* [pytest assersion] introduce assertion handling code (sonic-net#1592)

Signed-off-by: Ying Xie <ying.xie@microsoft.com>

* [advanced-reboot] Refactor prepare ssh keys into ssh_utils (sonic-net#1571)

The ssh keys preparation is also required by wr_arp test suite,
and so putting it into common utils file for sharing

signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>

* [tests/pc] Use test_po_update.py in config.yml directly (sonic-net#1589)

* [tests/pc] Use test_po_update.py in config.yml directly

- Replace origin logic of config.yml with test_po_update.py
- Refine the test_po_udpate test with recommand fixture usage
- Remove useless import
- Remove useless var

Co-authored-by: Blueve <jika@microsoft.com>

* [pytest] Convert control plane assisted warm-reboot test (wr_arp) (sonic-net#1572)

* [pytest] Convert control plane assisted warm-reboot test (wr_arp)

This PR converts Ansible control assisted warm-reboot (wr_arp) test
case to pytest.

signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>

* [recovery] introduce adaptive recover method (sonic-net#1583)

* [recovery] introduce adaptive recover method

- For interfaces down failure, try to bring up the interfaces.
- For service down failure, reload config to bring them back up.
- For all other failures, reboot to recover.
- Make adaptive the default recover method.
- reboot dut to recover if database service is down
- Only allow cover service to override None action

Signed-off-by: Ying Xie <ying.xie@microsoft.com>

* [tests/lag_2] Refactor test_lag_2.py (sonic-net#1582)

* [tests/lag_2] Refactor test_lag_2.py

- Refine the code structure by using class to share a static context
- Remove most useless code and defines
- Rename some vars
- Use fanouthosts fixture for single_lag test
- Fix existing LGMT alarm
- Use wait_until to check if a int has been shut
- Fix pytest warnings regarding Test case detector

Co-authored-by: Jing Kan <jika@microsoft.com>

* [ansible] Fix symlink to ferret.conf.j2 (sonic-net#1595)

Template file was moved to arp/files/ferret.conf.j2 and symlink update was missed

signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>

* [test/procdockerstatsd] Verify daemon is active/running and uploads data successfully (sonic-net#1548)

* [pfc_storm_icos.j2]: jinja2 template to start PFC storm on ICOS. (sonic-net#1590)

[pfc_storm_stop_icos.j2]: jinja2 template to stop PFC storm on ICOS.

Note: Jinja2 considers physical to logical mapping of interfaces as per ICOS user manual.
fpti1_0_1 0/1
fpti1_0_2 0/2
fpti1_0_3 0/3

Co-authored-by: Praveen Chaudhary <pchaudha@pchaudha-mn1.linkedin.biz>

* [pytest/hash] Add hash keys: 'src-mac', 'dst-mac', 'ip-proto', 'vlan-id'. (sonic-net#1512)

* Revise according to the following review comments:
1. Not enable src-mac, dst-mac and vlan-id hash keys by default.
2. Get all untag vlan ports move to a separate function.
3. setup vlans for hash_key vlan-id test when vlan-id in hash_keys.

* [pytest/default_route_check]: check various aspect related to default route learnt from bgp (sonic-net#1598)

- check if default route has correct set src address
- check if ipv6 default route use global nexthop or not

Signed-off-by: Guohan Lu <gulv@microsoft.com>

* Fix below issue: (sonic-net#1596)

add topo is giving error as part of VEOS task
TASK [eos : Get VM front panel interface number]
with error FAILED! => {"msg": "Incorrect sudo password"}

Instead of using ansible_sudo_password make it ansible_become_password
based on below thread.
ansible/ansible#62042

* [tests/lag_2] Use test_lag_2.py in lag_2.yml directly (sonic-net#1602)

- Remove lag_fallback.yml
- Remove single_lag_lacp_rate_test.yml
- Remove single_lag_test.yml
- Use pytest runner to run pytest lag_2 test in lag_2.yml

Co-authored-by: Jing Kan<jika@microsoft.com>

* [tests/sensors] Convert sensors_check.yml to pytest (sonic-net#1601)

* [tests/sensors] Convert sensors_check.yml to pytest

- Convert sensors_check.yml to test_sensors.py
- Skip test if platform not supported
- Replace origin playbook with pytest runner
- Use pytest_assert helper instead of native assert to reduce noise
- Refactor _platform_info method in SonicHost and expose it function as a getter

Co-authored-by: JIng Kan<jika@microsoft.com>

* [tests/telemetry] Verify default config parameters (sonic-net#1530)

* [pytest/test_default_route]: support 4.9 kernel 201911/201811 release (sonic-net#1600)


Signed-off-by: Guohan Lu <gulv@microsoft.com>

* [TestbedProcessing] Minor fixes to suite latest changes (sonic-net#1604)

* Remove witespases

* Add ptf_pb_ip parsing

* Fix testbed.csv generating

* Use ansible_become_pass

* [pytest]: reorg tests info subfolder (sonic-net#1613)

Signed-off-by: Guohan Lu <gulv@microsoft.com>

* [sanity_checks]: add critical process check in sanity checks (sonic-net#1617)

- first read the critical process list (/etc/supervisord/critical_process)
from the container, and then check if any of the process crashed.

- add snmp container to the critical service list

- add auto_recover support

Signed-off-by: Guohan Lu <gulv@microsoft.com>

* add comment

Signed-off-by: Guohan Lu <gulv@microsoft.com>

* add auto_recover support if process check failed

* [tests/sensors] Move test_sensors.py to domain specific folder (sonic-net#1616)

Co-authored-by: Jing Kan <jika@microsoft.com>

* Fix sensors_check link (sonic-net#1611)

* [pytest]: add get_ip_route_info in SonicHost (sonic-net#1618)

get_ip_route_info returns ip route info in the kernel
for a given destination ip

refactor test_default_route to use the new method

Signed-off-by: Guohan Lu <gulv@microsoft.com>

* [Mellanox/platform]Compare CPU's temperature against critical threshold (sonic-net#1585)

* [loganalyzer] Fail flag support in context manager and callback execution (sonic-net#1619)

* Fail flag support in context manager and callback execution

Signed-off-by: Neetha John <nejo@microsoft.com>

* [Mellanox] add test cases for dynamic minimum fan speed and psu fan speed policy (sonic-net#1552)

* [pytest/nbrhost]: add neighbor config info in nbrhost class (sonic-net#1621)

* [vxlan-decap]: Flush ptf rx buffer before ptf send packet. (sonic-net#1608)

* add links to virtual switch testbed setup

* [telemetry certs] deploy certs for telemetry in deploy-mg (sonic-net#1614)

* adding server and dsmsroot certs for telemetry
* adding support for ptfhost copy certs

* [pytest] Add support to populate DUT FDB entries (sonic-net#1593)

* [pytest] Add support to populate DUT FDB entries

This new PTF plugin and test case that populates DUT FDb entries.
It generates n Packets and sends them to DUT vlan IP. The number
of distinct MAC to distinct ip addresses is configurable.

signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>

* Revert "[telemetry certs] deploy certs for telemetry in deploy-mg (sonic-net#1614)" (sonic-net#1628)

This reverts commit 19e92b1.

* [Mellanox]Fix sensor data for 3700/3700c/3800 (sonic-net#1627)

* Ignore test in test_turn_on_off_psu_and_check_psustatus (sonic-net#1606)

* Fixed fixture scope conflict for DUT monitor (sonic-net#1630)

Signed-off-by: Yuriy Volynets <yuriyv@mellanox.com>

* [test plans] Moving test plans from sonic-wiki repo to sonic-mgmt repo (sonic-net#1631)

Signed-off-by: Ying Xie <ying.xie@microsoft.com>

* [pytest/bgp_gr_test]: add bgp graceful restart helper test (sonic-net#1623)

Signed-off-by: Guohan Lu <gulv@microsoft.com>

* [Mellanox] Adjust test cases for fan led support (sonic-net#1580)

* Fixes in test_drop_counters.py (sonic-net#1610)

- Added more attempts to read and verify if expected drop counter was changed to the specific - value. It was observed delays in counter refreshing, so this fix was added.
- Fixed 'test_not_expected_vlan_tag_drop', removed vlan 1, because port channels are members of vlan 1.
- Skipped 'test_loopback_filter', added description.

Signed-off-by: Yuriy Volynets <yuriyv@mellanox.com>

* [swap_syncd] Fix failures in swap syncd (sonic-net#1632)

Signed-off-by: Danny Allen <daall@microsoft.com>

* [pytest]: fix typo in parse default route output (sonic-net#1634)

* Update README.md

* Update README.md

* [pytest/ansible] Add support for multi-duts (sonic-net#1432)

* [pytest/ansible] Add support for multi-duts

* review comment: use duts instead of dut

* Fixing various test cases to use duthost instead of testbed_devices['dut']

* trim spaces if any when present in the duts list

signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>

* Pytest organization proposal (sonic-net#1605)

* Pytest organization proposal

Signed-off-by: Neetha John <nejo@microsoft.com>

* [test/dir_bcast] Convert dir_bcast.yml to pytest (sonic-net#1609)

* [test/dir_bcast] Convert dir_bcast.yml to pytest
* Force refresh router before testing and move to ipfwd folder
* Fix LGTM alerts

Co-authored-by: Jing Kan<jika@microsoft.com>

* [copp] Convert COPP test to pytest (sonic-net#1633)

Signed-off-by: Danny Allen <daall@microsoft.com>

* [doc]: Update document to specify non-empty password file (sonic-net#1638)

ansible 2.8 expects a non-empty file

* Revert "[copp] Convert COPP test to pytest (sonic-net#1633)" (sonic-net#1643)

This reverts commit 9a8fc1e.

* [pytest] Fix ptf_runner issue introduced by COPP test conversion (sonic-net#1644)

* [copp] Convert COPP test to pytest (sonic-net#1633)
* Fix missing space in ptf_runner

Signed-off-by: Danny Allen <daall@microsoft.com>

* [fast-reboot]: Fix IP range overlapping. (sonic-net#1637)

Signed-off-by: Nazarii Hnydyn <nazariig@mellanox.com>

* Add support of specifying multiple inventory files (sonic-net#1645)

The pytest-ansible plugin always assumes that the value
of cli argument '--inventory' is a single inventory file.
With this enhancement, we can pass in multiple inventory
files using the cli argument '--inventory'. The multiple
inventory files can be separated by comma ','.

For example:
        pytest --inventory "inventory1, inventory2" <other arguments>
        pytest --inventory inventory1,inventory2 <other arguments>

Signed-off-by: Xin Wang <xiwang5@microsoft.com>

* [pytest] Fix module import issue when running whole test suite (sonic-net#1642)

* [pytest] Fix module import issue when running whole test suite

When having multiple conftest, dir leading to this conftest has
to be Python package (presence of __init__.py.) Also, adding
pytest basedir to conftest.py

signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>

* [advanced-reboot] Fix IP range overalapping (sonic-net#1653)

This code ports PR 1637 to advanced reboot.

ported-pr: sonic-net#1637
signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>

* Deprecate ansible_host.py (sonic-net#1658)

* Deprecate ansible_host.py

The functionalities in ansible_host.py , including the
AnsibleHost class and some exceptions, have been
implemented in the common library. This file can
be deprecated in case people new to sonic-mgmt
waste time on old libraries.

Changes:
1. Remove ansible_host.py
2. Replace all the AnsibleHost related calls with
    more appropriate fixtures or functions.
3. Remove duplicated localhost fixture definition

Signed-off-by: Xin Wang <xiwang5@microsoft.com>

* [Mellanox] Fix sensor data for SN4700 (sonic-net#1660)

* Fix for SN4700 sensors

Signed-off-by: Shlomi Bitton <shlomibi@mellanox.com>

* Edit sensors file with correct labels

* [drop counters] Improve support for combined L2/L3 drop counters (sonic-net#1649)

- Get the correct interface for combined L2/L3 counters
- Add Arista and Dell SKUs to the combined counter list

Signed-off-by: Danny Allen <daall@microsoft.com>

* [pytest/ntp] Use local time when behind proxy (sonic-net#1640)

When testbed is behind a proxy then NTP inside the ptf container cant synchronize with public NTP servers and test hung on ntpd -gq command

* [recover] improve adaptive recover methods (sonic-net#1652)

- When process is missing, reload_config.
- When port channle or vlan link is down, reload_config.
- redirect output of config reload and load minigraph to /dev/null.

Signed-off-by: Ying Xie <ying.xie@microsoft.com>

* [sensors] Added support of Montara nad Maverics platforms to sku-sensors-data.yml (sonic-net#1646)

Signed-off-by: Vitaliy Senchyshyn <vsenchyshyn@barefootnetworks.com>

* [pytest] PFCWD config test (sonic-net#1620)

* Pytest PFCWD config test

Signed-off-by: Neetha John <nejo@microsoft.com>

* Address LGTM

Signed-off-by: Neetha John <nejo@microsoft.com>

* Rename marker to syslog_marker

Signed-off-by: Neetha John <nejo@microsoft.com>

* Separate out verbose comments

Signed-off-by: Neetha John <nejo@microsoft.com>

* [advanced-reboot] Fix testing hang when doing BGP shutdown on Arista VM (sonic-net#1579)

Fix warm-reboot-sad-bgp testing hang after vEOS upgraded to 4.20.15M.
It pops a confirm message when doing BGP shutdown.

* [file organization] Rename folder names to allow markers to work again (sonic-net#1671)

* Revert "[pytest] Fix module import issue when running whole test suite (sonic-net#1642)"

This reverts commit f88cff2.

* [file organization] Rename folder names to allow markers to work again

pytest has an issue dealing with sub-folders with the same name and being
included from. In this case, the problem was with several components has
'args' as sub-folder to hold argument parsing utilities.

This issue was initially caught when we were tryign to run the whole test
suite. Attemp was made to address this issue. However there are more in
pytest that cannot tolerate same name sub-folders.

Change 'args' to '<feature>_args' and change conftest.py accordingly.

Signed-off-by: Ying Xie <ying.xie@microsoft.com>

* Rename 'platform' directory to 'platform_tests' to prevent conflicts (sonic-net#1662)

* Deprecate fixture testbed_devices (sonic-net#1665)

The testbed_devices fixture is not a good design. Multiple testbed device
objects are initialized in this fixture. It is not common that a test
script needs all the devices. This fixture may cause unnecessary overhead.
It would be better for test scripts to use the fixtures for different
devices on needed basis.

Changes:
1. Remove the definition of fixture testbed_devices.
2. Replace the call to testbed_devices fixture with other fixtures that
   fit better.

Signed-off-by: Xin Wang <xiwang5@microsoft.com>

* [Mellanox] Fix sensor data for 2010 (sonic-net#1673)

* Update Th buffer params for Alpha change (sonic-net#1676)

Signed-off-by: Neetha John <nejo@microsoft.com>

* [QoS] Fix qos issues (sonic-net#1664)

Fix some issues found during QoS test.

- Support ARP populate for ptf topo
- Add debug info in WRRtest for the purpose of providing detail result in case of failure.

Co-authored-by: Ying Xie <yxieca@users.noreply.github.com>
Co-authored-by: Danny Allen <daall@microsoft.com>
Co-authored-by: Xin Wang <wangxin.wang@gmail.com>
Co-authored-by: Xin Wang <xinw@mellanox.com>
Co-authored-by: Sujin Kang <sujkang@microsoft.com>
Co-authored-by: Junchao-Mellanox <57339448+Junchao-Mellanox@users.noreply.github.com>
Co-authored-by: William-zx <47626856+William-zx@users.noreply.github.com>
Co-authored-by: Tamer Ahmed <tamer.ahmed@microsoft.com>
Co-authored-by: lguohan <lguohan@gmail.com>
Co-authored-by: Renuka Manavalan <47282725+renukamanavalan@users.noreply.github.com>
Co-authored-by: Ubuntu <remanava@remanava-kube-1.hblknyhzkmnujibhxvn3dmavjb.xx.internal.cloudapp.net>
Co-authored-by: Blueve <672454911@qq.com>
Co-authored-by: abdosi <58047199+abdosi@users.noreply.github.com>
Co-authored-by: Iris Hsu <iris_hsu@edge-core.com>
Co-authored-by: Blueve <jika@microsoft.com>
Co-authored-by: Rama Sasthri, Kristipati <rama.kristipati@gmail.com>
Co-authored-by: Rama Sasthri, Kristipati <rama.kristipati@broadcom.com>
Co-authored-by: Kebo Liu <kebol@mellanox.com>
Co-authored-by: pra-moh <49077256+pra-moh@users.noreply.github.com>
Co-authored-by: Praveen Chaudhary <pchaudhary@linkedin.com>
Co-authored-by: Praveen Chaudhary <pchaudha@pchaudha-mn1.linkedin.biz>
Co-authored-by: Myron Sosyak <49795530+msosyak@users.noreply.github.com>
Co-authored-by: Stephen Sun <5379172+stephenxs@users.noreply.github.com>
Co-authored-by: Neetha John <nejo@microsoft.com>
Co-authored-by: yvolynets-mlnx <50697593+yvolynets-mlnx@users.noreply.github.com>
Co-authored-by: Xin Liu <xinxliu@microsoft.com>
Co-authored-by: Mahesh Maddikayala <10645050+smaheshm@users.noreply.github.com>
Co-authored-by: Nazarii Hnydyn <nazariig@mellanox.com>
Co-authored-by: Xin Wang <xiwang5@microsoft.com>
Co-authored-by: shlomibitton <60430976+shlomibitton@users.noreply.github.com>
Co-authored-by: Vitaliy Senchyshyn <43479243+vsenchyshyn@users.noreply.github.com>
Co-authored-by: Vincent Chiang <47546216+vincentchiang-ec@users.noreply.github.com>
Co-authored-by: Joe LeVeque <jleveque@users.noreply.github.com>
  • Loading branch information
1 parent 4c24e68 commit c14ff80
Show file tree
Hide file tree
Showing 203 changed files with 9,492 additions and 3,136 deletions.
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,10 @@
# Description
Tools for managing, configuring and monitoring SONiC

# CII Best Practices
[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/3933/badge)](https://bestpractices.coreinfrastructure.org/projects/3933)

# Contribution guide

All contributors must sign a contribution license agreement before contributions can be accepted.
[How to become a contributer](https://github.com/Azure/SONiC/wiki/Becoming-a-contributor)

Expand Down
1 change: 1 addition & 0 deletions ansible/README.testbed.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

- [Overview](doc/README.testbed.Overview.md)
- [Setup](doc/README.testbed.Setup.md)
- [Virtual Switch Testbed Setup](doc/README.testbed.VsSetup.md)
- [Topology](doc/README.testbed.Topology.md)
- [Configuration](doc/README.testbed.Config.md)
- [Minigraph](doc/README.testbed.Minigraph.md)
Expand Down
50 changes: 26 additions & 24 deletions ansible/TestbedProcessing.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,14 @@
Requirement:
python version: 2.X
python package: PyYAML 3.12 (or later)
python package: PyYAML 3.12 (or later)
PyYaml Install Instructions:
[1] Download PyYAML from https://pyyaml.org/wiki/PyYAML
[1] Download PyYAML from https://pyyaml.org/wiki/PyYAML
[2] Unpack the archive
[3] Install the package by executing (python setup.py install)
[4] Test if installation was successful (python setup.py test)
[4] Test if installation was successful (python setup.py test)
Usage:
put TestbedProcessing.py and testbed.yaml under sonic-mgmt/ansible
python TestbedProcessing.py
Expand All @@ -25,12 +25,12 @@
Arguments:
-i : the testbed.yaml file to parse
-basedir : the basedir for the project
-backupdir : the backup directory for the files
-backupdir : the backup directory for the files
Script Procedure
[1] Backup the files we will be copying
[1] Backup the files we will be copying
[2] Load testbed.yaml into dictionaries for easy processing
[3] Generate the files via methods defined below
[3] Generate the files via methods defined below
"""

# ARGUMENTS TO PARSE
Expand Down Expand Up @@ -86,7 +86,7 @@

"""
represent_none(self, _)
modifies yaml to replace null values with blanks
modifies yaml to replace null values with blanks
SOURCE: https://stackoverflow.com/questions/37200150/can-i-dump-blank-instead-of-null-in-yaml-pyyaml/37201633#3720163
"""
def represent_none(self, _):
Expand All @@ -98,7 +98,7 @@ def represent_none(self, _):
generateDictionary(data, result, category)
@:parameter data - the dictionary to iterate through
@:parameter result - the resulting dictionary
Generates the dictionaries that are used when creating csv, yml, or text files
Generates the dictionaries that are used when creating csv, yml, or text files
"""
def generateDictionary(data, result, category):
for key, value in data[category].items():
Expand All @@ -108,7 +108,7 @@ def generateDictionary(data, result, category):
"""
makeMain(data, outfile)
@:parameter data - the dictionary to look through
@:parameter outfile - the file to write to
@:parameter outfile - the file to write to
makeMain generates the vm_host/main.yml file
it pulls two sets of information; dictionary data and proxy data
"""
Expand All @@ -122,7 +122,9 @@ def makeMain(data, outfile):
"skip_image_downloading": veos.get("skip_image_downloading"),
"vm_console_base": veos.get("vm_console_base"),
"memory": veos.get("memory"),
"max_fp_num": veos.get("max_fp_num")
"max_fp_num": veos.get("max_fp_num"),
"ptf_bp_ip": veos.get("ptf_bp_ip"),
"ptf_bp_ipv6": veos.get("ptf_bp_ipv6")
}
proxy = {
"proxy_env": {
Expand All @@ -141,21 +143,21 @@ def makeMain(data, outfile):
@:parameter data - the dictionary to look for (in this case: veos)
@:parameter outfile - the file to write to
generates /group_vars/vm_host/creds.yml
pulls ansible_user, ansible_password, ansible_sudo_pass from vm_host_ansible into a dictionary
pulls ansible_user, ansible_password, ansible_become_pass from vm_host_ansible into a dictionary
"""
def makeVMHostCreds(data, outfile):
veos = data
result = {
"ansible_user": veos.get("vm_host_ansible").get("ansible_user"),
"ansible_password": veos.get("vm_host_ansible").get("ansible_password"),
"ansible_sudo_password": veos.get("vm_host_ansible").get("ansible_sudo_pass")
"ansible_become_pass": veos.get("vm_host_ansible").get("ansible_become_pass")
}
with open(outfile, "w") as toWrite:
toWrite.write("---\n")
yaml.dump(result, stream=toWrite, default_flow_style=False)

"""
makeSonicLabDevices(data, outfile)
makeSonicLabDevices(data, outfile)
@:parameter data - the dictionary to look through (devices dictionary)
@:parameter outfile - the file to write to
generates files/sonic_lab_devices.csv by pulling hostname, managementIP, hwsku, and type
Expand Down Expand Up @@ -190,14 +192,14 @@ def makeSonicLabDevices(data, outfile):


"""
makeTestbed(data, outfile)
makeTestbed(data, outfile)
@:parameter data - the dictionary to look through (devices dictionary)
@:parameter outfile - the file to write to
generates /testbed.csv by pulling confName, groupName, topo, ptf_image_name, ptf_ip, server, vm_base, dut, and comment
error handling: checks if attribute values are None type or string "None"
"""
def makeTestbed(data, outfile):
csv_columns = "# conf-name,group-name,topo,ptf_image_name,ptf_ip,server,vm_base,dut,ptf,comment"
csv_columns = "# conf-name,group-name,topo,ptf_image_name,ptf,ptf_ip,server,vm_base,dut,comment"
topology = data
csv_file = outfile

Expand Down Expand Up @@ -236,7 +238,7 @@ def makeTestbed(data, outfile):
if not comment:
comment = ""

row = confName + "," + groupName + "," + topo + "," + ptf_image_name + "," + ptf_ip + "," + server + "," + vm_base + "," + dut + "," + ptf + "," + comment
row = confName + "," + groupName + "," + topo + "," + ptf_image_name + "," + ptf + "," + ptf_ip + "," + server + "," + vm_base + "," + dut + "," + comment
f.write(row + "\n")
except IOError:
print("I/O error: issue creating testbed.csv")
Expand All @@ -245,9 +247,9 @@ def makeTestbed(data, outfile):
"""
makeSonicLabLinks(data, outfile)
@:parameter data - the dictionary to look through (devices dictionary)
@:parameter outfile - the file to write to
@:parameter outfile - the file to write to
generates /files/sonic_lab_links.csv by pulling startPort, endPort, bandWidth, vlanID, vlanMode
error handling: checks if attribute values are None type or string "None"
error handling: checks if attribute values are None type or string "None"
"""
def makeSonicLabLinks(data, outfile):
csv_columns = "StartDevice,StartPort,EndDevice,EndPort,BandWidth,VlanID,VlanMode"
Expand Down Expand Up @@ -308,7 +310,7 @@ def makeEOSCreds(data, outfile):
"""
makeFanout_secrets(data, outfile)
@:parameter data - reads from devices dictionary
@:parameter outfile - the file to write to
@:parameter outfile - the file to write to
Makes /group_vars/fanout/secrets.yml
Finds the fanout secret credentials by using "fanout" as the value to search for under device_type
Under github and personal topology configuration, there is only one designated fanout switch credential
Expand Down Expand Up @@ -428,7 +430,7 @@ def makeLab(data, devices, testbed, outfile):
"""
makeVeos(data, veos, devices, outfile)
@:parameter data - reads from either veos-groups, this helps separate the function into 3 components; children, host, vars
@:parameter veos - reads from either veos
@:parameter veos - reads from either veos
@:parameter devices - reads from devices
@:parameter outfile - writes to veos
"""
Expand Down Expand Up @@ -487,8 +489,8 @@ def makeHostVar(data):

"""
updateDockerRegistry
@:parameter outfile - the file to write to
hard codes the docker registry to search locally rather than externally
@:parameter outfile - the file to write to
hard codes the docker registry to search locally rather than externally
"""
def updateDockerRegistry(docker_registry, outfile):
if (not docker_registry.get("docker_registry_host")) or (not docker_registry.get("docker_registry_username")) or (not docker_registry.get("docker_registry_password")):
Expand Down
50 changes: 49 additions & 1 deletion ansible/config_sonic_basedon_testbed.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,12 +41,17 @@
testbed_file: testbed.csv
when: testbed_file is not defined

- name: Set default dut index
set_fact:
dut_index: 0
when: dut_index is not defined

- name: Gathering testbed information
test_facts: testbed_name="{{ testbed_name }}" testbed_file="{{ testbed_file }}"
delegate_to: localhost

- fail: msg="The DUT you are trying to run test does not belongs to this testbed"
when: testbed_facts['dut'] != inventory_hostname
when: testbed_facts['duts'][dut_index] != inventory_hostname

- name: set testbed_type
set_fact:
Expand Down Expand Up @@ -144,6 +149,49 @@
become: true
when: stat_result.stat.exists is defined and stat_result.stat.exists

- name: Init account key and proxy
set_fact:
core_key: ""
core_proxy: ""

- name: read account key
set_fact:
core_key: "{{ corefile_uploader['azure_sonic_core_storage']['account_key'] }}"
when: corefile_uploader['azure_sonic_core_storage']['account_key'] is defined

- name: read https proxy
set_fact:
core_proxy: "{{ corefile_uploader['env']['https_proxy'] }}"
when: corefile_uploader['env']['https_proxy'] is defined

- name: Put secret in core_analyzer.rc.json
lineinfile:
name: /etc/sonic/core_analyzer.rc.json
regexp: '(^.*)account_key'
line: '\1account_key": "{{ core_key }}",'
backrefs: yes
become: true
when: core_key != ""

- name: Put https-proxy in core_analyzer.rc.json
lineinfile:
name: /etc/sonic/core_analyzer.rc.json
regexp: '(^.*)https_proxy'
line: '\1https_proxy": "{{ core_proxy }}"'
backrefs: yes
become: true
when: core_proxy != ""

- name: enable core uploader service
become: true
command: systemctl enable core_uploader.service
when: core_key != ""

- name: start core uploader service
become: true
command: systemctl start core_uploader.service
when: core_key != ""

- name: Replace snmp community string
lineinfile:
name: /etc/sonic/snmp.yml
Expand Down
2 changes: 1 addition & 1 deletion ansible/doc/README.testbed.VsSetup.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ from the ```sonic-mgmt``` container. Then, test you can sudo without password pr
```
$ ./testbed-cli.sh -m veos.vtb -n 4 start-vms server_1 password.txt
```
- please note: Here "password.txt" is the ansible vault password file name/path. Ansible allows user use ansible vault to encrypt password files. By default, this shell script require a password file. If you are not using ansible vault, just create an empty file and pass the filename to the command line. The file name and location is created and maintained by user.
- please note: Here "password.txt" is the ansible vault password file name/path. Ansible allows user use ansible vault to encrypt password files. By default, this shell script require a password file. If you are not using ansible vault, just create a file with a dummy pasword and pass the filename to the command line. The file name and location is created and maintained by user.

Check that all VMs are up and running, and the passwd is ```123456```
```
Expand Down
131 changes: 131 additions & 0 deletions ansible/doc/README.testbed.cEOS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
# cEOS

This document discusses how to use cEOS as DUT neighbor device.

cEOS is the container-based EOS. All the software running inside
the container. Compared with vEOS, cEOS has much smaller memory
footprint.

Follow [instruction](README.testbed.VsSetup.md) to setup cEOS testbed.

In below example, there are four cEOS containers.

```
lgh@jenkins-worker-15:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fe48c207a51c ceosimage:4.23.2F-1 "/sbin/init systemd.…" 8 days ago Up 8 days ceos_vms6-1_VM0103
52297010e66a ceosimage:4.23.2F-1 "/sbin/init systemd.…" 8 days ago Up 8 days ceos_vms6-1_VM0102
8dd95269b312 ceosimage:4.23.2F-1 "/sbin/init systemd.…" 8 days ago Up 8 days ceos_vms6-1_VM0101
3a50dd481bfb ceosimage:4.23.2F-1 "/sbin/init systemd.…" 8 days ago Up 8 days ceos_vms6-1_VM0100
b91b48145def debian:jessie "bash" 8 days ago Up 8 days net_vms6-1_VM0103
d1ff26d84249 debian:jessie "bash" 8 days ago Up 8 days net_vms6-1_VM0102
1489f52b9617 debian:jessie "bash" 8 days ago Up 8 days net_vms6-1_VM0101
ce1214a008ed debian:jessie "bash" 8 days ago Up 8 days net_vms6-1_VM0100
```

## Resource consumption

A cEOS containers consumes around 1G memory.

```
lgh@jenkins-worker-15:~$ docker stats --no-stream
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 6
fe48c207a51c ceos_vms6-1_VM0103 2.04% 970.9MiB / 125.9GiB 0.75% 0B / 0B 365MB / 55.8GB 138
52297010e66a ceos_vms6-1_VM0102 2.19% 965.4MiB / 125.9GiB 0.75% 0B / 0B 237MB / 55.6GB 139
8dd95269b312 ceos_vms6-1_VM0101 1.93% 980.9MiB / 125.9GiB 0.76% 0B / 0B 300MB / 55.9GB 138
3a50dd481bfb ceos_vms6-1_VM0100 2.05% 970.2MiB / 125.9GiB 0.75% 0B / 0B 365MB / 56.1GB 138
```

## Network Setup

We first create a base container `net_${testbed_name}_${vm_name}`, inject six ethernet ports into the base container,
and then start cEOS `ceos_${testbed_name}_${vm_name}` container on top of the base container. The six ethernet ports
are used for
- 1 management port
- 4 front panel ports to DUT
- 1 backplane port to PTF docker

```
+------------+ +----+
| cEOS Ma0 +--------- VM0100-m ---+ br |
| | +----+
| |
| | +--------------+
| Et1 +----------VM0100-t0---+ br-VM0100-0 |
| | +--------------+
| |
| | +--------------+
| Et2 +----------VM0100-t1---+ br-VM0100-1 |
| | +--------------+
| |
| | +--------------+
| Et3 +----------VM0100-t2---+ br-VM0100-2 |
| | +--------------+
| |
| | +--------------+
| Et4 +----------VM0100-t3---+ br-VM0100-3 |
| | +--------------+
| |
| | +--------------+
| Et5 +----------VM0100-back--+ br-b-vms6-1 |
| | +--------------+
+------------+
```

## Configuration

The `/mnt/flash` in cEOS container is mount to `/data/ceos/ceos_${testbed_name}_${vm_name}` on the host. The `/mnt/flash`
contiains the configuration file and logs.

```
lgh@jenkins-worker-15:~$ ls -l /data/ceos/ceos_vms6-1_VM0100/
total 40
-rw-rw-r--+ 1 root root 924 Mar 31 07:35 AsuFastPktTransmit.log
drwxrwxr-x+ 2 root root 4096 Mar 31 03:31 Fossil
-rw-rw-r--+ 1 root root 568 Mar 31 07:35 SsuRestore.log
-rw-rw-r--+ 1 root root 568 Mar 31 07:35 SsuRestoreLegacy.log
drwxr-xr-x+ 4 897 88 4096 Mar 31 07:35 archive
drwxrwx---+ 3 root root 4096 Mar 18 06:12 debug
drwxrwxr-x+ 2 root root 4096 Mar 18 06:12 fastpkttx.backup
-rw-rw-r--+ 1 root root 180 Mar 31 07:35 kickstart-config
drwxrwxr-x+ 3 root root 4096 Apr 8 09:11 persist
-rw-rwxr--+ 1 root root 1915 Mar 18 06:12 startup-config
```

## Login

There are two ways to get into cEOS container

1. docker exec
```
lgh@jenkins-worker-15:~$ docker exec -it ceos_vms6-1_VM0100 Cli
ARISTA01T1>show int status
Port Name Status Vlan Duplex Speed Type Flags Encapsulation
Et1 connected in Po1 full unconf EbraTestPhyPort
Et2 connected 1 full unconf EbraTestPhyPort
Et3 connected 1 full unconf EbraTestPhyPort
Et4 connected 1 full unconf EbraTestPhyPort
Et5 backplane connected routed full unconf EbraTestPhyPort
Ma0 connected routed full 10G 10/100/1000
Po1 connected routed full unconf N/A
ARISTA01T1>
```

2. ssh
```
lgh@jenkins-worker-15:~$ ssh admin@10.250.0.51
Password:
ARISTA01T1>show int status
Port Name Status Vlan Duplex Speed Type Flags Encapsulation
Et1 connected in Po1 full unconf EbraTestPhyPort
Et2 connected 1 full unconf EbraTestPhyPort
Et3 connected 1 full unconf EbraTestPhyPort
Et4 connected 1 full unconf EbraTestPhyPort
Et5 backplane connected routed full unconf EbraTestPhyPort
Ma0 connected routed full 10G 10/100/1000
Po1 connected routed full unconf N/A
ARISTA01T1>
```

Loading

0 comments on commit c14ff80

Please sign in to comment.