[config qos] QoS and Buffer config genration for multi ASIC platforms #978

smaheshm · 2020-07-09T01:49:10Z

- What I did
To support buffer and QoS config generation for multi ASIC platform. In case of multi ASIC platforms each ASIC has its own buffer and qos configuration template which is used to populate corresponding ASIC's config DB.

Following commands were modified for this purpose:

# config qos reload
# config qos clear

In multi ASIC platforms the command runs on all namespaces. In case of single ASIC platform the command runs in global namespace.

admin@str-nSonic-acs-2:~$ sudo config qos reload --help                                                                                                                                
Usage: config qos reload [OPTIONS]                                                                                                                                                    

  Reload QoS configuration

Options:
  -?, -h, --help        Show this message and exit.
admin@str-nSonic-acs-2:~$ sudo config qos clear --help                                                                                                                                 
Usage: config qos clear [OPTIONS]

  Clear QoS configuration

Options:
  -?, -h, --help        Show this message and exit.

- How I did it
Modified config generation script to generate configs for all ASICs in case of multi-ASIC platform.

- How to verify it

verified configs generated on both multi-ASIC and single ASIC platforms. Verified traffic with different priorities using queue counters.

Single ASIC Platform:

admin@sonic:/usr/local/lib/python2.7/dist-packages$ cd
admin@sonic:~$ show platform summary
Platform: x86_64-dell_s6000_s1220-r0
HwSKU: Force10-S6000
ASIC: broadcom
admin@sonic:~$ sudo config qos reload
Running command: /usr/local/bin/sonic-cfggen  -d -t /usr/share/sonic/device/x86_64-dell_s6000_s1220-r0/Force10-S6000/buffers.json.j2 > /tmp/buffers.json
Running command: /usr/local/bin/sonic-cfggen  -d -t /usr/share/sonic/device/x86_64-dell_s6000_s1220-r0/Force10-S6000/qos.json.j2 -y /etc/sonic/sonic_version.yml > /tmp/qos.json
Running command: /usr/local/bin/sonic-cfggen  -j /tmp/buffers.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen  -j /tmp/qos.json --write-to-db
admin@sonic:~$ sudo config qos clear

Multi ASIC Platform

Before generating configs:

admin@str-nSonic-acs-2:~$ docker exec -it database0 redis-cli -n 4
127.0.0.1:6379[4]> keys *BUFFER*
(empty list or set)
127.0.0.1:6379[4]>
admin@str-nSonic-acs-2:~$ docker exec -it database2 redis-cli -n 4
127.0.0.1:6379[4]> keys *BUFFER*
(empty list or set)
127.0.0.1:6379[4]>


admin@str-nSonic-acs-2:~$ sudo config qos reload
Running command: /usr/local/bin/sonic-cfggen -n asic0 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/0/buffers.json.j2 > /tmp/buffers0.json
Running command: /usr/local/bin/sonic-cfggen -n asic0 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/0/qos.json.j2 -y /etc/sonic/sonic_version.yml > /tmp/qos0.json
Running command: /usr/local/bin/sonic-cfggen -n asic0 -j /tmp/buffers0.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen -n asic0 -j /tmp/qos0.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen -n asic1 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/1/buffers.json.j2 > /tmp/buffers1.json
Running command: /usr/local/bin/sonic-cfggen -n asic1 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/1/qos.json.j2 -y /etc/sonic/sonic_version.yml > /tmp/qos1.json
Running command: /usr/local/bin/sonic-cfggen -n asic1 -j /tmp/buffers1.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen -n asic1 -j /tmp/qos1.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen -n asic2 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/2/buffers.json.j2 > /tmp/buffers2.json
Running command: /usr/local/bin/sonic-cfggen -n asic2 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/2/qos.json.j2 -y /etc/sonic/sonic_version.yml > /tmp/qos2.json
Running command: /usr/local/bin/sonic-cfggen -n asic2 -j /tmp/buffers2.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen -n asic2 -j /tmp/qos2.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen -n asic3 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/3/buffers.json.j2 > /tmp/buffers3.json
Running command: /usr/local/bin/sonic-cfggen -n asic3 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/3/qos.json.j2 -y /etc/sonic/sonic_version.yml > /tmp/qos3.json
Running command: /usr/local/bin/sonic-cfggen -n asic3 -j /tmp/buffers3.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen -n asic3 -j /tmp/qos3.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen -n asic4 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/4/buffers.json.j2 > /tmp/buffers4.json
Running command: /usr/local/bin/sonic-cfggen -n asic4 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/4/qos.json.j2 -y /etc/sonic/sonic_version.yml > /tmp/qos4.json
Running command: /usr/local/bin/sonic-cfggen -n asic4 -j /tmp/buffers4.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen -n asic4 -j /tmp/qos4.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen -n asic5 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/5/buffers.json.j2 > /tmp/buffers5.json
Running command: /usr/local/bin/sonic-cfggen -n asic5 -d -t /usr/share/sonic/device/x86_64-nSonic-r0/Nexus-Sonic/5/qos.json.j2 -y /etc/sonic/sonic_version.yml > /tmp/qos5.json
Running command: /usr/local/bin/sonic-cfggen -n asic5 -j /tmp/buffers5.json --write-to-db
Running command: /usr/local/bin/sonic-cfggen -n asic5 -j /tmp/qos5.json --write-to-db
admin@str-nSonic-acs-2:~$

After generating configs:

admin@str-nSonic-acs-2:/usr/local/lib/python2.7/dist-packages$ docker exec -it database2 redis-cli -n 4
127.0.0.1:6379[4]> keys *BUFFER*
  1) "BUFFER_PROFILE|pg_lossless_40000_300m_profile"
  2) "BUFFER_PG|Ethernet-BP184|3-4"
  3) "BUFFER_QUEUE|Ethernet148|0-2"
  4) "BUFFER_POOL|egress_lossless_pool"
  5) "BUFFER_QUEUE|Ethernet184|3-4"
  6) "BUFFER_QUEUE|Ethernet-BP172|3-4"
  7) "BUFFER_QUEUE|Ethernet-BP172|5-6"
  8) "BUFFER_PG|Ethernet140|3-4"
  9) "BUFFER_QUEUE|Ethernet168|3-4"
.
.


admin@str-nSonic-acs-2:~$ sudo ip netns exec asic0 show mmu
Pool: ingress_lossless_pool
----  --------
type  ingress
mode  dynamic
size  12766208
----  --------

Pool: egress_lossless_pool
----  --------
type  egress
mode  static
size  12766208
----  --------

Pool: egress_lossy_pool
----  -------
type  egress
mode  dynamic
size  7326924
----  -------

Profile: pg_lossless_40000_300m_profile
----------  -----------------------------------
xon_offset  2496
dynamic_th  -3
xon         18432
xoff        55120
pool        [BUFFER_POOL|ingress_lossless_pool]
size        56368
----------  -----------------------------------

Profile: pg_lossless_40000_5m_profile
----------  -----------------------------------
xon_offset  2496
dynamic_th  -3
xon         18432
xoff        55120
pool        [BUFFER_POOL|ingress_lossless_pool]
size        56368
----------  -----------------------------------

Profile: egress_lossy_profile
----------  -------------------------------
dynamic_th  3
pool        [BUFFER_POOL|egress_lossy_pool]
size        1518
----------  -------------------------------

Profile: ingress_lossy_profile
----------  -----------------------------------
dynamic_th  3
pool        [BUFFER_POOL|ingress_lossless_pool]
size        0
----------  -----------------------------------

Profile: egress_lossless_profile
---------  ----------------------------------
static_th  12766208
pool       [BUFFER_POOL|egress_lossless_pool]
size       0
---------  ----------------------------------


admin@str-nSonic-acs-2:~$ sudo ip netns exec asic0 show queue counters Ethernet-BP40
         Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
-------------  -----  --------------  ---------------  -----------  ------------
Ethernet-BP40    UC0             252            44240            0             0
Ethernet-BP40    UC1            3000           192000            0             0
Ethernet-BP40    UC2            5000           320000            0             0
Ethernet-BP40    UC3            1000            64000            0             0
Ethernet-BP40    UC4            1000            64000            0             0
Ethernet-BP40    UC5               0                0            0             0
Ethernet-BP40    UC6               0                0            0             0
Ethernet-BP40    UC7               0                0            0             0
Ethernet-BP40    MC8               0                0            0             0
Ethernet-BP40    MC9               0                0            0             0
Ethernet-BP40   MC10               0                0            0             0
Ethernet-BP40   MC11               0                0            0             0
Ethernet-BP40   MC12               0                0            0             0
Ethernet-BP40   MC13               0                0            0             0
Ethernet-BP40   MC14               0                0            0             0
Ethernet-BP40   MC15               0                0            0             0


admin@str-nSonic-acs-2:/etc/sonic$ grep "Ethernet-BP..|3-4" config_db0.json -A2
        "Ethernet-BP12|3-4": {
            "profile": "[BUFFER_PROFILE|pg_lossless_40000_5m_profile]"
        },
--
        "Ethernet-BP16|3-4": {
            "profile": "[BUFFER_PROFILE|pg_lossless_40000_5m_profile]"
        },
--
        "Ethernet-BP20|3-4": {
            "profile": "[BUFFER_PROFILE|pg_lossless_40000_5m_profile]"
        },
--
.
.

- Previous command output (if the output of a command-line utility has changed)

- New command output (if the output of a command-line utility has changed)

arlakshm

As comments

config/main.py

smaheshm · 2020-07-17T16:01:49Z

@arlakshm @neethajohn Let me know if I should add any other reviewer.

config/main.py

smaheshm · 2020-07-27T20:44:37Z

@arlakshm Ok with the changes?

config/main.py

abdosi · 2020-07-27T21:02:06Z

config/main.py

+                        format(ns),
+                    fg='yellow'
+                )
+                raise click.Abort()


we have sonic_device_util.get_all_namespaces() that provide the dictionary of front and back namespace with string as namespace value. Then no need to convert from id to string. We can see if this API can be used.

I need the ASIC ID. That API doesn't give the ASIC ID.

why can't we have directory name as "asic0" ? i feel that is better and correlate to namespace directory created by linux.
We can change port_config.ini to also be in similar folder

@judyjoseph @SuvarnaMeenakshi what do you think here ?

This is a temporary directory, it can be anything. I still need the ASIC ID to read, and matches that format.

/usr/share/sonic/device/0
/usr/share/sonic/device/1
...

ASIC IDs and ASIC namespaces are different. The configuration is for each ASIC ID rather than namespace. I think it's OK to specify ASIC ID for configuration.

A simple approach would be to get the num of asics present in the platform and do a for range loop. We are using the asic_index as asic_id's internally.
If we derive the asic_id from namespace name, could cause problems if we create a namespace for a different purpose in the switch ?

I'm using 'get_namespaces()' API, this should return the valid namespaces for ASICs. It's better to use common APIs so there's one place that calculates the ASIC ID and namespace. Using a range loop can be fragile if the ASIC ID allocation scheme changes.

This is a temporary directory, it can be anything. I still need the ASIC ID to read, and matches that format.

/usr/share/sonic/device/0
/usr/share/sonic/device/1
...

@smaheshm why can't asic_id_suffix be namespace name directly ?

because the directories use 'asic ID'.

/usr/share/sonic/device/0
/usr/share/sonic/device/1
...

smaheshm · 2020-08-03T16:19:44Z

retest this please

smaheshm · 2020-08-03T20:30:18Z

retest this please

smaheshm · 2020-08-04T00:13:34Z

retest this please

config/main.py

smaheshm · 2020-08-04T16:16:42Z

retest this please

smaheshm · 2020-08-05T00:56:44Z

retest this please

config/main.py

smaheshm · 2020-08-05T21:38:40Z

retest this please

smaheshm · 2020-08-06T18:05:07Z

retest this please

smaheshm · 2020-08-07T17:43:55Z

retest this please

arlakshm

lgtm

abdosi · 2020-09-03T23:01:15Z

@smaheshm Create PR for 201911

QoS and Buffer config genration for multi ASIC platforms

5d7b03d

smaheshm requested review from abdosi, arlakshm and neethajohn July 9, 2020 01:54

arlakshm requested changes Jul 10, 2020

View reviewed changes

config/main.py Outdated Show resolved Hide resolved

config/main.py Outdated Show resolved Hide resolved

* use get_npu_id_from_name(namespace) API

9df039f

neethajohn reviewed Jul 23, 2020

View reviewed changes

config/main.py Outdated Show resolved Hide resolved

config/main.py Outdated Show resolved Hide resolved

Updated print statements

13d235c

neethajohn previously approved these changes Jul 24, 2020

View reviewed changes

abdosi reviewed Jul 27, 2020

View reviewed changes

config/main.py Outdated Show resolved Hide resolved

abdosi reviewed Jul 27, 2020

View reviewed changes

removed the option of providing namespace for Qos commands

6b8591d

smaheshm dismissed neethajohn’s stale review via 6b8591d July 27, 2020 21:55

smaheshm requested a review from arlakshm August 3, 2020 18:14

judyjoseph reviewed Aug 4, 2020

View reviewed changes

config/main.py Show resolved Hide resolved

Merge branch 'master' into masic

a9e7347

removed extra space when not running under a namespace

ffd5d1a

smaheshm commented Aug 5, 2020

View reviewed changes

config/main.py Show resolved Hide resolved

smaheshm added 2 commits August 5, 2020 19:01

Merge remote-tracking branch 'upstream/master' into masic

d7a7ffe

remove extra spaces in load minigraph output in unit test

59148e8

arlakshm approved these changes Aug 8, 2020

View reviewed changes

smaheshm merged commit a80826d into sonic-net:master Aug 8, 2020

smaheshm added the Request for 201911 Branch label Aug 19, 2020

smaheshm mentioned this pull request Sep 4, 2020

[sonic-cfggen][QoS][multi ASIC] Multi ASIC QoS and Buffer config generation support, merge from master #1095

Merged

abdosi added the Included in 201911 Branch label Sep 19, 2020

smaheshm deleted the masic branch February 26, 2021 22:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[config qos] QoS and Buffer config genration for multi ASIC platforms #978

[config qos] QoS and Buffer config genration for multi ASIC platforms #978

smaheshm commented Jul 9, 2020 •

edited

Loading

arlakshm left a comment

smaheshm commented Jul 17, 2020

smaheshm commented Jul 27, 2020

abdosi Jul 27, 2020

smaheshm Jul 27, 2020

abdosi Jul 27, 2020

smaheshm Jul 27, 2020

smaheshm Jul 27, 2020 •

edited

Loading

judyjoseph Aug 4, 2020

smaheshm Aug 4, 2020

abdosi Aug 5, 2020

smaheshm Aug 5, 2020

smaheshm commented Aug 3, 2020

smaheshm commented Aug 3, 2020

smaheshm commented Aug 4, 2020

smaheshm commented Aug 4, 2020

smaheshm commented Aug 5, 2020

smaheshm commented Aug 5, 2020

smaheshm commented Aug 6, 2020

smaheshm commented Aug 7, 2020

arlakshm left a comment

abdosi commented Sep 3, 2020

[config qos] QoS and Buffer config genration for multi ASIC platforms #978

[config qos] QoS and Buffer config genration for multi ASIC platforms #978

Conversation

smaheshm commented Jul 9, 2020 • edited Loading

arlakshm left a comment

Choose a reason for hiding this comment

smaheshm commented Jul 17, 2020

smaheshm commented Jul 27, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smaheshm Jul 27, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smaheshm commented Aug 3, 2020

smaheshm commented Aug 3, 2020

smaheshm commented Aug 4, 2020

smaheshm commented Aug 4, 2020

smaheshm commented Aug 5, 2020

smaheshm commented Aug 5, 2020

smaheshm commented Aug 6, 2020

smaheshm commented Aug 7, 2020

arlakshm left a comment

Choose a reason for hiding this comment

abdosi commented Sep 3, 2020

smaheshm commented Jul 9, 2020 •

edited

Loading

smaheshm Jul 27, 2020 •

edited

Loading