From 9800a41ffe807a5796fd08b31aaa0d1e7a274412 Mon Sep 17 00:00:00 2001 From: Danny Allen Date: Wed, 24 Jul 2019 18:14:49 -0700 Subject: [PATCH 01/24] Create initial HLD for configurable drop counter feature --- doc/drop_counters/drop_counters_HLD.md | 57 ++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 doc/drop_counters/drop_counters_HLD.md diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md new file mode 100644 index 00000000000..ec4e97c0faa --- /dev/null +++ b/doc/drop_counters/drop_counters_HLD.md @@ -0,0 +1,57 @@ +# Configurable Drop Counters in SONiC + +# High Level Design Document +#### Rev 0.1 + +# Table of Contents +* [List of Tables](#list-of-tables) + +# List of Tables +Placeholder + +# Revision +| Rev | Date | Author | Change Description | +|:---:|:--------:|:-----------:|--------------------| +| 0.1 | 07/24/19 | Danny Allen | Initial version | + +# About this Manual +This document provides an overview of the implementation of configurable packet drop counters in SONiC. + +# Scope +This document describes the high level design of the configurable drop counter feature. + +# Definitions/Abbreviation +Placeholder + +# 1 Overview +The goal of this feature is to provide better packet drop visibility in SONiC by providing a mechanism to count and classify packet drops that occur due to different reasons. Because different types of packet drops are important to track in different use cases, it is also key for this feature to be easily configurable. + +## 1.1 Types of Drop Counters +| Drop Category | SAI Counter | Configurable? | +|------------------------------|----------------------------------------------|:-------------:| +| RX Layer-2 packet corruption | SAI_PORT_STAT_IF_IN_ERRORS | No | +| TX Layer-2 packet corruption | SAI_PORT_STAT_IF_OUT_ERRORS | No | +| RX MMU packet drop | SAI_PORT_STAT_IF_IN_DROPPED_PKTS | No | +| TX MMU packet drop | SAI_PORT_STAT_IF_OUT_DROPPED_PKTS | No | +| All RX pipeline drops | SAI_PORT_STAT_IF_IN_DISCARDS | No | +| All TX pipeline drops | SAI_PORT_STAT_IF_OUT_DISCARDS | No | +| Specified RX pipeline drops | SAI_DEBUG_COUNTER_TYPE_PORT_IN_DROP_REASONS | Yes | +| Specified TX pipeline drops | SAI_DEBUG_COUNTER_TYPE_PORT_OUT_DROP_REASONS | Yes | + +# 2 Requirements + +## 2.1 Functional Requirements (WIP) +1. Users must be able to configure debug counters to track one or more drop reason(s). + 1. Supported ingress drop reasons are specified in sai_port_in_drop_reason_t. + 2. Supported egress drop reasons are specified in sai_port_out_drop_reason_t. +2. Users must be able to access debug counter information via a CLI command + +## 2.2 Configuration and Management Requirements +Configuration of the drop counters can be done via: +* CLI +* JSON input + +## 2.3 Scalability Requirements +Users must be able to use all counters and drop reasons provided by the underlying hardware. + +# 3 Functionality (WIP) \ No newline at end of file From 276cc6090768d69455b9b96b864a592952db446f Mon Sep 17 00:00:00 2001 From: Danny Allen Date: Thu, 25 Jul 2019 17:27:03 -0700 Subject: [PATCH 02/24] Add design details --- doc/drop_counters/drop_counters_HLD.md | 54 ++++++++++++++++++++++++-- 1 file changed, 50 insertions(+), 4 deletions(-) diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index ec4e97c0faa..6ed7080ef3c 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -40,11 +40,14 @@ The goal of this feature is to provide better packet drop visibility in SONiC by # 2 Requirements -## 2.1 Functional Requirements (WIP) -1. Users must be able to configure debug counters to track one or more drop reason(s). +## 2.1 Functional Requirements +1. Users can configure debug counters to track one or more drop reason(s). 1. Supported ingress drop reasons are specified in sai_port_in_drop_reason_t. 2. Supported egress drop reasons are specified in sai_port_out_drop_reason_t. -2. Users must be able to access debug counter information via a CLI command +2. Users can access debug counter information via a CLI command + 1. Users can see what types of drops each counter contains + 2. Users can assign aliases to counters + 3. Users can clear counters ## 2.2 Configuration and Management Requirements Configuration of the drop counters can be done via: @@ -54,4 +57,47 @@ Configuration of the drop counters can be done via: ## 2.3 Scalability Requirements Users must be able to use all counters and drop reasons provided by the underlying hardware. -# 3 Functionality (WIP) \ No newline at end of file +# 3 Design + +## 3.1 Counters DB +The contents of the drop counters will be added to Counters DB by the port stat flex counters. + +## 3.2 Config DB +We'll add a new table to Config DB to save counters that have been configured by the user. + +### 3.2.1 DROP_COUNTER Table +Example: +``` +{ + "DROP_COUNTER": { + "LEGAL_RX_DROPS": { + "offset": "0", + "type": "ingress", + "reasons": [ + SAI_PORT_IN_DROP_REASON_SMAC_EQUALS_DMAC, + SAI_PORT_IN_DROP_REASON_INGRESS_VLAN_FILTER + ] + }, + "LEGAL_TX_DROPS": { + "offset": "1", + "type": "egress", + "reasons": [ + SAI_PORT_IN_DROP_REASON_EGRESS_VLAN_FILTER + ] + } + } +} +``` + +## 3.3 State DB +State DB will store information about: +* Whether drop counters are available on this device +* How many drop counters are available on this device +* What drop reasons are supported on this device + +## 3.4 SWSS (WIP) +Drop counter orch should be implemented with following functionality: +* Handle drop counter configuration +* Handle requests to clear drop counters + +## 3.5 CLI (WIP) \ No newline at end of file From 6d621c3764c00c6c2d203d98cd59e9dfb2588756 Mon Sep 17 00:00:00 2001 From: Danny Allen Date: Mon, 29 Jul 2019 11:26:22 -0700 Subject: [PATCH 03/24] Add CLI description --- doc/drop_counters/drop_counters_HLD.md | 93 ++++++++++++++++++++++++-- 1 file changed, 86 insertions(+), 7 deletions(-) diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index 6ed7080ef3c..06e13d592f4 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -60,7 +60,7 @@ Users must be able to use all counters and drop reasons provided by the underlyi # 3 Design ## 3.1 Counters DB -The contents of the drop counters will be added to Counters DB by the port stat flex counters. +The contents of the drop counters will be added to Counters DB by flex counters. ## 3.2 Config DB We'll add a new table to Config DB to save counters that have been configured by the user. @@ -93,11 +93,90 @@ Example: State DB will store information about: * Whether drop counters are available on this device * How many drop counters are available on this device -* What drop reasons are supported on this device -## 3.4 SWSS (WIP) -Drop counter orch should be implemented with following functionality: -* Handle drop counter configuration -* Handle requests to clear drop counters +## 3.4 SWSS +Portorch should be extended to support a variable number of SAI_PORT_STAT_IN/OUT_DROP_REASON counters. -## 3.5 CLI (WIP) \ No newline at end of file +Debugcountsorch should be implemented to handle debug counter creation and configuration. + +## 3.5 CLI +The CLI tool will provide the following functionality: +* Show drop counts: ```show drops``` +* Clear drop counters: ```clear drops``` +* See drop counter config: ```show drops config``` +* Initialize a new drop counter: ```config drops init``` +* Add drop reasons to a drop counter: ```config drops add``` +* Remove drop reasons from a drop counter: ```config drops remove``` +* Delete a drop counter: ```config drops delete``` + +### 3.5.1 CLI show + +``` +$ show drops + IFACE STATE RX_ERR RX_DRP RX_DISC RX_LEGAL TX_ERR TX_DRP TX_DISC TX_LEGAL +---------- ------- -------- -------- --------- ---------- -------- -------- --------- ---------- + Ethernet0 U 0 0 1500 1500 0 0 0 0 + Ethernet4 U 0 0 300 250 0 0 0 0 + Ethernet8 U 0 0 0 0 0 0 0 0 +Ethernet12 U 0 0 1200 400 0 0 0 0 + +$ show drops --include "LEGAL" + IFACE STATE RX_LEGAL TX_LEGAL +---------- ------- ---------- ---------- + Ethernet0 U 0 0 + Ethernet4 U 0 0 + Ethernet8 U 0 0 +Ethernet12 U 0 0 +``` + +### 3.5.2 CLI clear +``` +$ clear drops +$ clear drops RX_LEGAL +$ clear drops RX_LEGAL TX_LEGAL +``` + +### 3.5.3 CLI Configuration +``` +$ show drops config +Drop Counters: supported +Available Counters: 3 + +Name Type Reasons Description +-------- ------- ------------------- -------------- +RX_LEGAL ingress SMAC_EQUALS_DMAC Legal RX pipeline drops + INGRESS_VLAN_FILTER +TX_LEGAL egress EGRESS_VLAN_FILTER Legal TX pipeline drops +DEBUG_2 OPEN NONE Available debug counter + +$ config drops init --counter=DEBUG_2 --name=EXAMPLE --type=ingress --desc="example" +Initializing DEBUG_2 as EXAMPLE... +DONE! + +Name Type Reasons Description +-------- ------- ------------------- -------------- +EXAMPLE ingress NONE example + +$ config drops add --counter=EXAMPLE --reasons="SMAC_MULTICAST,DMAC_RESERVED" +Configuring EXAMPLE... +DONE! + +Name Type Reasons Description +-------- ------- ------------------- -------------- +EXAMPLE ingress SMAC_MULTICAST example + DMAC_RESERVED + +$ config drops remove --counter=EXAMPLE --reasons="DMAC_RESERVED" +Configuring EXAMPLE... +DONE! + +Name Type Reasons Description +-------- ------- ------------------- -------------- +EXAMPLE ingress SMAC_MULTICAST example + +$ config drops delete --counter=EXAMPLE +Deleting EXAMPLE... +DONE! +``` + +# 4 Flows \ No newline at end of file From a28e5a45f4947e31294c5ad472b4d7612e03464d Mon Sep 17 00:00:00 2001 From: Danny Allen Date: Mon, 29 Jul 2019 11:49:58 -0700 Subject: [PATCH 04/24] Make debug counter definition more flexible --- doc/drop_counters/drop_counters_HLD.md | 39 ++++++++++++++++++++++---- 1 file changed, 33 insertions(+), 6 deletions(-) diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index 06e13d592f4..0c5663b9a4e 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -63,15 +63,42 @@ Users must be able to use all counters and drop reasons provided by the underlyi The contents of the drop counters will be added to Counters DB by flex counters. ## 3.2 Config DB -We'll add a new table to Config DB to save counters that have been configured by the user. +We'll add two new tables to Config DB: +* DEBUG_COUNTERS to track which counters have been configured and for what purpose + * At this point the only supported type is PACKET_DROP +* PACKET_DROP_COUNTER to save drop counters that have been configured by the user -### 3.2.1 DROP_COUNTER Table +### 3.2.1 DEBUG_COUNTER Table Example: ``` { - "DROP_COUNTER": { + "DEBUG_COUNTER": { + "DEBUG_0": { + "configured": true, + "type": "PACKET_DROP" + }, + "DEBUG_1": { + "configured": true, + "type": "PACKET_DROP" + }, + "DEBUG_2": { + "configured": true, + "type": "PACKET_DROP" + }, + "DEBUG_3": { + "configured": false + } + } +} +``` + +### 3.2.2 PACKET_DROP_COUNTER Table +Example: +``` +{ + "PACKET_DROP_COUNTER": { "LEGAL_RX_DROPS": { - "offset": "0", + "counter": "DEBUG_0", "type": "ingress", "reasons": [ SAI_PORT_IN_DROP_REASON_SMAC_EQUALS_DMAC, @@ -79,7 +106,7 @@ Example: ] }, "LEGAL_TX_DROPS": { - "offset": "1", + "counter": "DEBUG_1", "type": "egress", "reasons": [ SAI_PORT_IN_DROP_REASON_EGRESS_VLAN_FILTER @@ -179,4 +206,4 @@ Deleting EXAMPLE... DONE! ``` -# 4 Flows \ No newline at end of file +# 4 Flows (WIP) \ No newline at end of file From f26ef9d29f9e10f43f072bd6452a4fa95f55c84f Mon Sep 17 00:00:00 2001 From: Danny Allen Date: Mon, 29 Jul 2019 17:19:06 -0700 Subject: [PATCH 05/24] Add basic flow diagram --- doc/drop_counters/drop_counters_HLD.md | 21 +++++++++++++++++- .../drop_counters_general_flow.png | Bin 0 -> 25065 bytes 2 files changed, 20 insertions(+), 1 deletion(-) create mode 100644 doc/drop_counters/drop_counters_general_flow.png diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index 0c5663b9a4e..9a8d3e02de3 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -206,4 +206,23 @@ Deleting EXAMPLE... DONE! ``` -# 4 Flows (WIP) \ No newline at end of file +# 4 Flows (WIP) +## 4.1 General Flow +![alt text](./drop_counters_general_flow.png) +The overall workflow is shown above in figure 1. + +(1) Users configure drop counters using the CLI. Configurations are stored in the PACKET_DROP_COUNTER Config DB table. + +(2&3&4) The Debug Counts orchagent subscribes to the Config DB table. Once the configuration changes, the orchagent uses the debug SAI API to configure the drop counters. It also publishes counter configs to the Flex Counter DB. + +(5&6&7) Syncd subscribes to Flex Counter DB and periodically queries the ASIC counters. Those values are written to the Counters DB. + +(8) CLI queries the Counters DB to satisfy user requests. + +**TODO:** add state DB to flow diagram, simplify some of the connections. + +# 5 Unit Tests +A separate test plan will be uploaded and reviewed by the community. + +# 6 Open Questions +None at this time. \ No newline at end of file diff --git a/doc/drop_counters/drop_counters_general_flow.png b/doc/drop_counters/drop_counters_general_flow.png new file mode 100644 index 0000000000000000000000000000000000000000..c72427d85ee8dda0de4fcc9d4d88dcc302194f5b GIT binary patch literal 25065 zcmdqJ2T)X5+b+5r2OUvx6ci=M3@T9sm5hK62&g1MLCJ#ToRQS+I0isq5F`gB3J8eg zj3OYCbIy`;j@^CV-HkKzedn)J_tvRfx9VD@NcY|=zVUgVwY-!SWvQtcs1O99z9A>A zf*_ReUy7!mDd5LdmYW#(fw5PS{T)efV48woelotMa1B8+{SWOvKp#D5BX{2(L1-$_ zznE(4cMlQdf#(hBYj<39=J3S8;~O6A+cWH*%}Uz`?u@Z$8L|iP_`EkCZ5-fM@6V6C z`&4r_y*}yvi^dCz=ecqVK0XhneyS2@5l>z4LOZCdOPxFDRgjCjkY0_E;h9^Xf1@~T zP|A7cBHQKPaeqF*T{GDfW$;~l^r^_vy=|h-CRF5dtH<`zJLB6Pg*$bJFU!Yxwk)j7 zM-1#k5FC{<4rcT2C(4uX%gz5^|H7e$(OGO~mmn+__y%>GY0r%F^k`ru9>hBi|j-A6sSJb-H-0F9w;{omg_oGEQx8YL{v2NtXq zaFnaC{Aet9#ci;w8N2Gfo6pWdWI~%Opty47lB?cg-#9ZnI?fXxa}b2@BIP|JF{%&p zJS)VqL9g^)ybE!8t=spbnF!YOJhK#R-7%P$6P;d|#>nO`l)`zJj(G?(Uxm-@;PY$6 z2ks^Q#s5=b=Dv(xVVAkhPqPK?Tgz#A1FqG~&RWGT7d0Kl83ir!rxGJvU0pv`BZ&Pq zOt6TYL+GY$1f_-*M>6tU>wr#aDKFu-w5d~Fg$s9f-Q;%o;u4#TRE>4=Z62(jMD#9j zlwUfqHA-|E`Zn1fJ0elu;%Y`I+$OsCQC7$AdBvHy2A%>n#8p*Q zoj8BbW~E*zM%0l@)6sG)uM9u8wK5@lk!EGQUbEO`i94hCm1SvSYAScf!a#|b+hK%n z2BsMif4gbHk7!nrB|F`THLWkjJ{sL!w$_dslfwjWx6u#%C2G@n$oipw2CUF*!NA+L zKSvwy2}-K~*-n>i=+KINsnELNIh7EiZ|LmojLxJ3m4mGk>sc4i zM&+z1{@PfyqCkNPb39E0@S!hkSO zmk~oc{dB*2?sj_5gCtKaTgR*6>{}w9A0J9d$VBp32~RB+#Ih2Ne-{CSI_Eq8Ok&fn zv=q%15&CquRW0(up@>5aNGLbehZjL-T6xO_4(xACU@LyOHo%_CBrfQ{*;~FAFN91y zyO^ZLRPWTDa@Q0e%%eH-VANzHF%4ZNj&fENzwAt;^Kj{-2l&g+7arLt@ekj7@w|IK zThpq(wXoA{SCS-ts*B%xpjgRcS~c^7%hE{I#Y2)%260 z_PfNSusIYEV6l&d|HhLv$Dv;YHfz{p?%*)E*xRHjHB)HWTTH6J)}Gfc!rpdXDD-^- z!!~5hXY+kvW%0ojZ<*J>zS^o(wmIUHe6YG=JCEuBs9UHeX_i^4ICDiB-R|r;7yjX= zsZOXc4>+^+DqAWrTBIHCys+ix>zlBZZfCtu4LU7IAdRq=GT+GMp~H$FZEfeqbz0BK zWGT&!Cj@0>y_<6Vryu*fIP=(plAU60psCv2PKj*pgqC}`X_pv` zA6vN^#5Zj-KfxeIz7*$K$G%3=4~0*bH2)d5PnOnCf~mIM^s}#{(DD8%jfe~D4_IxV-g zV2R_=38<7)Xj!`rh6#x8JET6RvHrC-B9~cG#Qj`vlWhy%c;DSVgK9q>&D>egivqPj zGt>qvedx=Uzj1eun=?FD+K1F#qP!>lE#MCr=l~&WMv&&h^t^VV6PAD2Kq}-Tb(H=$ z+WQIYuKV~9zUL$t@l6G1PIEZght0k5iCggdft@eGam=lMw#toH-Oor*O--%cG)DBS zI8pftO8;Aa;ttB{|90Al@n-52$7D^yQlk~_C!~&t)cNv>p>XI8Y@$QWbV>3-mXlXP zEIA_h=~t;sBAltBP9bmRz)wZ{3JBq%3zQ(->VnRz9I--J0ezOu%6+Oc|4L(F*_@19Y zH@u%Hi%HC<(@Qz;y3pZ~6ieP^1sBvJ>_^IjILhflC=Nexu(%?CiX;zdTMsILewmT_ zeHJ)ag4E>$R*?U+Ug7+AMMB}I&0}IL46+2vuTRBS znSfL{GGDp$kq}m~aV6gv7EZ}RM09ifsRaZ))a2$nuCMvBF$oBNvpH|_89{u>pK=Nd zI!t--$i^KcFW6%FPyxJjb&QiV+*Vt=&2KQ4ko5kT@Wu7g%ia4CT#5>g^8)3`9#(E0@A8_aT$Q zSpNBLQE;ogBF;4_9=9OahI9)+CsOJC&_R2iSoHvfvs%&Gp5UK zwW}!sovjFYwv-oP$<`W~N+&*v-%?xfnC(8Z#0R^s%td-vs#$u?mx8o}cSfGb%6z0o z5W_d5@zQlJloZZ!f>}^M&T7-zL#qr`6e-Oa<3`^7N^CP>DKL@*@?}v;zaqMt?76Z}!|~ z%`i6zG%hTOG&hG2CZGSd1!ZXKG~`#Q{#*5{pItCyM35c@7?jaFM7Pv^t3%VeWYkaW zf=+R0T46LbqhOyytAe<=c(;Mx{ylG^O6&L3B@`w@Qm*i~6jz=KMu@I_etIaYx?J9S z8Z|1npTr%OrIp}wxd7Y2Cl1RZKuhUjVPWy<4@#jy_myPTq>pzOd=$9V(_dQkWluUe zI@U)BS#QkuLr3GxjD>A0pBce7>bCV4S&hDdNyrDn;@3-KAWT@~$?I#mXY0=_wmzkV z742D&rrb;j(eYO3??+!h5ye#c5sZqVe{r!*dpcOx-kpYrFi)#L$xT>^5=7D>g{%ik zHU`9QW1_*`Kh9+&i~sVPNAsT$1~q|6fIu}1rr8+>%WBD6Zn|r4b=KZ2*N2|h&nV); ziXcz9Ld>B$Fio(Pj*A&4H>N7PKV=8m#I&wwf8%v4FLADZ|-`3L0c zInX2Z-a|GF74H!#O6HSS^8Ujjr5xOMRRC2r?((a>)TNJp6Bw2||D^&nP#Hs#^nI5n zKqDb6(f;Ua%|GnOc;Lw4=vpv>QDB$zdvtL16x96Q|1$BuY<<00L68XtCXh)pH%#=T zM7cm9z1{}qS%VW;1^$0blpYub=&Fg6ypVg+1Mz2XiJNoobhtHkr*}hVrMuBnkZWF6 zVmV;dW^KB?`!ua2{p1yLSo?y_9X92NxT6WU<~V~<(2_TGd;1dqpl(gwRyE7_weR$S zTe{Ah{7sJzo!{*&5llWW8K18XhmBPJ9j4mSi*?%CE(NEMGv-vnZT-Ws-cem0 z*fV5|#(qeg!h=4R*xFj{b44 z%a+g$`)h+uMURr2!X8fxlZ~bDJqU;NYnw@70CM#cnAzL->qF~01jX1}(=mf zPsFnjlKef1^G9y9x2zl}3)4Jv_v?`pSYgi@-K}}u(#)0T3#F|!t<$1k?p{_pPx!98 zJIsQwHYA=Q)LFjbiWhw@jML{b!JHP6*lax>)4krl1w(iL=%K|K-YaXnArob#Be{L% z&tBZ&vEH5VFWDSZ+8tpz-QY*V{OLDEX%*4sUk)hSJmoGwQ+;5xZE+Q!{``epu%&Jk z7A9p>{oQMC?%^Nxho`_r4#Mv=JmiiK| z7cUN7n_@ zt(C}~@d?=@R`m>~4lMz#3&kB`%bOu~MLFEo8N|=+jRJ)Et{Jz%Q@h=-OBy4IR_g8p zy1u@pYl~&|bA9>^gvumO$6<@@ZSw(U38#b&76v!Rl_;e$&!z6NZ*!dX$0_e=)SPE2 zqB{Nj)3NKA{h`cMANXq$(gtfh&vq)lW;J~z>gcc>-{`VbZEkL^f;la@GakfCVa#ry ze1}-O{8d@#9P zT#J}n#;gZh2{-SYpy&KU{Cz$AUzKO0OcN&jn7eg*SIkatKAanEt6gy735?7Wh~^es z;r4`7|LmN&Q-Xi6*oXKhD)=Omv0p+6&b&#b3A%Xw0TxrHb0Qn_eLYDpo@Q&hW2RmN zxs8bJ6JsRK(>yunO%|oCt1Sf@krrbeLWycxtwojLrFYMBXJ#z;N@hgIv=4ZyIiGv@ zeSwg1Pm<8FbdNddZRVBjX&+mL5b?bEpR%0_gPm?E-C)U`V&Vym$r~diDmW`*s}tum zj3fb(d9{AJ$m$)UG@3@W+ht)R z=44Es>6Y`!^SAc|P`gFAL9?uZ^S zSBaj1QS7cAoL`eUuIM=)vU91WAt$RQl|86a(P!{cTHaEm%J^9rP^c3NI-n$RcCjxC zHXqD(aq0&O+!yuk?a&qUYcQ$K@!i|gxZuKU_ha2%>JjPIcCJF7zNt|ywtGmpd=6%eus0gs$mYZdZP%@tAr*5EhK1%nzs-dJhO8D&B1g zm4RQq8cSNH?#!`T&dXmo7iDlf3z={#U0mI_B%noPWX(E8d5=xL$X{FN4lV=x!hHMD zATiuLL^JC!;k;T$^uVgBbc^~)mX)Z-QQ&*$`#LoKk?iHi@vNd-JCkf8ao?g8bMlah z24!}U5QVVaI^GSTD3b|?;IJJtL&aF6_N?hbrn%rD8`Vyp@__9dAJQ9&*FHTeR{4*{ zvdU*7C*}LD*fZypPztAc^-r^ZGIP}ly;KvX+jEDg9%?eNdV^imBdGf&#reSalIrws z?(+K5&~1J^-I~b8*RmqJA*?ojqJ36MZPBz6wo?ly)=`AU7XonYjE5kzKF+LzjLguyN(H- ziT9{Z*IfwqJ7}t(63}WZAF6>-4AC5o)5gDKr)l?d%;9A#$vWc^tK9qGmC|iDwQX+x z6I0vb%=>)+Mn!F1%N6w`brD;Rolja~kIdeO(?@&$T5@!w=k$W$eA56$)nisqbg>`o@*oSOCwoWff`f&l1)t|iEkQ4m#IEvzshAQN*{l$q(ew5tDhdp5z;DN zTU;3P(@tX7gT-&zq|BZX2^2e0%Ug zde8oi$+e1@vKAzp$3KY!2)KdkJV8N}LajEW7$11$aTuggwHIL|3K zsIL$lniK}N*rMnG{1AoY7PZ3({5LL#5Vq^>I@t;eR(A(Y%8q*hg2c>3UI@Ta0FZ0> zv^NCSG1%q*J73bJB4vwfV*cry_vX)Clm5Pn&m-zBJ^x?BgN@xabsc<<^|;P)X=xo@ z+g>GgcQ!GVpkN3!_!W$qMv$9t?@_u8$ufGD){PxKT9dR;I@q-lC0XM~moWAVJF7zL zsl6bebhjgUA_PA}%Pkr0_TZNQt_kSX8vo5YMr_&4DV~Gpx$sd_`RctUg_+!@cEsj) z=K9Q^ytpH>Qz}7hUmzGdz2p#q=V}rHX>5^7^?VAZb&r|Sn0_HFgk}ZmKOarjqj8^G zk0Oq(ER@=x6T2D|10!u~c?5&_gIga%lR?a$V8>o5gJHUOR9Y5?+CmO#|b2YyYFk9{^#6$?v7;oB776;!*K?c$U%`Uny#NJ`-!dkaHh!Ep$XCrF=5oEN`-4Uc{d=+t%Y0mu_` zPavWD$q!BgY@iK@BAWD!Zd8U(xIR~t*ft~OXa@$1|AHLCDiYF2&ho1X{JDd!Yf^DS zmSR>W{D}Nx(#MGJVc-2YPICUJ#|rH*s>ZO0&E0&oQ6D_4J&&Qxr65KoaKlRmk__GhApfRHd4~ce z{|RUy%*JGh8JcS9hSbrVjcT#0tLqQSLsq3*p)?vw{1&NcpN6g!1#Ni~`pBDu{8p9lgYn^t^ zb3Lz9tXF$cOg?$v^S*N}K?;Y00fpWXMcKXYOffXt+S@~iih%tSqYrFcS)VZ<_&Qgoiz#VD#itfZkWsa419H;IDj()|U~tvAB>t?vL*G8+&A`YN$L8$eE@`FzO8LH?8n zPSc|7lLroy$eUtKH^k2`%9RPd)v=K$e43F_^Ml@};zuh^3j!T=R1 zYcF&vLwoy$EIg4imnH`G_j#wMUl@(S7+td_c7wJ!r*&_SFHH=IjYk7ZV4DMR!nJ1!wc6 z0;CeAExktn7~mGY6HNbYb43@3I<0s6wjb9kBs4ep(z_KE7HSl@9#*!Y;=6u_+;$eH zjzcWo>3h>8;EpxVFLowYnw>8+)ymGp>@$Scm8V!g)RrN2Xf2C|ks<2BoT8^toYw{7 zGqq?MWp^q*N&!*9QinOu5G*EC6Ff-F3^UilAysZ_L0@|-*D%mRQK?L-#XmA6mX@%@ z!Rsnk#6I1L4!PH~)z4?;FnlvDO_TH0J8=gyrcQB;HceI)MA^5Tobnnt`ySaB1u=_o zgRQwLi`R=bb`t49Z~n0VORF9ycPN-Z-6Uz^sSv>QHG*^lj*PQsL$FUM4m<2et5QcP zPjUnSt`%f%vihA`u>Pocjjtb_-kZy$$|y(q)r+qU;hU_hy3V-l`)?jMyg}=6z(lyN zCdt1I<&{Ii{I0Kd5Z#SsYa5n-y~i23jG_c zJ$&SiR_?sm8mQmAMy5R=^^XA~>k)bFZ8gtj#g!MR+aKk=$$!=B$6(NM0l50uU!~%b zt_0k5jTJ)nF<;#ya@?E)gjIwER)1a5A!+c2gQMdq;y=@zJPhmZw$o^2ZhWitc2nT3 z(U^YZ;N)RaS(S>f=>}^1!Gzt1{bO@4ee1lU`1bNXx?r@z`2v?>vgV*{gg7AN zw8|n*oZrltAGG1iy8*cb-_;k>jFeqXCb;C=xO2D6%5JX+qHGd;#UP^`{h!$4orm~s zPH9Z=6r{}WI(xC}P=?{r%Bh7nfO%jiP)1CW6mup-Z60->IPly)g02~_rb}W&1s`gN`p1JOgn{(Cy16z?!wI1S7{#Kls5x`qhgiVorC>Rg2x1ha^8#7a> zk9hc(z1i0stFV+&$+C5)>36rC%{!mBKKIT&rRls8qaYSv6&ii~wJ_l&v;K^EHE(>> zjMxO;pIN@qyK~uVJz4vs;!b>Nl*G7D8BwDrN$_0}fjgOZYYpq;RT}-SDQ6Hi-PGpK$c{XjWf4}_y~Mb2!`#fb#5TxoTWd{MXEyNLqj}MF$Fk_EBNyzC z!CE_#z1TyLIX{Z?JjaLHEX;HG?p9YhQ%*1#UGPnCh<#lU?=gML)IC<~j6!A3dUAy4 zxdQ*z5h&}>pM?N|E|RHqxhqR{{*jG8ctIf81XJ0}7n&HH@zu)X5cE)EsO!6jgHUs) zGeW1gA1nBAqDgZtM*PwvOgi4T<;|EzOLekG`iZjBl=;6xj+Yn*u_r`oKoB|}ht#&m z&m;sE9}v!~INNUfswP&GH?lLT-)#A}Ch++|6k08?Pp#^b{I2~~*QDYer=a`9Fu17^ zqY?97o7GB#IYX09^L%8oMRj;jFOD%LwBw7_TA*ILlS$@BuJSR!(wbzJT25sU_6o^+ z6+=P9fuHhTI@{e(UZOfF)oW27OdVhHdrVmJq|r4bT3+7x(j(zUQ*wM}^m3Km{0Z4W z>^V*=yyn`gxnRwSTMbLHi;*qBz~V`uzrWtU+wML zK3h7}S0AWiPHh_M5RTz)bG4YE%wIoOS^a2klTyJ2prw6ve{8>Ph6GHApJ5W9G523n zi3Hu)Awx<3jR8haH~vm8RW`2jfXVVfwhU`VfS5GSu>X3re{FXUwTZ( zgnpY#&P$g*qj1BqgwSI9eRzSot#flz8M{p^R)LXp(mnzL(zY-pMOD8y-asM4JB$qd zco&Gvu2use;9k=olppL6<@BoHeL@?W`0GNj0~ry7|4i;Hoy1!-$}&_Rf_Re=;x4^) zd&c{cyIq^#6@392eGV>d>d`PY2*XEJXzE5>&?t+%tEyG}y4rBn+uD|k(4O}Y&~YHl zTQV~@;U_h92`zxbsOIn`!<$pJjM3uY+EF-tL{2JB9a29A2Ie&3UPReq&ox}dw1c&# zgl_&&ML9fZtM3K*T~T5bTG5`}=gj>R$IpB2^v!1MhRlutKLf}4AD>7A@4`*T#Q+Qx z?CMh#FhN~tQ1hoVXQ2L^__i79q7X5Xi6Y>V-xPy00$0z+&#=dC0-Jo`9knl7zt2?0 z%1Ytx|8T?(ufTnugqJ0Lba%$G-#KMx$rA=lW+*{l4j&*iQRwwcdfN=ed@~b%KB+w? zh2w&Dff?7>Wb&WpMcaGR0Z_BRTbDO0w4|8P3*$X*4r#lLKi5%xg69gEwzTC?9%Ti4=wSj zRVBJy=fCOQ&Hqoq9Lcr-!imm2q}DyxF^O*kEyV}`K8!Zb+C~9oA$FrGbqIBRmx>`B zF=%wremMqJRs%dZ<;*A7RI~g4D;J%PPrL2U$Y`go-no~5i0>cT4+V9)jAaO1M#l!4 zO!UAAdvoZtj~u^XX1Uuv!rixZ%iy+a-n`gxN?>VREf$qI=jZ>+nsBgzA%GEIXr5T> ze2s=sd2zTRug|Vly3?LhY43wd)H$$+h;W|HA z^#j?W*=_WR^Z}I9aQoS)L1MDN$k=5yiB2&(F^K7mlyF`Omk0$;wwv=s3s!x`R0~Y| z^^QSk!{WC70+b3;FRCP`Eun-ppK_tIH?1Mb;V7Sp5n(bE*10xnUm26h1fjTvugAxd zVZ66kv)s6b&^*-I&zA>E(&IQ&K^6cE0eOrn;}5n(Ok|*ZI?YQ=lwV=BnE^SDTdk@! zrW-=3frc*VhEaa`3``i`*GsRz&@5?0h(G4Kp>7*dipeKXShrFzv1uH+stMwm4WP$8 zJD8@?Bs8>tu=tmfJcvqxOms@I)v8kccRUTn5YW@=|1Czc+pVJwN!% zA_g)4EAhv#ZqeQsXu!RWIAtD7aizHx6ME2g`~#HnO-&r`=o4>HgO95`Zg@$P>$OXz z?*L;)&TU-loX>(rCj2XQwRL~5i@{mYO=Uq?Z51gu96B38Fs1kq_RvdqU6p4}8NGWz0E0l6`(;C{CTCX(i;mPBTHucTAnsluh z1r97vnBXXGO2vhUFE7W=54fBI!zCI`%1wDhA&(}#2%Ni{tyTpdwu}mPEJLoe)Q_W0 z#F6X2WW4ehcROpg2(za4eMUpdSQq&L)jC!pL}e51#NA)I2`I;xkG=Ft*TNSIoPhev z&r`@FL5~v&c_x!|KQ51ybw`l^v7xYE9EEOL-x=g6Uu(6xF79UtL)Buay0Ue*FOf3e zc9S2uZe+smzy-^qf-_bv&X!&12O0p~P*1id4AGl1k2gJRSy~O!BIK~L=+fs6a%;QqdMBvKzv#YepY{DmngnwJ){=KqkcEu|7OSxHsJjLIH({pED-gZLJ_Gy z?H4TRz4ZHyiq=$^m zMl8;dWM>Toyw&RN^m-*yCj#I9O8P#rWaK;fdk}0Cin!PPDUWymn_ows0%7ZFF#_of zf%}9}7->{>exMvAsYvLhuNJxaegOxIqRat%1g1wpDz5$kCJPJ|a1_UtVRIp-j72(? z(0Mc673nDQxQMhmknv*3N;8KfH6J9Buo-y_G-%!9bcM>s64i|Tdf_yuPoc_PS=nR~ zDgp)pSbMCuUBLo$O4==SPhD?PnlQe5|}J`5-0$u4jfECC&G1(ECOz) z`)JE3Z^34~SswXX2CQs;epiJ?+*>I=L(=f1-o=m<4M^#ud1p?H$aAPAqL$+18!9l@ zFa&79z=VGftmgCLD1n{xA`5~nHO>zb0(tW0+r-7r%SS}DMCcm4KC9D}=bODe92 zWdTeb^T2Q+Y3rUjopN9KJ@R-vSlBNM>i8crpsa#2iE^3Y+0l0ZkIR#h>CoHY?Qgrz zB=vh=PTDLG8h5G#~tg z{sH^HDChue^Z=UDGZ`syk}|(l1{H`SZS{pq@vsS~mptiwK$!_1@gvEOvOAbP$o9Nf zQnpR=rmb~RTGjf+!$#hqEVO0rhAgkU|-TAdDq= z*ms`x#jQ`D#b_ih#?^&%d2%?0=F$!I2Ej9S=xEO*~4t{`=8Z zKWc4_JX;$wsw>DJ&IR~Buui!yz>XJr*hWTHKKKMeX7tC~<~t}64=N9FFm{+KN>69H zz+kOw(Zhpde9g!(hdnuUg*2#rX{rCn0pF`4k4)1%MC0gzcF3<4rQ+g_F)}DXz>Rhk zT1t|E8ges<|G2X$gxu8Sv(r{V1JET~p@;4IpQb078J)lV9MliN5A4l7DCjWpQ2Qpf zk>p{&Q8$z#Vpls*d3elm>=0;t>D$X?T_O^n!=o|#ett{?MGq7u1Kd*iCz3RP+4n+s zIDB%wQbFk_KxGn7Fz*!69d>vLX-Q^e+SAYEatg;s5JkYtr7`I+J7+LVa+JcA33_h= zL6P_H4lv~~w?dQPk9cO>KMkzd^!q@aDcDw2tub(yWB!&3HS)uK8 zVTHQjH?&b#7X1$hrSBx|V2RI(Q*nc3J5nvlTcf;^T7#ad^Z4a6z^Hm?rIh@;=ZDRT zB~9WtXX3JNl%)Gc6nFRVn=|J(r)gP;8$|8zy7*6{#NHMb!VGb9qhL*C?90+`Vk&fK znWa)hKli67!|$6zP;NBpRi9YEf0I3e9nF1vH}43sn>}82r5d~JoHMY4^%%7-+eoVQ z>nA4nNvK3cmJ-)D(*}v{{`2>ost@Vz2vzLc`8fLfR;7s5Tuk8$ISxdg zD!QBBeuq>b3@ZQGXFWAPo|~it-c}qwx^(c|Ak`Rsswa@p<0X1aMTy@ZW9$5mLLJ5D zE0l@fo?GgD#d9rbbIQ9hf1R6k4JA}bZY`I&PK@++KZwo#&sKl7p4}bxGj#sm;Bj4g zoo6TQ3DMpXEtEpq!UoAXkJn(Z#X-@}lEk@~lH%F80$Sap0guPZa`25rssHu4Kb9I< zTli7RSU0c7d9;QLKppLM%(LJx*0+EPKYYMP(qfzYsA>4ZApY&F+-(iW=H!80q;af_ zr~8=0pTeK&qn5p2#P3y{_huujjJu8!x=2;%|MSms+Q(``tV`B9#OCuyM=E`~Q)aq9 zJg86*>Thdp4HSRI35P=dtGk?bx*m(^xP}YxjdJc=?@ZZZs`CYZ=I$6xUX=kD zoWe>qajyJz^Ac@E2z`3b>)x{{Rfx2L$sYf8CTVTJt=f-SGuNUU$iT-ch5MX(EtMqk z4!Lt4P_^14iLRf78s|TBuyyU!mDJ4Uywild%nFMWLwhuVrKy%qJ_)<+s7_be(?k`k zJjI)>aQ?2MI(?N;(M@zUz9kmpi3e)J>t0nh;Dgvl4DM97*#-Yzd$57Bz*7QWMN zr|t&bPy;}!d*h{Hkzl%>N6?s0*Dg|7D_Lm}TL8MT5_Y>ac|ROyGIrBiX}QHKkp3qW zRP#bLk^q4#ndv;m5IRQTZaU-K(s&_0<2Rnpl+C{HkCpsNnl}bEi_T!3bv;$ZUU?9! z2btYG7Q~1j#h2&3q^5XJbMuN5g*v+*neffI0cmY!Bd7+zC#LZY;;QjPLm92)m*G5F zK=}IYC(~9h;~S39Osq!&O)LuXa+2(*?7$%?KcgZz2&As_=$zXwL83tDpwe(u#b0qS zz_w&I&M-Kp(}>OE#t+%OUYMiC(&4j457`iQh~@0O*4RrPQqvKZtx>#J@K% zmlS=cerRyc4X9(9=lZH(tctsO+uL6kxgHjh##micy--dsIH1djWyF6Vh^J0*7Wjvy zFA4Irn5z18@@04q^!z$(zJRMQxsgA`ep6QkpCo6VjuPaSggvsC?Ad30<|iesZ_LcJ z=dIo-Js9wV<5sY+a(0lI%^7`n=OvX*kxDE@aiXyv6!ua6tN&mU5+Vc++b?Daz~8zw zr5l~k8S^?almrBV%d(Ah=LV|)a_TD^*2Al+oF)uhoh!TjZ16Nx&ZN`&WEhD9KGt8J zdn=$Y`Ifj8cj<~ZMIYwxxz@&g;3`uUod2JFgD~=$^p~ZoPB7 zx$WBkwjN~Jq|2O28}%K0%(&ga-!I~EN`Yv6jqPRum5pdzeY+lIeldDF)ANtp!f-zG zY#wLJDuYF#6_e79<3e*QjTt4(*s5>+cw=2L&)+=iv<@g5igi>sWNJWt=7^=D?vRlrsdbd_P^?h26CH`v5$x*M|j&lON@33nj?yX{aGh)Nk$A_|wEe!)N ziG2EHQmyo0j?&WOeu>bULpJj>rn?Ukh!idpIRjo2rThw~4%ysH+s|E|(6Hk6tff)t zn{nGvRXStDbhTVkgO*#b4Oh9T!d!u`HIHJW=vv2Xv;>gz8igB!v8frkjt&;Zxofw~ z*Au!6=!QPZ?iuhlHd=u$2SZ`O_V0<6aPS7AVylZeiQ%f;+R2se&Yu%4b8Yi_%Ex^q zHB^b171IJlsV&xi#`+yHkzyM7nwD+|Mm;qwJtk<#MVa3WELv=bs#~y@5KvoEB=4Ws z4u*u6U4mHOcau{+_C={DDl;QBg><{u+?2nN~C|qcCx)S&cIPax5Zs3ylHH0#7i` zN~6dB!q_PDE9G^{S!n@lu%i?%G?_LE3@L95WXB#^JgaM*ey+g?V};V*Hlo5zCVDE@ zx2;R%UtKyeq_L(jb9_kgX5n{DUE3Mnf$=IySq;u|-<$#oDjVZ}_9#uLE3wo2Un{Ue z(Kmb3LW0G@D^M#&o_bvrZ#mK^W~`3O*iZ;H!YD>eOV&z@B`NXO9Jy`Bsg7y5J1C8r zkB|STp({Ic%X(DKNF8Um%u(L-YSXeNxF`Pg{WpOQ)p1�*CnFjB;KmNe(cZ(UuYA&K_RrUL&nvJq0A|>6BQ-U;Q zKA~I{lGkuUStg9Ur%GD{S$q5Mjz`n@d6vSDEPS51VZtlZdmMtzULr~~$ z^q99-lGu9a8rnAgWb0FWcW3SiVn(ADQnk!$iNC$t5U+{*IB@qhMB6Hy&RYmwdlr z9231ud&A@A9UOZs62jli=TEJ@J-(rvnuZ?emDA=|7i-O*98Jm~Oz5>Hr=)A*T4ld_ zbO9oFNAi#ZzGL+PV+`xRqCO(eZDZb|eJ4lIefW@W{>Fe?>*0IDKDxX5xzz%2 zIEQTTIm!dxXOwOX6ea0qC=lpA26x%q&CLuNUnytL`sQ2@o}RV)NuhDOgmt(FSJ9Eh zpF)fc@a_FAU+$bq8nwQdLlK$&rAoGF`;-z>3O*ryMIk?cf~nCvgAboJ$~iU4Pbe`P ztlF;0YaxtIC@QYf2BCay_2k!eaQfY3I`2Vj^VdKyhZx1t$W~dC;v+Wrjvfyj)V;>m zXz7pt%Q{Gyp0vl`?K8>cE!`Y3mfY!XbE^O#YzxlK8b=IvZZDdKzJPcpQ4Mzn*pzy^ zX`|hFZoB;)kIyK%7vyqwj77gPd-%Ri`%+rtQ{N5QisJR78QzMU4dx|loo*TPE1w#B z^DEkcFIr1n&AT7Sgx$EgAW5QAhFi{13afBWsAouazHUL?8Y5Tv+Q_Ju`+5dL*=+@9 zzXg?DO7^Yvn+B*yM+>*WK{FE&-kxa0E~yy|zdznqCEBBp@%wUM3w%|Zg!-L%$Z*<( zzHf^ilZ4_<8cp8gQT8d&C~jz*1pTEg!8(5kq4!4dwP zBGy5@m=%pCI6ci)^!oc}C1h9wrGHpeHQ%^58Qi{Dr+sHh7a(}WBDaykR|`q`M$9Z!QnXTsV4Ax~J=Op4ZB&ro{5>oVV`E}aqy9ha_gmwC*8O+}A}{AU(J zSFN*6@Z0eebKd4=z1vWl(TH6u!iF)Q`6drE1X#|!b)fmUpt*scE!~nzwQ7v|$w!XT ze#ks;VV5@si>8j4*gCqQn-eqC=F-!Dztgb1t>ospT)nb0zvJ7f-WM#!cp`x7`=q!3 z8Wu04ySvQ39lg8KU{VH^*Fo)p4>I31ei*AJ+Y4omW-!C=N!$`E_#(-(taob<>6Y1V zRA7a#eBc#hb@+2|>xp%xNctU~7{avDw4360;^wv_-g<*DwBCq!gdErIU&-Xja2^c8 z=L}EJxZ!l`}w`)&Q2x*n?FKJ8{B#i>bLv=Q>t6(}&svjKM%yvs*{;herA zwb_@dA5jRa78n3Y6eR403HX+9ep$@}GM5~{OlA$P-6k`okS3kmT4$Q?#8u09DDVgg zbirxo>SZY^u~WJm_iF(=J4%<-qpLbd5# z&TT7IRg~7b#Ft`i(E~7X&t=rn*-c# zbae{QGy@Uz)Nv+fkLfIsspQt{asd#pe5(5ms!ssWA#c9L>c}{!DTwDRH><~WBqk>T zPwusbRSz5oPVQ=*RkhJ3!zlcdrdl4f_h`^{Xbb9jvi_E$Yj@ti zRA`NHr3$@7Ld7t+-_ct&96KFyP-4dQGzC?q>-r+E5~F({NS+mxQdPpJ?$OKmc3Rq0 z0o+opg?8G|`k6vv^dDsC8ZI69tyv=u9S>_VI=|SNDm&;F?3uojs?s%MGMt2j`th%| z0l}@=AVN{R9ayMl1(}*(+EkmC$N&+N&h6N_ot%Wqj9CF-F#gX?r;~w7Cggiweezf=>P*r+j9X&?j2%ijWtlr1K`>ad1mO?8n4;4av1TLA`Buar0FbXkXoz zG3aqaR5(IS3zCxxSr+^?BA|J=S;C+dRwy$G2BWgqOV0>E@RRBA+zH2xs#|kV^lquh z=2Q@O1lIGqJg$MvA4am%J*Shaqs3pDS+*u8x$ZPpMJGm6aLJK7`YPh`@NA(uEJT1y zx7j<(m;$}>1ZoEW@J}a2YY^*(+w4Xixl~~N&N}&ox>CIf zC4aBi0V05E>uMLs6p&sOd4xcBxD8Z6?p$uJKSXYl$1sRaz@vl<9#U5P5Q81BrSo2qM(=?xR9dwD&Y4Dq`g?$(Ink zL0JQFMesIy7lWB4bPh&IwHA~a$f$H~;jUD;ZZfKRwfTP#dW7Rw^7O{Pkc1u<;veGS$kA3RKqGI2Q9n*Tmbd8RX$5^5B)}zO>9hoIKtrG5m1KZ)$8a)q53Y4 znh0(d4}lyluo2~LfVRW2UA++OV?{fVTL;wKuuj?J!BsH#8bG7q?h=q?y&fozb7}wN zv_;Pm)zhX5)e3*EBDmtg%+gZt7Fc67lF2DQ4;uQ;;b_;G?we3x@ZIVcaJ4uKb{<-5 zvojC@v26q9253930Hx!CYJEJ1EBFf(9Vj6(X{aTI5)^79F;_nwFGOPg2b7sr<>@Y* zQ5nr6yj~!TF^XX~=f%`NR$%HvP12gRL4`q$f*G}ig35}KVP0vw%E3!8%^By?zKy*WL3#2UW$=i9|@6M^1-?Yi;0%U zt2n2{xSDmJ;Y|uIf-0g*D9jT|%5H3kGBx>-FBvS>JzDc|E_Fre^k((!$!j@dpl@i| z!yKFf>gK!ZLGG`Xd_Fz|6L@J}#;OtnXJ-h3yhOk@OQV?CCLN%F)U1sb1A-&KLW6C- zZ41@+%Q<~u{fl9ASkBNbQBDXcRa}` z?D;7Su*Tj~>^5R|jCpx?Oy<9+OFz`xvbH3y5aH4=4*ws{3rtskj=phrAJ{Iqx+8h4 z$|?XzGqYhq-~P8C?ta|;+wUnXcC%ja*L%HY!LHw8sr93ylmEv8fZ0uf8deKf%;_ut zB3dI^vG*u9{|l)8^gb0V-Z{Y(3lPX}mf%a7yudQR>;q=Q?v(!rizPuIxliU?Lya~o zP&oFe*ICsF-wJvS+D}5x-?&*{#UhQ_VE@5R$&w^<&ke}|?CM_2lh>wOn|6Fi$)kzB zY_rnB-&_ayDqTrH!iI~gE-R6#wN^f`MqmxWnralmEiI(T<&|JUa*{rAHjM9a)lwr5 zjWv2Z1m?#LYeOCAEeM6?V6~ti7O%0lp_y6{t|^d)&zeqiCZ6}jHbClpaYuXC6xRy}doeLAEHM-@x&&KMcJy$u))>qm2l5uX`t#3-6%YliI zYA(VR68rZqgK{u~vk^+yW2<8Kn6+6O1Skzk+3!#b3k2JQ;0Qq|AabqquhAJ^y#0ikacAwX9bLk52;Ai% z&n&#*q)T}ccRvxSeLn%G_CyxTJl7h=R;92Tt26X5_|#pAPCFFemY~NL3+8d!t@BE( zr5-ZI3J7j@R1gL8Zirs0grgp0zY9us99%b8!{&=EC&xPm=OcPmG5hs`;Uciq8)*Wb z9iwNG2Dd`y7T|~~elp@t-vJ4cbs60rQKyGk&r}Mn61sSV1(+NC0$}K1+OXF`V;BC3 zKEYs!YR*s#&V{+YU(0}>smrX6aJQLDle?zArdMzQt&n~0lIZ|rF$A}ATi zi^`VM&FvOblh!z)aaVC{eb>2Ygmct*@XC(v!uvs4OuEMOAsb*v@jtoq9lW}Qm52(nN|X>ndNqxnCTe;l`J>dcm|CjzawhTu7H-rB4b<&?83S6z8*|xDOE2*H=o_p4fLf zs;293q?;F;3OmdfW>RS7d|4VsZ-}#)3DmG6nJ$@1Ux=!V6d0XS0DvQ_xMbMhOdy{yZN)&`eLPcAZ`7!QOAY4#$(4M~*KJpGN*8b* zx_F_V>H_#*6%Z`Td8VhQFA+|Wwuy8djjXZOpIF7+B0POk14Sc;GlGbaAZ(AKFi6T^|`%(sso9G>Zs+elasUl93o0F;@`Wm!l8;0xuK^ zWrK?j1tvaiN>16?_FMzW`+w~M%4F~-aA@<9LuIv{ zaS1DH(n1*?{7dBDmt+2Y0#?!e`oB)VB5;itoT5FlYf{k|QUF$MYqfuaai?^JoqU64 zJhc122TIU;HsPs%4!TZd{Kr98xRErOOAAA@Q|dU_I3FRIJ-yy{?>>Em9Ezg;@(uLg`D6fz2IAi5f!lETN>-!jz6?HbZql#Dv73ggyN8xVN#I$_M zP;W+NMqP-=cf-W@|6lE$eM}Q)9LKLZF%iXVBeGT9#>5a_giR1ZqZos_idKhCuv9Qn zP(TVYhCrEunIN*3h=O3HHEw{6Py|725okgHg((vhg|@uNpcaGz(((@bUJsmR`)B)S zumAO~*XN#l?(TVhZ{JS|BK3oLsi)y`jA+6F6P>Yof`$pJ&IN@Eut&cmszo+)z#0n# ztHuRP!*dBTU}K@$FlP3?u{GJ*YHSh%BAXi4V~XO9Ef}&S3UlQTL4i(1DN0gc%E|OI}MfpUQSiqedOqVNa z-g1X6z;jQ;$deUvVx+hr(o%fVWj`fJUORlN%pA>mP|YcTbd-w6`zsdbu_dd!k6Xtj zM*X~kjPBp69%q&BTSyQP{6O2sq%q6-&)6!sn$jYHX{U|O5_kP48XS=vq)Py5WJL?Y zlosWlAnl^g8>+u_E14ua_ws{L2(pMx^AQ>mH(AIeRx%Q=s;^n}KSblKGYoBD$%kSY zzA?bf8IfU$Kx^5~fIwDS$jp z-3=xdmgE2pMnd}{qYy``gFCz$E<@~F`x#Vst) zw858h&icurh9Yx5%+jFJk!LZ$#1#uJfB%MPeTa*>s1bjiTrV?T-C7b$K4MNkVR?}em;KagOuP4^=0xh1DNU#lEZ zfo-beFTY^Y=Dx&c$8?BM9(%v%`%b~C^dJo&v?n}fR>qX3&$^2r`o={@)^A1*NZTI2 zb=x+&VQ}xY0kgU!CW>7s)^P`8(gAVK`?|sQ3VDzuN%L1HDhz$q!L*&0V(r}IhHpG}#@P(pdPTJ-yVpC%Dv;xU?HIb8 zeG2oWQY%Y8(dZxC8^7QNR80DZ#ZEB9R3=PLVk|#jmEG!4Y8t%C@B7f)fyl+b9+PU{ z>@==+xE^$q$KVy~y4o=%(hK%q&4GH3!Z{n*J&bbCU3jpJLlYOlRI!5Ai~1%r3Gva1 zIxR!#$UDuqNF!^2KhWQ&O)nmzSx>p@c zX83aXkG$J*KeX5|R_pCWfA6zeZKuSK#uu_nBHYfJJA5q;MZOd(7=xCQR^!qk3dj~! zcH<_8jj6K&);M Date: Tue, 30 Jul 2019 11:37:56 -0700 Subject: [PATCH 06/24] Update HLD --- doc/drop_counters/drop_counters_HLD.md | 60 ++++++++++++++---- .../drop_counters_general_flow.png | Bin 25065 -> 24762 bytes 2 files changed, 49 insertions(+), 11 deletions(-) diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index 9a8d3e02de3..cbf9e510841 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -5,9 +5,41 @@ # Table of Contents * [List of Tables](#list-of-tables) +* [List of Figures](#list-of-figures) +* [Revision](#revision) +* [About this Manual](#about-this-manual) +* [Scope](#scope) +* [Defintions/Abbreviation](#definitions/abbreviation) +* [1 Overview](#1-overview) + - [1.1 Types of Drop Counters](#1.1-types-of-drop-counters) +* [2 Requirements](#2-requirements) + - [2.1 Functional Requirements](#2.1-functional-requirements) + - [2.2 Configuration and Management Requirements](#2.2-configuration-and-management-requirements) + - [2.3 Scalability Requirements](#2.3-scalability-requirements) +* [3 Design](#3-design) + - [3.1 Counters DB](#3.1-counters-db) + - [3.2 Config DB](#3.2-config-db) + - [3.2.1 DEBUG_COUNTER Table](#3.2.1-debug_counter-table) + - [3.2.2 PACKET_DROP_COUNTER Table](#3.2.2-packet_drop_counter-table) + - [3.3 State DB](#3.3-state-db) + - [3.4 SWSS](#3.4-swss) + - [3.5 CLI](#3.5-CLI) + - [3.5.1 CLI show](#3.5.1-cli-show) + - [3.5.2 CLI clear](#3.5.2-cli-clear) + - [3.5.3 CLI configuration](#3.5.3-cli-configuration) +* [4 Flows](#4-flows) + - [4.1 General Flow](#4.1-general-flow) +* [5 Unit Tests](#5-unit-tests) +* [6 Open Questions](#6-open-questions) + + # List of Tables -Placeholder +* [Table 1: Abbreviations](#definitions/abbreviation) +* [Table 2: Types of Drop Counters](#1.1-types-of-drop-counters) + +# List of Figures +* [Figure 1: General Flow](#4.1-general-flow) # Revision | Rev | Date | Author | Change Description | @@ -21,7 +53,10 @@ This document provides an overview of the implementation of configurable packet This document describes the high level design of the configurable drop counter feature. # Definitions/Abbreviation -Placeholder +| Abbreviation | Description | +|--------------|-----------------| +| RX | Receive/ingress | +| TX | Transmit/egress | # 1 Overview The goal of this feature is to provide better packet drop visibility in SONiC by providing a mechanism to count and classify packet drops that occur due to different reasons. Because different types of packet drops are important to track in different use cases, it is also key for this feature to be easily configurable. @@ -101,15 +136,15 @@ Example: "counter": "DEBUG_0", "type": "ingress", "reasons": [ - SAI_PORT_IN_DROP_REASON_SMAC_EQUALS_DMAC, - SAI_PORT_IN_DROP_REASON_INGRESS_VLAN_FILTER + SMAC_EQUALS_DMAC, + INGRESS_VLAN_FILTER ] }, "LEGAL_TX_DROPS": { "counter": "DEBUG_1", "type": "egress", "reasons": [ - SAI_PORT_IN_DROP_REASON_EGRESS_VLAN_FILTER + EGRESS_VLAN_FILTER ] } } @@ -120,6 +155,7 @@ Example: State DB will store information about: * Whether drop counters are available on this device * How many drop counters are available on this device +* What drop reasons are supported by this device ## 3.4 SWSS Portorch should be extended to support a variable number of SAI_PORT_STAT_IN/OUT_DROP_REASON counters. @@ -206,23 +242,25 @@ Deleting EXAMPLE... DONE! ``` -# 4 Flows (WIP) +# 4 Flows ## 4.1 General Flow ![alt text](./drop_counters_general_flow.png) The overall workflow is shown above in figure 1. (1) Users configure drop counters using the CLI. Configurations are stored in the PACKET_DROP_COUNTER Config DB table. -(2&3&4) The Debug Counts orchagent subscribes to the Config DB table. Once the configuration changes, the orchagent uses the debug SAI API to configure the drop counters. It also publishes counter configs to the Flex Counter DB. +(2) The debug counts orchagent subscribes to the Config DB table. Once the configuration changes, the orchagent uses the debug SAI API to configure the drop counters. + +(3) The debug counts orchagent publishes counter configurations to Flex Counter DB. -(5&6&7) Syncd subscribes to Flex Counter DB and periodically queries the ASIC counters. Those values are written to the Counters DB. +(4) Syncd subscribes to Flex Counter DB and sets up flex counters. Flex counters periodically query ASIC counters and publishes data to Counters DB. -(8) CLI queries the Counters DB to satisfy user requests. +(5) CLI uses counters DB to satisfy CLI requests. -**TODO:** add state DB to flow diagram, simplify some of the connections. +(6) (not shown) CLI uses State DB to display hardware capabilities (e.g. how many counters are available, supported drop reasons, etc.) # 5 Unit Tests A separate test plan will be uploaded and reviewed by the community. # 6 Open Questions -None at this time. \ No newline at end of file +* There's still some debate in the SAI proposal over whether counter indices will be managed by the SAI or the user. This doc is currently written as if the user is managing the indices, but if this changes then the design (specifically, the Config DB schema) will need to be revised. \ No newline at end of file diff --git a/doc/drop_counters/drop_counters_general_flow.png b/doc/drop_counters/drop_counters_general_flow.png index c72427d85ee8dda0de4fcc9d4d88dcc302194f5b..c6620803601018ac823d25072790e361468a34c1 100644 GIT binary patch literal 24762 zcmdqJXIN8t_cpo#(Gf*P1wo38ItmB~3Ic)@m7*XW0qKhL-jSB5qu2mZI#QG-U3w?6 z017C*caYwDPtMv2&dl>Z@AW_J!?~{WVRTG(es$G*-D^KpQIeyjVWvS4g!cM1X*C3) zhX19k|BVvmw@t&wCAK|UO5RKKc$+>xny8E$pIu*3 zoTY-zP;=s8y&4ZF_rpI<{r~zO^r;Bn+C)>#+A8s?bo@C7udn{}2*Q7=;tbxwe^X3f zS6Z6+aFy@WHMm~&cgXW=7vCXN2)`oDhbKWNY4{PODgOWC4{P*B6jI&n#>HGLoG)!t zG5Odz*WtrW`X&@~F|*1(9wSr34O#P^ge|kYq~`PUn&YMDh2D5g$>S7~WOylPyT1=Z z9Q*b>FjsFfm*zvW9K~+LD1zQ7JknXRCB|n=x!nGPk9S+a#s-}qC%%!1nhC*Ls9{YG zyehWZ`hwfurYcfgR>dYr?3_jo>!11raUm~^4_hnO+s`Gq1vP*NOs(y+EyPDn|3+2Cv%| z2FIpfu$fFSnapM}X>nZnq%HawLGZg^>C>F}^Lbz9kJH%On*6z8FN%fFzrg1Y-RVBu zIBxNqv+*8-5}&!wd64$gZxL9s=#8zy@07f4!5Y}B8B~0_ZH&a1nydS*JH1a3#N)jY zBsLHZ?4{(!tS6tJ0q>a^DxT>iP3+ZlDn9S2Wp@!+s>5W~Owz`zLHTr`*{ani1QAR? zYP$dQdr*F!P{Uy;qKzZKMGfVy{l%#WPvI9W!)V`P8mZb~Sl# zRZNz(4-bM^qIa-0Bb;Ae>cqME5l196yk6EE4o(<+1Yk`Si9d9?Anb;-Q>cb552X# zN?d4daOwyk>NNBS*q_!X_%+kmD|sw_pw!C~Jksy_1^1HII|)wV3ohJbazn^kqBBK+ z3aOu`H#*2Am#B%IlsOE?Xs_xv`|5(H;B5p;qlT4ob=IKNeKNhE*JpEjv@WJj zQ*q=QM@;E>Vx)(MM|Ks0$dI(p@g$ACqDCJGF8u|OYJ0%}?>%zav@Jr9gd#;3KRw!) z^Usr4_t}3Rc)Jfy_g1!P-Mg9IyvaSQ2co|TY1Fm1w?ClXXIs8La+pIY{z7uM?5Q8$ z^iRx8OhgFV_4>1kp%XrOi)!|HG3vsRwYdb^$N<^H28E8B6VdnKRI+J<9s2S2b#03h z2s*au2xjAR`C0{^si)pzr&VT!L{CpohcSZvq(VB7VM;v#vV_1nzS}FgFhx)-QGVg? zXVxXEGqg4;ZE7mF+cFap`y1)b zo~dK4WUDHLrzHX-=CWP9SqhG^96S0wKUmL$%HEdA=?osXVuf=hUNBs**7DgJVi#r` zee+n^lZ78q5hR&BY99P@{!)}qkH81};gy7dvhv%D4MU$pvuTHePG?u^PbpH1w6;YG z+ZhZ6_a-XuLBgEKzvIJSDBDhI%V5`Q?kTzZJ=*+}ukYErEb<9AAF{D*wiwMCZVfVW zYo=Xe6IpdN1iQd_nj|gR^!*WK56#}~?c1FDu_I587X}0$7i|}8j;v^4DX@1)QpcrF zk#W>(w7YnePWrI_mn)ZQh3K_6X@#c4Jqtd7C!+_8cVhV3J+f3NZmgQ6}4HFRrPJKD2{30^1o*Yz9=g1CJZv-)*^B1kYM~S2}zu~}zf0>go+YEYR{z;*As^i1@7@SMTW0p7v z`la~BH8Wu;%3r%SH<|4Z2R&tUS(QCJb8q-7C~(K)qroE&$}`*+mEMB+sb!`P8&t#I zR@*$_pXN!xm(k7Ym0Gqn@f~CK8r93EK(M>XtJ{3HA$!=Lnbkz(2x^;TNuPeEXFXOt zxAffi-Cn+l8GW!`FrhUSA~no~p4h*9%!QPzckD*6DQZ}*v(!wc z(w3iKr@VSa-Y#`gf&<`ItyvJMVA6^27qB8<822&E;>>W@v3st>#2yZqIaZBysE=8G zPCd#Z>vX_rR!k*&DHMqFS<-d}WLel)&>Mz}hBJkW|9mp8LkPk8$06!^H2k#@&3}Oe z@xa_2#pULos(krOx)4P45hZ%S{2ZW@&rX-VFLC=8Ai z53sC#KAY={qd7NU-&^>`VOKn!7#$rwb*{d)_oIMS8(re*EjX0}boyU~5G3d|44Zcg zn^bSAZe~zue|v=~tiE4tv{f9_-|Opk)*x^4DdU04&*;G=!}hhrsNJottv^1X7Vh&} ze0rE8`zHGnMbPFaw%xg7t}ZU$!6H=ovvnB9(yDsLA#$|-=UIc@`#uJuBOE1YwKk{k*26GeaFfDkIExotehry&$UV7wqfXdIG&am^ty1C`2RL|XTO4YDE`T*FTq7JrcZM6H^NVD~9 z$!xiK{`h&jCe*OvgJcf(7B07gecY#I8*sW5yy(tVpM#aQ#Z!y8^i&EXbo=+mKiW%U zKQ=L8S?aO>M$_C^bHRasLIAnL5}vntz~AyGfdE)FQoiX$E`8)5dAiS@nQPhxZ~stw z);rma^TNQIK*)vLh>*Nm2@XJCe~|Zmr)3vKv5WSPhcZ?G6%u!Gc1`IE=xFGtT@I-11pCwrtaY z)BBOy10Z*f)J!QxpaABdRwYWEz#f8G9Gk0u1h;~}p-vL|GbVFUPLt7+FC*-EzMvP5 z->-`0%DBGo!>!o2CuNqsH!7~GK8)Khceoc^K8Pg0IC)xI7xw;$r~5u^B3D1M9V@P1 ztPcG9-c%0nMiNmwcbXS(Y;3&m2v69K-@Vzk^}TQF`*xexN=}U5vrQ{-NaF)3H&e64 zbtD-%W5Lxx98$cL)dmqyd9j(1LSHhvsAUz>qU^7I4x7vzOf+TTW@(`diFRHSRFMKr z0!OEtpTQR~_b)pNhj<>A+M+}ey&$bQ{eZxI-?nVq%jIkuEB(KBs)$q`D%z3MOtHbegr>i5P+4$*98^i2 zBFjN-5ID(yspmK&oy8|aMUiCms03``5|j7J-8p{Dlm)^xI4rPdg$|(Z?r?w4wWh&% zxNi4@DFAyH>$EfG1M^Q{OF&@I=*j#O$>AYVRdE0C>o7&IK@b4uXV`%U_(!>-u8{nf zk<$Kv7Z7vuM9lTn6o~&T;z6+I$)XjeYsUAYtXZsiTLUAADuJ;gNH*tj>aaWG(~@P@ z`iAoK{hirv!xT=O9uyY1*qAM44+F2nUs+|gt0QEZH& zPp-bXW}^P+Dyvi=Ll%PDuFm1Ztos*)*434nKDpN7_!H-~)xw8a7$we`yE>2kcEI7p ztHwZI5$pqQJ&I!&MVXQwbFe#$OG{^46DHWIRtd(gFE}syiX8lw@cMuQQlV8yx%GND z#^JWK^n{aj1&7bpFvewl%d_M5yIR&k=|4l2M88nb8)nhW1=>iq7cZ@J?|b;<>r4W! zXcm+H&7VGD_;-#IicLphMm0`N3c@IEyQwYIhi5hAj}zxVObHikbzmUT|6d*x5>b(k z7-@f|TsoH!U?%sh(ecUQz{K$vuuU$&2{6R+fCv^@c*Jwn>rV&>L$1Dgk}JQ5br6w% zt4%H9Pbg}mIv5yqKt#+LeyFL6r7-nHW6me=2OBFjlaVVY{)C!8{L%RF|NfV1L2o=~ z-WnQ|yD#^BTnpaEM|4TMxY&*p+wPl5>yt}ig>YF^goYnKrGibk_rq&(F>X`H#7|Ik z{Oh+GyJi1N%L(1K0>P`T-UAb(%((L_D;YHpCChRr*K_*DpVe*FWHq?4Zx<9rb=8hb z5|{YuqLd6V--8D}hk6j-W@!oct^e#AHC(Idjy|si`FT|BLRMjDlDZs+(#DHXOm1TSX@>s!rr8_oC+uC%hsrPGte@&#%qMpC4QVlk zb9CM_=bzRk-u%WYOA&HiV6!SCf1y69(toL9z~txbv9ET;AOAInMDo4I-9-(a%7oi* zKKI1F{@Q@?IVUkx8*QBMx@ha^?A4g!!r|~G#x$a{qMO8O_tmT}XZgZfBgmvhoE)bt zxt;W}Y`YZ>lJ8>S?OiRzbep(omWdcD6yMOm$hSFjOXd!^s7HAj6Gs8|2r{ z1rki9>;nCpV0))^!L!FU-i?wSw~BK9Hvq1twm`RMPV1UJ5< z@&v=}I}%J5tfwxn&UAMsJ$aO?>qRmCB*=4c_`Z!F#?ANRAsLF@VQe%Xyg0}8c$M-U zCt1|FGsb)FDE6HR32rZNtBBF6tFyF|Of5Nde!-1+u{8kaDX3XxVa^j!zHa%Z(xfh3 zj3PZMBcZ2Jz0uc3RIcXBs7(>hqA9AWFUz7Nq$7Zm=kQ(A0R6U`SPqFVGKzl7T5fCo z0tSTj5uLsts)u5O5~bupJAEZ{&;w3QHYm#;Nhufl>7O`=#7#Un3Glq z4u4F%K5F0)t%2$Iv+QOX;U`yAaYePry_Bf%3*u4oqqh~SoqBA?pW;M6Mp5C1=Gb%o&%u^Y#{ahG$czas? zr>m9Ijl9oBPXU>Os78CDapn45V?&$b@^9n%&x64ur*3+!{lynmmbYP<_9d1I-HxUlZGeB!n2Rkr37wmVovQ^>LpZTpISMPqw)hULCr+sq5ACKOgakezwXFs#w<$+U2(9^mtM@*BZ zSW-@Hw-&e4A!_Z@Uu%z*nC2kD5ZU=RD?e~?8A+$B8w&o9!J?>@9N#q+-y0E~s6Eo3 zd)hTMW>@O?z1!8h%zP}=w+-L6{1|_oFX|oC@nYw$bF{FL-shGrHn=97MzSz&7PG|i zjk?qhU&p^SyvFyHlJlV1xq0(~%5skj?0WU@)9qHLK15M&ef%~!{4pjmlRw01b>BCW z2LiZCClfQD3RIGQ%CeR?`X*NfuYYdN6TAqgT>KpuMwc|6WPU?5@q!QMKN9`(^^b3Z zdefh3F#0$`t+1q|tFs}qN$A;tFY78pH&WCuidn}YW_6-YrP~4@5;>gGGeQI<7dMS$ zTd&tim>hYB)O-p5XsjjrM>G4W!H0VK0#3e#*HYhBjJ8T@Z!YL~KkMw{Oc<{U5ZNEX zxnn4g(|Ml}6D}U9o*THf5<6qfCD!&L=bbZS^^#^TU5wIqBMSd7-|nOpZQPRh-LQSQ zz+XeX#>Adu-W(lFpz)I__EGlzI0*17s@#Hg zE#o)+r|W$9+mxc2Q%8zMkICD8Da)@4u4FUKN0Ph;FQvJfhIp?$ZYm7w3^yzjF}y_l zWT<2xq0L8VJsgl7f?nHC=E*6+p9xab#_lu0HrL#|G%)|Ftju?=bXhmHUjo9cHg9`}4_@ zH(qo3vwf=#Q`b~=AR1{DjBg;@?3#;vdvX;B{xgb;@KZK3Y&e>s1;t+{V8 zye{{or5DCM``g?wn*li|Wh^d!d3vqMx$^jdSo&u-v3%uVr4xu75JLPkV3LlnzPR3R z6Is5XYj&VKoQ+Gq=D3nHbJMw^%*u_)J`xm(NVX73q|hS8F7DiOYTN%QZ1mo+EXlv(5F#m2`&Z9>)0C z48JQlwk7E_nee%>1Li!zZBB83u}?f}vkQYpVe7r(?17f172EoNg+-MvEQ`5a+Sv1% zswC&+l`4lPud9d{w+hv>3+$nUy064-{61V$-l%C*6sp(<5t`IOYNkOReqG8j zX-E?+6AD5g_P!fQfelZDjLG3d7%zqMd&s#ayX@{l%nPW8Q6if4y!p*k0b#R*92ns1 zR>ppa^dSK{cC7g2GzC)2O2SR{OI<>1ahd!BoZ#8!sRzzV6a_32GAfgGcBTyCV@2kQYWv*Q5aQ|fS=vw zCeGHEIy2ukAolv?Pr{lZi?8FBJCfPS@RS;Tl#-mN<=O!zg5qD0+z!qQ!?MoJi#?-= zg?bF&a=Qx+irzLfJj0M0ayer`_Z>8|2JUNX7LZU!(vw&m>yC8 zT3O;=lcMnQ($d`kit0EY^Cc)>IclVOw!@+fnGP!Z^YPr9I`|o)a0WDw5md%1MVw_#ua>*Cq0LuT!kkuZ%_?YfSoDR^`57cWF0gB0%_;Ag?V6wbKPS+->eg{t=}5 z1}v(R51-tvBual#|CDWBSNy~9rY5KDqVLMIDr(DD<6?Z~UxF+IMW7&nA=W7_R5v^p zNm4v)_|+y)c?QY=Y~SFRMB$3j@4J#SkJCEZcD`q^{S%OE#*o?-+Uc;1j3s8XT&Z!-|ku5 zbHru!ER?$d#C*qO#)8DU!-=JCQ@+`KTv_EZ9lL}Vz>Jj=ihaplKO)^XH#eK2UNUp% zQttnK!dG^6CFVyfrBom)WH>Amn^R}f6%V30M21VdJ4{99|NXfb_ zq+eun;d+ZpOIu=S1bi!tiWdqzxA;)lUf8!3KP`O2W3$i0;)8R>WCS&{5hdUg?0~ug z);^>RLxt(?JS_U6C~BoEEXC>I$FgV@5A2I_ zsr?)o|EfI!3$er@!es%f6A1%Nv}k!J(ff3vyRPR3WBJ@85GqNRaTQP7#)%{Hh#*6Y zFpWJkGh1*w+?^>2KZW96O7?Q-Z7bS*)#9IG4sL$wndKW3W|Es7Qx}DMoJXV39L1?) zgf6%E@qYZ5{eHWRR^T$*Tp0SfYNdmVCj~h&ayP6J zz;Y=+keeGZAU9x&enL45*DRI`7$mvYM(ek`zZjN*KFAGN5n2g^f*%T*05C%fC=N)_ z1(Fq~2x|uDDerAVq^q*`9So8p1=GU@l=t zfYZ+Ur1aH~MR0Ws^eN8XS9(XXbx2dtW7$mit?wl8^FX=N*)Elv>gGAg0(^Rf{GR(& zOhH5%yD%PrsRwc5^MDaSvT&Ou+UGiO!pQ?rrWbl z!WcR*#(7bp^7kdeDR$m>6w~TfphEtBp?-{N&w*~o8s7Ab;qsMEtK{v+dK5~L#U&+B zq)bRa)I|VX`_9m>{n~~K8~&1onJrzUtG?kh6H%{3#0zYo;EVu@U_;^R#=y_BlqFmH z@P4mBR1NaD5A9>SBq(~bTk~YNRZn4RYHAjuyrqXd1kl{O4H5#yzYj&Jz`lWhgOY?3 zz{pXnn9T7rwoSrFn_<6=4)$v5GBOMYwS!slM-IwxblS-s@KdrwY(Z&!{8Ke$g!NCX z2pPpr(Hqg6(ck5|&WOEu?$Y_r+gIb7q&58njD}4sEpK9V#o1!@;1SeWg9HKs#$NzB z5AvYV%b1u*n?GN9jfTIz1A-(*ujO&&txe+bSB_nygfj)QDQ`q1?#j_%Z-qXVb(3s? zYBf|>j{3?bovSU{-Q|usNzF87P2ywdMs8e_j+Y>lG@Q(!-U8wP;AQdZKopdDFG_fE zb*Z@f(dLimYNNA}YM$WERlqT7WxQ?hqZLXIm&6dZU1kc0{I&0a0#-V1D+a(L0z1%* z)VW|2bLSzRdJC1MU9f{lYUZ*1J7f=)yJYLFLg6c`Y|q~Bk^We!K!;A{?62=>EX>|qGxWdRhO1J-?b^eJ6>ujkzRFZVL1ffu ze^A5R*D)~EH0K39eih&9DSqv$85~&H;@bv`KNsTxlx}WmG1(HRKl`s=Vc$@xzoZ?P~({Zn0|580X8IUew3z>|%j*4bEPfJt^6 zgX$GPrNY=>Pvk=xe8k>dP)g?4*)hB+wc%;U6A)%#Y=#h}6<;GLJqr_n-0%8`3 zrGZ^GmXKVDq7F&nY8)D%!D(|Ch%jaT0N6SMz&2bp)B&q~4$a~)OFt{Ft@LlNV#w@> z=q{p%dL!i)nP)N;JZ~tm_OrbEboh5jQ;w(D(HnuwNCXgyCwW`;yXV5Ahs%@f+cQ$d ze{99ERZX@j9|z0^Rw`8nI6M?&FrZgpiE@=GLQ>buB-h8ob3%vY=B8)azuG(l<_voz z6xwIk7h{Hn^tAv^QUaJv7UZW*9D|3WdCG-%yfD5nn>FxUvFc*dZYHtj35S$y$xeq$ zM|ve(CBzFf1JLQG^J#(PQMu1)<3k=LE4w{l!MUHGy765&QUk^CGGK=2-#akx*juQ) zy(d}A@PpR&HZGozPx@iH_bIMCoQ%h)Qz!50GVoZ=r)Rb;;r9NNMe{Bx8}lXLY=A8{ zmw^pjzhL*^@{i=>ZN(-j{=}_T&*V7So^*Y7CH6C&rq(?T7}4pM%*Gy?8iQ*Bu20hK z*u+C!GiKk=No4Vfa~ss?POF#Pjas?xE#_To=QEq0t8Am zF<`Mt_;ILrDxXa-q^A4p{ox!T9cWBzP$VWX#X_uX;dsB`oZ>$Fv~T)?N)Hp1OFff) zJ~%snZDz8ndlsPXsX22}Zt&sMQ4w{+^DL{^tt|b@9D*D-^p*_`re0Uyn-N`!H&h^jEd>VVp{5+ko6yu?i2TrsCp_OW#mL_K{Vm8#6V>sKc zsXL^5Pb3;%pBc(sU*^P9!wCG;sLuio0iR_7QVytUI^~;^vz_hNPQE@;kyt1HFq~!I z97NFs%g?%E8_II7GnfKGj@;?9_&l5{bFQAkhDcsr%Sc-s!U!Qb-We3KH+NGy|48f| zdZVHU=Q8I9tH`KpSgfcL|1IauM^23$>8FoT>T}~2x$vj_UJdAY8*cW14>uD^usUeIz4;b8$X5XIB)!x zXI1)^@U+3sYV!5k11Di+`Sc6%w;L^Xr>RV4O*8g)hSIZ-rpJv64xrM&zYnaPUAtE^ z%<`ENzwkC~V-xfALFo3SUJPcftU)jJhQ&?n$%gy5&51xAF~0y6mN(r#16~upd#u=x zPfR{ko_g**5ffHD4XP(|Qt`S=8KKePXT$7)=C-vp71TJXc6Ack4&S>ssj^3hagK6*P?VO?j6#cspokE#!X>kMjcO4U4yee| zdw&TuX*WRtlZlaU% z{Q5*1QZ!G4i`niiXJL>lC|-jOQf*F7ZJX-HAb$YMGxE#Vs2G5l1G#~TnYNbU*ChNeKRR=rHrAhJF1I0rO=7em;SAbzLc*-7v z%}3k?Rr_v3L0$W3b<4iF!5ITr$g^I{ni!HNUB70(S-R#qP2(Y9awE*$e8ne{ktLHD5wk=htZV$o=H zO}=+*YN{3RB*a{&z8<5pkAg_ziPk#gKl6@u=7RhtXyTND2)h8MGN>2g6=@2<5Pr|} zSA9BkQd-=9Si1wN`R%u%sN8Y5e}hfL{*Pz2+j-s7T?I)Ko)A^_Vu;&rJuX@)A5k9M zv9?kOfBm>A!XCJ%V}5pQTbMgTCSuJi<++n_b3;*7dKN$lbLV~(XfY9yu#V6_&}P>)j0m3qli_`up#9|JaX%utpm4blqjA}A}jA2+bd z;ZQjRp>;-wkUVwa{2ey3xfeUl9`>pH+;<*>cSSC>4J1} zrPYsr)9{jK#2=b5(cGLhhc8Mq3qU(aJ7Oc zfD;c`nCwxdCfkbVwmenOZPNnSKt{erOMOdtn#Me9>O4PKVNw&)16KiJ>(4gwJ#uv{ zF2J;FBG*F{5&i^06-w)PEGu_vC8k*lO_-?p@agdu1%r7)H1@^b^H8@h@^;u`-eS}g zJkYK|#sVltz5puD*+k1ICWCUi0K-OAx0Jh15GE#N!*NbvE!!QEYb zeK9kCHLq#jhML=f7RX*EX8aiMG~JUjL17DDhW^Bfq#pSKK7YoMwdYk={?sh?c5uk; zFw!lf)u!0ZNqIHy?I4OW(7*w1SQJB>e^Q#&6pvhuj>T$*j3VMtiXQJ!TQiEBB zNI7kHA}~zeEww6_@cBH=qzp4J~5Wt14NT_^us4$ z}@*JQ492-i?Qv- zy%VWR(ssyh&M5Z*&%s9z0Yng>^%L6uGboDmJDJ}TgnP=}0sVQmAGUMGb8}GeRnh!0 z-0O*niSvY`vCV@m_AYf&%*HS0wIZkoEi1U^cJK z!AjNo*Yc!XB$f)&g(G|zNSc66`B?VqI0-D!69v_9-W)Xl1b>UjJQ*JQmX%FqYxw4$ zEJej0vIWTdAh?s4Mv;L2A1@PoQgGf5Z_a+9Yny&fF^0f+z%U2!p}V6oSC?hd){xbV zZhWPJB%HfrdB~B+v(OKMg2gBMcI}EHLBIW$&l)2WQF%1THdWqIxV9s1=a5G(UTbQl z8UTqv)(-ye8EPKrjVwWWEla%}(q4QyU~9Nv>hhOVHRc_cufj?a27zQy{EDNp5w6^B zaCWFjQStj)UX2^d1DubOl7xVSM~HMiY1EaR*4!q^YjCmmImGD)eE%Ey`&$kR2NMc> zI@3F+pQCO&ZK@x}QTcuTN=~ua9|qXQdUGPfz8n;Oq&G5^11+NOCrkej>#0^ZJRAyV z5Qsdu@x-?cBOkE0$izY?PPkkE2UG#kCh1`c>ts(k6V^hM45ovo^Q_fr7B+wq0y7i~ z>LhF}!AxB!MBa!Z&@kg6cN@H8F4zt91CW086RI9f)6XH(`J>uC!nFpbE^vQ2F)QccjGooYZhdAO^Mu0yYs%an*MRkAJKJ zr%X0)6?SqwU&2+=ZlEURoD>64mf&z;gr<7)F!IIRri9Jp>ZXGLCiXbW(PI74DNzC! z7v8??-Ha@1`tsnor+)tyjYx&V765dIO!mMmKNsohZYS;s;OWOROHkawRdk7D8RSfc zcpS*oVD z=dr9h>Lr0vECc=rL=13T%&$kg%^^B~3oitAD;R7RW<>@vs>lF>l+sf!qV!|%Lq8ml z@PoGv8h?>yl!_zEK5BVc&obK?T6gkdCxcm-r!h?9mq=4lO{d1Va`%+40aAcx zzbt?l?N9RSP$ngM>ZuRH>+xj%SMP;G(r2V7y$^iLR z=x@R80c7+JX;Ur%$>}Dc|pxxCV>i{z0E}3cCPS&VL2W z-m}-AHYiygxR}N+sd^8^GFBA;lEcKT7>K&+H>XV%>Yw`mZwP4P&07tj?CEwmu2sx> zR?_UIhOKh)m2ym<+)x3*Z+5E!NLq!C>uRndW54Yy-&o=xY!iQ9*TNB-iCI7Q5H}6X4r~o>Tcxbs>g{FsY>5gXFQm!%*vj9 zQicKztGDv+D%4-@^%?F6#d%eA)r^iRJx@?yEHaVY_-yC3+9A2Vp-V@IDN(VL6U*qb(ZiZz zQZV5KLvB;lbp(EQXl1`A}lZ@ZzkA@6$%Kjo~J}k+}kCWe5GO zH$S!}6pa4xmXUVl+y3rNv{H19=j4S9Nk@j_asC%Oh5TJ?12M!EWkbV_tnR0oPrl-_ z`??p*h-kY4nq8&8L$2x$7M#+3;BT~*^F;7F%;-L#z+MO3w*;ucr*h)!L)L7*g~Xop zLfLHu`G2w7{`2?0a}*Y-8ypOzhv`w8S?(zx-Rx6izmfMwCb)GT!tr(-a^6K=;gP@t z{(o=!0eLi1+5gwlFOyG^RO|n>Oths1Rz+gS?HsS13fAn*Z2IOWK0ZTMu|kdlgZNTN zDkDaAP{gXod}krVD7Y?9PtElA5(>Y+AuK8r7rMu%N;F49jpJq|HtdXjJ9zMB*2j3V zM$Gq}Wn#<-M9+O&y}Cu-BY)D^n@R6nhv3`)IYX?4DE51^=*`^kWBR!2dOQzf|Gvob zC?D73-n14)niZp`>AWqcuJvsWX~`01?r$ZAUt8H`Cv)|P*{2+b+X+jI0l`3%n7rb z_c5NWJ@z?G1>1FFLgR2yEx-z9xTatq`=aXA-q#qW?VO=~_JmC$&_`_Vo%X&u5oyyD z1$7CPDIuSYs7mh2#xpb@YHy>>d0V?S50q-@E&^`b=k6&@5fc7O-jnmPva+c69qIO3 z{LZ2D=Bs8IkudPtH}tH&M-6X*^tl6zqji{>qS0&{b|BLK-F{^c;kZrWZ`2|sA8ysw zggL2UGf(Ls%CyT@bcDw zy2gB$XKyJgfAg+&_A9UmREq2@eSB|uv32)eL(e_dkaH?Zs&2fRkd)DJvS?cz_p2I@ z^NYD?VCXDIT(8J2H}F=7>5u$9+54IHwUP?PmzxsGbKA<1gSYbP-XnDy@)}sQkptR@ zAd8!t-$UQOV2ZXy9^N9gMRJXA*blXCEGH*cE`(~%W^8wf%+Dn3iRsoN1_jCC#8fuz zf(BO68n`BRK|>`9SgomKjaRSn8TMYx-RiRA#IyarqX%kMenT;F9ldUS-gc$)vL0b) z4^~MOrG7zkW{8@)O@Wq_vpVTBXWHC8FcDYm5uRc^ed{nLB-e?)qI;vVxtHUOgk6&x z-aw1OKfr1EIXiA4Rk*KJf6C8CceBgHu4Egf1V%4K&b?JW7}+b{xyDwQF?$1;grN;f zm?QBwt8ZePj@o4G=Ee^zLMzhz;TNCQf9~7k6nR21^JC&<<$REaLz4uGIM17(bf$?J1?+$}elBb8XnQF|c<0jDu|f$f-Amy7)4`z(^I1V|BYw4O8QfLawLXw&-p(}Et6!7+K zuQ)GRTHC4OtYn`T&J(qH1++<&Z(5t9yVhCYw8wF%o0sH3N|ct<*F*J zbaA;JZV^!m6lVeRAX|9e_|(E8g%ZDrBmMO)EiEE~%(kqkd5Ex7N_S2;qSd!IGQXp; z$C4_tj%e?)y{9&5nzB1ZX0+8%i{i)4rqkTI^!={od&h65I!2xkrtJQk*7zFb>MV0S zPpAQyi1>397p}`SEKlE15Mc6oP>nXj#xgB?T+Jr^kZ_rNcj5WfG`$G_FSZ+j3C_P< zTHecu>G;oKOl~zj z^Ic%`A*-RkGVOj_@^w?_^ZT~X-m~v7pgo3i(eDr?B^0KF#)*Gi;~g%e^jcl80RV#k z()kInpz)Ks-g(S=lkdhQuIE|h6!`{fs86yu{yD8VnNg(LRFvD9g1E{4b9AO+<2^4L zgL^CAb&WLL2AIsobdd50fVh#>4Ek%S1hj%;w^qu74w(Y|THcFj=C zL&{nnb!=Dd7iylyQbf4qP-LC2?%=LA4)upo@whxfsgo?J#{JbN| z8?-1)L%;-%6z{WtUVsxyY*pYWlL(inh-c4PAKi{MZEH;m2-$S@<7Ocm1iJox)O*W` zenh@lvYqkVR31&SdxyBGG4HdV-k4_J8g6--?C6&pMagP0DvmD#Sp9F`)%@uTt&n0M zgIKOdXy(pF>1pg!uXx&SKRn4^TlXgQ?&|}NN@CJ=Y2PKH97iH!*u{$ZyxW)V4H>se zDE#@IvZwvfB)65<%Tz0gQr!~jLYX!jVkG|~O0OM8seeEWckg$MHhZmv_&mp}jiC2P z!8;^Wuz{bBPJeA=wKpXlHRX!>f?8rtg=29XnS@cJrsNdp5*k1GQ(_95cQRwBi+!v% zKKLmIHvS5Nnm_HL73!RYjQ0K%4re&5B3Bj-J+D>C=ZYfw7p6Sw- zU%WS_erveo7vU3(a-?ZY^$b<_;DU7;u2`2u*xcep37f&~zX+SS=0E%Xn&iUIr>)U6 z^@&Eml297zOI@)^i}B)~mh>5MPmPHcgxX<|eEQwTV4c-*wCS=Zl|NWO(^uuiq=0&N z_Wq6ZyiFv=eYLTx>IXO8Bw!~rN`9o7BURAk^YB(|xC!cpbSV5$qW`>GfN9^%Jhv*P zTr|PaMm?=Y{8Y@ObQn?7m45H$xJ>E$-h%$dc3x~{^{Eti&(pA(k~ zZoOv3IVP{Nlh~f$^S3EklYh90aOwUaKu9^XeFaJ)uK!nYM`F|D~>M z<@wTehW>21ogk?BO4BA%(BboN6A^%Z(t{R&>B|Q`?fb$F@xqQ@K0-#S`R#g3*~DD$ zrb@ie+o9N)isiYV?N-?dD=U<3j-`1<^i*BD``~ZuuR!YYYNfG{+xXa+{#g05UmQ7V zRNIQ@N^=}~Bhv{rA5yfPoX4enH#mBX7GI0D6GPk_@_?g^vA|)2JJ9u4IAqD|clSpb zZ6TY!zJ8_T-)sIS%uU#P!T&+D1~YML_qE#d{}tU@TmkOPq=)b;6hG5f#mID{a=?Pv-=T;423&#(%uu!$j_;LaT6 zU7^@&Tm&zr(7=N6t%`=$LxAa9Q)Z#Y2O(RCHxK6-5^_U%ThH|uTi9R z1=>v37SOVc)q#=O7!9p)CUw@QT@~u|=hq?|d~@QL}zrlx(m*lx6aS zs%I2vgv?_}eYL8vr0SAzJly^Fj(O`TMC5EQ!ms)DQ zk%0DcT^>2$di)_IBj(pbDk5BrgKDGWK&T@blS1BSzYlHDw^CP5hpNZ>Jd25Bz>kwc zB86%Ti2= zeioc%v7xR^p-u)95fx52sf3w`j8+NYVv?42jYke<1>og*k8ORkIzEUDjUctk=r=X12WnWspA9a?MMBHXUYcc1p(qbaU=_FDo!~OMLmorQ1#n1Nx9^JLE+9@rUj2Yjsnw*% zcvzwZHvx*yovk~E!Rf&QlZO*^bS0NH{t4Ec|yX=&ERDN6ceKrxS}|<#!C{YhrG-o$;?J_ z(uw(JM{j}j^1y~fq08FyG^nynB0{Bd8P)=?ERjyP5nXOZOTf`^Vqq`H130GuD@4uB z?-Z-7ce6^3h6NMn|4mr%E2KoYaU1p}016D1%!4PJZltA^Z|9vB4O09yJi9QZNbJu`;i)>vIrK)4E+))17A@G^1*R zmLt+^t(?;e&I4P1q5o$xtwBby#%2JzBHt(Uk^BS;OP z$WKr$Lf`?5Qc~-$1KKd{=J)WnWl?e~bmIG+{0VNM>2ogE%2RQOfWJoXMSv9Yi^0hd z2onKcXsI1k0kyl$j-U&OI>7YO%Z&H+3DlzB&@!X`B%Ejb=N@1p-MI*Y3!pQIJk|nK z@4YjrVcFRdjtmNaDN{fVd7qbepBDrUm_0DmyjdMU|0k%LBGg)AlwMx5k~6xv zm6&5&W$ar?0p7FG0Z*H80?S0>)_;)0C?<;kOAWPimf-M{LeQ7gxQBs3gXRsmX%*T>-1w7nd*ZgEz3NXH9{iTQp6bTh zw;9lJ?tx{EqWChn?zmRennPrq6`7obU_Ia$&|7`9%||9)qDs?iu*#6264?|1I<=o$T`Y;4(g45BulsBrx5HoN^1Gl@9Z|n{qKI~pue409@#7CfWb+DXrFvbZ5X zJXrC(0Nxie4etwqRt)QwyU|_?-y&^`larD-Joh6x@3g7g*US!DL_icd|0z@=nvIJy z>>mpNW>p~^e<8i>4$5ZtKQ8^TF`!l#%&#e4{m8idYqBB*60Aw9MlENxOk{V zSO`Lh6CDD{*#8ANY`Om5>}%-$$fB`-(?6ngPexjezMln5w+0|hs#d6vhZo>XSxj)f z`X7kcD9}h^1__)1+m$e@@k=>TIwEKWj3K|>%W$>mN@CnL7%XwMklzz9ykv1B^FtZKuNdof>a;C)yzyIDA(w;;T7H(Z8F-rb;%V zI0moY>A3(ahT^ZK4H)qECG-4bcam$iawCj)oEER>d#bJBhXV7`vXqA2t4@+p4HWqc(Hv4;J6I))!|povPT}zEnfUkwD0#od=u; z`Z}fX3NL%rg7KFldTKZQ#ZiPfQCSt68Uin)=q-WZpy66ltK`sla!Fz;DmiEoJcOkN zwZIUS-6=5d>=3vw>X##xbh!a#IoTY6?vijyo}7qgQ@&{ri2{ijtD6^tyU-r~q!%$h zqiE&A2f%*IWIjBU37ht-1p?hJJC7DjN|uvl5jbBDsJLc*YgXsKaJYLcAmUG7n~=O>bUJ=-jMeIG)YPNL|SRfpFO zIszP}^>}wqjK}Ae{$}gRs!K_GvmE1MVVL3BnGzcarYk8~TZA_YcGW)mmHi`V`ai-m zE57avC?m#2*bhUfU+y){G2_DDRA8nQhyD$C%@1c;9%U)KTPBmhLOF7c`P-%OE-VZA zT~>HN{!IK|KNfCVZ5LE+1-UjXQ=Qv${bmt?aGl1!8nPo3?!xB@`cX8UlYZq4*dW-q=F0QYvGZGfGxY;)}nCIRo+P4h)%W);Mw2_{E z4iDo6;U#~b$?UzCBper@yE*p_3-L|lbWuom-W`w*AY>w8a3-b~cFJKs%V+!MxuK1> zWx}>*zK;D*{j*A9ocNo+#v-#PN$-tB-)KRe;Q!T!45$buD9}T$hwnzLVox44^K4sJ#nsG`NWNN?od9HJ-@*+ zQ%)S|pIcG(@Y#}cTN0m}wBw735o0+5uTH672w25z3ma@NaBN2ZANy(iR-B3tEnoWw*J69PP z(x9C<-~m#Q{i?v=a~w4cSC}o`{{IVb3`#Sw0VNuMTZwA|%Yh34e;t_x-mC^Zs}|*W zBVdwP2@KykcR)>gSXzK|7l3g9>`)3`1#cw-#+C^1$RRIqM*=9*1x#6xw08VyJTOw1 z0Z+X7->!Q9-;w?Q<&`wS<7U}QPIW$ZhUc=r;yd+wUaARBJO*AK4NPjlJ#-Wm3DBk( z@QkRn&vI)8ud zpFck*&b5w?J?d+_ZJ+f2oPU45Z_AszgJJ*1g2Z3bixZA^-8yyRp7r~h{NH7)I}XE6 z9X{h0w7=KP`pV_WVz0JeTgSg$N>(;g=fKwPW5PDxR=~xt;1h{Y^ZLKq4s23nZxiR2 zxdGgQ`1SXaq?$J;)nA8STfUKD`>%jYD`7Q2UPVx_@8Q|OlEB7T#Qi4L-rq-l-AQ;B z1#G7-nHy=PgD}$M>aSz7+nmkCu0K9+Gx?w75^zfr zVV4)4{tMhj{jS&il6~8qs|lA zf*_ReUy7!mDd5LdmYW#(fw5PS{T)efV48woelotMa1B8+{SWOvKp#D5BX{2(L1-$_ zznE(4cMlQdf#(hBYj<39=J3S8;~O6A+cWH*%}Uz`?u@Z$8L|iP_`EkCZ5-fM@6V6C z`&4r_y*}yvi^dCz=ecqVK0XhneyS2@5l>z4LOZCdOPxFDRgjCjkY0_E;h9^Xf1@~T zP|A7cBHQKPaeqF*T{GDfW$;~l^r^_vy=|h-CRF5dtH<`zJLB6Pg*$bJFU!Yxwk)j7 zM-1#k5FC{<4rcT2C(4uX%gz5^|H7e$(OGO~mmn+__y%>GY0r%F^k`ru9>hBi|j-A6sSJb-H-0F9w;{omg_oGEQx8YL{v2NtXq zaFnaC{Aet9#ci;w8N2Gfo6pWdWI~%Opty47lB?cg-#9ZnI?fXxa}b2@BIP|JF{%&p zJS)VqL9g^)ybE!8t=spbnF!YOJhK#R-7%P$6P;d|#>nO`l)`zJj(G?(Uxm-@;PY$6 z2ks^Q#s5=b=Dv(xVVAkhPqPK?Tgz#A1FqG~&RWGT7d0Kl83ir!rxGJvU0pv`BZ&Pq zOt6TYL+GY$1f_-*M>6tU>wr#aDKFu-w5d~Fg$s9f-Q;%o;u4#TRE>4=Z62(jMD#9j zlwUfqHA-|E`Zn1fJ0elu;%Y`I+$OsCQC7$AdBvHy2A%>n#8p*Q zoj8BbW~E*zM%0l@)6sG)uM9u8wK5@lk!EGQUbEO`i94hCm1SvSYAScf!a#|b+hK%n z2BsMif4gbHk7!nrB|F`THLWkjJ{sL!w$_dslfwjWx6u#%C2G@n$oipw2CUF*!NA+L zKSvwy2}-K~*-n>i=+KINsnELNIh7EiZ|LmojLxJ3m4mGk>sc4i zM&+z1{@PfyqCkNPb39E0@S!hkSO zmk~oc{dB*2?sj_5gCtKaTgR*6>{}w9A0J9d$VBp32~RB+#Ih2Ne-{CSI_Eq8Ok&fn zv=q%15&CquRW0(up@>5aNGLbehZjL-T6xO_4(xACU@LyOHo%_CBrfQ{*;~FAFN91y zyO^ZLRPWTDa@Q0e%%eH-VANzHF%4ZNj&fENzwAt;^Kj{-2l&g+7arLt@ekj7@w|IK zThpq(wXoA{SCS-ts*B%xpjgRcS~c^7%hE{I#Y2)%260 z_PfNSusIYEV6l&d|HhLv$Dv;YHfz{p?%*)E*xRHjHB)HWTTH6J)}Gfc!rpdXDD-^- z!!~5hXY+kvW%0ojZ<*J>zS^o(wmIUHe6YG=JCEuBs9UHeX_i^4ICDiB-R|r;7yjX= zsZOXc4>+^+DqAWrTBIHCys+ix>zlBZZfCtu4LU7IAdRq=GT+GMp~H$FZEfeqbz0BK zWGT&!Cj@0>y_<6Vryu*fIP=(plAU60psCv2PKj*pgqC}`X_pv` zA6vN^#5Zj-KfxeIz7*$K$G%3=4~0*bH2)d5PnOnCf~mIM^s}#{(DD8%jfe~D4_IxV-g zV2R_=38<7)Xj!`rh6#x8JET6RvHrC-B9~cG#Qj`vlWhy%c;DSVgK9q>&D>egivqPj zGt>qvedx=Uzj1eun=?FD+K1F#qP!>lE#MCr=l~&WMv&&h^t^VV6PAD2Kq}-Tb(H=$ z+WQIYuKV~9zUL$t@l6G1PIEZght0k5iCggdft@eGam=lMw#toH-Oor*O--%cG)DBS zI8pftO8;Aa;ttB{|90Al@n-52$7D^yQlk~_C!~&t)cNv>p>XI8Y@$QWbV>3-mXlXP zEIA_h=~t;sBAltBP9bmRz)wZ{3JBq%3zQ(->VnRz9I--J0ezOu%6+Oc|4L(F*_@19Y zH@u%Hi%HC<(@Qz;y3pZ~6ieP^1sBvJ>_^IjILhflC=Nexu(%?CiX;zdTMsILewmT_ zeHJ)ag4E>$R*?U+Ug7+AMMB}I&0}IL46+2vuTRBS znSfL{GGDp$kq}m~aV6gv7EZ}RM09ifsRaZ))a2$nuCMvBF$oBNvpH|_89{u>pK=Nd zI!t--$i^KcFW6%FPyxJjb&QiV+*Vt=&2KQ4ko5kT@Wu7g%ia4CT#5>g^8)3`9#(E0@A8_aT$Q zSpNBLQE;ogBF;4_9=9OahI9)+CsOJC&_R2iSoHvfvs%&Gp5UK zwW}!sovjFYwv-oP$<`W~N+&*v-%?xfnC(8Z#0R^s%td-vs#$u?mx8o}cSfGb%6z0o z5W_d5@zQlJloZZ!f>}^M&T7-zL#qr`6e-Oa<3`^7N^CP>DKL@*@?}v;zaqMt?76Z}!|~ z%`i6zG%hTOG&hG2CZGSd1!ZXKG~`#Q{#*5{pItCyM35c@7?jaFM7Pv^t3%VeWYkaW zf=+R0T46LbqhOyytAe<=c(;Mx{ylG^O6&L3B@`w@Qm*i~6jz=KMu@I_etIaYx?J9S z8Z|1npTr%OrIp}wxd7Y2Cl1RZKuhUjVPWy<4@#jy_myPTq>pzOd=$9V(_dQkWluUe zI@U)BS#QkuLr3GxjD>A0pBce7>bCV4S&hDdNyrDn;@3-KAWT@~$?I#mXY0=_wmzkV z742D&rrb;j(eYO3??+!h5ye#c5sZqVe{r!*dpcOx-kpYrFi)#L$xT>^5=7D>g{%ik zHU`9QW1_*`Kh9+&i~sVPNAsT$1~q|6fIu}1rr8+>%WBD6Zn|r4b=KZ2*N2|h&nV); ziXcz9Ld>B$Fio(Pj*A&4H>N7PKV=8m#I&wwf8%v4FLADZ|-`3L0c zInX2Z-a|GF74H!#O6HSS^8Ujjr5xOMRRC2r?((a>)TNJp6Bw2||D^&nP#Hs#^nI5n zKqDb6(f;Ua%|GnOc;Lw4=vpv>QDB$zdvtL16x96Q|1$BuY<<00L68XtCXh)pH%#=T zM7cm9z1{}qS%VW;1^$0blpYub=&Fg6ypVg+1Mz2XiJNoobhtHkr*}hVrMuBnkZWF6 zVmV;dW^KB?`!ua2{p1yLSo?y_9X92NxT6WU<~V~<(2_TGd;1dqpl(gwRyE7_weR$S zTe{Ah{7sJzo!{*&5llWW8K18XhmBPJ9j4mSi*?%CE(NEMGv-vnZT-Ws-cem0 z*fV5|#(qeg!h=4R*xFj{b44 z%a+g$`)h+uMURr2!X8fxlZ~bDJqU;NYnw@70CM#cnAzL->qF~01jX1}(=mf zPsFnjlKef1^G9y9x2zl}3)4Jv_v?`pSYgi@-K}}u(#)0T3#F|!t<$1k?p{_pPx!98 zJIsQwHYA=Q)LFjbiWhw@jML{b!JHP6*lax>)4krl1w(iL=%K|K-YaXnArob#Be{L% z&tBZ&vEH5VFWDSZ+8tpz-QY*V{OLDEX%*4sUk)hSJmoGwQ+;5xZE+Q!{``epu%&Jk z7A9p>{oQMC?%^Nxho`_r4#Mv=JmiiK| z7cUN7n_@ zt(C}~@d?=@R`m>~4lMz#3&kB`%bOu~MLFEo8N|=+jRJ)Et{Jz%Q@h=-OBy4IR_g8p zy1u@pYl~&|bA9>^gvumO$6<@@ZSw(U38#b&76v!Rl_;e$&!z6NZ*!dX$0_e=)SPE2 zqB{Nj)3NKA{h`cMANXq$(gtfh&vq)lW;J~z>gcc>-{`VbZEkL^f;la@GakfCVa#ry ze1}-O{8d@#9P zT#J}n#;gZh2{-SYpy&KU{Cz$AUzKO0OcN&jn7eg*SIkatKAanEt6gy735?7Wh~^es z;r4`7|LmN&Q-Xi6*oXKhD)=Omv0p+6&b&#b3A%Xw0TxrHb0Qn_eLYDpo@Q&hW2RmN zxs8bJ6JsRK(>yunO%|oCt1Sf@krrbeLWycxtwojLrFYMBXJ#z;N@hgIv=4ZyIiGv@ zeSwg1Pm<8FbdNddZRVBjX&+mL5b?bEpR%0_gPm?E-C)U`V&Vym$r~diDmW`*s}tum zj3fb(d9{AJ$m$)UG@3@W+ht)R z=44Es>6Y`!^SAc|P`gFAL9?uZ^S zSBaj1QS7cAoL`eUuIM=)vU91WAt$RQl|86a(P!{cTHaEm%J^9rP^c3NI-n$RcCjxC zHXqD(aq0&O+!yuk?a&qUYcQ$K@!i|gxZuKU_ha2%>JjPIcCJF7zNt|ywtGmpd=6%eus0gs$mYZdZP%@tAr*5EhK1%nzs-dJhO8D&B1g zm4RQq8cSNH?#!`T&dXmo7iDlf3z={#U0mI_B%noPWX(E8d5=xL$X{FN4lV=x!hHMD zATiuLL^JC!;k;T$^uVgBbc^~)mX)Z-QQ&*$`#LoKk?iHi@vNd-JCkf8ao?g8bMlah z24!}U5QVVaI^GSTD3b|?;IJJtL&aF6_N?hbrn%rD8`Vyp@__9dAJQ9&*FHTeR{4*{ zvdU*7C*}LD*fZypPztAc^-r^ZGIP}ly;KvX+jEDg9%?eNdV^imBdGf&#reSalIrws z?(+K5&~1J^-I~b8*RmqJA*?ojqJ36MZPBz6wo?ly)=`AU7XonYjE5kzKF+LzjLguyN(H- ziT9{Z*IfwqJ7}t(63}WZAF6>-4AC5o)5gDKr)l?d%;9A#$vWc^tK9qGmC|iDwQX+x z6I0vb%=>)+Mn!F1%N6w`brD;Rolja~kIdeO(?@&$T5@!w=k$W$eA56$)nisqbg>`o@*oSOCwoWff`f&l1)t|iEkQ4m#IEvzshAQN*{l$q(ew5tDhdp5z;DN zTU;3P(@tX7gT-&zq|BZX2^2e0%Ug zde8oi$+e1@vKAzp$3KY!2)KdkJV8N}LajEW7$11$aTuggwHIL|3K zsIL$lniK}N*rMnG{1AoY7PZ3({5LL#5Vq^>I@t;eR(A(Y%8q*hg2c>3UI@Ta0FZ0> zv^NCSG1%q*J73bJB4vwfV*cry_vX)Clm5Pn&m-zBJ^x?BgN@xabsc<<^|;P)X=xo@ z+g>GgcQ!GVpkN3!_!W$qMv$9t?@_u8$ufGD){PxKT9dR;I@q-lC0XM~moWAVJF7zL zsl6bebhjgUA_PA}%Pkr0_TZNQt_kSX8vo5YMr_&4DV~Gpx$sd_`RctUg_+!@cEsj) z=K9Q^ytpH>Qz}7hUmzGdz2p#q=V}rHX>5^7^?VAZb&r|Sn0_HFgk}ZmKOarjqj8^G zk0Oq(ER@=x6T2D|10!u~c?5&_gIga%lR?a$V8>o5gJHUOR9Y5?+CmO#|b2YyYFk9{^#6$?v7;oB776;!*K?c$U%`Uny#NJ`-!dkaHh!Ep$XCrF=5oEN`-4Uc{d=+t%Y0mu_` zPavWD$q!BgY@iK@BAWD!Zd8U(xIR~t*ft~OXa@$1|AHLCDiYF2&ho1X{JDd!Yf^DS zmSR>W{D}Nx(#MGJVc-2YPICUJ#|rH*s>ZO0&E0&oQ6D_4J&&Qxr65KoaKlRmk__GhApfRHd4~ce z{|RUy%*JGh8JcS9hSbrVjcT#0tLqQSLsq3*p)?vw{1&NcpN6g!1#Ni~`pBDu{8p9lgYn^t^ zb3Lz9tXF$cOg?$v^S*N}K?;Y00fpWXMcKXYOffXt+S@~iih%tSqYrFcS)VZ<_&Qgoiz#VD#itfZkWsa419H;IDj()|U~tvAB>t?vL*G8+&A`YN$L8$eE@`FzO8LH?8n zPSc|7lLroy$eUtKH^k2`%9RPd)v=K$e43F_^Ml@};zuh^3j!T=R1 zYcF&vLwoy$EIg4imnH`G_j#wMUl@(S7+td_c7wJ!r*&_SFHH=IjYk7ZV4DMR!nJ1!wc6 z0;CeAExktn7~mGY6HNbYb43@3I<0s6wjb9kBs4ep(z_KE7HSl@9#*!Y;=6u_+;$eH zjzcWo>3h>8;EpxVFLowYnw>8+)ymGp>@$Scm8V!g)RrN2Xf2C|ks<2BoT8^toYw{7 zGqq?MWp^q*N&!*9QinOu5G*EC6Ff-F3^UilAysZ_L0@|-*D%mRQK?L-#XmA6mX@%@ z!Rsnk#6I1L4!PH~)z4?;FnlvDO_TH0J8=gyrcQB;HceI)MA^5Tobnnt`ySaB1u=_o zgRQwLi`R=bb`t49Z~n0VORF9ycPN-Z-6Uz^sSv>QHG*^lj*PQsL$FUM4m<2et5QcP zPjUnSt`%f%vihA`u>Pocjjtb_-kZy$$|y(q)r+qU;hU_hy3V-l`)?jMyg}=6z(lyN zCdt1I<&{Ii{I0Kd5Z#SsYa5n-y~i23jG_c zJ$&SiR_?sm8mQmAMy5R=^^XA~>k)bFZ8gtj#g!MR+aKk=$$!=B$6(NM0l50uU!~%b zt_0k5jTJ)nF<;#ya@?E)gjIwER)1a5A!+c2gQMdq;y=@zJPhmZw$o^2ZhWitc2nT3 z(U^YZ;N)RaS(S>f=>}^1!Gzt1{bO@4ee1lU`1bNXx?r@z`2v?>vgV*{gg7AN zw8|n*oZrltAGG1iy8*cb-_;k>jFeqXCb;C=xO2D6%5JX+qHGd;#UP^`{h!$4orm~s zPH9Z=6r{}WI(xC}P=?{r%Bh7nfO%jiP)1CW6mup-Z60->IPly)g02~_rb}W&1s`gN`p1JOgn{(Cy16z?!wI1S7{#Kls5x`qhgiVorC>Rg2x1ha^8#7a> zk9hc(z1i0stFV+&$+C5)>36rC%{!mBKKIT&rRls8qaYSv6&ii~wJ_l&v;K^EHE(>> zjMxO;pIN@qyK~uVJz4vs;!b>Nl*G7D8BwDrN$_0}fjgOZYYpq;RT}-SDQ6Hi-PGpK$c{XjWf4}_y~Mb2!`#fb#5TxoTWd{MXEyNLqj}MF$Fk_EBNyzC z!CE_#z1TyLIX{Z?JjaLHEX;HG?p9YhQ%*1#UGPnCh<#lU?=gML)IC<~j6!A3dUAy4 zxdQ*z5h&}>pM?N|E|RHqxhqR{{*jG8ctIf81XJ0}7n&HH@zu)X5cE)EsO!6jgHUs) zGeW1gA1nBAqDgZtM*PwvOgi4T<;|EzOLekG`iZjBl=;6xj+Yn*u_r`oKoB|}ht#&m z&m;sE9}v!~INNUfswP&GH?lLT-)#A}Ch++|6k08?Pp#^b{I2~~*QDYer=a`9Fu17^ zqY?97o7GB#IYX09^L%8oMRj;jFOD%LwBw7_TA*ILlS$@BuJSR!(wbzJT25sU_6o^+ z6+=P9fuHhTI@{e(UZOfF)oW27OdVhHdrVmJq|r4bT3+7x(j(zUQ*wM}^m3Km{0Z4W z>^V*=yyn`gxnRwSTMbLHi;*qBz~V`uzrWtU+wML zK3h7}S0AWiPHh_M5RTz)bG4YE%wIoOS^a2klTyJ2prw6ve{8>Ph6GHApJ5W9G523n zi3Hu)Awx<3jR8haH~vm8RW`2jfXVVfwhU`VfS5GSu>X3re{FXUwTZ( zgnpY#&P$g*qj1BqgwSI9eRzSot#flz8M{p^R)LXp(mnzL(zY-pMOD8y-asM4JB$qd zco&Gvu2use;9k=olppL6<@BoHeL@?W`0GNj0~ry7|4i;Hoy1!-$}&_Rf_Re=;x4^) zd&c{cyIq^#6@392eGV>d>d`PY2*XEJXzE5>&?t+%tEyG}y4rBn+uD|k(4O}Y&~YHl zTQV~@;U_h92`zxbsOIn`!<$pJjM3uY+EF-tL{2JB9a29A2Ie&3UPReq&ox}dw1c&# zgl_&&ML9fZtM3K*T~T5bTG5`}=gj>R$IpB2^v!1MhRlutKLf}4AD>7A@4`*T#Q+Qx z?CMh#FhN~tQ1hoVXQ2L^__i79q7X5Xi6Y>V-xPy00$0z+&#=dC0-Jo`9knl7zt2?0 z%1Ytx|8T?(ufTnugqJ0Lba%$G-#KMx$rA=lW+*{l4j&*iQRwwcdfN=ed@~b%KB+w? zh2w&Dff?7>Wb&WpMcaGR0Z_BRTbDO0w4|8P3*$X*4r#lLKi5%xg69gEwzTC?9%Ti4=wSj zRVBJy=fCOQ&Hqoq9Lcr-!imm2q}DyxF^O*kEyV}`K8!Zb+C~9oA$FrGbqIBRmx>`B zF=%wremMqJRs%dZ<;*A7RI~g4D;J%PPrL2U$Y`go-no~5i0>cT4+V9)jAaO1M#l!4 zO!UAAdvoZtj~u^XX1Uuv!rixZ%iy+a-n`gxN?>VREf$qI=jZ>+nsBgzA%GEIXr5T> ze2s=sd2zTRug|Vly3?LhY43wd)H$$+h;W|HA z^#j?W*=_WR^Z}I9aQoS)L1MDN$k=5yiB2&(F^K7mlyF`Omk0$;wwv=s3s!x`R0~Y| z^^QSk!{WC70+b3;FRCP`Eun-ppK_tIH?1Mb;V7Sp5n(bE*10xnUm26h1fjTvugAxd zVZ66kv)s6b&^*-I&zA>E(&IQ&K^6cE0eOrn;}5n(Ok|*ZI?YQ=lwV=BnE^SDTdk@! zrW-=3frc*VhEaa`3``i`*GsRz&@5?0h(G4Kp>7*dipeKXShrFzv1uH+stMwm4WP$8 zJD8@?Bs8>tu=tmfJcvqxOms@I)v8kccRUTn5YW@=|1Czc+pVJwN!% zA_g)4EAhv#ZqeQsXu!RWIAtD7aizHx6ME2g`~#HnO-&r`=o4>HgO95`Zg@$P>$OXz z?*L;)&TU-loX>(rCj2XQwRL~5i@{mYO=Uq?Z51gu96B38Fs1kq_RvdqU6p4}8NGWz0E0l6`(;C{CTCX(i;mPBTHucTAnsluh z1r97vnBXXGO2vhUFE7W=54fBI!zCI`%1wDhA&(}#2%Ni{tyTpdwu}mPEJLoe)Q_W0 z#F6X2WW4ehcROpg2(za4eMUpdSQq&L)jC!pL}e51#NA)I2`I;xkG=Ft*TNSIoPhev z&r`@FL5~v&c_x!|KQ51ybw`l^v7xYE9EEOL-x=g6Uu(6xF79UtL)Buay0Ue*FOf3e zc9S2uZe+smzy-^qf-_bv&X!&12O0p~P*1id4AGl1k2gJRSy~O!BIK~L=+fs6a%;QqdMBvKzv#YepY{DmngnwJ){=KqkcEu|7OSxHsJjLIH({pED-gZLJ_Gy z?H4TRz4ZHyiq=$^m zMl8;dWM>Toyw&RN^m-*yCj#I9O8P#rWaK;fdk}0Cin!PPDUWymn_ows0%7ZFF#_of zf%}9}7->{>exMvAsYvLhuNJxaegOxIqRat%1g1wpDz5$kCJPJ|a1_UtVRIp-j72(? z(0Mc673nDQxQMhmknv*3N;8KfH6J9Buo-y_G-%!9bcM>s64i|Tdf_yuPoc_PS=nR~ zDgp)pSbMCuUBLo$O4==SPhD?PnlQe5|}J`5-0$u4jfECC&G1(ECOz) z`)JE3Z^34~SswXX2CQs;epiJ?+*>I=L(=f1-o=m<4M^#ud1p?H$aAPAqL$+18!9l@ zFa&79z=VGftmgCLD1n{xA`5~nHO>zb0(tW0+r-7r%SS}DMCcm4KC9D}=bODe92 zWdTeb^T2Q+Y3rUjopN9KJ@R-vSlBNM>i8crpsa#2iE^3Y+0l0ZkIR#h>CoHY?Qgrz zB=vh=PTDLG8h5G#~tg z{sH^HDChue^Z=UDGZ`syk}|(l1{H`SZS{pq@vsS~mptiwK$!_1@gvEOvOAbP$o9Nf zQnpR=rmb~RTGjf+!$#hqEVO0rhAgkU|-TAdDq= z*ms`x#jQ`D#b_ih#?^&%d2%?0=F$!I2Ej9S=xEO*~4t{`=8Z zKWc4_JX;$wsw>DJ&IR~Buui!yz>XJr*hWTHKKKMeX7tC~<~t}64=N9FFm{+KN>69H zz+kOw(Zhpde9g!(hdnuUg*2#rX{rCn0pF`4k4)1%MC0gzcF3<4rQ+g_F)}DXz>Rhk zT1t|E8ges<|G2X$gxu8Sv(r{V1JET~p@;4IpQb078J)lV9MliN5A4l7DCjWpQ2Qpf zk>p{&Q8$z#Vpls*d3elm>=0;t>D$X?T_O^n!=o|#ett{?MGq7u1Kd*iCz3RP+4n+s zIDB%wQbFk_KxGn7Fz*!69d>vLX-Q^e+SAYEatg;s5JkYtr7`I+J7+LVa+JcA33_h= zL6P_H4lv~~w?dQPk9cO>KMkzd^!q@aDcDw2tub(yWB!&3HS)uK8 zVTHQjH?&b#7X1$hrSBx|V2RI(Q*nc3J5nvlTcf;^T7#ad^Z4a6z^Hm?rIh@;=ZDRT zB~9WtXX3JNl%)Gc6nFRVn=|J(r)gP;8$|8zy7*6{#NHMb!VGb9qhL*C?90+`Vk&fK znWa)hKli67!|$6zP;NBpRi9YEf0I3e9nF1vH}43sn>}82r5d~JoHMY4^%%7-+eoVQ z>nA4nNvK3cmJ-)D(*}v{{`2>ost@Vz2vzLc`8fLfR;7s5Tuk8$ISxdg zD!QBBeuq>b3@ZQGXFWAPo|~it-c}qwx^(c|Ak`Rsswa@p<0X1aMTy@ZW9$5mLLJ5D zE0l@fo?GgD#d9rbbIQ9hf1R6k4JA}bZY`I&PK@++KZwo#&sKl7p4}bxGj#sm;Bj4g zoo6TQ3DMpXEtEpq!UoAXkJn(Z#X-@}lEk@~lH%F80$Sap0guPZa`25rssHu4Kb9I< zTli7RSU0c7d9;QLKppLM%(LJx*0+EPKYYMP(qfzYsA>4ZApY&F+-(iW=H!80q;af_ zr~8=0pTeK&qn5p2#P3y{_huujjJu8!x=2;%|MSms+Q(``tV`B9#OCuyM=E`~Q)aq9 zJg86*>Thdp4HSRI35P=dtGk?bx*m(^xP}YxjdJc=?@ZZZs`CYZ=I$6xUX=kD zoWe>qajyJz^Ac@E2z`3b>)x{{Rfx2L$sYf8CTVTJt=f-SGuNUU$iT-ch5MX(EtMqk z4!Lt4P_^14iLRf78s|TBuyyU!mDJ4Uywild%nFMWLwhuVrKy%qJ_)<+s7_be(?k`k zJjI)>aQ?2MI(?N;(M@zUz9kmpi3e)J>t0nh;Dgvl4DM97*#-Yzd$57Bz*7QWMN zr|t&bPy;}!d*h{Hkzl%>N6?s0*Dg|7D_Lm}TL8MT5_Y>ac|ROyGIrBiX}QHKkp3qW zRP#bLk^q4#ndv;m5IRQTZaU-K(s&_0<2Rnpl+C{HkCpsNnl}bEi_T!3bv;$ZUU?9! z2btYG7Q~1j#h2&3q^5XJbMuN5g*v+*neffI0cmY!Bd7+zC#LZY;;QjPLm92)m*G5F zK=}IYC(~9h;~S39Osq!&O)LuXa+2(*?7$%?KcgZz2&As_=$zXwL83tDpwe(u#b0qS zz_w&I&M-Kp(}>OE#t+%OUYMiC(&4j457`iQh~@0O*4RrPQqvKZtx>#J@K% zmlS=cerRyc4X9(9=lZH(tctsO+uL6kxgHjh##micy--dsIH1djWyF6Vh^J0*7Wjvy zFA4Irn5z18@@04q^!z$(zJRMQxsgA`ep6QkpCo6VjuPaSggvsC?Ad30<|iesZ_LcJ z=dIo-Js9wV<5sY+a(0lI%^7`n=OvX*kxDE@aiXyv6!ua6tN&mU5+Vc++b?Daz~8zw zr5l~k8S^?almrBV%d(Ah=LV|)a_TD^*2Al+oF)uhoh!TjZ16Nx&ZN`&WEhD9KGt8J zdn=$Y`Ifj8cj<~ZMIYwxxz@&g;3`uUod2JFgD~=$^p~ZoPB7 zx$WBkwjN~Jq|2O28}%K0%(&ga-!I~EN`Yv6jqPRum5pdzeY+lIeldDF)ANtp!f-zG zY#wLJDuYF#6_e79<3e*QjTt4(*s5>+cw=2L&)+=iv<@g5igi>sWNJWt=7^=D?vRlrsdbd_P^?h26CH`v5$x*M|j&lON@33nj?yX{aGh)Nk$A_|wEe!)N ziG2EHQmyo0j?&WOeu>bULpJj>rn?Ukh!idpIRjo2rThw~4%ysH+s|E|(6Hk6tff)t zn{nGvRXStDbhTVkgO*#b4Oh9T!d!u`HIHJW=vv2Xv;>gz8igB!v8frkjt&;Zxofw~ z*Au!6=!QPZ?iuhlHd=u$2SZ`O_V0<6aPS7AVylZeiQ%f;+R2se&Yu%4b8Yi_%Ex^q zHB^b171IJlsV&xi#`+yHkzyM7nwD+|Mm;qwJtk<#MVa3WELv=bs#~y@5KvoEB=4Ws z4u*u6U4mHOcau{+_C={DDl;QBg><{u+?2nN~C|qcCx)S&cIPax5Zs3ylHH0#7i` zN~6dB!q_PDE9G^{S!n@lu%i?%G?_LE3@L95WXB#^JgaM*ey+g?V};V*Hlo5zCVDE@ zx2;R%UtKyeq_L(jb9_kgX5n{DUE3Mnf$=IySq;u|-<$#oDjVZ}_9#uLE3wo2Un{Ue z(Kmb3LW0G@D^M#&o_bvrZ#mK^W~`3O*iZ;H!YD>eOV&z@B`NXO9Jy`Bsg7y5J1C8r zkB|STp({Ic%X(DKNF8Um%u(L-YSXeNxF`Pg{WpOQ)p1�*CnFjB;KmNe(cZ(UuYA&K_RrUL&nvJq0A|>6BQ-U;Q zKA~I{lGkuUStg9Ur%GD{S$q5Mjz`n@d6vSDEPS51VZtlZdmMtzULr~~$ z^q99-lGu9a8rnAgWb0FWcW3SiVn(ADQnk!$iNC$t5U+{*IB@qhMB6Hy&RYmwdlr z9231ud&A@A9UOZs62jli=TEJ@J-(rvnuZ?emDA=|7i-O*98Jm~Oz5>Hr=)A*T4ld_ zbO9oFNAi#ZzGL+PV+`xRqCO(eZDZb|eJ4lIefW@W{>Fe?>*0IDKDxX5xzz%2 zIEQTTIm!dxXOwOX6ea0qC=lpA26x%q&CLuNUnytL`sQ2@o}RV)NuhDOgmt(FSJ9Eh zpF)fc@a_FAU+$bq8nwQdLlK$&rAoGF`;-z>3O*ryMIk?cf~nCvgAboJ$~iU4Pbe`P ztlF;0YaxtIC@QYf2BCay_2k!eaQfY3I`2Vj^VdKyhZx1t$W~dC;v+Wrjvfyj)V;>m zXz7pt%Q{Gyp0vl`?K8>cE!`Y3mfY!XbE^O#YzxlK8b=IvZZDdKzJPcpQ4Mzn*pzy^ zX`|hFZoB;)kIyK%7vyqwj77gPd-%Ri`%+rtQ{N5QisJR78QzMU4dx|loo*TPE1w#B z^DEkcFIr1n&AT7Sgx$EgAW5QAhFi{13afBWsAouazHUL?8Y5Tv+Q_Ju`+5dL*=+@9 zzXg?DO7^Yvn+B*yM+>*WK{FE&-kxa0E~yy|zdznqCEBBp@%wUM3w%|Zg!-L%$Z*<( zzHf^ilZ4_<8cp8gQT8d&C~jz*1pTEg!8(5kq4!4dwP zBGy5@m=%pCI6ci)^!oc}C1h9wrGHpeHQ%^58Qi{Dr+sHh7a(}WBDaykR|`q`M$9Z!QnXTsV4Ax~J=Op4ZB&ro{5>oVV`E}aqy9ha_gmwC*8O+}A}{AU(J zSFN*6@Z0eebKd4=z1vWl(TH6u!iF)Q`6drE1X#|!b)fmUpt*scE!~nzwQ7v|$w!XT ze#ks;VV5@si>8j4*gCqQn-eqC=F-!Dztgb1t>ospT)nb0zvJ7f-WM#!cp`x7`=q!3 z8Wu04ySvQ39lg8KU{VH^*Fo)p4>I31ei*AJ+Y4omW-!C=N!$`E_#(-(taob<>6Y1V zRA7a#eBc#hb@+2|>xp%xNctU~7{avDw4360;^wv_-g<*DwBCq!gdErIU&-Xja2^c8 z=L}EJxZ!l`}w`)&Q2x*n?FKJ8{B#i>bLv=Q>t6(}&svjKM%yvs*{;herA zwb_@dA5jRa78n3Y6eR403HX+9ep$@}GM5~{OlA$P-6k`okS3kmT4$Q?#8u09DDVgg zbirxo>SZY^u~WJm_iF(=J4%<-qpLbd5# z&TT7IRg~7b#Ft`i(E~7X&t=rn*-c# zbae{QGy@Uz)Nv+fkLfIsspQt{asd#pe5(5ms!ssWA#c9L>c}{!DTwDRH><~WBqk>T zPwusbRSz5oPVQ=*RkhJ3!zlcdrdl4f_h`^{Xbb9jvi_E$Yj@ti zRA`NHr3$@7Ld7t+-_ct&96KFyP-4dQGzC?q>-r+E5~F({NS+mxQdPpJ?$OKmc3Rq0 z0o+opg?8G|`k6vv^dDsC8ZI69tyv=u9S>_VI=|SNDm&;F?3uojs?s%MGMt2j`th%| z0l}@=AVN{R9ayMl1(}*(+EkmC$N&+N&h6N_ot%Wqj9CF-F#gX?r;~w7Cggiweezf=>P*r+j9X&?j2%ijWtlr1K`>ad1mO?8n4;4av1TLA`Buar0FbXkXoz zG3aqaR5(IS3zCxxSr+^?BA|J=S;C+dRwy$G2BWgqOV0>E@RRBA+zH2xs#|kV^lquh z=2Q@O1lIGqJg$MvA4am%J*Shaqs3pDS+*u8x$ZPpMJGm6aLJK7`YPh`@NA(uEJT1y zx7j<(m;$}>1ZoEW@J}a2YY^*(+w4Xixl~~N&N}&ox>CIf zC4aBi0V05E>uMLs6p&sOd4xcBxD8Z6?p$uJKSXYl$1sRaz@vl<9#U5P5Q81BrSo2qM(=?xR9dwD&Y4Dq`g?$(Ink zL0JQFMesIy7lWB4bPh&IwHA~a$f$H~;jUD;ZZfKRwfTP#dW7Rw^7O{Pkc1u<;veGS$kA3RKqGI2Q9n*Tmbd8RX$5^5B)}zO>9hoIKtrG5m1KZ)$8a)q53Y4 znh0(d4}lyluo2~LfVRW2UA++OV?{fVTL;wKuuj?J!BsH#8bG7q?h=q?y&fozb7}wN zv_;Pm)zhX5)e3*EBDmtg%+gZt7Fc67lF2DQ4;uQ;;b_;G?we3x@ZIVcaJ4uKb{<-5 zvojC@v26q9253930Hx!CYJEJ1EBFf(9Vj6(X{aTI5)^79F;_nwFGOPg2b7sr<>@Y* zQ5nr6yj~!TF^XX~=f%`NR$%HvP12gRL4`q$f*G}ig35}KVP0vw%E3!8%^By?zKy*WL3#2UW$=i9|@6M^1-?Yi;0%U zt2n2{xSDmJ;Y|uIf-0g*D9jT|%5H3kGBx>-FBvS>JzDc|E_Fre^k((!$!j@dpl@i| z!yKFf>gK!ZLGG`Xd_Fz|6L@J}#;OtnXJ-h3yhOk@OQV?CCLN%F)U1sb1A-&KLW6C- zZ41@+%Q<~u{fl9ASkBNbQBDXcRa}` z?D;7Su*Tj~>^5R|jCpx?Oy<9+OFz`xvbH3y5aH4=4*ws{3rtskj=phrAJ{Iqx+8h4 z$|?XzGqYhq-~P8C?ta|;+wUnXcC%ja*L%HY!LHw8sr93ylmEv8fZ0uf8deKf%;_ut zB3dI^vG*u9{|l)8^gb0V-Z{Y(3lPX}mf%a7yudQR>;q=Q?v(!rizPuIxliU?Lya~o zP&oFe*ICsF-wJvS+D}5x-?&*{#UhQ_VE@5R$&w^<&ke}|?CM_2lh>wOn|6Fi$)kzB zY_rnB-&_ayDqTrH!iI~gE-R6#wN^f`MqmxWnralmEiI(T<&|JUa*{rAHjM9a)lwr5 zjWv2Z1m?#LYeOCAEeM6?V6~ti7O%0lp_y6{t|^d)&zeqiCZ6}jHbClpaYuXC6xRy}doeLAEHM-@x&&KMcJy$u))>qm2l5uX`t#3-6%YliI zYA(VR68rZqgK{u~vk^+yW2<8Kn6+6O1Skzk+3!#b3k2JQ;0Qq|AabqquhAJ^y#0ikacAwX9bLk52;Ai% z&n&#*q)T}ccRvxSeLn%G_CyxTJl7h=R;92Tt26X5_|#pAPCFFemY~NL3+8d!t@BE( zr5-ZI3J7j@R1gL8Zirs0grgp0zY9us99%b8!{&=EC&xPm=OcPmG5hs`;Uciq8)*Wb z9iwNG2Dd`y7T|~~elp@t-vJ4cbs60rQKyGk&r}Mn61sSV1(+NC0$}K1+OXF`V;BC3 zKEYs!YR*s#&V{+YU(0}>smrX6aJQLDle?zArdMzQt&n~0lIZ|rF$A}ATi zi^`VM&FvOblh!z)aaVC{eb>2Ygmct*@XC(v!uvs4OuEMOAsb*v@jtoq9lW}Qm52(nN|X>ndNqxnCTe;l`J>dcm|CjzawhTu7H-rB4b<&?83S6z8*|xDOE2*H=o_p4fLf zs;293q?;F;3OmdfW>RS7d|4VsZ-}#)3DmG6nJ$@1Ux=!V6d0XS0DvQ_xMbMhOdy{yZN)&`eLPcAZ`7!QOAY4#$(4M~*KJpGN*8b* zx_F_V>H_#*6%Z`Td8VhQFA+|Wwuy8djjXZOpIF7+B0POk14Sc;GlGbaAZ(AKFi6T^|`%(sso9G>Zs+elasUl93o0F;@`Wm!l8;0xuK^ zWrK?j1tvaiN>16?_FMzW`+w~M%4F~-aA@<9LuIv{ zaS1DH(n1*?{7dBDmt+2Y0#?!e`oB)VB5;itoT5FlYf{k|QUF$MYqfuaai?^JoqU64 zJhc122TIU;HsPs%4!TZd{Kr98xRErOOAAA@Q|dU_I3FRIJ-yy{?>>Em9Ezg;@(uLg`D6fz2IAi5f!lETN>-!jz6?HbZql#Dv73ggyN8xVN#I$_M zP;W+NMqP-=cf-W@|6lE$eM}Q)9LKLZF%iXVBeGT9#>5a_giR1ZqZos_idKhCuv9Qn zP(TVYhCrEunIN*3h=O3HHEw{6Py|725okgHg((vhg|@uNpcaGz(((@bUJsmR`)B)S zumAO~*XN#l?(TVhZ{JS|BK3oLsi)y`jA+6F6P>Yof`$pJ&IN@Eut&cmszo+)z#0n# ztHuRP!*dBTU}K@$FlP3?u{GJ*YHSh%BAXi4V~XO9Ef}&S3UlQTL4i(1DN0gc%E|OI}MfpUQSiqedOqVNa z-g1X6z;jQ;$deUvVx+hr(o%fVWj`fJUORlN%pA>mP|YcTbd-w6`zsdbu_dd!k6Xtj zM*X~kjPBp69%q&BTSyQP{6O2sq%q6-&)6!sn$jYHX{U|O5_kP48XS=vq)Py5WJL?Y zlosWlAnl^g8>+u_E14ua_ws{L2(pMx^AQ>mH(AIeRx%Q=s;^n}KSblKGYoBD$%kSY zzA?bf8IfU$Kx^5~fIwDS$jp z-3=xdmgE2pMnd}{qYy``gFCz$E<@~F`x#Vst) zw858h&icurh9Yx5%+jFJk!LZ$#1#uJfB%MPeTa*>s1bjiTrV?T-C7b$K4MNkVR?}em;KagOuP4^=0xh1DNU#lEZ zfo-beFTY^Y=Dx&c$8?BM9(%v%`%b~C^dJo&v?n}fR>qX3&$^2r`o={@)^A1*NZTI2 zb=x+&VQ}xY0kgU!CW>7s)^P`8(gAVK`?|sQ3VDzuN%L1HDhz$q!L*&0V(r}IhHpG}#@P(pdPTJ-yVpC%Dv;xU?HIb8 zeG2oWQY%Y8(dZxC8^7QNR80DZ#ZEB9R3=PLVk|#jmEG!4Y8t%C@B7f)fyl+b9+PU{ z>@==+xE^$q$KVy~y4o=%(hK%q&4GH3!Z{n*J&bbCU3jpJLlYOlRI!5Ai~1%r3Gva1 zIxR!#$UDuqNF!^2KhWQ&O)nmzSx>p@c zX83aXkG$J*KeX5|R_pCWfA6zeZKuSK#uu_nBHYfJJA5q;MZOd(7=xCQR^!qk3dj~! zcH<_8jj6K&);M Date: Tue, 30 Jul 2019 11:41:29 -0700 Subject: [PATCH 07/24] Fix TOC links --- doc/drop_counters/drop_counters_HLD.md | 34 +++++++++++++------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index cbf9e510841..32f0b139b89 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -9,26 +9,26 @@ * [Revision](#revision) * [About this Manual](#about-this-manual) * [Scope](#scope) -* [Defintions/Abbreviation](#definitions/abbreviation) +* [Defintions/Abbreviation](#definitionsabbreviation) * [1 Overview](#1-overview) - - [1.1 Types of Drop Counters](#1.1-types-of-drop-counters) + - [1.1 Types of Drop Counters](#11-types-of-drop-counters) * [2 Requirements](#2-requirements) - - [2.1 Functional Requirements](#2.1-functional-requirements) + - [2.1 Functional Requirements](#21-functional-requirements) - [2.2 Configuration and Management Requirements](#2.2-configuration-and-management-requirements) - - [2.3 Scalability Requirements](#2.3-scalability-requirements) + - [2.3 Scalability Requirements](#23-scalability-requirements) * [3 Design](#3-design) - - [3.1 Counters DB](#3.1-counters-db) - - [3.2 Config DB](#3.2-config-db) - - [3.2.1 DEBUG_COUNTER Table](#3.2.1-debug_counter-table) - - [3.2.2 PACKET_DROP_COUNTER Table](#3.2.2-packet_drop_counter-table) - - [3.3 State DB](#3.3-state-db) - - [3.4 SWSS](#3.4-swss) - - [3.5 CLI](#3.5-CLI) - - [3.5.1 CLI show](#3.5.1-cli-show) - - [3.5.2 CLI clear](#3.5.2-cli-clear) - - [3.5.3 CLI configuration](#3.5.3-cli-configuration) + - [3.1 Counters DB](#31-counters-db) + - [3.2 Config DB](#32-config-db) + - [3.2.1 DEBUG_COUNTER Table](#321-debug_counter-table) + - [3.2.2 PACKET_DROP_COUNTER Table](#322-packet_drop_counter-table) + - [3.3 State DB](#33-state-db) + - [3.4 SWSS](#34-swss) + - [3.5 CLI](#35-CLI) + - [3.5.1 CLI show](#351-cli-show) + - [3.5.2 CLI clear](#352-cli-clear) + - [3.5.3 CLI configuration](#353-cli-configuration) * [4 Flows](#4-flows) - - [4.1 General Flow](#4.1-general-flow) + - [4.1 General Flow](#41-general-flow) * [5 Unit Tests](#5-unit-tests) * [6 Open Questions](#6-open-questions) @@ -36,10 +36,10 @@ # List of Tables * [Table 1: Abbreviations](#definitions/abbreviation) -* [Table 2: Types of Drop Counters](#1.1-types-of-drop-counters) +* [Table 2: Types of Drop Counters](#11-types-of-drop-counters) # List of Figures -* [Figure 1: General Flow](#4.1-general-flow) +* [Figure 1: General Flow](#41-general-flow) # Revision | Rev | Date | Author | Change Description | From 87483156d7262f2de7febf73c4783dcec6c9c575 Mon Sep 17 00:00:00 2001 From: Danny Allen Date: Tue, 30 Jul 2019 14:04:33 -0700 Subject: [PATCH 08/24] Revise config example --- doc/drop_counters/drop_counters_HLD.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index 32f0b139b89..c6f4f56b0d0 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -35,7 +35,7 @@ # List of Tables -* [Table 1: Abbreviations](#definitions/abbreviation) +* [Table 1: Abbreviations](#definitionsabbreviation) * [Table 2: Types of Drop Counters](#11-types-of-drop-counters) # List of Figures @@ -44,7 +44,7 @@ # Revision | Rev | Date | Author | Change Description | |:---:|:--------:|:-----------:|--------------------| -| 0.1 | 07/24/19 | Danny Allen | Initial version | +| 0.1 | 07/30/19 | Danny Allen | Initial version | # About this Manual This document provides an overview of the implementation of configurable packet drop counters in SONiC. @@ -99,7 +99,7 @@ The contents of the drop counters will be added to Counters DB by flex counters. ## 3.2 Config DB We'll add two new tables to Config DB: -* DEBUG_COUNTERS to track which counters have been configured and for what purpose +* DEBUG_COUNTER to track which counters have been configured and for what purpose * At this point the only supported type is PACKET_DROP * PACKET_DROP_COUNTER to save drop counters that have been configured by the user @@ -138,14 +138,16 @@ Example: "reasons": [ SMAC_EQUALS_DMAC, INGRESS_VLAN_FILTER - ] + ], + "desc": "Legal RX pipeline drops" }, "LEGAL_TX_DROPS": { "counter": "DEBUG_1", "type": "egress", "reasons": [ EGRESS_VLAN_FILTER - ] + ], + "desc": "Legal TX pipeline drops" } } } @@ -263,4 +265,4 @@ The overall workflow is shown above in figure 1. A separate test plan will be uploaded and reviewed by the community. # 6 Open Questions -* There's still some debate in the SAI proposal over whether counter indices will be managed by the SAI or the user. This doc is currently written as if the user is managing the indices, but if this changes then the design (specifically, the Config DB schema) will need to be revised. \ No newline at end of file +* There's still an open question in the SAI proposal over whether counter indices will be managed by the SAI or the user. This doc is currently written as if the user is managing the indices, but if this changes then the design (specifically, the Config DB schema) will need to be revised. \ No newline at end of file From 78acd4fb0ccfeb5142f8b53b98f66d108803a640 Mon Sep 17 00:00:00 2001 From: Danny Allen Date: Tue, 30 Jul 2019 16:24:43 -0700 Subject: [PATCH 09/24] Decouple SAI requirement from SONiC requirement --- doc/drop_counters/drop_counters_HLD.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index c6f4f56b0d0..e4c1a674180 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -21,7 +21,7 @@ - [3.2 Config DB](#32-config-db) - [3.2.1 DEBUG_COUNTER Table](#321-debug_counter-table) - [3.2.2 PACKET_DROP_COUNTER Table](#322-packet_drop_counter-table) - - [3.3 State DB](#33-state-db) + - [3.3 App DB](#33-app-db) - [3.4 SWSS](#34-swss) - [3.5 CLI](#35-CLI) - [3.5.1 CLI show](#351-cli-show) @@ -77,8 +77,7 @@ The goal of this feature is to provide better packet drop visibility in SONiC by ## 2.1 Functional Requirements 1. Users can configure debug counters to track one or more drop reason(s). - 1. Supported ingress drop reasons are specified in sai_port_in_drop_reason_t. - 2. Supported egress drop reasons are specified in sai_port_out_drop_reason_t. + 1. Supported drop reasons will be available in App DB 2. Users can access debug counter information via a CLI command 1. Users can see what types of drops each counter contains 2. Users can assign aliases to counters @@ -153,8 +152,8 @@ Example: } ``` -## 3.3 State DB -State DB will store information about: +## 3.3 App DB +App DB will store information about: * Whether drop counters are available on this device * How many drop counters are available on this device * What drop reasons are supported by this device @@ -259,7 +258,7 @@ The overall workflow is shown above in figure 1. (5) CLI uses counters DB to satisfy CLI requests. -(6) (not shown) CLI uses State DB to display hardware capabilities (e.g. how many counters are available, supported drop reasons, etc.) +(6) (not shown) CLI uses App DB to display hardware capabilities (e.g. how many counters are available, supported drop reasons, etc.) # 5 Unit Tests A separate test plan will be uploaded and reviewed by the community. From 46b8525565e50f6b1ed670a40519d9047a2fa2d6 Mon Sep 17 00:00:00 2001 From: Danny Allen Date: Wed, 31 Jul 2019 11:28:42 -0700 Subject: [PATCH 10/24] Incorporate community feedback --- doc/drop_counters/drop_counters_HLD.md | 42 ++++++++++++++++---------- 1 file changed, 26 insertions(+), 16 deletions(-) diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index e4c1a674180..ee60a3043c9 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -29,8 +29,9 @@ - [3.5.3 CLI configuration](#353-cli-configuration) * [4 Flows](#4-flows) - [4.1 General Flow](#41-general-flow) -* [5 Unit Tests](#5-unit-tests) -* [6 Open Questions](#6-open-questions) +* [5 Warm Reboot Support](#5-warm-reboot-support) +* [6 Unit Tests](#6-unit-tests) +* [7 Open Questions](#7-open-questions) @@ -97,10 +98,11 @@ Users must be able to use all counters and drop reasons provided by the underlyi The contents of the drop counters will be added to Counters DB by flex counters. ## 3.2 Config DB -We'll add two new tables to Config DB: +We'll add three new tables to Config DB: * DEBUG_COUNTER to track which counters have been configured and for what purpose * At this point the only supported type is PACKET_DROP * PACKET_DROP_COUNTER to save drop counters that have been configured by the user +* PACKET_DROP_COUNTER_RULE to save rules that are associated with user configured drop counters ### 3.2.1 DEBUG_COUNTER Table Example: @@ -134,24 +136,29 @@ Example: "LEGAL_RX_DROPS": { "counter": "DEBUG_0", "type": "ingress", - "reasons": [ - SMAC_EQUALS_DMAC, - INGRESS_VLAN_FILTER - ], "desc": "Legal RX pipeline drops" }, "LEGAL_TX_DROPS": { "counter": "DEBUG_1", "type": "egress", - "reasons": [ - EGRESS_VLAN_FILTER - ], "desc": "Legal TX pipeline drops" } } } ``` +### 3.2.3 PACKET_DROP_COUNTER_RULE Table +Example: +``` +{ + "PACKET_DROP_COUNTER_RULE": { + "LEGAL_RX_DROPS|SMAC_EQUALS_DMAC": {}, + "LEGAL_RX_DROPS|INGRESS_VLAN_FILTER": {}, + "LEGAL_TX_DROPS|EGRESS_VLAN_FILTER": {} + } +} +``` + ## 3.3 App DB App DB will store information about: * Whether drop counters are available on this device @@ -166,7 +173,7 @@ Debugcountsorch should be implemented to handle debug counter creation and confi ## 3.5 CLI The CLI tool will provide the following functionality: * Show drop counts: ```show drops``` -* Clear drop counters: ```clear drops``` +* Clear drop counters: ```sonic-clear drops``` * See drop counter config: ```show drops config``` * Initialize a new drop counter: ```config drops init``` * Add drop reasons to a drop counter: ```config drops add``` @@ -195,9 +202,9 @@ Ethernet12 U 0 0 ### 3.5.2 CLI clear ``` -$ clear drops -$ clear drops RX_LEGAL -$ clear drops RX_LEGAL TX_LEGAL +$ sonic-clear drops +$ sonic-clear drops RX_LEGAL +$ sonic-clear drops RX_LEGAL TX_LEGAL ``` ### 3.5.3 CLI Configuration @@ -260,8 +267,11 @@ The overall workflow is shown above in figure 1. (6) (not shown) CLI uses App DB to display hardware capabilities (e.g. how many counters are available, supported drop reasons, etc.) -# 5 Unit Tests +# 5 Warm Reboot Support +The debug counts orchagent will restore counter configurations after a warm reboot. It will get the current configuration from the SAI and check this against the configuration saved in config DB, reconfiguring the debug counters if necessary. + +# 6 Unit Tests A separate test plan will be uploaded and reviewed by the community. -# 6 Open Questions +# 7 Open Questions * There's still an open question in the SAI proposal over whether counter indices will be managed by the SAI or the user. This doc is currently written as if the user is managing the indices, but if this changes then the design (specifically, the Config DB schema) will need to be revised. \ No newline at end of file From 8aa485702ce5829af1ea94dde21a0754f3241524 Mon Sep 17 00:00:00 2001 From: Danny Allen Date: Wed, 31 Jul 2019 11:31:21 -0700 Subject: [PATCH 11/24] Fix typo in counter reason table --- doc/drop_counters/drop_counters_HLD.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index ee60a3043c9..6e23d885558 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -147,11 +147,11 @@ Example: } ``` -### 3.2.3 PACKET_DROP_COUNTER_RULE Table +### 3.2.3 PACKET_DROP_COUNTER_REASON Table Example: ``` { - "PACKET_DROP_COUNTER_RULE": { + "PACKET_DROP_COUNTER_REASON": { "LEGAL_RX_DROPS|SMAC_EQUALS_DMAC": {}, "LEGAL_RX_DROPS|INGRESS_VLAN_FILTER": {}, "LEGAL_TX_DROPS|EGRESS_VLAN_FILTER": {} From 049b4942b29ee61b1325515232a8ff1f66ad6481 Mon Sep 17 00:00:00 2001 From: Danny Allen Date: Wed, 31 Jul 2019 14:28:55 -0700 Subject: [PATCH 12/24] Clarify example usage --- doc/drop_counters/drop_counters_HLD.md | 28 +++++++++++++------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index 6e23d885558..6a0fe2ff331 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -133,15 +133,15 @@ Example: ``` { "PACKET_DROP_COUNTER": { - "LEGAL_RX_DROPS": { + "RX_LEGIT": { "counter": "DEBUG_0", "type": "ingress", - "desc": "Legal RX pipeline drops" + "desc": "Legitimate RX pipeline drops" }, - "LEGAL_TX_DROPS": { + "TX_LEGIT": { "counter": "DEBUG_1", "type": "egress", - "desc": "Legal TX pipeline drops" + "desc": "Legitimate TX pipeline drops" } } } @@ -152,9 +152,9 @@ Example: ``` { "PACKET_DROP_COUNTER_REASON": { - "LEGAL_RX_DROPS|SMAC_EQUALS_DMAC": {}, - "LEGAL_RX_DROPS|INGRESS_VLAN_FILTER": {}, - "LEGAL_TX_DROPS|EGRESS_VLAN_FILTER": {} + "RX_LEGIT|SMAC_EQUALS_DMAC": {}, + "RX_LEGIT|INGRESS_VLAN_FILTER": {}, + "TX_LEGIT|EGRESS_VLAN_FILTER": {} } } ``` @@ -184,15 +184,15 @@ The CLI tool will provide the following functionality: ``` $ show drops - IFACE STATE RX_ERR RX_DRP RX_DISC RX_LEGAL TX_ERR TX_DRP TX_DISC TX_LEGAL + IFACE STATE RX_ERR RX_DRP RX_DISC RX_LEGIT TX_ERR TX_DRP TX_DISC TX_LEGIT ---------- ------- -------- -------- --------- ---------- -------- -------- --------- ---------- Ethernet0 U 0 0 1500 1500 0 0 0 0 Ethernet4 U 0 0 300 250 0 0 0 0 Ethernet8 U 0 0 0 0 0 0 0 0 Ethernet12 U 0 0 1200 400 0 0 0 0 -$ show drops --include "LEGAL" - IFACE STATE RX_LEGAL TX_LEGAL +$ show drops --include "LEGIT" + IFACE STATE RX_LEGIT TX_LEGIT ---------- ------- ---------- ---------- Ethernet0 U 0 0 Ethernet4 U 0 0 @@ -203,8 +203,8 @@ Ethernet12 U 0 0 ### 3.5.2 CLI clear ``` $ sonic-clear drops -$ sonic-clear drops RX_LEGAL -$ sonic-clear drops RX_LEGAL TX_LEGAL +$ sonic-clear drops RX_LEGIT +$ sonic-clear drops RX_LEGIT TX_LEGIT ``` ### 3.5.3 CLI Configuration @@ -215,9 +215,9 @@ Available Counters: 3 Name Type Reasons Description -------- ------- ------------------- -------------- -RX_LEGAL ingress SMAC_EQUALS_DMAC Legal RX pipeline drops +RX_LEGIT ingress SMAC_EQUALS_DMAC Legitimate RX pipeline drops INGRESS_VLAN_FILTER -TX_LEGAL egress EGRESS_VLAN_FILTER Legal TX pipeline drops +TX_LEGIT egress EGRESS_VLAN_FILTER Legitimate TX pipeline drops DEBUG_2 OPEN NONE Available debug counter $ config drops init --counter=DEBUG_2 --name=EXAMPLE --type=ingress --desc="example" From 9b9d1d964b2d184fdd25b2a3d16f34703d824d0c Mon Sep 17 00:00:00 2001 From: Danny Allen Date: Tue, 6 Aug 2019 10:29:49 -0700 Subject: [PATCH 13/24] Address open question --- doc/drop_counters/drop_counters_HLD.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index 6a0fe2ff331..81ec3c9653f 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -274,4 +274,5 @@ The debug counts orchagent will restore counter configurations after a warm rebo A separate test plan will be uploaded and reviewed by the community. # 7 Open Questions -* There's still an open question in the SAI proposal over whether counter indices will be managed by the SAI or the user. This doc is currently written as if the user is managing the indices, but if this changes then the design (specifically, the Config DB schema) will need to be revised. \ No newline at end of file +* There's still an open question in the SAI proposal over whether counter indices will be managed by the SAI or the user. This doc is currently written as if the user is managing the indices, but if this changes then the design (specifically, the Config DB schema) will need to be revised. + - Update: This has been addressed and counters are SAI managed. We'll stick with the current design because it still may be useful to have a table of configured counters in the future in case more counter types are added. \ No newline at end of file From 57f83d1f9a50a57bdfe53767fd4c7100a2582f4f Mon Sep 17 00:00:00 2001 From: Danny Allen Date: Wed, 7 Aug 2019 12:28:27 -0700 Subject: [PATCH 14/24] Add description of SAI APIs used --- doc/drop_counters/drop_counters_HLD.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index 81ec3c9653f..69257f918f0 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -23,6 +23,7 @@ - [3.2.2 PACKET_DROP_COUNTER Table](#322-packet_drop_counter-table) - [3.3 App DB](#33-app-db) - [3.4 SWSS](#34-swss) + - [3.4.1 SAI APIs](#341-sai-apis) - [3.5 CLI](#35-CLI) - [3.5.1 CLI show](#351-cli-show) - [3.5.2 CLI clear](#352-cli-clear) @@ -166,10 +167,15 @@ App DB will store information about: * What drop reasons are supported by this device ## 3.4 SWSS -Portorch should be extended to support a variable number of SAI_PORT_STAT_IN/OUT_DROP_REASON counters. - Debugcountsorch should be implemented to handle debug counter creation and configuration. +### 3.4.1 SAI APIs +This orchestrator will interact with the following SAI APIs (all via sai_debug_counter_api_t): +* `sai_create_debug_counter_fn` to create/configure new drop counters. +* `sai_remove_debug_counter_fn` to delete/free up drop counters that are no longer being used. +* `sai_get_debug_counter_attribute_fn` to gather information about counters that have been configured (e.g. index, drop reasons, etc.). +* `sai_set_debug_counter_attribute_fn` to re-configure drop reasons for counters that have already been created. + ## 3.5 CLI The CLI tool will provide the following functionality: * Show drop counts: ```show drops``` From 3cf6c2f25676d1c322afe305487e320d35b50d5f Mon Sep 17 00:00:00 2001 From: Danny Allen Date: Fri, 9 Aug 2019 16:36:17 -0700 Subject: [PATCH 15/24] Simplify config schema --- doc/drop_counters/drop_counters_HLD.md | 70 ++++++++++++++------------ 1 file changed, 37 insertions(+), 33 deletions(-) diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index 69257f918f0..be0b94f6170 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -90,6 +90,10 @@ Configuration of the drop counters can be done via: * CLI * JSON input +At this stage, allocation of the debug counters will be restricted to config_db.json updates or minigraph updates. We may revise this in the future as more debug reasons are added. + +Debug counters that have been allocated for counting packet drops can be incrementally configured via the command line, which will be described later. + ## 2.3 Scalability Requirements Users must be able to use all counters and drop reasons provided by the underlying hardware. @@ -101,7 +105,7 @@ The contents of the drop counters will be added to Counters DB by flex counters. ## 3.2 Config DB We'll add three new tables to Config DB: * DEBUG_COUNTER to track which counters have been configured and for what purpose - * At this point the only supported type is PACKET_DROP + * At this point the only supported type is PACKET_DROP_COUNTER * PACKET_DROP_COUNTER to save drop counters that have been configured by the user * PACKET_DROP_COUNTER_RULE to save rules that are associated with user configured drop counters @@ -111,19 +115,13 @@ Example: { "DEBUG_COUNTER": { "DEBUG_0": { - "configured": true, - "type": "PACKET_DROP" + "type": "PACKET_DROP_COUNTER" }, "DEBUG_1": { - "configured": true, - "type": "PACKET_DROP" + "type": "PACKET_DROP_COUNTER" }, "DEBUG_2": { - "configured": true, - "type": "PACKET_DROP" - }, - "DEBUG_3": { - "configured": false + "type": "PACKET_DROP_COUNTER" } } } @@ -136,12 +134,12 @@ Example: "PACKET_DROP_COUNTER": { "RX_LEGIT": { "counter": "DEBUG_0", - "type": "ingress", + "type": "PORT_INGRESS", "desc": "Legitimate RX pipeline drops" }, "TX_LEGIT": { "counter": "DEBUG_1", - "type": "egress", + "type": "PORT_EGRESS", "desc": "Legitimate TX pipeline drops" } } @@ -197,7 +195,7 @@ $ show drops Ethernet8 U 0 0 0 0 0 0 0 0 Ethernet12 U 0 0 1200 400 0 0 0 0 -$ show drops --include "LEGIT" +$ show drops --contains "LEGIT" IFACE STATE RX_LEGIT TX_LEGIT ---------- ------- ---------- ---------- Ethernet0 U 0 0 @@ -209,8 +207,6 @@ Ethernet12 U 0 0 ### 3.5.2 CLI clear ``` $ sonic-clear drops -$ sonic-clear drops RX_LEGIT -$ sonic-clear drops RX_LEGIT TX_LEGIT ``` ### 3.5.3 CLI Configuration @@ -219,37 +215,45 @@ $ show drops config Drop Counters: supported Available Counters: 3 -Name Type Reasons Description --------- ------- ------------------- -------------- -RX_LEGIT ingress SMAC_EQUALS_DMAC Legitimate RX pipeline drops - INGRESS_VLAN_FILTER -TX_LEGIT egress EGRESS_VLAN_FILTER Legitimate TX pipeline drops -DEBUG_2 OPEN NONE Available debug counter +Name Type Reasons Description +-------- ------------ ------------------- -------------- +RX_LEGIT PORT_INGRESS SMAC_EQUALS_DMAC Legitimate RX pipeline drops + INGRESS_VLAN_FILTER +TX_LEGIT PORT_EGRESS EGRESS_VLAN_FILTER Legitimate TX pipeline drops +DEBUG_2 OPEN NONE Available drop counter -$ config drops init --counter=DEBUG_2 --name=EXAMPLE --type=ingress --desc="example" +$ config drops init --counter="DEBUG_2" --name="EXAMPLE" --type="PORT_INGRESS" --desc="example" Initializing DEBUG_2 as EXAMPLE... DONE! -Name Type Reasons Description --------- ------- ------------------- -------------- -EXAMPLE ingress NONE example +Name Type Reasons Description +-------- ------------ ------------------- -------------- +EXAMPLE PORT_INGRESS NONE example + +$ config drops add --counter=EXAMPLE --reason="SMAC_MULTICAST" +Configuring EXAMPLE... +DONE! + +Name Type Reasons Description +-------- ------------ ------------------- -------------- +EXAMPLE PORT_INGRESS SMAC_MULTICAST example -$ config drops add --counter=EXAMPLE --reasons="SMAC_MULTICAST,DMAC_RESERVED" +$ config drops add --counter=EXAMPLE --reason="DMAC_RESERVED" Configuring EXAMPLE... DONE! -Name Type Reasons Description --------- ------- ------------------- -------------- -EXAMPLE ingress SMAC_MULTICAST example +Name Type Reasons Description +-------- ------------ ------------------- -------------- +EXAMPLE PORT_INGRESS SMAC_MULTICAST example DMAC_RESERVED -$ config drops remove --counter=EXAMPLE --reasons="DMAC_RESERVED" +$ config drops remove --counter=EXAMPLE --reason="DMAC_RESERVED" Configuring EXAMPLE... DONE! -Name Type Reasons Description --------- ------- ------------------- -------------- -EXAMPLE ingress SMAC_MULTICAST example +Name Type Reasons Description +-------- ------------ ------------------- -------------- +EXAMPLE PORT_INGRESS SMAC_MULTICAST example $ config drops delete --counter=EXAMPLE Deleting EXAMPLE... From f80ab71988743866453d5b91fe072deb4eb2bcfc Mon Sep 17 00:00:00 2001 From: Danny Allen Date: Thu, 29 Aug 2019 12:15:39 -0700 Subject: [PATCH 16/24] Incorporate feedback --- doc/drop_counters/drop_counters_HLD.md | 47 +++++++++----------------- 1 file changed, 16 insertions(+), 31 deletions(-) diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index be0b94f6170..0d426ae9115 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -20,7 +20,7 @@ - [3.1 Counters DB](#31-counters-db) - [3.2 Config DB](#32-config-db) - [3.2.1 DEBUG_COUNTER Table](#321-debug_counter-table) - - [3.2.2 PACKET_DROP_COUNTER Table](#322-packet_drop_counter-table) + - [3.2.2 PACKET_DROP_COUNTER_REASON Table](#322-packet_drop_counter_reason-table) - [3.3 App DB](#33-app-db) - [3.4 SWSS](#34-swss) - [3.4.1 SAI APIs](#341-sai-apis) @@ -104,49 +104,29 @@ The contents of the drop counters will be added to Counters DB by flex counters. ## 3.2 Config DB We'll add three new tables to Config DB: -* DEBUG_COUNTER to track which counters have been configured and for what purpose - * At this point the only supported type is PACKET_DROP_COUNTER -* PACKET_DROP_COUNTER to save drop counters that have been configured by the user -* PACKET_DROP_COUNTER_RULE to save rules that are associated with user configured drop counters +* DEBUG_COUNTER to save debug counters have been configured by the user +* PACKET_DROP_COUNTER_REASON to save reasons that are associated with user configured packet drop counters ### 3.2.1 DEBUG_COUNTER Table Example: ``` -{ - "DEBUG_COUNTER": { - "DEBUG_0": { - "type": "PACKET_DROP_COUNTER" - }, - "DEBUG_1": { - "type": "PACKET_DROP_COUNTER" - }, - "DEBUG_2": { - "type": "PACKET_DROP_COUNTER" - } - } -} -``` - -### 3.2.2 PACKET_DROP_COUNTER Table -Example: -``` { "PACKET_DROP_COUNTER": { - "RX_LEGIT": { - "counter": "DEBUG_0", - "type": "PORT_INGRESS", + "DEBUG_0": { + "alias": "RX_LEGIT", + "type": "PORT_INGRESS_DROPS", "desc": "Legitimate RX pipeline drops" }, - "TX_LEGIT": { - "counter": "DEBUG_1", - "type": "PORT_EGRESS", + "DEBUG_1": { + "counter": "TX_LEGIT", + "type": "PORT_EGRESS_DROPS", "desc": "Legitimate TX pipeline drops" } } } ``` -### 3.2.3 PACKET_DROP_COUNTER_REASON Table +### 3.2.2 PACKET_DROP_COUNTER_REASON Table Example: ``` { @@ -164,11 +144,16 @@ App DB will store information about: * How many drop counters are available on this device * What drop reasons are supported by this device +### 3.3.1 SAI APIs +We will use the following SAI APIs to get this information: +* `sai_query_attribute_enum_values_capability` to query support for different types of counters +* `sai_object_type_get_availability` to query the amount of available debug counters + ## 3.4 SWSS Debugcountsorch should be implemented to handle debug counter creation and configuration. ### 3.4.1 SAI APIs -This orchestrator will interact with the following SAI APIs (all via sai_debug_counter_api_t): +This orchestrator will interact with the following SAI Debug Counter APIs: * `sai_create_debug_counter_fn` to create/configure new drop counters. * `sai_remove_debug_counter_fn` to delete/free up drop counters that are no longer being used. * `sai_get_debug_counter_attribute_fn` to gather information about counters that have been configured (e.g. index, drop reasons, etc.). From f77bfce6da2d6a5d65d7e0f088cb80fb77d457e3 Mon Sep 17 00:00:00 2001 From: Danny Allen Date: Thu, 29 Aug 2019 12:16:28 -0700 Subject: [PATCH 17/24] Fix TOC --- doc/drop_counters/drop_counters_HLD.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index 0d426ae9115..aa1c2d3c200 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -22,6 +22,7 @@ - [3.2.1 DEBUG_COUNTER Table](#321-debug_counter-table) - [3.2.2 PACKET_DROP_COUNTER_REASON Table](#322-packet_drop_counter_reason-table) - [3.3 App DB](#33-app-db) + - [3.3.1 SAI APIs](#331-sai-apis) - [3.4 SWSS](#34-swss) - [3.4.1 SAI APIs](#341-sai-apis) - [3.5 CLI](#35-CLI) From eea7e3e13910ef759bb188c32202f6edc7eba233 Mon Sep 17 00:00:00 2001 From: Danny Allen Date: Tue, 3 Sep 2019 23:39:09 -0700 Subject: [PATCH 18/24] Add details for switch-level counters --- doc/drop_counters/drop_counters_HLD.md | 157 +++++++++++++------------ 1 file changed, 83 insertions(+), 74 deletions(-) diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index aa1c2d3c200..d49496fdd68 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -11,7 +11,6 @@ * [Scope](#scope) * [Defintions/Abbreviation](#definitionsabbreviation) * [1 Overview](#1-overview) - - [1.1 Types of Drop Counters](#11-types-of-drop-counters) * [2 Requirements](#2-requirements) - [2.1 Functional Requirements](#21-functional-requirements) - [2.2 Configuration and Management Requirements](#2.2-configuration-and-management-requirements) @@ -21,14 +20,15 @@ - [3.2 Config DB](#32-config-db) - [3.2.1 DEBUG_COUNTER Table](#321-debug_counter-table) - [3.2.2 PACKET_DROP_COUNTER_REASON Table](#322-packet_drop_counter_reason-table) - - [3.3 App DB](#33-app-db) + - [3.3 State DB](#33-state-db) - [3.3.1 SAI APIs](#331-sai-apis) - [3.4 SWSS](#34-swss) - [3.4.1 SAI APIs](#341-sai-apis) - - [3.5 CLI](#35-CLI) - - [3.5.1 CLI show](#351-cli-show) - - [3.5.2 CLI clear](#352-cli-clear) - - [3.5.3 CLI configuration](#353-cli-configuration) + - [3.5 syncd](#35-syncd) + - [3.6 CLI](#36-CLI) + - [3.6.1 CLI show](#361-cli-show) + - [3.6.2 CLI clear](#362-cli-clear) + - [3.6.3 CLI configuration](#363-cli-configuration) * [4 Flows](#4-flows) - [4.1 General Flow](#41-general-flow) * [5 Warm Reboot Support](#5-warm-reboot-support) @@ -48,6 +48,7 @@ | Rev | Date | Author | Change Description | |:---:|:--------:|:-----------:|--------------------| | 0.1 | 07/30/19 | Danny Allen | Initial version | +| 0.2 | 09/03/19 | Danny Allen | Review updates | # About this Manual This document provides an overview of the implementation of configurable packet drop counters in SONiC. @@ -64,23 +65,11 @@ This document describes the high level design of the configurable drop counter f # 1 Overview The goal of this feature is to provide better packet drop visibility in SONiC by providing a mechanism to count and classify packet drops that occur due to different reasons. Because different types of packet drops are important to track in different use cases, it is also key for this feature to be easily configurable. -## 1.1 Types of Drop Counters -| Drop Category | SAI Counter | Configurable? | -|------------------------------|----------------------------------------------|:-------------:| -| RX Layer-2 packet corruption | SAI_PORT_STAT_IF_IN_ERRORS | No | -| TX Layer-2 packet corruption | SAI_PORT_STAT_IF_OUT_ERRORS | No | -| RX MMU packet drop | SAI_PORT_STAT_IF_IN_DROPPED_PKTS | No | -| TX MMU packet drop | SAI_PORT_STAT_IF_OUT_DROPPED_PKTS | No | -| All RX pipeline drops | SAI_PORT_STAT_IF_IN_DISCARDS | No | -| All TX pipeline drops | SAI_PORT_STAT_IF_OUT_DISCARDS | No | -| Specified RX pipeline drops | SAI_DEBUG_COUNTER_TYPE_PORT_IN_DROP_REASONS | Yes | -| Specified TX pipeline drops | SAI_DEBUG_COUNTER_TYPE_PORT_OUT_DROP_REASONS | Yes | - # 2 Requirements ## 2.1 Functional Requirements 1. Users can configure debug counters to track one or more drop reason(s). - 1. Supported drop reasons will be available in App DB + 1. Supported drop reasons will be available in State DB 2. Users can access debug counter information via a CLI command 1. Users can see what types of drops each counter contains 2. Users can assign aliases to counters @@ -91,10 +80,6 @@ Configuration of the drop counters can be done via: * CLI * JSON input -At this stage, allocation of the debug counters will be restricted to config_db.json updates or minigraph updates. We may revise this in the future as more debug reasons are added. - -Debug counters that have been allocated for counting packet drops can be incrementally configured via the command line, which will be described later. - ## 2.3 Scalability Requirements Users must be able to use all counters and drop reasons provided by the underlying hardware. @@ -104,24 +89,29 @@ Users must be able to use all counters and drop reasons provided by the underlyi The contents of the drop counters will be added to Counters DB by flex counters. ## 3.2 Config DB -We'll add three new tables to Config DB: -* DEBUG_COUNTER to save debug counters have been configured by the user -* PACKET_DROP_COUNTER_REASON to save reasons that are associated with user configured packet drop counters +Two new tables will be added to Config DB: +* DEBUG_COUNTER to store general debug counter metadata +* PACKET_DROP_COUNTER_REASON to store drop reasons for debug counters that have been configured to track packet drops ### 3.2.1 DEBUG_COUNTER Table Example: ``` { - "PACKET_DROP_COUNTER": { + "DEBUG_COUNTER": { "DEBUG_0": { - "alias": "RX_LEGIT", + "alias": "PORT_RX_LEGIT", "type": "PORT_INGRESS_DROPS", - "desc": "Legitimate RX pipeline drops" + "desc": "Legitimate port-level RX pipeline drops" }, "DEBUG_1": { - "counter": "TX_LEGIT", + "alias": "PORT_TX_LEGIT", "type": "PORT_EGRESS_DROPS", - "desc": "Legitimate TX pipeline drops" + "desc": "Legitimate port-level TX pipeline drops" + }, + "DEBUG_2": { + "alias": "SWITCH_RX_LEGIT", + "type": "SWITCH_INGRESS_DROPS", + "desc": "Legitimate switch-level RX pipeline drops" } } } @@ -132,15 +122,16 @@ Example: ``` { "PACKET_DROP_COUNTER_REASON": { - "RX_LEGIT|SMAC_EQUALS_DMAC": {}, - "RX_LEGIT|INGRESS_VLAN_FILTER": {}, - "TX_LEGIT|EGRESS_VLAN_FILTER": {} + "DEBUG_0|SMAC_EQUALS_DMAC": {}, + "DEBUG_0|INGRESS_VLAN_FILTER": {}, + "DEBUG_1|EGRESS_VLAN_FILTER": {}, + "DEBUG_2|TTL": {}, } } ``` -## 3.3 App DB -App DB will store information about: +## 3.3 State DB +State DB will store information about: * Whether drop counters are available on this device * How many drop counters are available on this device * What drop reasons are supported by this device @@ -151,7 +142,7 @@ We will use the following SAI APIs to get this information: * `sai_object_type_get_availability` to query the amount of available debug counters ## 3.4 SWSS -Debugcountsorch should be implemented to handle debug counter creation and configuration. +A new orchestrator will be created to handle debug counter creation and configuration. ### 3.4.1 SAI APIs This orchestrator will interact with the following SAI Debug Counter APIs: @@ -160,7 +151,10 @@ This orchestrator will interact with the following SAI Debug Counter APIs: * `sai_get_debug_counter_attribute_fn` to gather information about counters that have been configured (e.g. index, drop reasons, etc.). * `sai_set_debug_counter_attribute_fn` to re-configure drop reasons for counters that have already been created. -## 3.5 CLI +## 3.5 syncd +Flex counter will be extended to support switch-level SAI counters. + +## 3.6 CLI The CLI tool will provide the following functionality: * Show drop counts: ```show drops``` * Clear drop counters: ```sonic-clear drops``` @@ -170,10 +164,22 @@ The CLI tool will provide the following functionality: * Remove drop reasons from a drop counter: ```config drops remove``` * Delete a drop counter: ```config drops delete``` -### 3.5.1 CLI show +### 3.6.1 CLI show ``` $ show drops + IFACE STATE RX_ERR RX_DRP RX_DISC RX_LEGIT TX_ERR TX_DRP TX_DISC TX_LEGIT +--------------- ------- ---------- -------- --------- ---------- -------- -------- --------- ---------- + Ethernet0 U 0 0 1500 1500 0 0 0 0 + Ethernet4 U 0 0 300 250 0 0 0 0 + Ethernet8 U 0 0 0 0 0 0 0 0 + Ethernet12 U 0 0 1200 400 0 0 0 0 + + DEVICE STATE RX_LEGIT +--------------- ------- ---------- +ABCDEFG-123-XYZ U 2000 + +$ show drops --type=PORT IFACE STATE RX_ERR RX_DRP RX_DISC RX_LEGIT TX_ERR TX_DRP TX_DISC TX_LEGIT ---------- ------- -------- -------- --------- ---------- -------- -------- --------- ---------- Ethernet0 U 0 0 1500 1500 0 0 0 0 @@ -182,64 +188,68 @@ $ show drops Ethernet12 U 0 0 1200 400 0 0 0 0 $ show drops --contains "LEGIT" - IFACE STATE RX_LEGIT TX_LEGIT ----------- ------- ---------- ---------- - Ethernet0 U 0 0 - Ethernet4 U 0 0 - Ethernet8 U 0 0 -Ethernet12 U 0 0 + IFACE STATE RX_LEGIT TX_LEGIT +--------------- ------- ---------- ---------- + Ethernet0 U 0 0 + Ethernet4 U 0 0 + Ethernet8 U 0 0 + Ethernet12 U 0 0 + + DEVICE STATE RX_LEGIT +--------------- ------- ---------- +ABCDEFG-123-XYZ U 2000 ``` -### 3.5.2 CLI clear +### 3.6.2 CLI clear ``` $ sonic-clear drops ``` -### 3.5.3 CLI Configuration +### 3.6.3 CLI Configuration ``` $ show drops config Drop Counters: supported -Available Counters: 3 - -Name Type Reasons Description --------- ------------ ------------------- -------------- -RX_LEGIT PORT_INGRESS SMAC_EQUALS_DMAC Legitimate RX pipeline drops - INGRESS_VLAN_FILTER -TX_LEGIT PORT_EGRESS EGRESS_VLAN_FILTER Legitimate TX pipeline drops -DEBUG_2 OPEN NONE Available drop counter +Available Counters: 4 + +Name Type Reasons Description +-------- ------------ ------------------- -------------- +RX_LEGIT PORT_INGRESS SMAC_EQUALS_DMAC Legitimate port-level RX pipeline drops + INGRESS_VLAN_FILTER +TX_LEGIT PORT_EGRESS EGRESS_VLAN_FILTER Legitimate port-level TX pipeline drops +RX_LEGIT SWITCH_INGRESS TTL Legitimate switch-level RX pipeline drops -$ config drops init --counter="DEBUG_2" --name="EXAMPLE" --type="PORT_INGRESS" --desc="example" -Initializing DEBUG_2 as EXAMPLE... +$ config drops init --counter="DEBUG_3" --name="EXAMPLE" --type="SWITCH_EGRESS" --desc="example" +Initializing DEBUG_3 as EXAMPLE... DONE! -Name Type Reasons Description --------- ------------ ------------------- -------------- -EXAMPLE PORT_INGRESS NONE example +Name Type Reasons Description +-------- ------------- ------------------- -------------- +EXAMPLE SWITCH_EGRESS NONE example $ config drops add --counter=EXAMPLE --reason="SMAC_MULTICAST" Configuring EXAMPLE... DONE! -Name Type Reasons Description --------- ------------ ------------------- -------------- -EXAMPLE PORT_INGRESS SMAC_MULTICAST example +Name Type Reasons Description +-------- ------------- ------------------- -------------- +EXAMPLE SWITCH_EGRESS SMAC_MULTICAST example $ config drops add --counter=EXAMPLE --reason="DMAC_RESERVED" Configuring EXAMPLE... DONE! -Name Type Reasons Description --------- ------------ ------------------- -------------- -EXAMPLE PORT_INGRESS SMAC_MULTICAST example - DMAC_RESERVED +Name Type Reasons Description +-------- ------------- ------------------- -------------- +EXAMPLE SWITCH_EGRESS SMAC_MULTICAST example + DMAC_RESERVED $ config drops remove --counter=EXAMPLE --reason="DMAC_RESERVED" Configuring EXAMPLE... DONE! -Name Type Reasons Description --------- ------------ ------------------- -------------- -EXAMPLE PORT_INGRESS SMAC_MULTICAST example +Name Type Reasons Description +-------- ------------- ------------------- -------------- +EXAMPLE SWITCH_EGRESS SMAC_MULTICAST example $ config drops delete --counter=EXAMPLE Deleting EXAMPLE... @@ -251,7 +261,7 @@ DONE! ![alt text](./drop_counters_general_flow.png) The overall workflow is shown above in figure 1. -(1) Users configure drop counters using the CLI. Configurations are stored in the PACKET_DROP_COUNTER Config DB table. +(1) Users configure drop counters using the CLI. Configurations are stored in the DEBUG_COUNTER Config DB table. (2) The debug counts orchagent subscribes to the Config DB table. Once the configuration changes, the orchagent uses the debug SAI API to configure the drop counters. @@ -261,7 +271,7 @@ The overall workflow is shown above in figure 1. (5) CLI uses counters DB to satisfy CLI requests. -(6) (not shown) CLI uses App DB to display hardware capabilities (e.g. how many counters are available, supported drop reasons, etc.) +(6) (not shown) CLI uses State DB to display hardware capabilities (e.g. how many counters are available, supported drop reasons, etc.) # 5 Warm Reboot Support The debug counts orchagent will restore counter configurations after a warm reboot. It will get the current configuration from the SAI and check this against the configuration saved in config DB, reconfiguring the debug counters if necessary. @@ -270,5 +280,4 @@ The debug counts orchagent will restore counter configurations after a warm rebo A separate test plan will be uploaded and reviewed by the community. # 7 Open Questions -* There's still an open question in the SAI proposal over whether counter indices will be managed by the SAI or the user. This doc is currently written as if the user is managing the indices, but if this changes then the design (specifically, the Config DB schema) will need to be revised. - - Update: This has been addressed and counters are SAI managed. We'll stick with the current design because it still may be useful to have a table of configured counters in the future in case more counter types are added. \ No newline at end of file +None at this time. \ No newline at end of file From 768bb4a64cdc7a1de878cd994b6e7441c1724251 Mon Sep 17 00:00:00 2001 From: Danny Allen Date: Wed, 4 Sep 2019 00:31:53 -0700 Subject: [PATCH 19/24] Add details --- doc/drop_counters/drop_counters_HLD.md | 85 ++++++++++++++++---------- 1 file changed, 53 insertions(+), 32 deletions(-) diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index d49496fdd68..2ece410f020 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -1,7 +1,7 @@ # Configurable Drop Counters in SONiC # High Level Design Document -#### Rev 0.1 +#### Rev 0.2 # Table of Contents * [List of Tables](#list-of-tables) @@ -15,16 +15,17 @@ - [2.1 Functional Requirements](#21-functional-requirements) - [2.2 Configuration and Management Requirements](#2.2-configuration-and-management-requirements) - [2.3 Scalability Requirements](#23-scalability-requirements) + - [2.4 Supported Debug Counters](#24-supported-debug-counters) * [3 Design](#3-design) - - [3.1 Counters DB](#31-counters-db) - - [3.2 Config DB](#32-config-db) - - [3.2.1 DEBUG_COUNTER Table](#321-debug_counter-table) - - [3.2.2 PACKET_DROP_COUNTER_REASON Table](#322-packet_drop_counter_reason-table) - - [3.3 State DB](#33-state-db) + - [3.1 Config DB](#31-config-db) + - [3.1.1 DEBUG_COUNTER Table](#311-debug_counter-table) + - [3.1.2 PACKET_DROP_COUNTER_REASON Table](#312-packet_drop_counter_reason-table) + - [3.2 State DB](#32-state-db) + - [3.2.1 SAI APIs](#321-sai-apis) + - [3.3 SWSS](#33-swss) - [3.3.1 SAI APIs](#331-sai-apis) - - [3.4 SWSS](#34-swss) - - [3.4.1 SAI APIs](#341-sai-apis) - - [3.5 syncd](#35-syncd) + - [3.4 syncd](#34-syncd) + - [3.5 Counters DB](#35-counters-db) - [3.6 CLI](#36-CLI) - [3.6.1 CLI show](#361-cli-show) - [3.6.2 CLI clear](#362-cli-clear) @@ -65,15 +66,23 @@ This document describes the high level design of the configurable drop counter f # 1 Overview The goal of this feature is to provide better packet drop visibility in SONiC by providing a mechanism to count and classify packet drops that occur due to different reasons. Because different types of packet drops are important to track in different use cases, it is also key for this feature to be easily configurable. +We will do this by adding support for SAI debug counters in SONiC. Support for creating and configuring port-level and switch-level debug counters will be added to orchagent and syncd. We will also provide a CLI tool for users create these counters to track drop reasons. + # 2 Requirements ## 2.1 Functional Requirements -1. Users can configure debug counters to track one or more drop reason(s). - 1. Supported drop reasons will be available in State DB -2. Users can access debug counter information via a CLI command - 1. Users can see what types of drops each counter contains - 2. Users can assign aliases to counters - 3. Users can clear counters +1. CONFIG_DB can be configured to create debug counters +2. STATE_DB can be queried for debug counter capabilities +3. Users can access drop counter information via a CLI tool + 1. Users can see what capabilities are available to them + 1. Types of counters (i.e. port-level and/or switch-level) + 2. Number of counters + 3. Supported drop reasons + 2. Users can see what types of drops each configured counter contains + 3. Users can add and remove drop reasons from each counter + 4. Users can read the current value of each counter + 5. Users can assign aliases to counters + 6. Users can clear counters ## 2.2 Configuration and Management Requirements Configuration of the drop counters can be done via: @@ -83,17 +92,20 @@ Configuration of the drop counters can be done via: ## 2.3 Scalability Requirements Users must be able to use all counters and drop reasons provided by the underlying hardware. -# 3 Design +## 2.4 Supported Debug Counters +* PORT_INGRESS_DROPS: port-level ingress drop counters +* PORT_EGRESS_DROPS: port-level egress drop counters +* SWITCH_INGRESS_DROPS: switch-level ingress drop counters +* SWITCH_EGRESS_DROPS: switch-level egress drop counters -## 3.1 Counters DB -The contents of the drop counters will be added to Counters DB by flex counters. +# 3 Design -## 3.2 Config DB +## 3.1 Config DB Two new tables will be added to Config DB: * DEBUG_COUNTER to store general debug counter metadata -* PACKET_DROP_COUNTER_REASON to store drop reasons for debug counters that have been configured to track packet drops +* DEBUG_COUNTER_DROP_REASON to store drop reasons for debug counters that have been configured to track packet drops -### 3.2.1 DEBUG_COUNTER Table +### 3.1.1 DEBUG_COUNTER Table Example: ``` { @@ -117,11 +129,11 @@ Example: } ``` -### 3.2.2 PACKET_DROP_COUNTER_REASON Table +### 3.1.2 DEBUG_COUNTER_DROP_REASON Table Example: ``` { - "PACKET_DROP_COUNTER_REASON": { + "DEBUG_COUNTER_DROP_REASON": { "DEBUG_0|SMAC_EQUALS_DMAC": {}, "DEBUG_0|INGRESS_VLAN_FILTER": {}, "DEBUG_1|EGRESS_VLAN_FILTER": {}, @@ -130,30 +142,39 @@ Example: } ``` -## 3.3 State DB +## 3.2 State DB State DB will store information about: * Whether drop counters are available on this device * How many drop counters are available on this device * What drop reasons are supported by this device -### 3.3.1 SAI APIs +This information will be populated by the orchestrator (described later) on startup. + +### 3.2.1 SAI APIs We will use the following SAI APIs to get this information: * `sai_query_attribute_enum_values_capability` to query support for different types of counters * `sai_object_type_get_availability` to query the amount of available debug counters -## 3.4 SWSS -A new orchestrator will be created to handle debug counter creation and configuration. +## 3.3 SWSS +A new orchestrator will be created to handle debug counter creation and configuration. Specifically, this orchestrator will support: +* Creating a new counter +* Deleting existing counters +* Adding drop reasons to an existing counter +* Removing a drop reason from a counter -### 3.4.1 SAI APIs +### 3.3.1 SAI APIs This orchestrator will interact with the following SAI Debug Counter APIs: * `sai_create_debug_counter_fn` to create/configure new drop counters. * `sai_remove_debug_counter_fn` to delete/free up drop counters that are no longer being used. * `sai_get_debug_counter_attribute_fn` to gather information about counters that have been configured (e.g. index, drop reasons, etc.). * `sai_set_debug_counter_attribute_fn` to re-configure drop reasons for counters that have already been created. -## 3.5 syncd +## 3.4 syncd Flex counter will be extended to support switch-level SAI counters. +## 3.5 Counters DB +The contents of the drop counters will be added to Counters DB by flex counters. + ## 3.6 CLI The CLI tool will provide the following functionality: * Show drop counts: ```show drops``` @@ -274,10 +295,10 @@ The overall workflow is shown above in figure 1. (6) (not shown) CLI uses State DB to display hardware capabilities (e.g. how many counters are available, supported drop reasons, etc.) # 5 Warm Reboot Support -The debug counts orchagent will restore counter configurations after a warm reboot. It will get the current configuration from the SAI and check this against the configuration saved in config DB, reconfiguring the debug counters if necessary. +At this stage, debug counters will be deleted prior to warm reboot and re-installed when orchagent starts back up. This is intended to simplify upgrade behavior and conserve hardware resources during the warm reboot. # 6 Unit Tests -A separate test plan will be uploaded and reviewed by the community. +A separate test plan will be uploaded and reviewed by the community. This will include both virtual switch tests to verify that ASIC_DB is configured correctly as well as pytest to verify overall system correctness. # 7 Open Questions -None at this time. \ No newline at end of file +- How common of an operation is configuring a drop counter? Is this something that will usually only be done on startup, or something people will be updating frequently? \ No newline at end of file From 11959b204f2ca2d06dff1d7e3ba204900e7b5013 Mon Sep 17 00:00:00 2001 From: Danny Allen Date: Thu, 5 Sep 2019 18:27:01 -0700 Subject: [PATCH 20/24] Clarify usage and design considerations --- doc/drop_counters/drop_counters_HLD.md | 311 ++++++++++++++----------- 1 file changed, 180 insertions(+), 131 deletions(-) diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index 2ece410f020..5f0fd4ad68e 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -17,19 +17,21 @@ - [2.3 Scalability Requirements](#23-scalability-requirements) - [2.4 Supported Debug Counters](#24-supported-debug-counters) * [3 Design](#3-design) - - [3.1 Config DB](#31-config-db) - - [3.1.1 DEBUG_COUNTER Table](#311-debug_counter-table) - - [3.1.2 PACKET_DROP_COUNTER_REASON Table](#312-packet_drop_counter_reason-table) - - [3.2 State DB](#32-state-db) - - [3.2.1 SAI APIs](#321-sai-apis) - - [3.3 SWSS](#33-swss) - - [3.3.1 SAI APIs](#331-sai-apis) - - [3.4 syncd](#34-syncd) - - [3.5 Counters DB](#35-counters-db) - - [3.6 CLI](#36-CLI) - - [3.6.1 CLI show](#361-cli-show) - - [3.6.2 CLI clear](#362-cli-clear) - - [3.6.3 CLI configuration](#363-cli-configuration) + - [3.1 CLI (and usage example)](#31-cli-and-usage-example) + - [3.1.1 Displaying available counter capabilities](#311-displaying-available-counter-capabilities) + - [3.1.2 Displaying current counter configuration](#312-displaying-current-counter-configuration) + - [3.1.3 Displaying the current counts](#313-displaying-the-current-counts) + - [3.1.4 Clearing the counts](#314-clearing-the-counts) + - [3.1.5 Configuring counters from the CLI](#315-configuring-counters-from-the-CLI) + - [3.2 Config DB](#32-config-db) + - [3.2.1 DEBUG_COUNTER Table](#321-debug_counter-table) + - [3.2.2 PACKET_DROP_COUNTER_REASON Table](#322-packet_drop_counter_reason-table) + - [3.3 State DB](#32-state-db) + - [3.3.1 SAI APIs](#321-sai-apis) + - [3.4 Counters DB](#34-counters-db) + - [3.5 SWSS](#35-swss) + - [3.5.1 SAI APIs](#351-sai-apis) + - [3.6 syncd](#34-syncd) * [4 Flows](#4-flows) - [4.1 General Flow](#41-general-flow) * [5 Warm Reboot Support](#5-warm-reboot-support) @@ -40,7 +42,6 @@ # List of Tables * [Table 1: Abbreviations](#definitionsabbreviation) -* [Table 2: Types of Drop Counters](#11-types-of-drop-counters) # List of Figures * [Figure 1: General Flow](#41-general-flow) @@ -64,9 +65,13 @@ This document describes the high level design of the configurable drop counter f | TX | Transmit/egress | # 1 Overview -The goal of this feature is to provide better packet drop visibility in SONiC by providing a mechanism to count and classify packet drops that occur due to different reasons. Because different types of packet drops are important to track in different use cases, it is also key for this feature to be easily configurable. +The main goal of this feature is to provide better packet drop visibility in SONiC by providing a mechanism to count and classify packet drops that occur due to different reasons. -We will do this by adding support for SAI debug counters in SONiC. Support for creating and configuring port-level and switch-level debug counters will be added to orchagent and syncd. We will also provide a CLI tool for users create these counters to track drop reasons. +The other goal of this feature is for users to be able to track the types of drop reasons that are important for their scenario. Because different users have different priorities, and because priorities change over time, it is important for this feature to be easily configurable. + +We will accomplish both goals by adding support for SAI debug counters to SONiC. +* Support for creating and configuring port-level and switch-level debug counters will be added to orchagent and syncd. +* A CLI tool will be provided for users to manage and configure their own drop counters # 2 Requirements @@ -86,8 +91,9 @@ We will do this by adding support for SAI debug counters in SONiC. Support for c ## 2.2 Configuration and Management Requirements Configuration of the drop counters can be done via: +* config_db.json +* minigraph.xml * CLI -* JSON input ## 2.3 Scalability Requirements Users must be able to use all counters and drop reasons provided by the underlying hardware. @@ -100,12 +106,157 @@ Users must be able to use all counters and drop reasons provided by the underlyi # 3 Design -## 3.1 Config DB +## 3.1 CLI (and usage example) +The CLI tool will provide the following functionality: +* See available drop counter capabilities: `show drops available` +* See drop counter config: `show drops config` +* Show drop counts: `show drops` +* Clear drop counters: `sonic-clear drops` +* Initialize a new drop counter: `config drops init` +* Add drop reasons to a drop counter: `config drops add` +* Remove drop reasons from a drop counter: `config drops remove` +* Delete a drop counter: `config drops delete` + +### 3.1.1 Displaying available counter capabilities +``` +$ show drops available + TYPE FREE IN-USE +-------------- ---- ------ + PORT_INGRESS 2 1 + PORT_EGRESS 2 1 +SWITCH_INGRESS 1 1 + SWITCH_EGRESS 2 0 + +$ show drops reasons +PORT_INGRESS: + L2_ANY + SMAC_MULTICAST + SMAC_EQUALS_DMAC + INGRESS_VLAN_FILTER + EXCEEDS_L2_MTU + SIP_CLASS_E + SIP_LINK_LOCAL + DIP_LINK_LOCAL + UNRESOLVED_NEXT_HOP + DECAP_ERROR + +PORT_EGRESS: + L2_ANY + L3_ANY + A_CUSTOM_REASON + +SWITCH_INGRESS: + L2_ANY + SMAC_MULTICAST + SMAC_EQUALS_DMAC + SIP_CLASS_E + SIP_LINK_LOCAL + DIP_LINK_LOCAL + +SWITCH_EGRESS: + L2_ANY + L3_ANY + A_CUSTOM_REASON + ANOTHER_CUSTOM_REASON + +$ show drops reasons --type=PORT_EGRESS +PORT_EGRESS: + L2_ANY + L3_ANY + A_CUSTOM_REASON + +``` + +### 3.1.2 Displaying current counter configuration +``` +$ show drops config +Counter Alias Type Reasons Description +-------- -------- -------------- ------------------- -------------- +DEBUG_0 RX_LEGIT PORT_INGRESS SMAC_EQUALS_DMAC Legitimate port-level RX pipeline drops + INGRESS_VLAN_FILTER +DEBUG_1 TX_LEGIT PORT_EGRESS EGRESS_VLAN_FILTER Legitimate port-level TX pipeline drops +DEBUG_2 RX_LEGIT SWITCH_INGRESS TTL Legitimate switch-level RX pipeline drops +``` + +### 3.1.3 Displaying the current counts + +``` +$ show drops + IFACE STATE RX_ERR RX_DRP RX_DISC RX_LEGIT TX_ERR TX_DRP TX_DISC TX_LEGIT +--------------- ------- ---------- -------- --------- ---------- -------- -------- --------- ---------- + Ethernet0 U 0 0 1500 1500 0 0 0 0 + Ethernet4 U 0 0 300 250 0 0 0 0 + Ethernet8 U 0 0 0 0 0 0 0 0 + Ethernet12 U 0 0 1200 400 0 0 0 0 + + DEVICE STATE RX_LEGIT +--------------- ------- ---------- +ABCDEFG-123-XYZ U 2000 + +$ show drops --type=PORT + IFACE STATE RX_ERR RX_DRP RX_DISC RX_LEGIT TX_ERR TX_DRP TX_DISC TX_LEGIT +---------- ------- -------- -------- --------- ---------- -------- -------- --------- ---------- + Ethernet0 U 0 0 1500 1500 0 0 0 0 + Ethernet4 U 0 0 300 250 0 0 0 0 + Ethernet8 U 0 0 0 0 0 0 0 0 +Ethernet12 U 0 0 1200 400 0 0 0 0 + +$ show drops --contains "LEGIT" + IFACE STATE RX_LEGIT TX_LEGIT +--------------- ------- ---------- ---------- + Ethernet0 U 0 0 + Ethernet4 U 0 0 + Ethernet8 U 0 0 + Ethernet12 U 0 0 + + DEVICE STATE RX_LEGIT +--------------- ------- ---------- +ABCDEFG-123-XYZ U 2000 +``` + +### 3.1.4 Clearing the counts +``` +$ sonic-clear drops +``` + +### 3.1.5 Configuring counters from the CLI +``` +$ config drops init --counter="DEBUG_3" --alias="EXAMPLE" --type="SWITCH_EGRESS" --desc="example" --reasons=["L2_ANY", "L3_ANY"] +Initializing DEBUG_3 as TX_LEGIT... + +Counter Alias Type Reasons Description +------- -------- ------------- ------- ----------- +DEBUG_3 TX_LEGIT SWITCH_EGRESS L2_ANY Legitimate switch-level TX pipeline drops + L3_ANY + +$ config drops add --counter="DEBUG_3" --reasons=["A_CUSTOM_REASON", "ANOTHER_CUSTOM_REASON"] +Configuring DEBUG_3... + +Counter Alias Type Reasons Description +------- -------- ------------- --------------------- ----------- +DEBUG_3 TX_LEGIT SWITCH_EGRESS L2_ANY Legitimate switch-level TX pipeline drops + L3_ANY + A_CUSTOM_REASON + ANOTHER_CUSTOM_REASON + +$ config drops remove --counter="DEBUG_3" --reasons=["A_CUSTOM_REASON"] +Configuring DEBUG_3... + +Counter Alias Type Reasons Description +------- -------- ------------- --------------------- ----------- +DEBUG_3 TX_LEGIT SWITCH_EGRESS L2_ANY Legitimate switch-level TX pipeline drops + L3_ANY + ANOTHER_CUSTOM_REASON + +$ config drops delete --counter="DEBUG_3" +``` + +## 3.2 Config DB Two new tables will be added to Config DB: * DEBUG_COUNTER to store general debug counter metadata * DEBUG_COUNTER_DROP_REASON to store drop reasons for debug counters that have been configured to track packet drops -### 3.1.1 DEBUG_COUNTER Table +### 3.2.1 DEBUG_COUNTER Table Example: ``` { @@ -129,7 +280,7 @@ Example: } ``` -### 3.1.2 DEBUG_COUNTER_DROP_REASON Table +### 3.2.2 DEBUG_COUNTER_DROP_REASON Table Example: ``` { @@ -142,141 +293,39 @@ Example: } ``` -## 3.2 State DB +## 3.3 State DB State DB will store information about: -* Whether drop counters are available on this device +* What types of drop counters are available on this device * How many drop counters are available on this device * What drop reasons are supported by this device This information will be populated by the orchestrator (described later) on startup. -### 3.2.1 SAI APIs +### 3.3.1 SAI APIs We will use the following SAI APIs to get this information: * `sai_query_attribute_enum_values_capability` to query support for different types of counters * `sai_object_type_get_availability` to query the amount of available debug counters -## 3.3 SWSS +## 3.4 Counters DB +The contents of the drop counters will be added to Counters DB by flex counters. + +## 3.5 SWSS A new orchestrator will be created to handle debug counter creation and configuration. Specifically, this orchestrator will support: * Creating a new counter * Deleting existing counters * Adding drop reasons to an existing counter * Removing a drop reason from a counter -### 3.3.1 SAI APIs +### 3.5.1 SAI APIs This orchestrator will interact with the following SAI Debug Counter APIs: * `sai_create_debug_counter_fn` to create/configure new drop counters. * `sai_remove_debug_counter_fn` to delete/free up drop counters that are no longer being used. * `sai_get_debug_counter_attribute_fn` to gather information about counters that have been configured (e.g. index, drop reasons, etc.). * `sai_set_debug_counter_attribute_fn` to re-configure drop reasons for counters that have already been created. -## 3.4 syncd +## 3.6 syncd Flex counter will be extended to support switch-level SAI counters. -## 3.5 Counters DB -The contents of the drop counters will be added to Counters DB by flex counters. - -## 3.6 CLI -The CLI tool will provide the following functionality: -* Show drop counts: ```show drops``` -* Clear drop counters: ```sonic-clear drops``` -* See drop counter config: ```show drops config``` -* Initialize a new drop counter: ```config drops init``` -* Add drop reasons to a drop counter: ```config drops add``` -* Remove drop reasons from a drop counter: ```config drops remove``` -* Delete a drop counter: ```config drops delete``` - -### 3.6.1 CLI show - -``` -$ show drops - IFACE STATE RX_ERR RX_DRP RX_DISC RX_LEGIT TX_ERR TX_DRP TX_DISC TX_LEGIT ---------------- ------- ---------- -------- --------- ---------- -------- -------- --------- ---------- - Ethernet0 U 0 0 1500 1500 0 0 0 0 - Ethernet4 U 0 0 300 250 0 0 0 0 - Ethernet8 U 0 0 0 0 0 0 0 0 - Ethernet12 U 0 0 1200 400 0 0 0 0 - - DEVICE STATE RX_LEGIT ---------------- ------- ---------- -ABCDEFG-123-XYZ U 2000 - -$ show drops --type=PORT - IFACE STATE RX_ERR RX_DRP RX_DISC RX_LEGIT TX_ERR TX_DRP TX_DISC TX_LEGIT ----------- ------- -------- -------- --------- ---------- -------- -------- --------- ---------- - Ethernet0 U 0 0 1500 1500 0 0 0 0 - Ethernet4 U 0 0 300 250 0 0 0 0 - Ethernet8 U 0 0 0 0 0 0 0 0 -Ethernet12 U 0 0 1200 400 0 0 0 0 - -$ show drops --contains "LEGIT" - IFACE STATE RX_LEGIT TX_LEGIT ---------------- ------- ---------- ---------- - Ethernet0 U 0 0 - Ethernet4 U 0 0 - Ethernet8 U 0 0 - Ethernet12 U 0 0 - - DEVICE STATE RX_LEGIT ---------------- ------- ---------- -ABCDEFG-123-XYZ U 2000 -``` - -### 3.6.2 CLI clear -``` -$ sonic-clear drops -``` - -### 3.6.3 CLI Configuration -``` -$ show drops config -Drop Counters: supported -Available Counters: 4 - -Name Type Reasons Description --------- ------------ ------------------- -------------- -RX_LEGIT PORT_INGRESS SMAC_EQUALS_DMAC Legitimate port-level RX pipeline drops - INGRESS_VLAN_FILTER -TX_LEGIT PORT_EGRESS EGRESS_VLAN_FILTER Legitimate port-level TX pipeline drops -RX_LEGIT SWITCH_INGRESS TTL Legitimate switch-level RX pipeline drops - -$ config drops init --counter="DEBUG_3" --name="EXAMPLE" --type="SWITCH_EGRESS" --desc="example" -Initializing DEBUG_3 as EXAMPLE... -DONE! - -Name Type Reasons Description --------- ------------- ------------------- -------------- -EXAMPLE SWITCH_EGRESS NONE example - -$ config drops add --counter=EXAMPLE --reason="SMAC_MULTICAST" -Configuring EXAMPLE... -DONE! - -Name Type Reasons Description --------- ------------- ------------------- -------------- -EXAMPLE SWITCH_EGRESS SMAC_MULTICAST example - -$ config drops add --counter=EXAMPLE --reason="DMAC_RESERVED" -Configuring EXAMPLE... -DONE! - -Name Type Reasons Description --------- ------------- ------------------- -------------- -EXAMPLE SWITCH_EGRESS SMAC_MULTICAST example - DMAC_RESERVED - -$ config drops remove --counter=EXAMPLE --reason="DMAC_RESERVED" -Configuring EXAMPLE... -DONE! - -Name Type Reasons Description --------- ------------- ------------------- -------------- -EXAMPLE SWITCH_EGRESS SMAC_MULTICAST example - -$ config drops delete --counter=EXAMPLE -Deleting EXAMPLE... -DONE! -``` - # 4 Flows ## 4.1 General Flow ![alt text](./drop_counters_general_flow.png) From be27435dc9590a79ed6c116a74b6ff2ec4df1f6e Mon Sep 17 00:00:00 2001 From: Danny Allen Date: Tue, 10 Sep 2019 07:53:37 -0700 Subject: [PATCH 21/24] Fix usage example --- doc/drop_counters/drop_counters_HLD.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index 5f0fd4ad68e..587113c5f82 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -221,7 +221,7 @@ $ sonic-clear drops ### 3.1.5 Configuring counters from the CLI ``` -$ config drops init --counter="DEBUG_3" --alias="EXAMPLE" --type="SWITCH_EGRESS" --desc="example" --reasons=["L2_ANY", "L3_ANY"] +$ config drops init --counter="DEBUG_3" --alias="TX_LEGIT" --type="SWITCH_EGRESS" --desc="Legitimate switch-level TX pipeline drops" --reasons=["L2_ANY", "L3_ANY"] Initializing DEBUG_3 as TX_LEGIT... Counter Alias Type Reasons Description From 15a9508fa72cde5a49090641b114513688de8aef Mon Sep 17 00:00:00 2001 From: Danny Allen Date: Wed, 11 Sep 2019 13:03:40 -0700 Subject: [PATCH 22/24] Incorporate community feedback --- doc/drop_counters/drop_counters_HLD.md | 30 +++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-) diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index 587113c5f82..f2e4815dc4a 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -26,8 +26,9 @@ - [3.2 Config DB](#32-config-db) - [3.2.1 DEBUG_COUNTER Table](#321-debug_counter-table) - [3.2.2 PACKET_DROP_COUNTER_REASON Table](#322-packet_drop_counter_reason-table) - - [3.3 State DB](#32-state-db) - - [3.3.1 SAI APIs](#321-sai-apis) + - [3.3 State DB](#33-state-db) + - [3.3.1 DEBUG_COUNTER_CAPABILITIES Table](#331-debug-counter-capabilities-table) + - [3.3.2 SAI APIs](#332-sai-apis) - [3.4 Counters DB](#34-counters-db) - [3.5 SWSS](#35-swss) - [3.5.1 SAI APIs](#351-sai-apis) @@ -96,7 +97,9 @@ Configuration of the drop counters can be done via: * CLI ## 2.3 Scalability Requirements -Users must be able to use all counters and drop reasons provided by the underlying hardware. +Users must be able to use all debug counters and drop reasons provided by the underlying hardware. + +Interacting with debug counters will not interfere with existing hardware counters (e.g. portstat). Likewise, interacting with existing hardware counters will not interfere with debug counter behavior. ## 2.4 Supported Debug Counters * PORT_INGRESS_DROPS: port-level ingress drop counters @@ -299,9 +302,26 @@ State DB will store information about: * How many drop counters are available on this device * What drop reasons are supported by this device +### 3.3.1 DEBUG_COUNTER_CAPABILITIES Table +Example: +``` +{ + "DEBUG_COUNTER_CAPABILITIES": { + "SWITCH_INGRESS_DROPS": { + "count": 2, + "reasons": [L2_ANY, L3_ANY, SMAC_EQUALS_DMAC] + }, + "SWITCH_EGRESS_DROPS": { + "count": 2, + "reasons": [L2_ANY, L3_ANY] + } + } +} +``` + This information will be populated by the orchestrator (described later) on startup. -### 3.3.1 SAI APIs +### 3.3.2 SAI APIs We will use the following SAI APIs to get this information: * `sai_query_attribute_enum_values_capability` to query support for different types of counters * `sai_object_type_get_availability` to query the amount of available debug counters @@ -344,7 +364,7 @@ The overall workflow is shown above in figure 1. (6) (not shown) CLI uses State DB to display hardware capabilities (e.g. how many counters are available, supported drop reasons, etc.) # 5 Warm Reboot Support -At this stage, debug counters will be deleted prior to warm reboot and re-installed when orchagent starts back up. This is intended to simplify upgrade behavior and conserve hardware resources during the warm reboot. +On resource-constrained platforms, debug counters will be deleted prior to warm reboot and re-installed when orchagent starts back up. This is intended to conserve hardware resources during the warm reboot. # 6 Unit Tests A separate test plan will be uploaded and reviewed by the community. This will include both virtual switch tests to verify that ASIC_DB is configured correctly as well as pytest to verify overall system correctness. From a01af8fdd45475b0942791b3da6f10465ed157db Mon Sep 17 00:00:00 2001 From: Danny Allen Date: Fri, 13 Sep 2019 16:58:48 -0700 Subject: [PATCH 23/24] Add support for groups of counters and querying counter DB --- doc/drop_counters/drop_counters_HLD.md | 53 ++++++++++++++------------ 1 file changed, 29 insertions(+), 24 deletions(-) diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index f2e4815dc4a..61249d0e814 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -173,12 +173,12 @@ PORT_EGRESS: ### 3.1.2 Displaying current counter configuration ``` $ show drops config -Counter Alias Type Reasons Description --------- -------- -------------- ------------------- -------------- -DEBUG_0 RX_LEGIT PORT_INGRESS SMAC_EQUALS_DMAC Legitimate port-level RX pipeline drops - INGRESS_VLAN_FILTER -DEBUG_1 TX_LEGIT PORT_EGRESS EGRESS_VLAN_FILTER Legitimate port-level TX pipeline drops -DEBUG_2 RX_LEGIT SWITCH_INGRESS TTL Legitimate switch-level RX pipeline drops +Counter Alias Group Type Reasons Description +-------- -------- ----- -------------- ------------------- -------------- +DEBUG_0 RX_LEGIT LEGIT PORT_INGRESS SMAC_EQUALS_DMAC Legitimate port-level RX pipeline drops + INGRESS_VLAN_FILTER +DEBUG_1 TX_LEGIT LEGIT PORT_EGRESS EGRESS_VLAN_FILTER Legitimate port-level TX pipeline drops +DEBUG_2 RX_LEGIT LEGIT SWITCH_INGRESS TTL Legitimate switch-level RX pipeline drops ``` ### 3.1.3 Displaying the current counts @@ -204,7 +204,7 @@ $ show drops --type=PORT Ethernet8 U 0 0 0 0 0 0 0 0 Ethernet12 U 0 0 1200 400 0 0 0 0 -$ show drops --contains "LEGIT" +$ show drops --group "LEGIT" IFACE STATE RX_LEGIT TX_LEGIT --------------- ------- ---------- ---------- Ethernet0 U 0 0 @@ -224,32 +224,32 @@ $ sonic-clear drops ### 3.1.5 Configuring counters from the CLI ``` -$ config drops init --counter="DEBUG_3" --alias="TX_LEGIT" --type="SWITCH_EGRESS" --desc="Legitimate switch-level TX pipeline drops" --reasons=["L2_ANY", "L3_ANY"] +$ config drops init --counter="DEBUG_3" --alias="TX_LEGIT" --group="LEGIT" --type="SWITCH_EGRESS" --desc="Legitimate switch-level TX pipeline drops" --reasons=["L2_ANY", "L3_ANY"] Initializing DEBUG_3 as TX_LEGIT... -Counter Alias Type Reasons Description -------- -------- ------------- ------- ----------- -DEBUG_3 TX_LEGIT SWITCH_EGRESS L2_ANY Legitimate switch-level TX pipeline drops - L3_ANY +Counter Alias Group Type Reasons Description +------- -------- ----- ------------- ------- ----------- +DEBUG_3 TX_LEGIT LEGIT SWITCH_EGRESS L2_ANY Legitimate switch-level TX pipeline drops + L3_ANY $ config drops add --counter="DEBUG_3" --reasons=["A_CUSTOM_REASON", "ANOTHER_CUSTOM_REASON"] Configuring DEBUG_3... -Counter Alias Type Reasons Description -------- -------- ------------- --------------------- ----------- -DEBUG_3 TX_LEGIT SWITCH_EGRESS L2_ANY Legitimate switch-level TX pipeline drops - L3_ANY - A_CUSTOM_REASON - ANOTHER_CUSTOM_REASON +Counter Alias Group Type Reasons Description +------- -------- ----- ------------- ------- ----------- +DEBUG_3 TX_LEGIT LEGIT SWITCH_EGRESS L2_ANY Legitimate switch-level TX pipeline drops + L3_ANY + A_CUSTOM_REASON + ANOTHER_CUSTOM_REASON $ config drops remove --counter="DEBUG_3" --reasons=["A_CUSTOM_REASON"] Configuring DEBUG_3... -Counter Alias Type Reasons Description -------- -------- ------------- --------------------- ----------- -DEBUG_3 TX_LEGIT SWITCH_EGRESS L2_ANY Legitimate switch-level TX pipeline drops - L3_ANY - ANOTHER_CUSTOM_REASON +Counter Alias Group Type Reasons Description +------- -------- ----- ------------- ------- ----------- +DEBUG_3 TX_LEGIT LEGIT SWITCH_EGRESS L2_ANY Legitimate switch-level TX pipeline drops + L3_ANY + ANOTHER_CUSTOM_REASON $ config drops delete --counter="DEBUG_3" ``` @@ -267,17 +267,20 @@ Example: "DEBUG_0": { "alias": "PORT_RX_LEGIT", "type": "PORT_INGRESS_DROPS", - "desc": "Legitimate port-level RX pipeline drops" + "desc": "Legitimate port-level RX pipeline drops", + "group": "LEGIT" }, "DEBUG_1": { "alias": "PORT_TX_LEGIT", "type": "PORT_EGRESS_DROPS", "desc": "Legitimate port-level TX pipeline drops" + "group": "LEGIT" }, "DEBUG_2": { "alias": "SWITCH_RX_LEGIT", "type": "SWITCH_INGRESS_DROPS", "desc": "Legitimate switch-level RX pipeline drops" + "group": "LEGIT" } } } @@ -346,6 +349,8 @@ This orchestrator will interact with the following SAI Debug Counter APIs: ## 3.6 syncd Flex counter will be extended to support switch-level SAI counters. +Additionally, we will add a mapping from debug counter names to the appropriate port/switch stat index called COUNTERS_DEBUG_NAME_INDEX_MAP. + # 4 Flows ## 4.1 General Flow ![alt text](./drop_counters_general_flow.png) From 4bc3e0182da8f863e0c1c5f705a3d6ae54ec1f47 Mon Sep 17 00:00:00 2001 From: Danny Allen Date: Thu, 19 Sep 2019 16:40:19 -0700 Subject: [PATCH 24/24] Add another mapping to support CLI --- doc/drop_counters/drop_counters_HLD.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/doc/drop_counters/drop_counters_HLD.md b/doc/drop_counters/drop_counters_HLD.md index 61249d0e814..fd3105076ee 100644 --- a/doc/drop_counters/drop_counters_HLD.md +++ b/doc/drop_counters/drop_counters_HLD.md @@ -130,7 +130,6 @@ $ show drops available SWITCH_INGRESS 1 1 SWITCH_EGRESS 2 0 -$ show drops reasons PORT_INGRESS: L2_ANY SMAC_MULTICAST @@ -162,7 +161,11 @@ SWITCH_EGRESS: A_CUSTOM_REASON ANOTHER_CUSTOM_REASON -$ show drops reasons --type=PORT_EGRESS +$ show drops available --type=PORT_EGRESS + TYPE TOTAL FREE IN-USE +-------------- ----- ---- ------ + PORT_EGRESS 3 2 1 + PORT_EGRESS: L2_ANY L3_ANY @@ -311,11 +314,13 @@ Example: { "DEBUG_COUNTER_CAPABILITIES": { "SWITCH_INGRESS_DROPS": { - "count": 2, + "total": 3, + "used": 1, "reasons": [L2_ANY, L3_ANY, SMAC_EQUALS_DMAC] }, "SWITCH_EGRESS_DROPS": { - "count": 2, + "total": 3, + "used": 1, "reasons": [L2_ANY, L3_ANY] } } @@ -332,6 +337,8 @@ We will use the following SAI APIs to get this information: ## 3.4 Counters DB The contents of the drop counters will be added to Counters DB by flex counters. +Additionally, we will add a mapping from debug counter names to the appropriate port or switch stat index called COUNTERS_DEBUG_NAME_PORT_STAT_MAP and COUNTERS_DEBUG_NAME_SWITCH_STAT_MAP respectively. + ## 3.5 SWSS A new orchestrator will be created to handle debug counter creation and configuration. Specifically, this orchestrator will support: * Creating a new counter @@ -349,8 +356,6 @@ This orchestrator will interact with the following SAI Debug Counter APIs: ## 3.6 syncd Flex counter will be extended to support switch-level SAI counters. -Additionally, we will add a mapping from debug counter names to the appropriate port/switch stat index called COUNTERS_DEBUG_NAME_INDEX_MAP. - # 4 Flows ## 4.1 General Flow ![alt text](./drop_counters_general_flow.png)