Fixed orchagent crash in VM with the Qos BUFFER_QUEUE|system-port|Queue-id-range config #3050
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ue-id-range config Signed-off-by: saksarav <sakthivadivu.saravanaraj@nokia.com>
|
@judyjoseph @deepak-singhal0408 for viz |
deepak-singhal0408
approved these changes
Feb 15, 2024
|
@vmittal-msft for viz |
|
@saksarav-nokia, this crash is seen only when used as sonicVM for virtual chassis - but the fix is a generic fix, Do we need to add some unit test for this if not already covered |
|
@vmittal-msft , can you comment on the unit test. I remember you added some tests. |
|
Looks good. I am assuming this is working fine on switch as well. Please add VS test to cover this. It can be inside test_virtual_chassis.py. |
|
Yes. i tested the fix in our hardware. I will look at the vs tests |
|
@saksarav-nokia, we would need a vstest to improve coverage - could you add one. |
…R_QUEUE config is added/deleted Signed-off-by: saksarav <sakthivadivu.saravanaraj@nokia.com>
|
@
i am looking at |
|
@vmittal-msft @deepak-singhal0408 , based on our discussion with Bhaskar, Added the (gMySwitchType != "voq") check in processQueue function to not call createPortBufferQueueCounters or removePortBufferQueueCounters |
…port|Queue-id-range config" This reverts commit a27ec92.
vmittal-msft
approved these changes
Feb 22, 2024
Signed-off-by: saksarav <sakthivadivu.saravanaraj@nokia.com>
|
@judyjoseph , since i reverted the first commit which is not required with the latest fix, the code coverage check passed. |
prsunny
approved these changes
Feb 24, 2024
|
@vmittal-msft , please include MSFT ADO# for this PR so it can be reviewed/picked for the requested branches. |
|
ADO # 27162663 for porting to 202205/202305 branches. |
cscarpitta
pushed a commit
to cscarpitta/sonic-swss
that referenced
this pull request
Apr 5, 2024
…ue-id-range config (sonic-net#3050) * Fixed orchagent crash in VM with the Qos BUFFER_QUEUE|system-port|Queue-id-range config
|
hi @saksarav-nokia could you run test with 202035 image? @gechiang |
|
@yxieca , can you approve this for 202205 branch? This is a bug fix for Chassis and should have no impact to pizzabox as the added check is specifically for VOQ only. |
|
@gechiang , please mark for all future releases unless this fix is already there. I see you only tagged 202205 and removed 202305 |
Was thinking we may not be using 202405 for chassis but for Community that may not be the case so added back the label as it is a bug fix. |
Hi @StormLiangMS , This change is specifically adding a check to skip the port queue counter if dealing with VOQ (Chassis). It should have zero impact to pizzabox platforms. I am not sure if Nokia is able to test this with 202305 image as we don;t actively build the 202305 based image but we tagged it because it is a bug and we wanted it for 202205 bug fix and have to include all future branches as well (tried to skip it but caught by @prsunny :=) So adding it back. The one line fix should only impact VOQ chassis only. Thanks! |
superchild
pushed a commit
to superchild/sonic-swss
that referenced
this pull request
Apr 24, 2024
* Fixes mock test failure * Fixes mock test run failure fixes pipeline run failure FAIL: p4orch_tests_usan ======================= ../../../orchagent/vrforch.cpp:113:41: runtime error: member call on null pointer of type 'struct RouteOrch' ../../../orchagent/vrforch.cpp:113:41: runtime error: member access within null pointer of type 'struct RouteOrch' FAIL p4orch_tests_usan (exit status: 139) * Fixed orchagent crash in VM with the Qos BUFFER_QUEUE|system-port|Queue-id-range config (sonic-net#3050) * Fixed orchagent crash in VM with the Qos BUFFER_QUEUE|system-port|Queue-id-range config * [intfsorch] Enable ipv6 proxy ndp along with proxy arp (sonic-net#3045) * [intfsorch] Enable ipv6 proxy ndp along with proxy arp setting SAI_VLAN_ATTR_UNKNOWN_MULTICAST_FLOOD_CONTROL_TYPE to SAI_VLAN_FLOOD_CONTROL_TYPE_NONE when proxy arp is enabled. This fixes a bug where ipv6 NS packets were flooding ports with duplicate packets. We now set multicast flood type to none. * Fix multi VLAN neighbor learning (sonic-net#3049) What I did When adding a new neighbor, check if the neighbor IP has already been learned on a different VLAN. If it has, remove the old neighbor entry before adding the new one. Why I did it On Gemini devices, if a neighbor IP moves from an active port in one VLAN to a second VLAN, then back to the first VLAN (with 3 different MAC addresses), orchagent will crash. Even though the MAC address of the last move is different from the first MAC address, orchagent believes the last MAC address to already be programmed in the hardware and tries to set an attribute of the entry which doesn't exist. * [asan] Disable the "maybe-uninitialized" warning when compiled with ASAN enabled. * Set HOST_TX_READY_NOTIFY attribute only after query capabilities(sonic-net#3070) *Set HOST_TX_READY_NOTIFY attribute only after query capabilities * [EVPN] Skip EVPN routes with invalid VNI or router mac field (sonic-net#3073) * Skip EVPN routes with invalid VNI or router mac field * Add port flap count and last flap timestamp to APPL_DB (sonic-net#3052) * Add port flap count and last flap timestamp * Add basic fabric link monitoring counters and states handling. (sonic-net#2988) * Add basic fabric link monitoring counters and states handling. * [Mellanox] Fix inconsistence in the shared headroom pool initialization (sonic-net#3057) * Fix inconsistence in the shared headroom pool initialization * Why I did it During initialization, if SHP is enabled the buffer pool sizes, xoff have initialized to 0, which means SHP is disabled but the buffer profiles already indicate SHP later on the buffer pool sizes are updated with off being non-zero In case the orchagent starts handling buffer configuration between 2 and 3, it is inconsistent between buffer pools and profiles, which fails Mellanox SAI sanity check. To avoid it, it indicates SHP enabled by setting a very small buffer pool and SHP sizes * [acl] Add IN_PORTS qualifier for L3 table (sonic-net#3078) * Apply IN_PORTS qualifiier for L3 table Why I did it IN_PORTS qualifier was allowed for L3 table in 202012 release and below. Changes in sonic-net#1982 removed that support leading to regression in some of our testcases. The following error was observed ERR swss#orchagent: :- validateAclRuleMatch: Match SAI_ACL_ENTRY_ATTR_FIELD_IN_PORTS in rule RULE_1 is not supported by table DATAACL * [bulker] add support for neighbor bulking (sonic-net#2768) Adding support for sai_neighbor_api_t bulking in bulker.h * [buffermgrd] Move switch-statement outside of if-statement in BufferMgr::doTask (sonic-net#3055) * [buffermgr] Moved switch statement outside of if-statmement in Buffermgr::doTask The switch statement which would normally erase buffer events was moved to be inside the if-statement which would only enter if the event is a SET event. This was introduced in commit e5329c39. This would cause an infinite loop, since non-set events would never be erased. The switch statement has now been moved to occur outside the if, allowing for non-set commands to be processed. * [portsorch] process only updated APP_DB fields when port is already created (sonic-net#3025) * [portsorch] process only updated APP_DB fields when port is already created What I did Fixing an issue when setting some port attribute in APPL_DB triggers serdes parameters to be re-programmed with port toggling. Made portsorch to handle only those attributes that were pushed to APPL_DB, so that serdes programming happens only by xcvrd's request to do so. * [Copp]Refactor coppmgr tests (sonic-net#3093) What I did Refactoring coppmgr mock tests Why I did it After migration to bookworm, coppmgr tests started failing due to the use of sudo commands. * Revert "[acl] Add IN_PORTS qualifier for L3 table (sonic-net#3078)" (sonic-net#3092) This reverts commit 9d4a3ad. *Revert "[acl] Add IN_PORTS qualifier for L3 table" * [orchagent] TWAMP Light orchagent implementation (sonic-net#2927) * [orchagent] TWAMP Light orchagent implementation. (sonic-net#2927) * What I did Implemented the TWAMP Light feature according to the SONiC TWAMP Light HLD(sonic-net/SONiC#1320). * Clang format change. (sonic-net#3080) What I did This PR has no real code change. It is purely clang formatting. It only applies to the P4Orch codes. Commands that I run: find orchagent/p4orch -name *.h -o -name .cpp | xargs clang-format -i -style="{BasedOnStyle: Microsoft, DerivePointerAlignment: false}" find orchagent -name response_publisher -o -name return_code.h | xargs clang-format -i -style="{BasedOnStyle: Microsoft, DerivePointerAlignment: false}" * T2-VOQ-VS: Fix iBGP bringup issue (sonic-net#3053) * Fix iBGP bringup issue T2-vswitch * On T2-VOQ chassis Emulation with multi-asic linecards, iBGP sessions dont come up. Related Issue: sonic-net/sonic-buildimage#18129 * [Fdbsyncd] Adding extern_learn flag with fdb entry so Kernel doesn't age out (sonic-net#2985) * Adding extern_learn flag with fdb entry so that Kernel doesn't age out the MAC * [Fdbsyncd] Adding extern_learn flag with fdb entry so Kernel doesn't age out What I did extern_learn flag is added while programming the fdb entry into the Kernel. This will make sure that kernel doesn't age out the fdb entry. (#15004) How I did it A flag extern_learn will be passed while programing the fdb entry. (#15004) How to verify it Tested MAC add/del to the Kernel from the local FDB entry. (#15004) Signed-off-by: kishore.kunal@broadcom.com --------- Signed-off-by: kishore.kunal@broadcom.com Co-authored-by: Sudharsan Dhamal Gopalarathnam <sudharsand@nvidia.com> * Fix oper FEC retrieval after warmboot (sonic-net#3100) Updating oper FEC status in state_db after warm-reboot as part of refresh port status call * [EVPN]Fix fpmsyncd crash when EVPN type5 is received with bgp fib suppression enabled (sonic-net#3101) * [EVPN]Fix fpmsyncd crash when EVPN type5 is received with bgp fib suppression enabled * [portsorch] Handle TRANSCEIVER_INFO table on warm boot (sonic-net#3087) * Add existing data from TRANSCEIVER_INFO table * Introduce a new role for DPU-NPU Interconnect Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com> Co-authored-by: Sudharsan Dhamal Gopalarathnam <sudharsand@nvidia.com> * [p4orch] Clang format change. (sonic-net#3096) What I did [p4orch] This PR has no real code change. It is purely clang formatting. It does the same as sonic-net#3080. * [dash] fix ENI admin state update (sonic-net#3081) * [dash] fix ENI admin state update * Add force option for fabric port unisolate command (sonic-net#3089) What I did Add force option to the unisolate link command, so users can make the links not isolate if they want. depends on sonic-net/sonic-buildimage#18447 * [twamporch] Explicitly initialize local variable (sonic-net#3115) What I did Explicitly initialized local variable. Why I did it We met below error message in sonic-buildimage armhf build (sonic-net/sonic-buildimage#18334) * Add bookworm build to the PR checkers (sonic-net#3114) What I did Add a Bookworm build to the PR checkers. Also fix some Bookworm build errors that crept in. Why I did it Buildimage now builds swss for Bookworm, so the build needs to succeed. * [ACL] Remove flex counter when updating ACL rule (sonic-net#3118) What I did This PR is to fix sonic-net/sonic-buildimage#18719 When ACL rule is created for the first time, a flex counter is created and registered. When the same ACL rule is being updated, the FlexCounter created before is not removed, and another FlexCounter is created and registered. Why I did it Fix the issue that FlexCounter is duplicated when updating existing ACL rule. --------- Signed-off-by: kishore.kunal@broadcom.com Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com> Co-authored-by: saksarav-nokia <sakthivadivu.saravanaraj@nokia.com> Co-authored-by: Nikola Dancejic <26731235+Ndancejic@users.noreply.github.com> Co-authored-by: Lawrence Lee <lawlee@microsoft.com> Co-authored-by: Oleksandr Ivantsiv <oivantsiv@nvidia.com> Co-authored-by: noaOrMlnx <58519608+noaOrMlnx@users.noreply.github.com> Co-authored-by: Lior Avramov <73036155+liorghub@users.noreply.github.com> Co-authored-by: Prince George <45705344+prgeor@users.noreply.github.com> Co-authored-by: jfeng-arista <98421150+jfeng-arista@users.noreply.github.com> Co-authored-by: Stephen Sun <5379172+stephenxs@users.noreply.github.com> Co-authored-by: Neetha John <nejo@microsoft.com> Co-authored-by: Amir <mazora@marvell.com> Co-authored-by: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Co-authored-by: Sudharsan Dhamal Gopalarathnam <sudharsand@nvidia.com> Co-authored-by: xiaodong hu <32903206+huseratgithub@users.noreply.github.com> Co-authored-by: mint570 <70396898+mint570@users.noreply.github.com> Co-authored-by: Deepak Singhal <115033986+deepak-singhal0408@users.noreply.github.com> Co-authored-by: KISHORE KUNAL <64033340+kishorekunal01@users.noreply.github.com> Co-authored-by: Vivek <vivekreddykarri98@gmail.com> Co-authored-by: Yakiv Huryk <62013282+Yakiv-Huryk@users.noreply.github.com> Co-authored-by: Saikrishna Arcot <sarcot@microsoft.com> Co-authored-by: bingwang-ms <66248323+bingwang-ms@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What I did
Modified BufferOrch::processQueue in orchagent to pass the correct token for queue-id range if the switch_type is voq to gPortsOrch->createPortBufferQueueCounters.
Why I did it
VM crashes with Qos "BUFFER_QUEUE|system-port|Queue-id-range" config for system ports. The crash was not seen in hardware somehow and queue & voq stats are working with the different code path adding the QUEUE stats counters to FLEX_COUNTER_DB
FlexCounterOrch::doTask -> generateQueueMap -> generateQueueMapPerPort (true) -> addQueueFlexCountersPerPortPerQueueIndex
How I verified it
verified that the correct queue id-range is passed to gPortsOrch->createPortBufferQueueCounters. and not crashing.
Details if related