test_portstat failed due to ValueError

[ysmanman]

Description

Observed the following error when running test_portstat with T2 topo:

    @pytest.mark.parametrize('command', ['portstat -c', 'portstat --clear'])                                                                                                                                                                        \
     def test_portstat_clear(duthosts, enum_rand_one_per_hwsku_frontend_hostname, command):                                                                                                                                                         \
         duthost = duthosts[enum_rand_one_per_hwsku_frontend_hostname]                                                                                                                                                                              \
         wait(30, 'Wait for DUT to receive/send some packets')                                                                                                                                                                                      \
         before_portstat = parse_portstat(duthost.command('portstat')['stdout_lines'])                                                                                                                                                              \
         pytest_assert(before_portstat, 'No parsed command output')                                                                                                                                                                                 \
                                                                                                                                                                                                                                                    \
         duthost.command(command)                                                                                                                                                                                                                   \
         wait(1, 'Wait for portstat counters to refresh')                                                                                                                                                                                           \
                                                                                                                                                                                                                                                    \
         after_portstat = parse_portstat(duthost.command('portstat')['stdout_lines'])                                                                                                                                                               \
         pytest_assert(after_portstat, 'No parsed command output')                                                                                                                                                                                  \
                                                                                                                                                                                                                                                    \
         """                                                                                                                                                                                                                                        \
         Assert only when rx/tx count is no smaller than COUNT_THRES because DUT may send or receive                                                                                                                                                \
         some packets during test after port status are clear                                                                                                                                                                                       \
         """                                                                                                                                                                                                                                        \
         COUNT_THRES = 10                                                                                                                                                                                                                           \
         for intf in before_portstat:                                                                                                                                                                                                               \
            tmp_ok_cnt = before_portstat[intf]['rx_ok'].replace(',','')
>            rx_ok_before = int(0 if tmp_ok_cnt == 'N/A' else tmp_ok_cnt)
E            ValueError: invalid literal for int() with base 10: ''

The is is because parse_portstat failed to parse the output of portstat and returned some garbage. E.g., the invalid port keys like '' and 'Reminder: P' returned by parse_portstat in the following output:

(Pdb) before_portstat.keys()
[u'', u'Ethernet180', u'Ethernet8', u'Ethernet184', u'Ethernet188', u'Ethernet0', u'Reminder: P', u'Ethernet148', u'Ethernet4', u'Ethernet108', u'Ethernet100', u'Ethernet164', u'Ethernet104', u'Ethernet96', u'Ethernet124', u'Ethernet120', u'Ethernet92', u'Ethernet28', u'Ethernet144', u'Ethernet52', u'Ethernet160', u'Ethernet140', u'Ethernet56', u'Ethernet128', u'Ethernet76', u'Ethernet72', u'Ethernet32', u'Ethernet168', u'Ethernet16', u'Ethernet36', u'Ethernet12', u'Ethernet88', u'Ethernet24', u'Ethernet116', u'Ethernet80', u'Ethernet112', u'Ethernet84', u'Ethernet152', u'Ethernet136', u'Ethernet156', u'Ethernet132', u'Ethernet48', u'Ethernet172', u'Ethernet44', u'Ethernet176', u'Ethernet40', u'Ethernet64', u'Ethernet60', u'Ethernet20', u'Ethernet68']
(Pdb) before_portstat['']
{u'rx_bps': u'', u'rx_err': u'', u'tx_err': u'', u'rx_ovr': u'', u'tx_ovr': u'', u'tx_ok': u'', u'rx_util': u'', u'tx_util': u'', u'state': u'', u'tx_bps': u'', u'tx_drp': u'', u'rx_ok': u'', u'rx_drp': u''}
(Pdb) before_portstat['Reminder: P']                                                                                                                                                                                                                
{u'rx_bps': u'interface', u'rx_err': u"ll' to i", u'tx_err': u'', u'rx_ovr': u'nal link', u'tx_ovr': u'', u'tx_ok': u'', u'rx_util': u'unters -d', u'tx_util': u'', u'state': u'ase exe', u'tx_bps': u'', u'tx_drp': u'', u'rx_ok': u"te 'sho", u'rx_drp': u'lude int'}

Steps to reproduce the issue:
1.
2.
3.

Describe the results you received:

Describe the results you expected:

Additional information you deem important:

**Output of `show version`:**

```
(paste your output here)
```

**Attach debug file `sudo generate_dump`:**

```
(paste your output here)
```

[ysmanman]

#7561 fix this issue.

[wenyiz2021]

#7537 will help

[ysmanman]

Hi @wenyiz2021, I am still seeing the issue in t2 testing with #7537. How did we verify the fix?

[wenyiz2021]

Hi @wenyiz2021, I am still seeing the issue in t2 testing with #7537. How did we verify the fix?

This fix was verified manually by me and Suvarna, did your test branch has this PR fix? asking because our internal test branch hasn't have this change yet.

[ysmanman]

Hi @wenyiz2021, I am still seeing the issue in t2 testing with #7537. How did we verify the fix?

This fix was verified manually by me and Suvarna, did your test branch has this PR fix? asking because our internal test branch hasn't have this change yet.

Yes, we patched the PR in our internal testing and still observed the issue.

[ysmanman]

Created #7657 to fix this.