Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[chassis][sfm][swss] swss.sh shows backtrace for all empty SFM slots when shutdown or remove a SFM on Supervisor #18384

Closed
mlok-nokia opened this issue Mar 18, 2024 · 0 comments · Fixed by #18393
Assignees
Labels
Triaged this issue has been triaged

Comments

@mlok-nokia
Copy link
Contributor

Description

In the 202205 branch. on Supervisor card of a VOQ chassis, when remove or shutdown a Fabric card, swss.sh shows Stacktrace for all related empty SFM slots in the syslog file. Based on the investigation, this issue should be in all branches.

Mar 14 21:58:56.624631 ixre-cpm-chassis17 INFO swss.sh[1351668]: Exception in thread Thread-1:
Mar 14 21:58:56.624699 ixre-cpm-chassis17 INFO swss.sh[1351668]: Traceback (most recent call last):
Mar 14 21:58:56.624736 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 268, in _raise_for_status
Mar 14 21:58:56.624778 ixre-cpm-chassis17 INFO swss.sh[1351668]:     response.raise_for_status()
Mar 14 21:58:56.624809 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 1021, in raise_for_status
Mar 14 21:58:56.624836 ixre-cpm-chassis17 INFO swss.sh[1351668]:     raise HTTPError(http_error_msg, response=self)
Mar 14 21:58:56.625653 ixre-cpm-chassis17 INFO swss.sh[1351668]: Exception in thread Thread-2:
Mar 14 21:58:56.625694 ixre-cpm-chassis17 INFO swss.sh[1351668]: requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/swss5/wait
Mar 14 21:58:56.625724 ixre-cpm-chassis17 INFO swss.sh[1351668]: Traceback (most recent call last):
Mar 14 21:58:56.625750 ixre-cpm-chassis17 INFO swss.sh[1351668]: The above exception was the direct cause of the following exception:
Mar 14 21:58:56.625786 ixre-cpm-chassis17 INFO swss.sh[1351668]: Traceback (most recent call last):
Mar 14 21:58:56.625814 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 268, in _raise_for_status
Mar 14 21:58:56.625841 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
Mar 14 21:58:56.626035 ixre-cpm-chassis17 INFO swss.sh[1351668]:     self.run()
Mar 14 21:58:56.626083 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/lib/python3.9/threading.py", line 892, in run
Mar 14 21:58:56.626231 ixre-cpm-chassis17 INFO swss.sh[1351668]:     self._target(*self._args, **self._kwargs)
Mar 14 21:58:56.626268 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/bin/docker-wait-any", line 52, in wait_for_container
Mar 14 21:58:56.626296 ixre-cpm-chassis17 INFO swss.sh[1351668]:     response.raise_for_status()
Mar 14 21:58:56.626332 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 1021, in raise_for_status
Mar 14 21:58:56.626368 ixre-cpm-chassis17 INFO swss.sh[1351668]:     docker_client.wait(container_name)
Mar 14 21:58:56.626395 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/utils/decorators.py", line 19, in wrapped
Mar 14 21:58:56.626531 ixre-cpm-chassis17 INFO swss.sh[1351668]:     return f(self, resource_id, *args, **kwargs)
Mar 14 21:58:56.626605 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/container.py", line 1339, in wait
Mar 14 21:58:56.627013 ixre-cpm-chassis17 INFO swss.sh[1351668]:     raise HTTPError(http_error_msg, response=self)
Mar 14 21:58:56.627059 ixre-cpm-chassis17 INFO swss.sh[1351668]:     return self._result(res, True)
Mar 14 21:58:56.627090 ixre-cpm-chassis17 INFO swss.sh[1351668]: requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/syncd5/wait
Mar 14 21:58:56.627118 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 274, in _result
Mar 14 21:58:56.627154 ixre-cpm-chassis17 INFO swss.sh[1351668]: The above exception was the direct cause of the following exception:
Mar 14 21:58:56.627187 ixre-cpm-chassis17 INFO swss.sh[1351668]: Traceback (most recent call last):
Mar 14 21:58:56.627223 ixre-cpm-chassis17 INFO swss.sh[1351668]:     self._raise_for_status(response)
Mar 14 21:58:56.627252 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 270, in _raise_for_status
Mar 14 21:58:56.627280 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
Mar 14 21:58:56.627479 ixre-cpm-chassis17 INFO swss.sh[1351669]: Exception in thread Thread-1:
Mar 14 21:58:56.627515 ixre-cpm-chassis17 INFO swss.sh[1351669]: Traceback (most recent call last):
Mar 14 21:58:56.627542 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 268, in _raise_for_status
Mar 14 21:58:56.627580 ixre-cpm-chassis17 INFO swss.sh[1351668]:     raise create_api_error_from_http_exception(e) from e
Mar 14 21:58:56.627610 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
Mar 14 21:58:56.627643 ixre-cpm-chassis17 INFO swss.sh[1351668]:     raise cls(e, response=response, explanation=explanation) from e
Mar 14 21:58:56.627679 ixre-cpm-chassis17 INFO swss.sh[1351668]: docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.41/containers/swss5/wait: Not Found ("No such container: swss5")
Mar 14 21:58:56.627926 ixre-cpm-chassis17 INFO swss.sh[1351668]:     self.run()
Mar 14 21:58:56.627962 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/lib/python3.9/threading.py", line 892, in run
Mar 14 21:58:56.628366 ixre-cpm-chassis17 INFO swss.sh[1351669]: Exception in thread Thread-2:
Mar 14 21:58:56.628415 ixre-cpm-chassis17 INFO swss.sh[1351668]:     self._target(*self._args, **self._kwargs)
Mar 14 21:58:56.628451 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/bin/docker-wait-any", line 52, in wait_for_container
Mar 14 21:58:56.628499 ixre-cpm-chassis17 INFO swss.sh[1351669]:     response.raise_for_status()
Mar 14 21:58:56.628534 ixre-cpm-chassis17 INFO swss.sh[1351669]: Traceback (most recent call last):
Mar 14 21:58:56.628571 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 1021, in raise_for_status
Mar 14 21:58:56.628607 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 268, in _raise_for_status
Mar 14 21:58:56.628643 ixre-cpm-chassis17 INFO swss.sh[1351668]:     docker_client.wait(container_name)
Mar 14 21:58:56.628672 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/utils/decorators.py", line 19, in wrapped
Mar 14 21:58:56.628713 ixre-cpm-chassis17 INFO swss.sh[1351668]:     return f(self, resource_id, *args, **kwargs)
Mar 14 21:58:56.628752 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/container.py", line 1339, in wait
Mar 14 21:58:56.628791 ixre-cpm-chassis17 INFO swss.sh[1351669]:     raise HTTPError(http_error_msg, response=self)
Mar 14 21:58:56.628819 ixre-cpm-chassis17 INFO swss.sh[1351669]: requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/swss4/wait
Mar 14 21:58:56.628847 ixre-cpm-chassis17 INFO swss.sh[1351669]: The above exception was the direct cause of the following exception:
Mar 14 21:58:56.628873 ixre-cpm-chassis17 INFO swss.sh[1351669]: Traceback (most recent call last):
Mar 14 21:58:56.629028 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
Mar 14 21:58:56.629064 ixre-cpm-chassis17 INFO swss.sh[1351669]:     response.raise_for_status()
Mar 14 21:58:56.629237 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 1021, in raise_for_status
Mar 14 21:58:56.629273 ixre-cpm-chassis17 INFO swss.sh[1351669]:     self.run()
Mar 14 21:58:56.629306 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/lib/python3.9/threading.py", line 892, in run
Mar 14 21:58:56.629342 ixre-cpm-chassis17 INFO swss.sh[1351668]:     return self._result(res, True)
Mar 14 21:58:56.629371 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 274, in _result
Mar 14 21:58:56.629524 ixre-cpm-chassis17 INFO swss.sh[1351668]:     self._raise_for_status(response)
Mar 14 21:58:56.629559 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 270, in _raise_for_status
Mar 14 21:58:56.629622 ixre-cpm-chassis17 INFO swss.sh[1351669]:     self._target(*self._args, **self._kwargs)
Mar 14 21:58:56.629737 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/bin/docker-wait-any", line 52, in wait_for_container
Mar 14 21:58:56.629773 ixre-cpm-chassis17 INFO swss.sh[1351668]:     raise create_api_error_from_http_exception(e) from e
Mar 14 21:58:56.629802 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
Mar 14 21:58:56.629868 ixre-cpm-chassis17 INFO swss.sh[1351668]:     raise cls(e, response=response, explanation=explanation) from e
Mar 14 21:58:56.629906 ixre-cpm-chassis17 INFO swss.sh[1351668]: docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.41/containers/syncd5/wait: Not Found ("No such container: syncd5")
Mar 14 21:58:56.629979 ixre-cpm-chassis17 INFO swss.sh[1351669]:     raise HTTPError(http_error_msg, response=self)
Mar 14 21:58:56.630073 ixre-cpm-chassis17 INFO swss.sh[1351669]:     docker_client.wait(container_name)
Mar 14 21:58:56.630102 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/docker/utils/decorators.py", line 19, in wrapped
Mar 14 21:58:56.630138 ixre-cpm-chassis17 INFO swss.sh[1351669]: requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/syncd4/wait
Mar 14 21:58:56.630168 ixre-cpm-chassis17 INFO swss.sh[1351669]: The above exception was the direct cause of the following exception:
Mar 14 21:58:56.630195 ixre-cpm-chassis17 INFO swss.sh[1351669]: Traceback (most recent call last):
Mar 14 21:58:56.630223 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
Mar 14 21:58:56.630258 ixre-cpm-chassis17 INFO swss.sh[1351669]:     return f(self, resource_id, *args, **kwargs)
Mar 14 21:58:56.630289 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/container.py", line 1339, in wait
Mar 14 21:58:56.630551 ixre-cpm-chassis17 INFO swss.sh[1351669]:     self.run()
Mar 14 21:58:56.630586 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/lib/python3.9/threading.py", line 892, in run
Mar 14 21:58:56.630796 ixre-cpm-chassis17 INFO swss.sh[1351669]:     return self._result(res, True)
Mar 14 21:58:56.630832 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 274, in _result
Mar 14 21:58:56.631150 ixre-cpm-chassis17 INFO swss.sh[1351669]:     self._target(*self._args, **self._kwargs)
Mar 14 21:58:56.631187 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/bin/docker-wait-any", line 52, in wait_for_container
Mar 14 21:58:56.631374 ixre-cpm-chassis17 INFO swss.sh[1351669]:     docker_client.wait(container_name)
Mar 14 21:58:56.631485 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/docker/utils/decorators.py", line 19, in wrapped
Mar 14 21:58:56.631534 ixre-cpm-chassis17 INFO swss.sh[1351669]:     self._raise_for_status(response)

Steps to reproduce the issue:

  1. In a running VOQ chassis, remove a or shutdown a Fabric card.
  2. Check the syslog, the swss.sh shows Satcktrace for all related empty SFM slots as below.
Mar 14 21:58:56.624631 ixre-cpm-chassis17 INFO swss.sh[1351668]: Exception in thread Thread-1:
Mar 14 21:58:56.624699 ixre-cpm-chassis17 INFO swss.sh[1351668]: Traceback (most recent call last):
Mar 14 21:58:56.624736 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 268, in _raise_for_status
Mar 14 21:58:56.624778 ixre-cpm-chassis17 INFO swss.sh[1351668]:     response.raise_for_status()
Mar 14 21:58:56.624809 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 1021, in raise_for_status
Mar 14 21:58:56.624836 ixre-cpm-chassis17 INFO swss.sh[1351668]:     raise HTTPError(http_error_msg, response=self)
Mar 14 21:58:56.625653 ixre-cpm-chassis17 INFO swss.sh[1351668]: Exception in thread Thread-2:
Mar 14 21:58:56.625694 ixre-cpm-chassis17 INFO swss.sh[1351668]: requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/swss5/wait
Mar 14 21:58:56.625724 ixre-cpm-chassis17 INFO swss.sh[1351668]: Traceback (most recent call last):
Mar 14 21:58:56.625750 ixre-cpm-chassis17 INFO swss.sh[1351668]: The above exception was the direct cause of the following exception:
Mar 14 21:58:56.625786 ixre-cpm-chassis17 INFO swss.sh[1351668]: Traceback (most recent call last):
Mar 14 21:58:56.625814 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 268, in _raise_for_status
Mar 14 21:58:56.625841 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
Mar 14 21:58:56.626035 ixre-cpm-chassis17 INFO swss.sh[1351668]:     self.run()
Mar 14 21:58:56.626083 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/lib/python3.9/threading.py", line 892, in run
Mar 14 21:58:56.626231 ixre-cpm-chassis17 INFO swss.sh[1351668]:     self._target(*self._args, **self._kwargs)
Mar 14 21:58:56.626268 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/bin/docker-wait-any", line 52, in wait_for_container
Mar 14 21:58:56.626296 ixre-cpm-chassis17 INFO swss.sh[1351668]:     response.raise_for_status()
Mar 14 21:58:56.626332 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 1021, in raise_for_status
Mar 14 21:58:56.626368 ixre-cpm-chassis17 INFO swss.sh[1351668]:     docker_client.wait(container_name)
Mar 14 21:58:56.626395 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/utils/decorators.py", line 19, in wrapped
Mar 14 21:58:56.626531 ixre-cpm-chassis17 INFO swss.sh[1351668]:     return f(self, resource_id, *args, **kwargs)
Mar 14 21:58:56.626605 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/container.py", line 1339, in wait
Mar 14 21:58:56.627013 ixre-cpm-chassis17 INFO swss.sh[1351668]:     raise HTTPError(http_error_msg, response=self)
Mar 14 21:58:56.627059 ixre-cpm-chassis17 INFO swss.sh[1351668]:     return self._result(res, True)
Mar 14 21:58:56.627090 ixre-cpm-chassis17 INFO swss.sh[1351668]: requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/syncd5/wait
Mar 14 21:58:56.627118 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 274, in _result
Mar 14 21:58:56.627154 ixre-cpm-chassis17 INFO swss.sh[1351668]: The above exception was the direct cause of the following exception:
Mar 14 21:58:56.627187 ixre-cpm-chassis17 INFO swss.sh[1351668]: Traceback (most recent call last):
Mar 14 21:58:56.627223 ixre-cpm-chassis17 INFO swss.sh[1351668]:     self._raise_for_status(response)
Mar 14 21:58:56.627252 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 270, in _raise_for_status
Mar 14 21:58:56.627280 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
Mar 14 21:58:56.627479 ixre-cpm-chassis17 INFO swss.sh[1351669]: Exception in thread Thread-1:
Mar 14 21:58:56.627515 ixre-cpm-chassis17 INFO swss.sh[1351669]: Traceback (most recent call last):
Mar 14 21:58:56.627542 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 268, in _raise_for_status
Mar 14 21:58:56.627580 ixre-cpm-chassis17 INFO swss.sh[1351668]:     raise create_api_error_from_http_exception(e) from e
Mar 14 21:58:56.627610 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
Mar 14 21:58:56.627643 ixre-cpm-chassis17 INFO swss.sh[1351668]:     raise cls(e, response=response, explanation=explanation) from e
Mar 14 21:58:56.627679 ixre-cpm-chassis17 INFO swss.sh[1351668]: docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.41/containers/swss5/wait: Not Found ("No such container: swss5")
Mar 14 21:58:56.627926 ixre-cpm-chassis17 INFO swss.sh[1351668]:     self.run()
Mar 14 21:58:56.627962 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/lib/python3.9/threading.py", line 892, in run
Mar 14 21:58:56.628366 ixre-cpm-chassis17 INFO swss.sh[1351669]: Exception in thread Thread-2:
Mar 14 21:58:56.628415 ixre-cpm-chassis17 INFO swss.sh[1351668]:     self._target(*self._args, **self._kwargs)
Mar 14 21:58:56.628451 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/bin/docker-wait-any", line 52, in wait_for_container
Mar 14 21:58:56.628499 ixre-cpm-chassis17 INFO swss.sh[1351669]:     response.raise_for_status()
Mar 14 21:58:56.628534 ixre-cpm-chassis17 INFO swss.sh[1351669]: Traceback (most recent call last):
Mar 14 21:58:56.628571 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 1021, in raise_for_status
Mar 14 21:58:56.628607 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 268, in _raise_for_status
Mar 14 21:58:56.628643 ixre-cpm-chassis17 INFO swss.sh[1351668]:     docker_client.wait(container_name)
Mar 14 21:58:56.628672 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/utils/decorators.py", line 19, in wrapped
Mar 14 21:58:56.628713 ixre-cpm-chassis17 INFO swss.sh[1351668]:     return f(self, resource_id, *args, **kwargs)
Mar 14 21:58:56.628752 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/container.py", line 1339, in wait
Mar 14 21:58:56.628791 ixre-cpm-chassis17 INFO swss.sh[1351669]:     raise HTTPError(http_error_msg, response=self)
Mar 14 21:58:56.628819 ixre-cpm-chassis17 INFO swss.sh[1351669]: requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/swss4/wait
Mar 14 21:58:56.628847 ixre-cpm-chassis17 INFO swss.sh[1351669]: The above exception was the direct cause of the following exception:
Mar 14 21:58:56.628873 ixre-cpm-chassis17 INFO swss.sh[1351669]: Traceback (most recent call last):
Mar 14 21:58:56.629028 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
Mar 14 21:58:56.629064 ixre-cpm-chassis17 INFO swss.sh[1351669]:     response.raise_for_status()
Mar 14 21:58:56.629237 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 1021, in raise_for_status
Mar 14 21:58:56.629273 ixre-cpm-chassis17 INFO swss.sh[1351669]:     self.run()
Mar 14 21:58:56.629306 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/lib/python3.9/threading.py", line 892, in run
Mar 14 21:58:56.629342 ixre-cpm-chassis17 INFO swss.sh[1351668]:     return self._result(res, True)
Mar 14 21:58:56.629371 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 274, in _result
Mar 14 21:58:56.629524 ixre-cpm-chassis17 INFO swss.sh[1351668]:     self._raise_for_status(response)
Mar 14 21:58:56.629559 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 270, in _raise_for_status
Mar 14 21:58:56.629622 ixre-cpm-chassis17 INFO swss.sh[1351669]:     self._target(*self._args, **self._kwargs)
Mar 14 21:58:56.629737 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/bin/docker-wait-any", line 52, in wait_for_container
Mar 14 21:58:56.629773 ixre-cpm-chassis17 INFO swss.sh[1351668]:     raise create_api_error_from_http_exception(e) from e
Mar 14 21:58:56.629802 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
Mar 14 21:58:56.629868 ixre-cpm-chassis17 INFO swss.sh[1351668]:     raise cls(e, response=response, explanation=explanation) from e
Mar 14 21:58:56.629906 ixre-cpm-chassis17 INFO swss.sh[1351668]: docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.41/containers/syncd5/wait: Not Found ("No such container: syncd5")
Mar 14 21:58:56.629979 ixre-cpm-chassis17 INFO swss.sh[1351669]:     raise HTTPError(http_error_msg, response=self)
Mar 14 21:58:56.630073 ixre-cpm-chassis17 INFO swss.sh[1351669]:     docker_client.wait(container_name)
Mar 14 21:58:56.630102 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/docker/utils/decorators.py", line 19, in wrapped
Mar 14 21:58:56.630138 ixre-cpm-chassis17 INFO swss.sh[1351669]: requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/syncd4/wait
Mar 14 21:58:56.630168 ixre-cpm-chassis17 INFO swss.sh[1351669]: The above exception was the direct cause of the following exception:
Mar 14 21:58:56.630195 ixre-cpm-chassis17 INFO swss.sh[1351669]: Traceback (most recent call last):
Mar 14 21:58:56.630223 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
Mar 14 21:58:56.630258 ixre-cpm-chassis17 INFO swss.sh[1351669]:     return f(self, resource_id, *args, **kwargs)
Mar 14 21:58:56.630289 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/container.py", line 1339, in wait
Mar 14 21:58:56.630551 ixre-cpm-chassis17 INFO swss.sh[1351669]:     self.run()
Mar 14 21:58:56.630586 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/lib/python3.9/threading.py", line 892, in run
Mar 14 21:58:56.630796 ixre-cpm-chassis17 INFO swss.sh[1351669]:     return self._result(res, True)
Mar 14 21:58:56.630832 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 274, in _result
Mar 14 21:58:56.631150 ixre-cpm-chassis17 INFO swss.sh[1351669]:     self._target(*self._args, **self._kwargs)
Mar 14 21:58:56.631187 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/bin/docker-wait-any", line 52, in wait_for_container
Mar 14 21:58:56.631374 ixre-cpm-chassis17 INFO swss.sh[1351669]:     docker_client.wait(container_name)
Mar 14 21:58:56.631485 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/docker/utils/decorators.py", line 19, in wrapped
Mar 14 21:58:56.631534 ixre-cpm-chassis17 INFO swss.sh[1351669]:     self._raise_for_status(response)

Describe the results you received:

Mar 14 21:58:56.624631 ixre-cpm-chassis17 INFO swss.sh[1351668]: Exception in thread Thread-1:
Mar 14 21:58:56.624699 ixre-cpm-chassis17 INFO swss.sh[1351668]: Traceback (most recent call last):
Mar 14 21:58:56.624736 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 268, in _raise_for_status
Mar 14 21:58:56.624778 ixre-cpm-chassis17 INFO swss.sh[1351668]:     response.raise_for_status()
Mar 14 21:58:56.624809 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 1021, in raise_for_status
Mar 14 21:58:56.624836 ixre-cpm-chassis17 INFO swss.sh[1351668]:     raise HTTPError(http_error_msg, response=self)
Mar 14 21:58:56.625653 ixre-cpm-chassis17 INFO swss.sh[1351668]: Exception in thread Thread-2:
Mar 14 21:58:56.625694 ixre-cpm-chassis17 INFO swss.sh[1351668]: requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/swss5/wait
Mar 14 21:58:56.625724 ixre-cpm-chassis17 INFO swss.sh[1351668]: Traceback (most recent call last):
Mar 14 21:58:56.625750 ixre-cpm-chassis17 INFO swss.sh[1351668]: The above exception was the direct cause of the following exception:
Mar 14 21:58:56.625786 ixre-cpm-chassis17 INFO swss.sh[1351668]: Traceback (most recent call last):
Mar 14 21:58:56.625814 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 268, in _raise_for_status
Mar 14 21:58:56.625841 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
Mar 14 21:58:56.626035 ixre-cpm-chassis17 INFO swss.sh[1351668]:     self.run()
Mar 14 21:58:56.626083 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/lib/python3.9/threading.py", line 892, in run
Mar 14 21:58:56.626231 ixre-cpm-chassis17 INFO swss.sh[1351668]:     self._target(*self._args, **self._kwargs)
Mar 14 21:58:56.626268 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/bin/docker-wait-any", line 52, in wait_for_container
Mar 14 21:58:56.626296 ixre-cpm-chassis17 INFO swss.sh[1351668]:     response.raise_for_status()
Mar 14 21:58:56.626332 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 1021, in raise_for_status
Mar 14 21:58:56.626368 ixre-cpm-chassis17 INFO swss.sh[1351668]:     docker_client.wait(container_name)
Mar 14 21:58:56.626395 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/utils/decorators.py", line 19, in wrapped
Mar 14 21:58:56.626531 ixre-cpm-chassis17 INFO swss.sh[1351668]:     return f(self, resource_id, *args, **kwargs)
Mar 14 21:58:56.626605 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/container.py", line 1339, in wait
Mar 14 21:58:56.627013 ixre-cpm-chassis17 INFO swss.sh[1351668]:     raise HTTPError(http_error_msg, response=self)
Mar 14 21:58:56.627059 ixre-cpm-chassis17 INFO swss.sh[1351668]:     return self._result(res, True)
Mar 14 21:58:56.627090 ixre-cpm-chassis17 INFO swss.sh[1351668]: requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/syncd5/wait
Mar 14 21:58:56.627118 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 274, in _result
Mar 14 21:58:56.627154 ixre-cpm-chassis17 INFO swss.sh[1351668]: The above exception was the direct cause of the following exception:
Mar 14 21:58:56.627187 ixre-cpm-chassis17 INFO swss.sh[1351668]: Traceback (most recent call last):
Mar 14 21:58:56.627223 ixre-cpm-chassis17 INFO swss.sh[1351668]:     self._raise_for_status(response)
Mar 14 21:58:56.627252 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 270, in _raise_for_status
Mar 14 21:58:56.627280 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
Mar 14 21:58:56.627479 ixre-cpm-chassis17 INFO swss.sh[1351669]: Exception in thread Thread-1:
Mar 14 21:58:56.627515 ixre-cpm-chassis17 INFO swss.sh[1351669]: Traceback (most recent call last):
Mar 14 21:58:56.627542 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 268, in _raise_for_status
Mar 14 21:58:56.627580 ixre-cpm-chassis17 INFO swss.sh[1351668]:     raise create_api_error_from_http_exception(e) from e
Mar 14 21:58:56.627610 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
Mar 14 21:58:56.627643 ixre-cpm-chassis17 INFO swss.sh[1351668]:     raise cls(e, response=response, explanation=explanation) from e
Mar 14 21:58:56.627679 ixre-cpm-chassis17 INFO swss.sh[1351668]: docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.41/containers/swss5/wait: Not Found ("No such container: swss5")
Mar 14 21:58:56.627926 ixre-cpm-chassis17 INFO swss.sh[1351668]:     self.run()
Mar 14 21:58:56.627962 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/lib/python3.9/threading.py", line 892, in run
Mar 14 21:58:56.628366 ixre-cpm-chassis17 INFO swss.sh[1351669]: Exception in thread Thread-2:
Mar 14 21:58:56.628415 ixre-cpm-chassis17 INFO swss.sh[1351668]:     self._target(*self._args, **self._kwargs)
Mar 14 21:58:56.628451 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/bin/docker-wait-any", line 52, in wait_for_container
Mar 14 21:58:56.628499 ixre-cpm-chassis17 INFO swss.sh[1351669]:     response.raise_for_status()
Mar 14 21:58:56.628534 ixre-cpm-chassis17 INFO swss.sh[1351669]: Traceback (most recent call last):
Mar 14 21:58:56.628571 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 1021, in raise_for_status
Mar 14 21:58:56.628607 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 268, in _raise_for_status
Mar 14 21:58:56.628643 ixre-cpm-chassis17 INFO swss.sh[1351668]:     docker_client.wait(container_name)
Mar 14 21:58:56.628672 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/utils/decorators.py", line 19, in wrapped
Mar 14 21:58:56.628713 ixre-cpm-chassis17 INFO swss.sh[1351668]:     return f(self, resource_id, *args, **kwargs)
Mar 14 21:58:56.628752 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/container.py", line 1339, in wait
Mar 14 21:58:56.628791 ixre-cpm-chassis17 INFO swss.sh[1351669]:     raise HTTPError(http_error_msg, response=self)
Mar 14 21:58:56.628819 ixre-cpm-chassis17 INFO swss.sh[1351669]: requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/swss4/wait
Mar 14 21:58:56.628847 ixre-cpm-chassis17 INFO swss.sh[1351669]: The above exception was the direct cause of the following exception:
Mar 14 21:58:56.628873 ixre-cpm-chassis17 INFO swss.sh[1351669]: Traceback (most recent call last):
Mar 14 21:58:56.629028 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
Mar 14 21:58:56.629064 ixre-cpm-chassis17 INFO swss.sh[1351669]:     response.raise_for_status()
Mar 14 21:58:56.629237 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 1021, in raise_for_status
Mar 14 21:58:56.629273 ixre-cpm-chassis17 INFO swss.sh[1351669]:     self.run()
Mar 14 21:58:56.629306 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/lib/python3.9/threading.py", line 892, in run
Mar 14 21:58:56.629342 ixre-cpm-chassis17 INFO swss.sh[1351668]:     return self._result(res, True)
Mar 14 21:58:56.629371 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 274, in _result
Mar 14 21:58:56.629524 ixre-cpm-chassis17 INFO swss.sh[1351668]:     self._raise_for_status(response)
Mar 14 21:58:56.629559 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 270, in _raise_for_status
Mar 14 21:58:56.629622 ixre-cpm-chassis17 INFO swss.sh[1351669]:     self._target(*self._args, **self._kwargs)
Mar 14 21:58:56.629737 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/bin/docker-wait-any", line 52, in wait_for_container
Mar 14 21:58:56.629773 ixre-cpm-chassis17 INFO swss.sh[1351668]:     raise create_api_error_from_http_exception(e) from e
Mar 14 21:58:56.629802 ixre-cpm-chassis17 INFO swss.sh[1351668]:   File "/usr/local/lib/python3.9/dist-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
Mar 14 21:58:56.629868 ixre-cpm-chassis17 INFO swss.sh[1351668]:     raise cls(e, response=response, explanation=explanation) from e
Mar 14 21:58:56.629906 ixre-cpm-chassis17 INFO swss.sh[1351668]: docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.41/containers/syncd5/wait: Not Found ("No such container: syncd5")
Mar 14 21:58:56.629979 ixre-cpm-chassis17 INFO swss.sh[1351669]:     raise HTTPError(http_error_msg, response=self)
Mar 14 21:58:56.630073 ixre-cpm-chassis17 INFO swss.sh[1351669]:     docker_client.wait(container_name)
Mar 14 21:58:56.630102 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/docker/utils/decorators.py", line 19, in wrapped
Mar 14 21:58:56.630138 ixre-cpm-chassis17 INFO swss.sh[1351669]: requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/containers/syncd4/wait
Mar 14 21:58:56.630168 ixre-cpm-chassis17 INFO swss.sh[1351669]: The above exception was the direct cause of the following exception:
Mar 14 21:58:56.630195 ixre-cpm-chassis17 INFO swss.sh[1351669]: Traceback (most recent call last):
Mar 14 21:58:56.630223 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
Mar 14 21:58:56.630258 ixre-cpm-chassis17 INFO swss.sh[1351669]:     return f(self, resource_id, *args, **kwargs)
Mar 14 21:58:56.630289 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/container.py", line 1339, in wait
Mar 14 21:58:56.630551 ixre-cpm-chassis17 INFO swss.sh[1351669]:     self.run()
Mar 14 21:58:56.630586 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/lib/python3.9/threading.py", line 892, in run
Mar 14 21:58:56.630796 ixre-cpm-chassis17 INFO swss.sh[1351669]:     return self._result(res, True)
Mar 14 21:58:56.630832 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 274, in _result
Mar 14 21:58:56.631150 ixre-cpm-chassis17 INFO swss.sh[1351669]:     self._target(*self._args, **self._kwargs)
Mar 14 21:58:56.631187 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/bin/docker-wait-any", line 52, in wait_for_container
Mar 14 21:58:56.631374 ixre-cpm-chassis17 INFO swss.sh[1351669]:     docker_client.wait(container_name)
Mar 14 21:58:56.631485 ixre-cpm-chassis17 INFO swss.sh[1351669]:   File "/usr/local/lib/python3.9/dist-packages/docker/utils/decorators.py", line 19, in wrapped
Mar 14 21:58:56.631534 ixre-cpm-chassis17 INFO swss.sh[1351669]:     self._raise_for_status(response)

Describe the results you expected:

swss.sh should not show any stacktrace for this operation.

Output of show version:

Although the issue is shown on 202205 image, we believe it is in all branches.
admin@ixre-cpm-chassis17:~$ show version

SONiC Software Version: SONiC.20220532.55
SONiC OS Version: 11
Distribution: Debian 11.8
Kernel: 5.10.0-23-2-amd64
Build commit: 99f2e7b93b
Build date: Sat Feb 10 09:05:02 UTC 2024
Built by: cloudtest@65620ad2c000000

Platform: x86_64-nokia_ixr7250e_sup-r0
HwSKU: Nokia-IXR7250E-SUP-10
ASIC: broadcom
ASIC Count: 10
Serial Number: NS214510028
Model Number: N/A
Hardware Revision: 56
Uptime: 13:59:38 up 5 days, 15:01, 1 user, load average: 1.94, 1.98, 2.02
Date: Mon 18 Mar 2024 13:59:38

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

@mlok-nokia mlok-nokia changed the title [chassis][sfm][swss] Backtrace of swss shows in syslog on all empty SFM slot when shutdown or remove a SFM on Supervisor [chassis][sfm][swss] swss.sh shows backtrace for all empty SFM slots when shutdown or remove a SFM on Supervisor Mar 18, 2024
@prgeor prgeor added the Triaged this issue has been triaged label Mar 27, 2024
lguohan pushed a commit that referenced this issue Mar 30, 2024
…wn a SFM. (#18393)

On a Supervisor card of a VOQ chassis, when remove or shutdown a Fabric card, swss.sh shows Stacktrace for all related empty SFM slots in the syslog file. This PR fixes #18384

How I did it
In the asic_status.py, all empty SFM slots related swss.sh is in the waiting state to wait for the presence event of SFM -- SET operation. The subscriber event handler also includes the "DEL" operation when a SFM is shutdown/removal. When a SFM is shutdown, all empty slot's swss.sh also get the "DEL" event although it is not for them. In the "DEL" operation, the current implementation doesn't check if this "DEL" operation for them, and then they exit the wait state and proceed to docker-wait-any with wrong operation in the wrong slot. docker-wait0any raise the backtarce.

How to verify it
In a chassis which has some empty SMF slot, remove or shutdown a SFM. There should not be related stacktrace shown in syslog

Signed-off-by: mlok <marty.lok@nokia.com>
mlok-nokia added a commit to mlok-nokia/sonic-buildimage that referenced this issue Jun 5, 2024
…wn a SFM. (sonic-net#18393)

On a Supervisor card of a VOQ chassis, when remove or shutdown a Fabric card, swss.sh shows Stacktrace for all related empty SFM slots in the syslog file. This PR fixes sonic-net#18384

How I did it
In the asic_status.py, all empty SFM slots related swss.sh is in the waiting state to wait for the presence event of SFM -- SET operation. The subscriber event handler also includes the "DEL" operation when a SFM is shutdown/removal. When a SFM is shutdown, all empty slot's swss.sh also get the "DEL" event although it is not for them. In the "DEL" operation, the current implementation doesn't check if this "DEL" operation for them, and then they exit the wait state and proceed to docker-wait-any with wrong operation in the wrong slot. docker-wait0any raise the backtarce.

How to verify it
In a chassis which has some empty SMF slot, remove or shutdown a SFM. There should not be related stacktrace shown in syslog

Signed-off-by: mlok <marty.lok@nokia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Triaged this issue has been triaged
Projects
Status: Done
2 participants