[Monit] Monitor multiple processes with the same name but using different arguments. #4257

yozhao101 · 2020-03-12T16:44:40Z

- What I did
This script is used to monitor teamd process and dhcrelay process in teamd and dhcp_relay
docker container respectively. Since Monit can only monitor the process with unique name,
it is unable to do this monitoring for teamd and dhcrelay processes. Usually there will be
multiple teamd and dhcrelay processes which executes a same commad but with different arguments.

- How I did it
The number of teamd processes is decided by the number of port channels in Config_DB and
the number of dhcrelay processes is determined by Vlans which have non-empry list of dhcp servers. As such, we let Monit to monitor this script which will read number of port channles and
vlans with no-empty list of dhcp servers form Config_DB, then find whether there exist a
process in Linux corresponding to a port channel or a vlan. If this script fails to find
such process, it will write an alert message into syslog file.

- How to verify it
We can explicitly kill a teamd process or dhcrelay process and then check whether there
will be an alert message written in syslog file.

different arguments. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

name is valid or not. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

jleveque · 2020-03-12T17:36:00Z

files/image_config/monit/monit_multiprocesses.py

@@ -0,0 +1,88 @@
+#!/usr/bin/python


I think this file should be broken into two separate files, one for teamd and one for dhcp_relay. In the repo, the files should reside in the directories of their respective dockers.

Will break it into two separate files and place each one into their docker directories in the repo.

teamd and dhcrelay processes. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

check_teamd_processes. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

jleveque · 2020-03-13T17:52:10Z

dockers/docker-dhcp-relay/monit_dhcrelay_processes

+Since Monit can only monitor the process with unique name, it is unable to do 
+this monitoring for dhcrelay processes. Usually there will be multiple dhcrelay 
+processes which executes a same commad but with different arguments. The number 
+of dhcrelay processes is determined by Vlans which have non-empry list of dhcp servers.


s/non-empry/non-empty/

jleveque · 2020-03-13T17:52:40Z

dockers/docker-dhcp-relay/monit_dhcrelay_processes

+#!/usr/bin/python
+'''
+This script is used to monitor dhcrelay processes in dhcp_relay docker container. 
+Since Monit can only monitor the process with unique name, it is unable to do 


s/the process with unique name/processes with unique names/

jleveque · 2020-03-13T17:53:30Z

dockers/docker-dhcp-relay/monit_dhcrelay_processes

+'''
+This script is used to monitor dhcrelay processes in dhcp_relay docker container. 
+Since Monit can only monitor the process with unique name, it is unable to do 
+this monitoring for dhcrelay processes. Usually there will be multiple dhcrelay 


s/Usually there will be multiple/There can exist multiple/

jleveque · 2020-03-13T17:54:10Z

dockers/docker-dhcp-relay/monit_dhcrelay_processes

+This script is used to monitor dhcrelay processes in dhcp_relay docker container. 
+Since Monit can only monitor the process with unique name, it is unable to do 
+this monitoring for dhcrelay processes. Usually there will be multiple dhcrelay 
+processes which executes a same commad but with different arguments. The number 


s/commad/command

jleveque · 2020-03-13T17:54:55Z

dockers/docker-dhcp-relay/monit_dhcrelay_processes

+processes which executes a same commad but with different arguments. The number 
+of dhcrelay processes is determined by Vlans which have non-empry list of dhcp servers.
+As such, we let Monit to monitor this script which will read number of vlans with 
+no-empty list of dhcp servers form Config_DB, then find whether there exist a 


s/no-empry/non-empty/

jleveque · 2020-03-13T17:55:12Z

dockers/docker-dhcp-relay/monit_dhcrelay_processes

+processes which executes a same commad but with different arguments. The number 
+of dhcrelay processes is determined by Vlans which have non-empry list of dhcp servers.
+As such, we let Monit to monitor this script which will read number of vlans with 
+no-empty list of dhcp servers form Config_DB, then find whether there exist a 


s/exist/exists/

jleveque · 2020-03-13T17:55:26Z

dockers/docker-dhcp-relay/monit_dhcrelay_processes

+of dhcrelay processes is determined by Vlans which have non-empry list of dhcp servers.
+As such, we let Monit to monitor this script which will read number of vlans with 
+no-empty list of dhcp servers form Config_DB, then find whether there exist a 
+process in Linux corresponding to a vlan. If this script fails to find  such process, 


Remove extra space before "such"

Removed extra space.

jleveque · 2020-03-13T18:16:36Z

dockers/docker-teamd/monit_teamd_processes

+
+def check_teamd_processes():
+    port_channels = retrieve_portchannels()
+    cmd = "sudo monit procmatch '/usr/bin/teamd -r -t '"


Rather than making a call to monit, I'd prefer if we use a Python library like psutil.

Good suggestion! I will do that. I also found psutil library is not installed by default in host image.

I used psutil library to check whether one of teamd processes is running or not. Please help me review.

jleveque · 2020-03-13T18:16:55Z

dockers/docker-dhcp-relay/monit_dhcrelay_processes

+
+def check_dhcrelay_processes():
+    vlans = retrieve_vlans()
+    cmd = "sudo monit procmatch '/usr/sbin/dhcrelay -d -m discard'"


Rather than making a call to monit, I'd prefer if we use a Python library like psutil.

I used psutil library to check whether one of dhcrelay processes is running or not. Please help me review.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

…relay processes is running or not. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

lguohan · 2020-03-16T12:36:48Z

dockers/docker-dhcp-relay/base_image_files/monit_dhcrelay_processes

+
+from swsssdk import ConfigDBConnector
+
+def retrieve_vlans():


the approach is complicated. suggest to use supervisor ctl to check

yozhao101 added 3 commits March 12, 2020 09:19

[Monit] We use Monit to monitor multiprocess with the same name but with

2c6a35f

different arguments. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[Monit] Add new Monit configuration files for teamd and dhcp_relay.

448cb6b

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[Monit] Add an else statement in script to decide whether the container

da8fbd6

name is valid or not. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

yozhao101 requested a review from jleveque March 12, 2020 16:44

jleveque suggested changes Mar 12, 2020

View reviewed changes

yozhao101 added 3 commits March 13, 2020 09:55

[Monitoring] Split the single script file into two separate files for

52a7efd

teamd and dhcrelay processes. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[Monitoring] Change the functions names from check_teamd_process to

97765c1

check_teamd_processes. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[Monitoring] Change the lines in docker-teamd.mk file.

0249e6a

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

jleveque suggested changes Mar 13, 2020

View reviewed changes

jleveque added Enhancement ➕ Request for 201911 Branch labels Mar 13, 2020

yozhao101 added 3 commits March 13, 2020 13:41

[Monitoring] Reword the comments in two scripts files.

828a7ae

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[Monitoring] Use psutil library to decide whether one of teamd or dhc…

bb2c63f

…relay processes is running or not. Signed-off-by: Yong Zhao <yozhao@microsoft.com>

[Monitoring] Delete extra blank lines.

976906b

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

lguohan reviewed Mar 16, 2020

View reviewed changes

lguohan force-pushed the master branch from 3690c1a to 512eb6b Compare December 25, 2020 18:34

yozhao101 requested review from qiluo-msft and xumia as code owners February 6, 2021 20:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Monit] Monitor multiple processes with the same name but using different arguments. #4257

[Monit] Monitor multiple processes with the same name but using different arguments. #4257

yozhao101 commented Mar 12, 2020

jleveque Mar 12, 2020

yozhao101 Mar 12, 2020

jleveque Mar 13, 2020

yozhao101 Mar 13, 2020

jleveque Mar 13, 2020

yozhao101 Mar 13, 2020

jleveque Mar 13, 2020

yozhao101 Mar 13, 2020

jleveque Mar 13, 2020

yozhao101 Mar 13, 2020

jleveque Mar 13, 2020

yozhao101 Mar 13, 2020

jleveque Mar 13, 2020

yozhao101 Mar 13, 2020

jleveque Mar 13, 2020

yozhao101 Mar 13, 2020

jleveque Mar 13, 2020

yozhao101 Mar 13, 2020 •

edited

Loading

yozhao101 Mar 15, 2020

jleveque Mar 13, 2020

yozhao101 Mar 15, 2020

lguohan Mar 16, 2020

[Monit] Monitor multiple processes with the same name but using different arguments. #4257

Are you sure you want to change the base?

[Monit] Monitor multiple processes with the same name but using different arguments. #4257

Conversation

yozhao101 commented Mar 12, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yozhao101 Mar 13, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yozhao101 Mar 13, 2020 •

edited

Loading