A tiny command line tool that provides a dynamic real-time view of the active threads for a given process with stats of CPU, disk and scheduling.
The information comes from pidstat (provided by systat) and /proc/{pid}/schedstat. Also if the target process is a java process and the jstack Oracle tool is available it will try to use it to extract additional information such as thread name and stack trace.
The output can be viewed in full screen (similar to top
or watch
) or it can be printed on the terminal.
On each iteration, the following stats are displayed:
- CPU usage: total, %usr, %system, %guest and %wait
- Disk usege: kB read per second and kB written per second
- Scheduler stats: time spent on the cpu, time spent waiting on a run queue (runqueue latency) and number of timeslices run on the current CPU.
- Java details: in case the target is a Java process that can be attached with
jstack
, some extra details is showed such as thread name and stack traces.
Python 3
- systat
It's can be used only on linux
platform.
wget -O top_threads.py 'https://github.com/cspinetta/top-threads/releases/download/0.0.1/top_threads.py' \
&& chmod +x top_threads.py
# watch <pid>'s threads with default values
./top_threads.py -p <pid>
# print output in the terminal
./top_threads.py -p <pid> --display terminal
# sorting by run queue latency
./top_threads.py -p <pid> --sort rq
# in case a java process, change the number of stack traces to display
./top_threads.py -p <pid> --max-stack-depth 10
# enable debug log for troubleshooting
./top_threads.py -p <pid> --debug
Notes:
- The first output is with stats from the first execution of the process.
--display refresh
provides a view similar totop
orwatch
(the default) whileterminal
prints the output on each iteration in the terminal likepidstat
.
usage: top_threads.py [-h] -p PID [-n [NUMBER]]
[--max-stack-depth [STACK_SIZE]]
[--sort [{cpu,rq,disk,disk-rd,disk-wr}]]
[--display [{terminal,refresh}]] [--no-jstack] [--debug]
Tool for analysing active Threads
optional arguments:
-h, --help show this help message and exit
-p PID Process ID
-n [NUMBER] Number of threads to show per sample. Default: 10
--max-stack-depth [STACK_SIZE], -m [STACK_SIZE]
Max number of stack frames (only when jstack can be
used). Default: 1
--sort [{cpu,rq,disk,disk-rd,disk-wr}], -s [{cpu,rq,disk,disk-rd,disk-wr}]
Field used for sorting. Default: cpu
--display [{terminal,refresh}], -d [{terminal,refresh}]
Select the way to display the info: terminal or
refresh. Default: refresh
--no-jstack Turn off usage of jstack to retrieve thread info like
name and stack
--debug Turn on logs for debugging purposes
This tool comes from the need to want to see the time each thread spends in the runqueue waiting to be able to start running. That is a really useful metric to understand if the process is being slow down because the CPU is saturated. For this reason this tool emerged. It gets information from pidstat, /proc/{pid}/schedstat and jstack:
- pidstat to get cpu and disk usage metrics from each thread in time interval.
- /proc/{pid}/schedstat to gets metrics from the runqueue.
- jstack is used in case the process that is beaing monitored is an attachable java process, to obtain information such as thread name and stack traces.
I often use this script when I have to analyze a performance problem at thread level and I want to inspect the dynamic usage of the cpu or the disk.
Some questions this tool help me to answer:
- Which thread is eating the entire CPU?
- How long are the threads waiting to take the CPU?
- What threads are using the disk right now?
The run queue latency is the metric I usually look at first because it's difficult to get from other traditional system tools and this script displays it at thread level.
The importance of this metric comes from the fact that the run queue latency is an excellent metric to identify CPU saturation.
In case you suspect you are being limited by a CPU saturation, you may look for a tool that help you to analyze the runqueue in more details. If you have root privileges you can try with BCC: linux-bcc-runqlat
- With
--display refresh
(the default):
- With
--display terminal
: