forked from iovisor/bcc
-
Notifications
You must be signed in to change notification settings - Fork 0
/
fdleak_example.txt
156 lines (141 loc) · 6.1 KB
/
fdleak_example.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
Demonstrations of fdleak.
Trace and display fd alloc/close to detect FD leaks processes.
The tool's purpose is to know whether it is normal to call open/close in pairs during use..
If the survival time is longer than the minimum allowable survival time,
and the fd number opened by the process is keeps growing, That will trigger
fdleak to collect allocator information: Process name, PID, Thread name, TID, Backtrace, Trace hits
`In UNIX, everything is a file`, file leakage problems may occur when
most resources are used improperly, eg:
file, socket, eventfd, pipe, ashmem, dmabuf, syncfence ...
usage: fdleak [-h] [-p PID] [--lsof] [-i INTERVAL] [-m MIN_ALLOW] [-D]
----
For example:
# ./fdleak
Trace and display fd alloc/close to detect FD leaks process,Hit Ctrl-C to exit.
[23:32:27] NAME-PID LIMIT MAX_SURVIVAL(s) MAX_FD THREADs NAME-TID list
test_fd_leak-15456 -1 31 99 1 fdleak_0-15470
BackTrace:
- Stack ID:53126 Hits:2
- User Stack
open+0xd4 [libpthread-2.31.so]
[unknown]
[23:32:57] NAME-PID LIMIT MAX_SURVIVAL(s) MAX_FD THREADs NAME-TID list
test_fd_leak-15456 -1 61 130 1 fdleak_0-15470
BackTrace:
- Stack ID:53126 Hits:33
- User Stack
open+0xd4 [libpthread-2.31.so]
[unknown]
^C
Bye~Bye...
---->
^ 1 theads in `test_fd_leak-15456` triggered fd leak monitor
----
Simple info print is difficult to find the rootcause of file leakage .
you need to obtain the file type used by the defective process to help analysis.
Use the '--lsof' switch:
----
# ./fdleak --lsof
Trace and display over-threshold fds to detect FD leaks process, ctrl-c to exit.
[23:40:35] NAME-PID LIMIT MAX_SURVIVAL(s) MAX_FD THREADs NAME-TID list
test_fd_leak-19851 -1 30 93 1 fdleak_0-19854
BackTrace:
- Stack ID:32511 Hits:6
- User Stack
open+0xd4 [libpthread-2.31.so]
[unknown]
total 0
dr-x------ 2 vachel vachel 0 oct 26 23:51 .
dr-xr-xr-x 9 vachel vachel 0 oct 26 23:50 ..
lrwx------ 1 vachel vachel 64 oct 26 23:51 0 -> /dev/pts/2
lrwx------ 1 vachel vachel 64 oct 26 23:51 1 -> /dev/pts/2
lr-x------ 1 vachel vachel 64 oct 26 23:51 10 -> /dev/null
lr-x------ 1 vachel vachel 64 oct 26 23:51 103 -> /usr/share/code/v8_context_snapshot.bin
lr-x------ 1 vachel vachel 64 oct 26 23:51 11 -> /dev/null
lr-x------ 1 vachel vachel 64 oct 26 23:51 12 -> /dev/null
lr-x------ 1 vachel vachel 64 oct 26 23:51 13 -> /dev/null
lr-x------ 1 vachel vachel 64 oct 26 23:51 14 -> /dev/null
lr-x------ 1 vachel vachel 64 oct 26 23:51 15 -> /dev/null
lr-x------ 1 vachel vachel 64 oct 26 23:51 16 -> /dev/null
lr-x------ 1 vachel vachel 64 oct 26 23:51 17 -> /dev/null
lr-x------ 1 vachel vachel 64 oct 26 23:51 18 -> /dev/null
lr-x------ 1 vachel vachel 64 oct 26 23:51 19 -> /dev/null
lrwx------ 1 vachel vachel 64 oct 26 23:51 2 -> /dev/pts/2
lr-x------ 1 vachel vachel 64 oct 26 23:51 20 -> /dev/null
lr-x------ 1 vachel vachel 64 oct 26 23:51 21 -> /usr/share/code/v8_context_snapshot.bin
lr-x------ 1 vachel vachel 64 oct 26 23:51 22 -> /dev/null
lr-x------ 1 vachel vachel 64 oct 26 23:51 23 -> /dev/null
lr-x------ 1 vachel vachel 64 oct 26 23:51 24 -> /dev/null
lr-x------ 1 vachel vachel 64 oct 26 23:51 25 -> /dev/null
lr-x------ 1 vachel vachel 64 oct 26 23:51 26 -> /dev/null
---->
^ fdleak thread open too much `/dev/null`
----
If you need to observe the data monitored by fdleak, please use the `--csv` option
# ./fdleak --csv > /tmp/fdleak.csv
Trace and display over-threshold fds to detect FD leaks process, ctrl-c to exit.
Time,Name-PID,FDs
23:44:21,python_18466,2
23:44:21,test_fd_leak_17673,10
23:44:31,python_18466,2
23:44:31,test_fd_leak_17673,20
23:44:41,python_18466,2
23:44:41,v2ray_6703,1
23:44:41,test_fd_leak_17673,30
23:44:51,python_18466,2
23:44:51,v2ray_6703,1
23:44:51,test_fd_leak_17673,40
23:45:01,python_18466,2
23:45:01,v2ray_6703,1
23:45:01,test_fd_leak_17673,50
23:45:11,python_18466,2
23:45:11,v2ray_6703,1
23:45:11,test_fd_leak_17673,60
23:45:21,python_18466,2
23:45:21,v2ray_6703,1
23:45:21,test_fd_leak_17673,70
let me use seaborn to draw a simple line chart
```
import pandas as pd
import seaborn as sns
data = pd.read_csv('/tmp/fdleak.csv', skiprows=1,error_bad_lines=False)
sns.lineplot(data=data,x='Time',y='FDs',hue='Name-PID',style='Name-PID')
```
----
You can use the `-p` option to filter the process you want to monitor,
The purpose is to know whether it is normal to call open/close in pairs during use..
If you have a method to improve, pls contact me. thank you.
Use the '-p' switch:
# ./fdleak -p `pidof test_fd_leak`
Trace and display fd alloc/close to detect FD leaks process, ctrl-c to exit.
[21:17:59] stack id-6031 in 33 allocations from stack
PNAME-PID LIMIT MAX_SURVIVAL(s) MAX_FD THREADs NAME-TID list
test_fd_leak-89710 unlimited 33 96 1 test_fd_leak-89710
Backtrace:
open+0x5b [libpthread-2.31.so]
[unknown]
----
USAGE message:
usage: fdleak.py [-h] [-p PID] [--csv] [--lsof] [-i INTERVAL]
Trace FD leak
optional arguments:
-h, --help show this help message and exit
-p PID, --pid PID trace this PID only
--csv just print fields: comma-separated values
--lsof list the files opened by the monitor process
-i INTERVAL, --interval INTERVAL
interval (in seconds) to scan for outstanding
allocations
EXAMPLES:
./fdleak
Trace all process alloc/close file descriptor.
default internal inspection frequency is 10 seconds,
default minimim allowable survival time is 30 seconds.
./fdleak -p $(pidof allocs)
Only monitor the allocation and close files of filtered pid
./fdleak --csv
Print fields: Time,Name-PID,FDs
./fdleak --lsof
List the files opened by the monitor process
./fdleak -i 60
Set the internal inspection frequency to 60 seconds