Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

linkerd-tcp connection reuse issue #42

Open
stevej opened this issue Apr 20, 2017 · 1 comment
Open

linkerd-tcp connection reuse issue #42

stevej opened this issue Apr 20, 2017 · 1 comment

Comments

@stevej
Copy link

stevej commented Apr 20, 2017 •
edited
Loading

Running a load test without connection reuse (this is part of my testing out the tacho OrderMap changes) and linkerd-tcp starts rejecting traffic after a thousand requests.

stevej@netty-test-8:~$ ./slow_cooker_linux_amd64 -noreuse -host 'default' -totalRequests 1000000 -qps 100 -concurrency 100 http://proxy-test-4d:7474/
# sending 10000 req/s with concurrency=100 to http://proxy-test-4d:7474/ ...
#                      good/b/f t   good%   min [p50 p95 p99  p999]  max change
2017-04-20T21:47:03Z  14105/0/0 100000  14% 10s   5 [ 44  47  48   54 ]   54 +
2017-04-20T21:47:13Z      0/0/0 100000   0% 10s   0 [  0   0   0    0 ]    0 -
2017-04-20T21:47:23Z   1039/0/0 100000   1% 10s  40 [ 54 15319 15327 15335 ] 15329 +++
2017-04-20T21:47:33Z    780/0/0 100000   0% 10s  40 [1060 3081 3083 3083 ] 3083
2017-04-20T21:47:43Z    659/0/0 100000   0% 10s  44 [1048 3073 3075 4073 ] 4073

CPU was at 1% and traffic was trickling through. Here's some example strace output

recvfrom(78, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(70, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(58, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(70, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(65, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(51, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(65, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(60, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(46, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(60, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(53, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(39, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(53, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(48, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(33, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(48, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(41, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(27, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(41, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(36, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(20, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(36, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(30, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(17, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(30, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(23, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(14, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(23, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)

and an lsof report

stevej@proxy-test-4d:~$ sudo lsof -p 14178
COMMAND     PID   USER   FD      TYPE    DEVICE SIZE/OFF      NODE NAME
linkerd-t 14178 stevej  cwd       DIR       8,1     4096   1812484 /home/stevej/src/linkerd-tcp
linkerd-t 14178 stevej  rtd       DIR       8,1     4096         2 /
linkerd-t 14178 stevej  txt       REG       8,1  9851408   1816829 /home/stevej/src/linkerd-tcp/target/release/
linkerd-tcp
linkerd-t 14178 stevej  mem       REG       8,1  1088952      1907 /lib/x86_64-linux-gnu/libm-2.24.so
linkerd-t 14178 stevej  mem       REG       8,1  1856752      1903 /lib/x86_64-linux-gnu/libc-2.24.so
linkerd-t 14178 stevej  mem       REG       8,1    92552      1931 /lib/x86_64-linux-gnu/libgcc_s.so.1
linkerd-t 14178 stevej  mem       REG       8,1   142400      1918 /lib/x86_64-linux-gnu/libpthread-2.24.so
linkerd-t 14178 stevej  mem       REG       8,1    31712      1920 /lib/x86_64-linux-gnu/librt-2.24.so
linkerd-t 14178 stevej  mem       REG       8,1    14608      1906 /lib/x86_64-linux-gnu/libdl-2.24.so
linkerd-t 14178 stevej  mem       REG       8,1   158512      1899 /lib/x86_64-linux-gnu/ld-2.24.so
linkerd-t 14178 stevej    0u      CHR     136,3      0t0         6 /dev/pts/3
linkerd-t 14178 stevej    1u      CHR     136,3      0t0         6 /dev/pts/3
linkerd-t 14178 stevej    2u      CHR     136,3      0t0         6 /dev/pts/3
linkerd-t 14178 stevej    3u  a_inode      0,11        0      8469 [eventpoll]
linkerd-t 14178 stevej    4r     FIFO      0,10      0t0 908911732 pipe
linkerd-t 14178 stevej    5w     FIFO      0,10      0t0 908911732 pipe
linkerd-t 14178 stevej    6u     IPv4 908911733      0t0       TCP *:7474 (LISTEN)
linkerd-t 14178 stevej    7u  a_inode      0,11        0      8469 [eventpoll]
linkerd-t 14178 stevej    8r     FIFO      0,10      0t0 908910932 pipe
linkerd-t 14178 stevej    9w     FIFO      0,10      0t0 908910932 pipe
linkerd-t 14178 stevej   10u     IPv4 908911215      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33709 (ESTABLISHED)
linkerd-t 14178 stevej   11u     IPv4 908911205      0t0       TCP proxy-test-4d:54924->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej   12u     IPv4 908910933      0t0       TCP *:9989 (LISTEN)
linkerd-t 14178 stevej   13u     IPv4 908910934      0t0       TCP localhost:50620->localhost:4180 (ESTABLISHED
)
linkerd-t 14178 stevej   14u     IPv4 908910257      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33616 (ESTABLISHED)
linkerd-t 14178 stevej   15u     IPv4 908910258      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33604 (ESTABLISHED)
linkerd-t 14178 stevej   16u     IPv4 908910259      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33609 (ESTABLISHED)
linkerd-t 14178 stevej   17u     IPv4 908910260      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33622 (ESTABLISHED)
linkerd-t 14178 stevej   18u     IPv4 908910261      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33624 (ESTABLISHED)
linkerd-t 14178 stevej   19u     IPv4 908910262      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33620 (ESTABLISHED)
[cutting hundreds of other sockets that are open both upstream and downstream]
linkerd-t 14178 stevej  327u     IPv4 908911328      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33773 (ESTABLISHED)
linkerd-t 14178 stevej  328u     IPv4 908911331      0t0       TCP proxy-test-4d:42162->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  329u     IPv4 908911332      0t0       TCP proxy-test-4d:7474->netty-test-8.c[124/1893]osted.internal:33770 (ESTABLISHED)
linkerd-t 14178 stevej  330u     IPv4 908911333      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33772 (ESTABLISHED)
linkerd-t 14178 stevej  331u     IPv4 908911334      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33775 (ESTABLISHED)
linkerd-t 14178 stevej  332u     IPv4 908911335      0t0       TCP proxy-test-4d:38944->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  333u     IPv4 908911336      0t0       TCP proxy-test-4d:55042->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  334u     IPv4 908911337      0t0       TCP proxy-test-4d:42168->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  335u     IPv4 908911338      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33769 (ESTABLISHED)
linkerd-t 14178 stevej  336u     IPv4 908911339      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33771 (ESTABLISHED)
linkerd-t 14178 stevej  337u     IPv4 908911340      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33765 (ESTABLISHED)
linkerd-t 14178 stevej  338u     IPv4 908911343      0t0       TCP proxy-test-4d:38954->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  339u     IPv4 908911344      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33777 (ESTABLISHED)
linkerd-t 14178 stevej  340u     IPv4 908911345      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33778 (ESTABLISHED)
linkerd-t 14178 stevej  341u     IPv4 908911346      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33776 (ESTABLISHED)
linkerd-t 14178 stevej  342u     IPv4 908911347      0t0       TCP proxy-test-4d:38956->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  343u     IPv4 908911348      0t0       TCP proxy-test-4d:55054->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  344u     IPv4 908911349      0t0       TCP proxy-test-4d:42180->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  345u     IPv4 908911350      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33780 (ESTABLISHED)
linkerd-t 14178 stevej  346u     IPv4 908911351      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33782 (ESTABLISHED)
linkerd-t 14178 stevej  347u     IPv4 908911352      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33781 (ESTABLISHED)
linkerd-t 14178 stevej  348u     IPv4 908911353      0t0       TCP proxy-test-4d:42182->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  349u     IPv4 908911354      0t0       TCP proxy-test-4d:55060->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  350u     IPv4 908911355      0t0       TCP proxy-test-4d:38966->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  351u     IPv4 908911356      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33779 (ESTABLISHED)
linkerd-t 14178 stevej  352u     IPv4 908911357      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33785 (ESTABLISHED)
linkerd-t 14178 stevej  353u     IPv4 908911358      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33783 (ESTABLISHED)
linkerd-t 14178 stevej  354u     IPv4 908911359      0t0       TCP proxy-test-4d:38968->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  355u     IPv4 908911360      0t0       TCP proxy-test-4d:55066->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  356u     IPv4 908911361      0t0       TCP proxy-test-4d:42192->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  357u     IPv4 908911362      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33784 (ESTABLISHED)
linkerd-t 14178 stevej  358u     IPv4 908911363      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33790 (ESTABLISHED)
linkerd-t 14178 stevej  359u     IPv4 908911364      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33787 (ESTABLISHED)
linkerd-t 14178 stevej  360u     IPv4 908911365      0t0       TCP proxy-test-4d:55070->perf-target-4.[62/1893]-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  361u     IPv4 908911366      0t0       TCP proxy-test-4d:38976->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  362u     IPv4 908911367      0t0       TCP proxy-test-4d:42198->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  363u     IPv4 908911368      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33788 (ESTABLISHED)
linkerd-t 14178 stevej  364u     IPv4 908911369      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33789 (ESTABLISHED)
linkerd-t 14178 stevej  365u     IPv4 908911370      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33774 (ESTABLISHED)
linkerd-t 14178 stevej  366u     IPv4 908911371      0t0       TCP proxy-test-4d:55076->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  367u     IPv4 908911372      0t0       TCP proxy-test-4d:38982->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  368u     IPv4 908911373      0t0       TCP proxy-test-4d:42204->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  369u     IPv4 908911374      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33793 (ESTABLISHED)
linkerd-t 14178 stevej  370u     IPv4 908911375      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33792 (ESTABLISHED)
linkerd-t 14178 stevej  371u     IPv4 908911376      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33786 (ESTABLISHED)
linkerd-t 14178 stevej  372u     IPv4 908911377      0t0       TCP proxy-test-4d:55082->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  373u     IPv4 908911378      0t0       TCP proxy-test-4d:38988->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  374u     IPv4 908911379      0t0       TCP proxy-test-4d:42210->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  375u     IPv4 908911380      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33794 (ESTABLISHED)
linkerd-t 14178 stevej  376u     IPv4 908911381      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33797 (ESTABLISHED)
linkerd-t 14178 stevej  377u     IPv4 908911382      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33798 (ESTABLISHED)
linkerd-t 14178 stevej  378u     IPv4 908911383      0t0       TCP proxy-test-4d:42212->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  379u     IPv4 908911384      0t0       TCP proxy-test-4d:55090->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  380u     IPv4 908911385      0t0       TCP proxy-test-4d:38996->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  381u     IPv4 908911386      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33801 (ESTABLISHED)
linkerd-t 14178 stevej  382u     IPv4 908911387      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33799 (ESTABLISHED)
linkerd-t 14178 stevej  383u     IPv4 908911388      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33795 (ESTABLISHED)
linkerd-t 14178 stevej  384u     IPv4 908911389      0t0       TCP proxy-test-4d:38998->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  385u     IPv4 908911390      0t0       TCP proxy-test-4d:55096->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  386u     IPv4 908911391      0t0       TCP proxy-test-4d:42222->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  387u     IPv4 908911392      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33791 (ESTABLISHED)
linkerd-t 14178 stevej  388u     IPv4 908911393      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33796 (ESTABLISHED)
linkerd-t 14178 stevej  389u     IPv4 908911394      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33800 (ESTABLISHED)
linkerd-t 14178 stevej  390u     IPv4 908911395      0t0       TCP proxy-test-4d:55100->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  391u     IPv4 908911396      0t0       TCP proxy-test-4d:39006->perf-target-6.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  392u     IPv4 908911397      0t0       TCP proxy-test-4d:42228->perf-target-5.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  393u     IPv4 908911398      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:33802 (ESTABLISHED)
linkerd-t 14178 stevej  394u     IPv4 908911399      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:33807 (ESTABLISHED)
linkerd-t 14178 stevej  395u     IPv4 908911400      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:33803 (ESTABLISHED)
linkerd-t 14178 stevej  396u     IPv4 908911401      0t0       TCP proxy-test-4d:55106->perf-target-4.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  397u     IPv4 908911402      0t0       TCP proxy-test-4d:39012->perf-target-6.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  398u     IPv4 908911403      0t0       TCP proxy-test-4d:42234->perf-target-5.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  399u     IPv4 908911404      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:33805 (ESTABLISHED)
linkerd-t 14178 stevej  400u     IPv4 908911405      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:33806 (ESTABLISHED)
linkerd-t 14178 stevej  401u     IPv4 908911406      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:33804 (ESTABLISHED)
linkerd-t 14178 stevej  402u     IPv4 908911407      0t0       TCP proxy-test-4d:55112->perf-target-4.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  403u     IPv4 908911408      0t0       TCP proxy-test-4d:39018->perf-target-6.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  404u     IPv4 908911409      0t0       TCP proxy-test-4d:42240->perf-target-5.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  405u     IPv4 908911410      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:33808 (ESTABLISHED)
linkerd-t 14178 stevej  406u     IPv4 909244034      0t0       TCP proxy-test-4d:44928->perf-target-5.c.buoyant-hosted.internal:http-alt (SYN_SENT)
linkerd-t 14178 stevej  407u     sock       0,8      0t0 909253650 protocol: TCP
linkerd-t 14178 stevej  408u     IPv4 908911413      0t0       TCP proxy-test-4d:39022->perf-target-6.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  409u     IPv4 908911414      0t0       TCP proxy-test-4d:55120->perf-target-4.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  410u     IPv4 908911415      0t0       TCP proxy-test-4d:42246->perf-target-5.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  411u     IPv4 908911417      0t0       TCP proxy-test-4d:42248->perf-target-5.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  412u     IPv4 908911418      0t0       TCP proxy-test-4d:55126->perf-target-4.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  413u     IPv4 908911419      0t0       TCP proxy-test-4d:39032->perf-target-6.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  414u     sock       0,8      0t0 909253652 protocol: TCP
linkerd-t 14178 stevej  415u     sock       0,8      0t0 909253654 protocol: TCP
linkerd-t 14178 stevej  416u     sock       0,8      0t0 909253637 protocol: TCP
linkerd-t 14178 stevej  417u  unknown                              /proc/14178/fd/417 (readlink: No such file or directory)
linkerd-t 14178 stevej  418u     sock       0,8      0t0 909253639 protocol: TCP
linkerd-t 14178 stevej  419u     sock       0,8      0t0 909252607 protocol: TCP
linkerd-t 14178 stevej  420u     sock       0,8      0t0 909253647 protocol: TCP
linkerd-t 14178 stevej  421u     sock       0,8      0t0 909253645 protocol: TCP
linkerd-t 14178 stevej  422u     sock       0,8      0t0 909253649 protocol: TCP
linkerd-t 14178 stevej  423u     IPv4 909235806      0t0       TCP proxy-test-4d:33498->perf-target-6.c.buoyant-hosted.internal:http-alt (SYN_SENT)
linkerd-t 14178 stevej  424u     sock       0,8      0t0 909253641 protocol: TCP
linkerd-t 14178 stevej  425u     sock       0,8      0t0 909253643 protocol: TCP
linkerd-t 14178 stevej  427u     sock       0,8      0t0 909253653 protocol: TCP
linkerd-t 14178 stevej  428u     IPv4 909253633      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:39840 (ESTABLISHED)
linkerd-t 14178 stevej  429u     sock       0,8      0t0 909253635 protocol: TCP
@stevej
Copy link
Author

stevej commented Apr 20, 2017

Good news: the same process will happily route requests after being in this state

stevej@netty-test-8:~$ ./slow_cooker_linux_amd64 -host 'default' -totalRequests 1000000 -qps 100 -concurrency 100 http://proxy-test-4d:7474/
# sending 10000 req/s with concurrency=100 to http://proxy-test-4d:7474/ ...
#                      good/b/f t   good%   min [p50 p95 p99  p999]  max change
2017-04-20T22:02:32Z  99667/0/0 100000  99% 10s   0 [  2   4   4   74 ]   74 +
2017-04-20T22:02:42Z 100000/0/0 100000 100% 10s   0 [  2   4   5   13 ]   13 +
2017-04-20T22:02:52Z 100002/0/0 100000 100% 10s   0 [  2   4   4   14 ]   14
2017-04-20T22:03:02Z 100000/0/0 100000 100% 10s   0 [  2   4   4    9 ]    9
2017-04-20T22:03:12Z 100000/0/0 100000 100% 10s   0 [  2   4   4   12 ]   12
2017-04-20T22:03:22Z 100000/0/0 100000 100% 10s   0 [  2   4   5   11 ]   11
2017-04-20T22:03:32Z 100000/0/0 100000 100% 10s   0 [  2   4   5   11 ]   11
2017-04-20T22:03:42Z 100002/0/0 100000 100% 10s   0 [  2   4   5    9 ]    9
2017-04-20T22:03:52Z 100044/0/0 100000 100% 10s   0 [  2   4   5    9 ]    9
2017-04-20T22:04:02Z  99975/0/0 100000  99% 10s   0 [  2   4   5    9 ]    9
2017-04-20T22:04:12Z 100010/0/0 100000 100% 10s   0 [  2   4   5    8 ]    8
FROM    TO #REQUESTS
   0     2 167367
   2     8 932083
   8    32 196
  32    64 41
  64   128 13
 128   256 0
 256   512 0
 512  1024 0
1024  4096 0
4096 16384 0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@stevej