memory usage explosion with provided examples #113

acgreek · 2021-10-01T17:54:46Z

I used your client to stream in 1.5 MB messages at high throughput and I am seeing high memory usage. I'm going to look into optimizing for this scenario, but was wondering if you have any thought of how I could ago about do this.

I guess I could also go about creating a simple example, I'll do that first

acgreek · 2021-10-02T01:55:22Z

Ok, I think this is a memory leak

diff --git a/nbhttp/websocket/upgrader.go b/nbhttp/websocket/upgrader.go
index aa682bb..9facc7c 100644
--- a/nbhttp/websocket/upgrader.go
+++ b/nbhttp/websocket/upgrader.go
@@ -447,6 +455,7 @@ func (u *Upgrader) handleMessage(p *nbhttp.Parser, opcode MessageType, body []by
 
        p.Execute(func() {
                u.handleWsMessage(u.conn, opcode, body)
+               mempool.Free(body)
        })
 
 }

lesismal · 2021-10-02T02:58:11Z

It's not a memory leak.

Try to set ReleaseWebsocketPayload = true:
https://github.com/lesismal/nbio/blob/master/examples/websocket_1m/server/server.go#L48

That will release the buffer:
https://github.com/lesismal/nbio/blob/master/nbhttp/websocket/upgrader.go#L127
https://github.com/lesismal/nbio/blob/master/nbhttp/websocket/upgrader.go#L139

For HTTP, we can release the Body after the Handler is done because it's one-request-one-response:
https://github.com/lesismal/nbio/blob/master/nbhttp/processor.go#L254
https://github.com/lesismal/nbio/blob/master/nbhttp/processor.go#L265
https://github.com/lesismal/nbio/blob/master/nbhttp/processor.go#L52

But For Websocket, the user may hold the data and handle it in another goroutine, if we release the buffer afther the Handler is done, the buffer may be Re-Get by other goroutine and be dirty:

u.OnMessage(func(c *websocket.Conn, messageType websocket.MessageType, data []byte) {
	go func() {
		// do something
		c.WriteMessage(messageType, data)
	}()
})

So we use this configurable field ReleaseWebsocketPayload, let the user decide whether to release it automatically after the Handler is done.

lesismal · 2021-10-02T03:07:47Z

Ok, I think this is a memory leak
diff --git a/nbhttp/websocket/upgrader.go b/nbhttp/websocket/upgrader.go
index aa682bb..9facc7c 100644
--- a/nbhttp/websocket/upgrader.go
+++ b/nbhttp/websocket/upgrader.go
@@ -447,6 +455,7 @@ func (u *Upgrader) handleMessage(p *nbhttp.Parser, opcode MessageType, body []by
 
        p.Execute(func() {
                u.handleWsMessage(u.conn, opcode, body)
+               mempool.Free(body)
        })
 
 }
Try to set ReleaseWebsocketPayload = true: https://github.com/lesismal/nbio/blob/master/examples/websocket_1m/server/server.go#L48

That will release the buffer: https://github.com/lesismal/nbio/blob/master/nbhttp/websocket/upgrader.go#L127 https://github.com/lesismal/nbio/blob/master/nbhttp/websocket/upgrader.go#L139

For HTTP, we can release the Body after the Handler is done because it's one-request-one-response. But For Websocket, the user may hold the data and handle it in another goroutine, if we release the buffer afther the Handler is done, the buffer may be Re-Get by other goroutine and be dirty:
u.OnMessage(func(c *websocket.Conn, messageType websocket.MessageType, data []byte) {
	go func() {
		// do something
		c.WriteMessage(messageType, data)
	}()
})
So we use this configurable field ReleaseWebsocketPayload, let the user decide whether to release it automatically after the Handler is done.

It's not a memory leak and it seems there's no difference between the readAppend/messageAppend and the previous code, so we don't need to merge https://github.com/lesismal/nbio/pull/114/files.

But your issue reminds me that we don't release the Ping/Pong/Close messages' buffer:
https://github.com/lesismal/nbio/blob/master/nbhttp/websocket/upgrader.go#L100
https://github.com/lesismal/nbio/blob/master/nbhttp/websocket/upgrader.go#L107
https://github.com/lesismal/nbio/blob/master/nbhttp/websocket/upgrader.go#L114
https://github.com/lesismal/nbio/blob/master/nbhttp/websocket/upgrader.go#L464

Although it matters little for the release of Ping/Pong/Close's buffer, we should add release logic for them like this:
https://github.com/lesismal/nbio/blob/master/nbhttp/websocket/upgrader.go#L126
at here:
https://github.com/lesismal/nbio/blob/master/nbhttp/websocket/upgrader.go#L464

lesismal · 2021-10-02T03:15:36Z

If we use time.AfterFunc in the message handler and hold the buffer, it will execute in another goroutine, then we should not set ReleaseWebsocketPayload=true, but we can release the buffer like this:

u.OnMessage(func(c *websocket.Conn, messageType websocket.MessageType, data []byte) {
	// echo
	time.AfterFunc(time.Second, func() {
		c.WriteMessage(messageType, data)
		mempool.Free(data)
	})
}

acgreek · 2021-10-02T12:47:10Z

ok, thanks. That seemed to have corrected half of my high memory usage. After running a lot of data through and then stopping the clients and waiting a while, I still see 3+ GBs held within the connection

acgreek · 2021-10-02T14:01:14Z

#115

When I set the TLSAllocator to the mempool.NativeAllocator and push through a lot of data, I don't see memory growth growing endlessly as I do when I use the DefaultAllocator. There seems to be leak in the TLS connection code.

lesismal · 2021-10-02T14:14:22Z

Are they the sample you use for testing?
https://github.com/lesismal/nbio/blob/large-message-flood/examples/websocket/server/server.go
https://github.com/lesismal/nbio/blob/large-message-flood/examples/websocket/nbioclient/client.go

acgreek · 2021-10-02T14:29:45Z

No sorry, haven't gotten back to using the examples yet. Would need to add TLS, but should be the same otherwise.

acgreek · 2021-10-02T17:44:06Z

golang/go#23199

lesismal · 2021-10-03T02:35:33Z

golang/go#23199

golang/go#48535 (comment)

nbio is for general scenarios, and specific optimizations can be made for specific scenarios.
Please provide me the samples, I'll look into it.

lesismal · 2021-10-03T11:13:38Z

I tried some test, did not found that huge memory cost:

ENV

root@ubuntu:~/dev/gopath/src/github.com# go version
go version go1.16.6 linux/amd64
root@ubuntu:~/dev/gopath/src/github.com# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.3 LTS
Release:	18.04
Codename:	bionic
root@ubuntu:~/dev/gopath/src/github.com# cat /proc/cpuinfo | grep name | cut -f2 -d: | uniq -c
      8  Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz

Server Cost

root@ubuntu:~/dev/gopath/src/github.com/lesismal/nbio# top -d 1 | grep server
 44371 root      20   0 2470128 212804   6416 R 338.9  2.6   0:32.11 server                               
 44371 root      20   0 2470128 237756   6416 S 381.4  2.9   0:36.00 server                               
 44371 root      20   0 2470128 238044   6416 R 378.4  2.9   0:39.86 server                               
 44371 root      20   0 2470128 237676   6416 S 371.8  2.9   0:43.69 server                               
 44371 root      20   0 2470128 237396   6416 R 379.2  2.9   0:47.52 server                               
 44371 root      20   0 2470128 229720   6416 S 373.5  2.8   0:51.33 server                               
 44371 root      20   0 2470128 229492   6416 R 373.5  2.8   0:55.14 server                               
 44371 root      20   0 2470128 216920   6416 S 372.8  2.7   0:58.98 server                               
 44371 root      20   0 2412.2m 229.5m   6.3m R 379.7  2.9   1:01.79 server                               
 44371 root      20   0 2412.2m 225.8m   6.3m S 377.5  2.8   1:05.64 server                               
 44371 root      20   0 2412.2m 222.8m   6.3m S 372.8  2.8   1:09.48 server                               
 44371 root      20   0 2412.2m 222.7m   6.3m S 371.8  2.8   1:13.31 server                               
 44371 root      20   0 2412.2m 237.4m   6.3m S 370.9  3.0   1:17.13 server                               
 44371 root      20   0 2412.2m 237.2m   6.3m R 379.4  3.0   1:21.00 server                               
 44371 root      20   0 2412.2m 236.9m   6.3m S 373.1  3.0   1:24.88 server                               
 44371 root      20   0 2412.2m 236.6m   6.3m S 372.5  3.0   1:28.68 server                               
 44371 root      20   0 2412.2m 236.1m   6.3m S 381.4  3.0   1:32.57 server                               
 44371 root      20   0 2412.2m 213.9m   6.3m S 367.3  2.7   1:36.39 server                               
 44371 root      20   0 2412.2m 237.6m   6.3m R 381.2  3.0   1:40.24 server                               
 44371 root      20   0 2412.2m 207.7m   6.3m R 367.3  2.6   1:44.06 server                               
 44371 root      20   0 2412.2m 233.6m   6.3m R 378.2  2.9   1:47.88 server                               
 44371 root      20   0 2412.2m 207.8m   6.3m R 373.8  2.6   1:51.73 server                               
 44371 root      20   0 2412.2m 222.5m   6.3m S 369.6  2.8   1:55.50 server                               
 44371 root      20   0 2412.2m 222.7m   6.3m S 371.3  2.8   1:59.25 server                               
 44371 root      20   0 2412.2m 195.9m   6.3m S 202.0  2.5   2:01.31 server                               
 44371 root      20   0 2412.2m 186.7m   6.3m S 186.0  2.3   2:03.17 server                               
 44371 root      20   0 2412.2m 196.6m   6.3m S 363.1  2.5   2:06.91 server                               
 44371 root      20   0 2412.2m 210.1m   6.3m S 364.7  2.6   2:10.63 server                               
 44371 root      20   0 2412.2m 210.1m   6.3m R 360.2  2.6   2:14.34 server                               
 44371 root      20   0 2412.2m 225.5m   6.3m R 371.8  2.8   2:18.17 server                               
 44371 root      20   0 2412.2m 240.6m   6.3m S 370.9  3.0   2:21.99 server                               
 44371 root      20   0 2412.2m 240.5m   6.3m S 367.6  3.0   2:25.74 server                               
 44371 root      20   0 2412.2m 240.1m   6.3m S 368.9  3.0   2:29.54 server                               
 44371 root      20   0 2412.2m 198.0m   6.3m R 365.7  2.5   2:33.27 server                               
 44371 root      20   0 2412.2m 180.2m   6.3m S 165.3  2.3   2:34.94 server                               
 44371 root      20   0 2412.2m 213.1m   6.3m S 195.0  2.7   2:36.91 server                               
 44371 root      20   0 2412.2m 230.1m   6.3m R 370.3  2.9   2:40.65 server                               
 44371 root      20   0 2412.2m 230.6m   6.3m R 362.7  2.9   2:44.35 server                               
 44371 root      20   0 2412.2m 244.1m   6.3m R 364.7  3.1   2:48.07 server                               
 44371 root      20   0 2478.3m 259.4m   6.3m S 376.2  3.3   2:51.87 server                               
 44371 root      20   0 2478.3m 263.2m   6.3m S 364.1  3.3   2:55.62 server                               
 44371 root      20   0 2478.3m 262.9m   6.3m S 375.5  3.3   2:59.45 server                               
 44371 root      20   0 2478.3m 262.8m   6.3m R 376.5  3.3   3:03.29 server                               
 44371 root      20   0 2478.3m 262.3m   6.3m S 365.4  3.3   3:07.09 server                               
 44371 root      20   0 2478.3m 214.3m   6.3m S 370.6  2.7   3:10.87 server                               
 44371 root      20   0 2478.3m 215.7m   6.3m R 381.2  2.7   3:14.72 server                               
 44371 root      20   0 2478.3m 243.1m   6.3m S 366.3  3.1   3:18.53 server                               
 44371 root      20   0 2478.3m 234.3m   6.3m S 254.9  2.9   3:21.13 server                               
 44371 root      20   0 2478.3m 202.7m   6.3m S 188.0  2.5   3:23.01 server                               
 44371 root      20   0 2478.3m 240.6m   6.3m S 365.0  3.0   3:26.77 server                               
 44371 root      20   0 2478.3m 240.6m   6.3m R 370.3  3.0   3:30.51 server                               
 44371 root      20   0 2478.3m 258.7m   6.3m S 372.3  3.3   3:34.27 server                               
 44371 root      20   0 2478.3m 273.8m   6.3m R 365.0  3.4   3:38.03 server                               
 44371 root      20   0 2478.3m 273.5m   6.3m R 377.2  3.4   3:41.84 server                               
 44371 root      20   0 2478.3m 273.3m   6.3m R 372.5  3.4   3:45.64 server                               
 44371 root      20   0 2478.3m 272.9m   6.3m S 380.2  3.4   3:49.48 server                               
 44371 root      20   0 2478.3m 260.5m   6.3m R 368.6  3.3   3:53.24 server                               
 44371 root      20   0 2478.3m 260.1m   6.3m S 366.7  3.3   3:56.98 server                               
 44371 root      20   0 2478.3m 259.7m   6.3m S 370.3  3.3   4:00.72 server                               
 44371 root      20   0 2478.3m 259.3m   6.3m R 373.8  3.3   4:04.57 server                               
 44371 root      20   0 2478.3m 258.9m   6.3m S 370.9  3.3   4:08.39 server                               
 44371 root      20   0 2478.3m 246.6m   6.3m R 367.0  3.1   4:12.17 server                               
 44371 root      20   0 2478.3m 224.1m   6.3m S 380.2  2.8   4:16.01 server                               
 44371 root      20   0 2478.3m 226.2m   6.3m S 357.7  2.8   4:19.73 server                               
 44371 root      20   0 2478.3m 226.7m   6.3m S 366.7  2.8   4:23.47 server                               
 44371 root      20   0 2478.3m 226.6m   6.3m R 364.7  2.8   4:27.19 server                               
 44371 root      20   0 2478.3m 226.3m   6.3m S 374.5  2.8   4:31.01 server                               
 44371 root      20   0 2478.3m 277.8m   6.3m R 367.6  3.5   4:34.76 server                               
 44371 root      20   0 2478.3m 278.9m   6.3m S 374.8  3.5   4:38.62 server                               
 44371 root      20   0 2478.3m 244.6m   6.3m S 360.6  3.1   4:42.37 server                               
 44371 root      20   0 2478.3m 238.5m   6.3m S 370.3  3.0   4:46.11 server                               
 44371 root      20   0 2478.3m 250.8m   6.3m S 353.4  3.2   4:49.75 server                               
 44371 root      20   0 2478.3m 236.8m   6.3m R 367.3  3.0   4:53.46 server                               
 44371 root      20   0 2478.3m 229.4m   6.3m R 360.2  2.9   4:57.17 server                               
 44371 root      20   0 2478.3m 259.2m   6.3m R 368.6  3.3   5:00.93 server                               
 44371 root      20   0 2478.3m 266.6m   6.3m R 360.2  3.4   5:04.64 server                               
 44371 root      20   0 2478.3m 266.5m   6.3m S 360.4  3.4   5:08.28 server                               
 44371 root      20   0 2478.3m 266.2m   6.3m R 364.4  3.3   5:11.96 server                               
 44371 root      20   0 2478.3m 271.0m   6.3m R 355.9  3.4   5:15.59 server                               
 44371 root      20   0 2478.3m 271.0m   6.3m S 365.3  3.4   5:19.28 server                               
 44371 root      20   0 2478.3m 256.4m   6.3m R 367.0  3.2   5:23.06 server                               
 44371 root      20   0 2478.3m 252.7m   6.3m S 368.3  3.2   5:26.78 server                               
 44371 root      20   0 2478.3m 237.7m   6.3m S 359.8  3.0   5:30.45 server                               
 44371 root      20   0 2478.3m 249.0m   6.3m R 363.4  3.1   5:34.12 server                               
 44371 root      20   0 2478.3m 248.9m   6.3m S 364.7  3.1   5:37.84 server                               
 44371 root      20   0 2478.3m 248.6m   6.3m S 358.8  3.1   5:41.50 server                               
 44371 root      20   0 2478.3m 248.3m   6.3m S 370.9  3.1   5:45.32 server                               
 44371 root      20   0 2478.3m 247.8m   6.3m S 353.4  3.1   5:48.96 server                               
 44371 root      20   0 2478.3m 247.5m   6.3m R 358.8  3.1   5:52.62 server                               
 44371 root      20   0 2478.3m 272.5m   6.3m S 368.3  3.4   5:56.34 server                               
 44371 root      20   0 2478.3m 273.1m   6.3m R 364.1  3.4   6:00.09 server                               
 44371 root      20   0 2478.3m 270.5m   6.3m S 359.8  3.4   6:03.76 server                               
 44371 root      20   0 2478.3m 270.2m   6.3m S 359.8  3.4   6:07.43 server                               
 44371 root      20   0 2478.3m 269.8m   6.3m S 360.8  3.4   6:11.11 server                               
 44371 root      20   0 2478.3m 269.4m   6.3m S 362.7  3.4   6:14.81 server                               
 44371 root      20   0 2478.3m 269.0m   6.3m R 361.8  3.4   6:18.50 server                               
 44371 root      20   0 2478.3m 262.6m   6.3m S 359.8  3.3   6:22.17 server                               
 44371 root      20   0 2478.3m 262.2m   6.3m S 359.2  3.3   6:25.87 server                               
 44371 root      20   0 2478.3m 267.7m   6.3m S 362.7  3.4   6:29.57 server                               
 44371 root      20   0 2478.3m 268.0m   6.3m R 353.4  3.4   6:33.21 server                               
 44371 root      20   0 2478.3m 267.6m   6.3m S 366.7  3.4   6:36.95 server                               
 44371 root      20   0 2478.3m 267.2m   6.3m R 367.0  3.4   6:40.73 server                               
 44371 root      20   0 2478.3m 270.2m   6.3m R 366.7  3.4   6:44.47 server                               
 44371 root      20   0 2478.3m 269.9m   6.3m S 362.1  3.4   6:48.20 server                               
 44371 root      20   0 2478.3m 269.5m   6.3m S 370.3  3.4   6:51.94 server                               
 44371 root      20   0 2478.3m 269.1m   6.3m S 368.9  3.4   6:55.74 server                               
 44371 root      20   0 2478.3m 256.7m   6.3m S 361.2  3.2   6:59.46 server                               
 44371 root      20   0 2478.3m 236.2m   6.3m S 364.1  3.0   7:03.21 server                               
 44371 root      20   0 2478.3m 237.2m   6.3m R 362.7  3.0   7:06.91 server                               
 44371 root      20   0 2478.3m 236.8m   6.3m R 365.7  3.0   7:10.64 server                               
 44371 root      20   0 2478.3m 198.7m   6.3m S 363.4  2.5   7:14.31 server                               
 44371 root      20   0 2478.3m 229.5m   6.3m R 376.5  2.9   7:18.15 server                               
 44371 root      20   0 2478.3m 229.2m   6.3m S 351.5  2.9   7:21.77 server                               
 44371 root      20   0 2478.3m 228.8m   6.3m S 361.8  2.9   7:25.46 server                               
 44371 root      20   0 2478.3m 211.7m   6.3m S 368.6  2.7   7:29.22 server                               
 44371 root      20   0 2478.3m 228.6m   6.3m R 367.6  2.9   7:32.97 server                               
 44371 root      20   0 2478.3m 263.0m   6.3m S 367.3  3.3   7:36.68 server                               
 44371 root      20   0 2478.3m 292.7m   6.3m R 362.7  3.7   7:40.38 server                               
 44371 root      20   0 2478.3m 292.3m   6.3m R 370.3  3.7   7:44.12 server                               
 44371 root      20   0 2478.3m 291.9m   6.3m R 370.6  3.7   7:47.90 server                               
 44371 root      20   0 2478.3m 291.4m   6.3m R 373.8  3.7   7:51.75 server                               
 44371 root      20   0 2478.3m 253.2m   6.3m S 357.3  3.2   7:55.43 server                               
 44371 root      20   0 2478.3m 250.4m   6.3m S 364.7  3.1   7:59.15 server                               
 44371 root      20   0 2478.3m 250.2m   6.3m R 360.2  3.1   8:02.86 server                               
 44371 root      20   0 2478.3m 249.8m   6.3m R 366.0  3.1   8:06.63 server                               
 44371 root      20   0 2478.3m 249.4m   6.3m S 360.4  3.1   8:10.27 server                               
 44371 root      20   0 2478.3m 233.2m   6.3m R 357.8  2.9   8:13.92 server                               
 44371 root      20   0 2478.3m 207.1m   6.3m S 366.3  2.6   8:17.62 server                               
 44371 root      20   0 2478.3m 219.0m   6.3m S 357.4  2.8   8:21.23 server                               
 44371 root      20   0 2478.3m 224.5m   6.3m R 355.9  2.8   8:24.86 server                               
 44371 root      20   0 2478.3m 232.7m   6.3m S 370.3  2.9   8:28.60 server                               
 44371 root      20   0 2478.3m 251.1m   6.3m S 360.2  3.2   8:32.31 server                               
 44371 root      20   0 2478.3m 251.8m   6.3m R 373.3  3.2   8:36.08 server                               
 44371 root      20   0 2478.3m 271.1m   6.3m S 363.7  3.4   8:39.79 server                               
 44371 root      20   0 2478.3m 271.4m   6.3m S 368.3  3.4   8:43.51 server                               
 44371 root      20   0 2478.3m 217.2m   6.3m S 371.6  2.7   8:47.30 server                               
 44371 root      20   0 2478.3m 209.2m   6.3m R 363.1  2.6   8:51.04 server                               
 44371 root      20   0 2478.3m 237.4m   6.3m S 364.7  3.0   8:54.76 server                               
 44371 root      20   0 2478.3m 248.2m   6.3m R 371.3  3.1   8:58.51 server                               
 44371 root      20   0 2478.3m 256.2m   6.3m R 367.6  3.2   9:02.26 server                               
 44371 root      20   0 2478.3m 256.0m   6.3m S 365.0  3.2   9:06.02 server                               
 44371 root      20   0 2478.3m 255.7m   6.3m S 373.5  3.2   9:09.83 server                               
 44371 root      20   0 2478.3m 255.3m   6.3m S 369.6  3.2   9:13.60 server                               
 44371 root      20   0 2478.3m 227.2m   6.3m S 369.6  2.9   9:17.37 server                               
 44371 root      20   0 2478.3m 246.0m   6.3m S 368.6  3.1   9:21.13 server                               
 44371 root      20   0 2478.3m 246.0m   6.3m R 373.3  3.1   9:24.90 server                               
 44371 root      20   0 2478.3m 247.8m   6.3m S 357.8  3.1   9:28.55 server                               
 44371 root      20   0 2478.3m 247.6m   6.3m S 364.7  3.1   9:32.27 server                               
 44371 root      20   0 2478.3m 247.3m   6.3m S 369.3  3.1   9:36.00 server                               
 44371 root      20   0 2478.3m 230.9m   6.3m S 356.9  2.9   9:39.64 server                               
 44371 root      20   0 2478.3m 230.6m   6.3m S 371.6  2.9   9:43.43 server                               
 44371 root      20   0 2478.3m 236.6m   6.3m R 365.3  3.0   9:47.12 server                               
 44371 root      20   0 2478.3m 244.8m   6.3m R 364.7  3.1   9:50.84 server                               
 44371 root      20   0 2478.3m 244.6m   6.3m R 375.2  3.1   9:54.63 server                               
 44371 root      20   0 2478.3m 263.1m   6.3m R 370.6  3.3   9:58.41 server                               
 44371 root      20   0 2478.3m 265.1m   6.3m S 357.7  3.3  10:02.13 server                               
 44371 root      20   0 2478.3m 247.3m   6.3m R 367.3  3.1  10:05.84 server                               
 44371 root      20   0 2478.3m 241.3m   6.3m R 369.3  3.0  10:09.57 server                               
 44371 root      20   0 2478.3m 241.7m   6.3m S 350.5  3.0  10:13.18 server                               
 44371 root      20   0 2478.3m 241.4m   6.3m R 365.3  3.0  10:16.87 server                               
 44371 root      20   0 2550.3m 224.2m   6.3m S 370.6  2.8  10:20.65 server                               
 44371 root      20   0 2550.3m 251.8m   6.3m S 357.7  3.2  10:24.37 server                               
 44371 root      20   0 2550.3m 251.8m   6.3m S 362.1  3.2  10:28.10 server                               
 44371 root      20   0 2550.3m 257.6m   6.3m S 365.7  3.2  10:31.83 server                               
 44371 root      20   0 2550.3m 257.3m   6.3m S 364.7  3.2  10:35.55 server                               
 44371 root      20   0 2550.3m 229.1m   6.3m S 365.0  2.9  10:39.31 server                               
 44371 root      20   0 2550.3m 211.2m   6.3m S 360.8  2.7  10:42.99 server                               
 44371 root      20   0 2550.3m 241.1m   6.3m S 367.3  3.0  10:46.81 server                               
 44371 root      20   0 2550.3m 246.0m   6.3m S 365.0  3.1  10:50.57 server                               
 44371 root      20   0 2550.3m 253.1m   6.3m S 375.2  3.2  10:54.36 server                               
 44371 root      20   0 2550.3m 227.1m   6.3m S 366.7  2.9  10:58.10 server                               
 44371 root      20   0 2550.3m 202.1m   6.3m S 360.8  2.5  11:01.78 server                               
 44371 root      20   0 2550.3m 212.7m   6.3m S 368.9  2.7  11:05.58 server                               
 44371 root      20   0 2550.3m 274.1m   6.3m S 369.9  3.4  11:09.39 server                               
 44371 root      20   0 2550.3m 274.0m   6.3m S 368.6  3.4  11:13.15 server                               
 44371 root      20   0 2550.3m 273.8m   6.3m S 369.6  3.4  11:16.92 server                               
 44371 root      20   0 2550.3m 273.5m   6.3m S 370.6  3.4  11:20.70 server                               
 44371 root      20   0 2550.3m 265.1m   6.3m S 371.6  3.3  11:24.49 server                               
 44371 root      20   0 2550.3m 258.8m   6.3m S 369.9  3.3  11:28.30 server                               
 44371 root      20   0 2550.3m 258.5m   6.3m S 367.6  3.2  11:32.05 server                               
 44371 root      20   0 2550.3m 258.1m   6.3m S 371.6  3.2  11:35.84 server                               
 44371 root      20   0 2550.3m 257.7m   6.3m S 364.7  3.2  11:39.56 server                               
 44371 root      20   0 2550.3m 257.3m   6.3m S 373.3  3.2  11:43.33 server                               
 44371 root      20   0 2550.3m 215.1m   6.3m S 366.7  2.7  11:47.07 server                               
 44371 root      20   0 2550.3m 232.4m   6.3m S 375.2  2.9  11:50.86 server                               
 44371 root      20   0 2550.3m 214.4m   6.3m S 365.0  2.7  11:54.62 server                               
 44371 root      20   0 2550.3m 229.8m   6.3m S 374.5  2.9  11:58.44 server                               
 44371 root      20   0 2550.3m 226.7m   6.3m S 377.2  2.8  12:02.25 server                               
 44371 root      20   0 2550.3m 225.3m   6.3m S 367.6  2.8  12:06.00 server                               
 44371 root      20   0 2550.3m 232.9m   6.3m S 372.8  2.9  12:09.84 server                               
 44371 root      20   0 2550.3m 266.4m   6.3m S 376.5  3.4  12:13.68 server                               
 44371 root      20   0 2550.3m 266.3m   6.3m S 365.0  3.3  12:17.44 server                               
 44371 root      20   0 2550.3m 265.9m   6.3m S 365.7  3.3  12:21.17 server                               
 44371 root      20   0 2550.3m 265.5m   6.3m S 371.3  3.3  12:24.92 server                               
 44371 root      20   0 2550.3m 257.1m   6.3m S 369.6  3.2  12:28.69 server                               
 44371 root      20   0 2550.3m 246.1m   6.3m S 376.7  3.1  12:32.57 server                               
 44371 root      20   0 2550.3m 246.1m   6.3m S 368.6  3.1  12:36.33 server                               
 44371 root      20   0 2550.3m 245.9m   6.3m S 375.5  3.1  12:40.16 server                               
 44371 root      20   0 2550.3m 245.4m   6.3m S 372.5  3.1  12:43.96 server                               
 44371 root      20   0 2550.3m 272.0m   6.3m S 377.5  3.4  12:47.81 server                               
 44371 root      20   0 2550.3m 271.5m   6.3m S 366.0  3.4  12:51.58 server                               
 44371 root      20   0 2550.3m 269.1m   6.3m S 387.3  3.4  12:55.53 server                               
 44371 root      20   0 2550.3m 233.0m   6.3m S 361.2  2.9  12:59.25 server                               
 44371 root      20   0 2550.3m 232.6m   6.3m S 380.4  2.9  13:03.13 server                               
 44371 root      20   0 2550.3m 248.6m   6.3m S 372.5  3.1  13:06.93 server                               
 44371 root      20   0 2550.3m 248.5m   6.3m S 379.4  3.1  13:10.80 server                               
 44371 root      20   0 2550.3m 229.5m   6.3m S  72.8  2.9  13:11.55 server                               
 44371 root      20   0 2550.3m 219.1m   6.3m S  27.0  2.8  13:11.82 server                               
 44371 root      20   0 2550.3m 223.6m   6.3m S 372.5  2.8  13:15.62 server                               
 44371 root      20   0 2550.3m 223.5m   6.3m S 373.8  2.8  13:19.47 server                               
 44371 root      20   0 2550.3m 226.6m   6.3m S 365.7  2.8  13:23.20 server                               
 44371 root      20   0 2550.3m 231.3m   6.3m S 370.9  2.9  13:27.02 server                               
 44371 root      20   0 2550.3m 235.7m   6.3m S 365.0  3.0  13:30.78 server                               
 44371 root      20   0 2550.3m 235.3m   6.3m S 372.3  3.0  13:34.54 server                               
 44371 root      20   0 2550.3m 209.2m   6.3m S 383.3  2.6  13:38.45 server                               
 44371 root      20   0 2550.3m 240.0m   6.3m S 375.5  3.0  13:42.28 server                               
 44371 root      20   0 2550.3m 261.5m   6.3m S 370.6  3.3  13:46.06 server                               
 44371 root      20   0 2550.3m 261.7m   6.3m S 380.2  3.3  13:49.90 server                               
 44371 root      20   0 2550.3m 261.3m   6.3m S 368.6  3.3  13:53.66 server                               
 44371 root      20   0 2550.3m 199.2m   6.3m S 381.4  2.5  13:57.55 server                               
 44371 root      20   0 2550.3m 244.6m   6.3m S 368.9  3.1  14:01.35 server                               
 44371 root      20   0 2550.3m 244.8m   6.3m S 369.6  3.1  14:05.12 server                               
 44371 root      20   0 2550.3m 244.6m   6.3m S 370.6  3.1  14:08.90 server                               
 44371 root      20   0 2550.3m 244.2m   6.3m S 373.5  3.1  14:12.71 server                               
 44371 root      20   0 2550.3m 243.3m   6.3m S 311.9  3.1  14:15.86 server

Client Cost

root@ubuntu:~/dev/gopath/src/github.com# top -d 1 | grep client
 44489 root      20   0 2470024 187260   6280 S 331.6  2.3   0:42.69 client                               
 44489 root      20   0 2470024 187796   6280 S 352.4  2.3   0:46.32 client                               
 44489 root      20   0 2470024 210584   6280 S 351.0  2.6   0:49.90 client                               
 44489 root      20   0 2412.1m 205.6m   6.1m R 347.4  2.6   0:52.54 client                               
 44489 root      20   0 2412.1m 205.3m   6.1m R 357.8  2.6   0:56.19 client                               
 44489 root      20   0 2412.1m 205.0m   6.1m S 349.5  2.6   0:59.72 client                               
 44489 root      20   0 2412.1m 204.7m   6.1m S 343.7  2.6   1:03.26 client                               
 44489 root      20   0 2412.1m 230.6m   6.1m S 344.7  2.9   1:06.81 client                               
 44489 root      20   0 2412.1m 231.6m   6.1m S 358.4  2.9   1:10.43 client                               
 44489 root      20   0 2412.1m 231.6m   6.1m S 349.5  2.9   1:14.03 client                               
 44489 root      20   0 2412.1m 231.4m   6.1m S 354.5  2.9   1:17.61 client                               
 44489 root      20   0 2412.1m 231.0m   6.1m R 345.1  2.9   1:21.13 client                               
 44489 root      20   0 2412.1m 182.9m   6.1m S 344.1  2.3   1:24.64 client                               
 44489 root      20   0 2412.1m 178.5m   6.1m S 356.9  2.2   1:28.28 client                               
 44489 root      20   0 2412.1m 203.6m   6.1m R 339.8  2.6   1:31.78 client                               
 44489 root      20   0 2412.1m 203.4m   6.1m S 343.1  2.6   1:35.28 client                               
 44489 root      20   0 2412.1m 203.1m   6.1m S 342.2  2.6   1:38.77 client                               
 44489 root      20   0 2412.1m 196.8m   6.1m S 334.3  2.5   1:42.18 client                               
 44489 root      20   0 2412.1m 201.8m   6.1m R 343.6  2.5   1:45.65 client                               
 44489 root      20   0 2412.1m 201.8m   6.1m S 346.6  2.5   1:49.22 client                               
 44621 root      20   0 1928.0m  94.2m   6.2m S  23.8  1.2   0:00.24 client                               
 44621 root      20   0 2412.1m 193.0m   6.2m S 353.5  2.4   0:03.81 client                               
 44621 root      20   0 2412.1m 197.1m   6.2m S 349.5  2.5   0:07.41 client                               
 44621 root      20   0 2412.1m 197.1m   6.2m R 348.0  2.5   0:10.96 client                               
 44621 root      20   0 2412.1m 195.5m   6.2m R 353.9  2.5   0:14.57 client                               
 44621 root      20   0 2412.1m 195.6m   6.2m R 342.7  2.5   0:18.10 client                               
 44621 root      20   0 2412.1m 196.6m   6.2m S 351.0  2.5   0:21.68 client                               
 44621 root      20   0 2412.1m 210.2m   6.2m R 349.5  2.6   0:25.21 client                               
 44621 root      20   0 2412.1m 206.1m   6.2m S 344.7  2.6   0:28.76 client                               
 44737 root      20   0 1920.0m  84.1m   6.2m R  25.7  1.1   0:00.26 client                               
 44737 root      20   0 2346.1m 168.6m   6.2m S 342.2  2.1   0:03.75 client                               
 44737 root      20   0 2412.1m 198.3m   6.2m S 344.1  2.5   0:07.26 client                               
 44737 root      20   0 2484.1m 223.4m   6.2m S 355.9  2.8   0:10.89 client                               
 44737 root      20   0 2484.1m 223.3m   6.2m S 346.1  2.8   0:14.42 client                               
 44737 root      20   0 2484.1m 224.3m   6.2m R 341.2  2.8   0:17.90 client                               
 44737 root      20   0 2484.1m 186.2m   6.2m R 350.0  2.3   0:21.47 client                               
 44737 root      20   0 2484.1m 184.1m   6.2m S 348.0  2.3   0:25.02 client                               
 44737 root      20   0 2484.1m 198.9m   6.2m S 347.1  2.5   0:28.56 client                               
 44737 root      20   0 2484.1m 215.9m   6.2m R 341.7  2.7   0:32.08 client                               
 44737 root      20   0 2484.1m 216.6m   6.2m R 352.5  2.7   0:35.64 client                               
 44737 root      20   0 2484.1m 166.5m   6.2m R 346.6  2.1   0:39.21 client                               
 44737 root      20   0 2484.1m 201.6m   6.2m R 356.4  2.5   0:42.81 client                               
 44848 root      20   0 1920.0m  75.2m   6.1m S  23.5  0.9   0:00.24 client                               
 44848 root      20   0 2412.1m 201.6m   6.1m S 345.1  2.5   0:03.76 client                               
 44848 root      20   0 2484.1m 202.2m   6.1m S 354.9  2.5   0:07.38 client                               
 44848 root      20   0 2484.1m 201.8m   6.1m S 339.8  2.5   0:10.88 client                               
 44848 root      20   0 2484.1m 201.6m   6.1m S 350.0  2.5   0:14.45 client                               
 44848 root      20   0 2484.1m 201.4m   6.1m S 347.1  2.5   0:17.99 client                               
 44848 root      20   0 2484.1m 209.3m   6.1m S 347.6  2.6   0:21.57 client                               
 44848 root      20   0 2484.1m 209.8m   6.1m S 352.9  2.6   0:25.17 client                               
 44848 root      20   0 2484.1m 213.3m   6.1m S 349.5  2.7   0:28.70 client                               
 44848 root      20   0 2484.1m 213.6m   6.1m S 353.8  2.7   0:32.38 client                               
 44848 root      20   0 2484.1m 213.2m   6.1m S 346.6  2.7   0:35.95 client                               
 44848 root      20   0 2484.1m 212.8m   6.1m S 334.0  2.7   0:39.39 client                               
 44848 root      20   0 2484.1m 212.6m   6.1m S 349.5  2.7   0:42.92 client                               
 44848 root      20   0 2484.1m 182.5m   6.1m S 348.0  2.3   0:46.47 client                               
 44848 root      20   0 2484.1m 217.1m   6.1m S 347.1  2.7   0:50.01 client                               
 44848 root      20   0 2484.1m 217.9m   6.1m S 353.5  2.7   0:53.58 client                               
 44848 root      20   0 2484.1m 217.6m   6.1m S 339.8  2.7   0:57.08 client                               
 44848 root      20   0 2484.1m 217.3m   6.1m S 355.9  2.7   1:00.71 client                               
 44848 root      20   0 2484.1m 216.9m   6.1m S 364.4  2.7   1:04.39 client                               
 44848 root      20   0 2484.1m 212.7m   6.1m S 334.0  2.7   1:07.93 client                               
 44848 root      20   0 2484.1m 196.6m   6.1m S 339.6  2.5   1:11.36 client                               
 44848 root      20   0 2484.1m 196.4m   6.1m S 338.2  2.5   1:14.81 client                               
 44848 root      20   0 2484.1m 205.5m   6.1m S 337.3  2.6   1:18.25 client                               
 44848 root      20   0 2484.1m 210.9m   6.1m S 342.6  2.7   1:21.71 client                               
 44848 root      20   0 2484.1m 221.0m   6.1m S 339.2  2.8   1:25.17 client                               
 44848 root      20   0 2484.1m 220.9m   6.1m S 339.6  2.8   1:28.60 client                               
 44848 root      20   0 2484.1m 182.7m   6.1m S 344.1  2.3   1:32.11 client                               
 44848 root      20   0 2484.1m 225.9m   6.1m S 349.0  2.8   1:35.67 client                               
 44848 root      20   0 2484.1m 225.7m   6.1m S 354.5  2.8   1:39.25 client                               
 44848 root      20   0 2484.1m 225.3m   6.1m S 339.8  2.8   1:42.75 client                               
 44848 root      20   0 2484.1m 224.9m   6.1m S 348.5  2.8   1:46.27 client                               
 44848 root      20   0 2484.1m 212.7m   6.1m S 339.8  2.7   1:49.77 client                               
 44848 root      20   0 2484.1m 212.4m   6.1m S 339.8  2.7   1:53.27 client                               
 44848 root      20   0 2484.1m 212.1m   6.1m S 341.6  2.7   1:56.72 client                               
 44848 root      20   0 2484.1m 211.7m   6.1m S 340.2  2.7   2:00.19 client                               
 44848 root      20   0 2484.1m 211.3m   6.1m S 348.5  2.7   2:03.71 client                               
 44848 root      20   0 2484.1m 211.0m   6.1m S 343.1  2.7   2:07.21 client                               
 44848 root      20   0 2484.1m 206.6m   6.1m S 331.1  2.6   2:10.62 client                               
 44848 root      20   0 2484.1m 211.3m   6.1m S 336.9  2.7   2:14.09 client                               
 44848 root      20   0 2484.1m 211.4m   6.1m S 343.6  2.7   2:17.56 client                               
 44848 root      20   0 2484.1m 211.1m   6.1m S 336.3  2.7   2:20.99 client                               
 44848 root      20   0 2484.1m 180.9m   6.1m S 343.1  2.3   2:24.49 client                               
 44848 root      20   0 2484.1m 192.2m   6.1m S 339.6  2.4   2:27.92 client                               
 44848 root      20   0 2484.1m 201.8m   6.1m S 339.2  2.5   2:31.38 client                               
 44848 root      20   0 2484.1m 201.7m   6.1m S 344.7  2.5   2:34.93 client                               
 44848 root      20   0 2484.1m 201.3m   6.1m S 332.4  2.5   2:38.32 client                               
 44848 root      20   0 2484.1m 211.9m   6.1m S 354.5  2.7   2:41.90 client                               
 44848 root      20   0 2484.1m 212.2m   6.1m S 339.8  2.7   2:45.40 client                               
 44848 root      20   0 2484.1m 217.5m   6.1m S 345.1  2.7   2:48.92 client                               
 44848 root      20   0 2484.1m 217.4m   6.1m S 346.5  2.7   2:52.42 client                               
 44848 root      20   0 2484.1m 217.1m   6.1m S 334.3  2.7   2:55.83 client                               
 44848 root      20   0 2484.1m 218.8m   6.1m S 343.6  2.8   2:59.30 client                               
 44848 root      20   0 2484.1m 190.7m   6.1m S 343.1  2.4   3:02.80 client                               
 44848 root      20   0 2484.1m 190.4m   6.1m S 331.4  2.4   3:06.18 client                               
 44848 root      20   0 2484.1m 196.4m   6.1m S 348.0  2.5   3:09.73 client                               
 44848 root      20   0 2484.1m 199.8m   6.1m S 343.7  2.5   3:13.27 client                               
 44848 root      20   0 2484.1m 205.0m   6.1m S 342.6  2.6   3:16.73 client                               
 44848 root      20   0 2484.1m 213.3m   6.1m S 346.1  2.7   3:20.26 client                               
 44848 root      20   0 2484.1m 213.5m   6.1m S 340.2  2.7   3:23.73 client                               
 44848 root      20   0 2484.1m 213.1m   6.1m S 338.8  2.7   3:27.22 client                               
 44848 root      20   0 2484.1m 192.9m   6.1m S 343.1  2.4   3:30.72 client                               
 44848 root      20   0 2484.1m 216.4m   6.1m S 340.8  2.7   3:34.23 client                               
 44848 root      20   0 2484.1m 216.3m   6.1m S 344.6  2.7   3:37.71 client                               
 44848 root      20   0 2484.1m 216.0m   6.1m S 340.2  2.7   3:41.18 client                               
 44848 root      20   0 2484.1m 212.9m   6.1m S 342.2  2.7   3:44.67 client                               
 44848 root      20   0 2484.1m 150.9m   6.1m S 344.1  1.9   3:48.18 client                               
 44848 root      20   0 2484.1m 211.0m   6.1m S 345.1  2.7   3:51.70 client                               
 44848 root      20   0 2484.1m 211.5m   6.1m S 342.6  2.7   3:55.16 client                               
 44848 root      20   0 2484.1m 216.5m   6.1m S 335.3  2.7   3:58.58 client                               
 44848 root      20   0 2484.1m 216.2m   6.1m S 337.3  2.7   4:02.02 client                               
 44848 root      20   0 2550.2m 248.7m   6.1m S 340.8  3.1   4:05.53 client                               
 44848 root      20   0 2550.2m 248.3m   6.1m S 349.5  3.1   4:09.06 client                               
 44848 root      20   0 2550.2m 247.8m   6.1m S 351.5  3.1   4:12.68 client                               
 44848 root      20   0 2550.2m 213.7m   6.1m S 332.7  2.7   4:16.14 client                               
 44848 root      20   0 2550.2m 157.7m   6.1m S 337.3  2.0   4:19.58 client                               
 44848 root      20   0 2550.2m 207.0m   6.1m S 342.2  2.6   4:23.07 client                               
 44848 root      20   0 2550.2m 208.3m   6.1m S 351.5  2.6   4:26.62 client                               
 44848 root      20   0 2550.2m 208.0m   6.1m S 348.0  2.6   4:30.17 client                               
 44848 root      20   0 2550.2m 207.7m   6.1m S 349.5  2.6   4:33.70 client                               
 44848 root      20   0 2550.2m 167.7m   6.1m S 344.7  2.1   4:37.25 client                               
 44848 root      20   0 2550.2m 198.8m   6.1m S 340.2  2.5   4:40.72 client                               
 44848 root      20   0 2550.2m 213.8m   6.1m S 346.6  2.7   4:44.29 client                               
 44848 root      20   0 2550.2m 214.0m   6.1m S 344.1  2.7   4:47.80 client                               
 44848 root      20   0 2550.2m 207.6m   6.1m S 348.0  2.6   4:51.35 client                               
 44848 root      20   0 2550.2m 207.3m   6.1m S 352.9  2.6   4:54.95 client                               
 44848 root      20   0 2550.2m 207.0m   6.1m S 344.1  2.6   4:58.46 client                               
 44848 root      20   0 2550.2m 206.7m   6.1m S 346.6  2.6   5:02.03 client                               
 44848 root      20   0 2550.2m 214.0m   6.1m S 348.0  2.7   5:05.58 client                               
 44848 root      20   0 2550.2m 209.6m   6.1m S 342.6  2.6   5:09.04 client                               
 44848 root      20   0 2550.2m 193.4m   6.1m S 339.2  2.4   5:12.50 client                               
 44848 root      20   0 2550.2m 193.9m   6.1m S 345.5  2.4   5:15.99 client                               
 44848 root      20   0 2550.2m 193.9m   6.1m S 354.4  2.4   5:19.64 client                               
 44848 root      20   0 2550.2m 181.7m   6.1m S 337.9  2.3   5:23.12 client                               
 44848 root      20   0 2550.2m 195.0m   6.1m S 349.0  2.5   5:26.68 client                               
 44848 root      20   0 2550.2m 194.7m   6.1m S 352.5  2.4   5:30.24 client                               
 44848 root      20   0 2550.2m 201.6m   6.1m S 346.1  2.5   5:33.77 client                               
 44848 root      20   0 2550.2m 201.5m   6.1m S 342.2  2.5   5:37.26 client                               
 44848 root      20   0 2550.2m 193.3m   6.1m S 352.9  2.4   5:40.86 client                               
 44848 root      20   0 2550.2m 205.4m   6.1m S 353.9  2.6   5:44.47 client                               
 44848 root      20   0 2550.2m 181.5m   6.1m S 342.2  2.3   5:47.96 client                               
 44848 root      20   0 2550.2m 198.1m   6.1m S 344.1  2.5   5:51.47 client                               
 44848 root      20   0 2550.2m 201.7m   6.1m S 354.5  2.5   5:55.05 client                               
 44848 root      20   0 2550.2m 193.5m   6.1m S 348.0  2.4   5:58.60 client                               
 44848 root      20   0 2550.2m 193.1m   6.1m S 341.3  2.4   6:02.15 client                               
 44848 root      20   0 2550.2m 198.4m   6.1m S 349.5  2.5   6:05.68 client                               
 44848 root      20   0 2550.2m 198.3m   6.1m S 339.2  2.5   6:09.14 client                               
 44848 root      20   0 2550.2m 201.4m   6.1m S 347.6  2.5   6:12.72 client                               
 44848 root      20   0 2550.2m 195.4m   6.1m S 352.9  2.5   6:16.32 client                               
 44848 root      20   0 2550.2m 199.6m   6.1m S 346.2  2.5   6:19.92 client                               
 44848 root      20   0 2550.2m 201.9m   6.1m S 344.7  2.5   6:23.47 client                               
 44848 root      20   0 2550.2m 229.3m   6.1m S 349.5  2.9   6:27.07 client                               
 44848 root      20   0 2550.2m 181.3m   6.1m S 346.6  2.3   6:30.64 client                               
 44848 root      20   0 2550.2m 180.6m   6.1m S 346.5  2.3   6:34.14 client                               
 44848 root      20   0 2550.2m 180.8m   6.1m S 349.0  2.3   6:37.70 client                               
 44848 root      20   0 2550.2m 212.2m   6.1m S 351.5  2.7   6:41.25 client                               
 44848 root      20   0 2550.2m 212.2m   6.1m S 341.2  2.7   6:44.73 client                               
 44848 root      20   0 2550.2m 211.8m   6.1m S 345.1  2.7   6:48.25 client                               
 44848 root      20   0 2550.2m 185.7m   6.1m S 345.1  2.3   6:51.77 client                               
 44848 root      20   0 2550.2m 196.3m   6.1m S 342.2  2.5   6:55.26 client                               
 44848 root      20   0 2550.2m 203.3m   6.1m S 356.9  2.6   6:58.90 client                               
 44848 root      20   0 2622.2m 206.1m   6.1m S 350.0  2.6   7:02.47 client                               
 44848 root      20   0 2622.2m 206.5m   6.1m S 357.4  2.6   7:06.08 client                               
 44848 root      20   0 2622.2m 208.0m   6.1m S 345.2  2.6   7:09.67 client                               
 44848 root      20   0 2622.2m 207.9m   6.1m S 350.0  2.6   7:13.24 client                               
 44848 root      20   0 2622.2m 179.7m   6.1m S 343.6  2.3   7:16.71 client                               
 44848 root      20   0 2622.2m 212.4m   6.1m S 346.6  2.7   7:20.28 client                               
 44848 root      20   0 2622.2m 212.6m   6.1m S 347.1  2.7   7:23.82 client                               
 44848 root      20   0 2622.2m 214.6m   6.1m S 344.7  2.7   7:27.37 client                               
 44848 root      20   0 2622.2m 214.9m   6.1m S 363.4  2.7   7:31.04 client                               
 44848 root      20   0 2622.2m 214.5m   6.1m S 340.4  2.7   7:34.58 client                               
 44848 root      20   0 2622.2m 214.1m   6.1m S 344.7  2.7   7:38.13 client                               
 44848 root      20   0 2622.2m 213.8m   6.1m S 350.5  2.7   7:41.67 client                               
 44848 root      20   0 2622.2m 219.3m   6.1m S 353.9  2.8   7:45.28 client                               
 44848 root      20   0 2622.2m 219.2m   6.1m S 344.7  2.8   7:48.83 client                               
 44848 root      20   0 2622.2m 219.0m   6.1m S 352.0  2.8   7:52.42 client                               
 44848 root      20   0 2622.2m 218.6m   6.1m S 342.7  2.7   7:55.95 client                               
 44848 root      20   0 2622.2m 218.2m   6.1m S 351.0  2.7   7:59.53 client                               
 44848 root      20   0 2622.2m 218.7m   6.1m S 349.5  2.8   8:03.13 client                               
 44848 root      20   0 2622.2m 218.4m   6.1m S 352.0  2.7   8:06.72 client                               
 44848 root      20   0 2622.2m 218.0m   6.1m S 348.5  2.7   8:10.24 client                               
 44848 root      20   0 2622.2m 209.7m   6.1m S 349.0  2.6   8:13.80 client                               
 44848 root      20   0 2622.2m 185.4m   6.1m S 352.9  2.3   8:17.40 client                               
 44848 root      20   0 2622.2m 200.9m   6.1m S 352.9  2.5   8:21.00 client                               
 44848 root      20   0 2622.2m 169.4m   6.1m S 341.7  2.1   8:24.52 client                               
 44848 root      20   0 2622.2m 208.6m   6.1m S 361.8  2.6   8:28.21 client                               
 44848 root      20   0 2622.2m 193.9m   6.1m S 349.5  2.4   8:31.81 client                               
 44848 root      20   0 2622.2m 221.6m   6.1m S 353.4  2.8   8:35.45 client                               
 44848 root      20   0 2622.2m 241.3m   6.1m S 345.6  3.0   8:39.01 client                               
 44848 root      20   0 2622.2m 239.7m   6.1m S 346.1  3.0   8:42.54 client                               
 44848 root      20   0 2622.2m 177.7m   6.1m S 349.5  2.2   8:46.07 client                               
 44848 root      20   0 2622.2m 179.3m   6.1m S 347.6  2.3   8:49.65 client                               
 44848 root      20   0 2622.2m 185.2m   6.1m S 354.9  2.3   8:53.27 client                               
 44848 root      20   0 2622.2m 215.8m   6.1m S 345.6  2.7   8:56.83 client                               
 44848 root      20   0 2622.2m 215.7m   6.1m S 351.0  2.7   9:00.41 client                               
 44848 root      20   0 2622.2m 185.7m   6.1m S 357.8  2.3   9:04.06 client                               
 44848 root      20   0 2622.2m 176.7m   6.1m S 340.8  2.2   9:07.57 client                               
 44848 root      20   0 2622.2m 203.0m   6.1m S 355.4  2.6   9:11.16 client                               
 44848 root      20   0 2622.2m 203.3m   6.1m S 361.8  2.6   9:14.85 client                               
 44968 root      20   0 2276.1m 192.1m   6.2m S 232.0  2.4   0:02.32 client                               
 44968 root      20   0 2412.1m 195.2m   6.2m R 354.4  2.5   0:05.97 client                               
 44968 root      20   0 2412.1m 195.2m   6.2m S 353.9  2.5   0:09.58 client                               
 44968 root      20   0 2412.1m 224.0m   6.2m S 356.9  2.8   0:13.22 client                               
 44968 root      20   0 2412.1m 224.0m   6.2m S 359.8  2.8   0:16.89 client                               
 44968 root      20   0 2412.1m 223.6m   6.2m S 349.5  2.8   0:20.49 client                               
 44968 root      20   0 2412.1m 223.3m   6.2m S 353.5  2.8   0:24.06 client                               
 44968 root      20   0 2412.1m 223.0m   6.2m R 356.9  2.8   0:27.70 client                               
 44968 root      20   0 2412.1m 231.1m   6.2m R 356.3  2.9   0:31.37 client                               
 44968 root      20   0 2412.1m 220.9m   6.2m S 352.9  2.8   0:34.97 client                               
 44968 root      20   0 2412.1m 220.5m   6.2m S 357.8  2.8   0:38.62 client                               
 44968 root      20   0 2412.1m 222.8m   6.2m R 352.9  2.8   0:42.22 client                               
 44968 root      20   0 2412.1m 211.0m   6.2m R 345.2  2.7   0:45.81 client                               
 44968 root      20   0 2412.1m 217.8m   6.2m S 350.0  2.7   0:49.38 client                               
 44968 root      20   0 2556.1m 217.7m   6.2m S 363.4  2.7   0:53.05 client                               
 44968 root      20   0 2556.1m 217.3m   6.2m S 348.0  2.7   0:56.60 client                               
 44968 root      20   0 2556.1m 217.0m   6.2m S 359.2  2.7   1:00.30 client

Example Code

// server.go
package main

import (
	"flag"
	"fmt"
	"log"
	"net/http"
	"os"
	"os/signal"
	"sync/atomic"
	"time"

	"github.com/lesismal/llib/std/crypto/tls"
	"github.com/lesismal/nbio"
	"github.com/lesismal/nbio/nbhttp"
	"github.com/lesismal/nbio/nbhttp/websocket"
)

var (
	qps       int64 = 0
	qpsTotal  int64 = 0
	flow      int64 = 0
	flowTotal int64 = 0

	svr *nbhttp.Server
)

func newUpgrader() *websocket.Upgrader {
	u := websocket.NewUpgrader()
	u.OnMessage(func(c *websocket.Conn, messageType websocket.MessageType, data []byte) {
		atomic.AddInt64(&qps, 1)
		atomic.AddInt64(&flow, int64(len(data)))
		// echo
		// time.AfterFunc(1000, func() {
		c.WriteMessage(messageType, data)
		// })
	})
	u.OnClose(func(c *websocket.Conn, err error) {
		fmt.Println("OnClose:", c.RemoteAddr().String(), err)
	})

	return u
}

func onWebsocket(w http.ResponseWriter, r *http.Request) {
	// time.Sleep(time.Second * 5)
	upgrader := newUpgrader()
	conn, err := upgrader.Upgrade(w, r, nil)
	if err != nil {
		panic(err)
	}
	conn.SetReadDeadline(time.Time{})
	wsConn := conn.(*websocket.Conn)
	fmt.Println("OnOpen:", wsConn.RemoteAddr().String())
}

func main() {
	flag.Parse()

	cert, err := tls.X509KeyPair(rsaCertPEM, rsaKeyPEM)
	if err != nil {
		log.Fatalf("tls.X509KeyPair failed: %v", err)
	}
	tlsConfig := &tls.Config{
		Certificates:       []tls.Certificate{cert},
		InsecureSkipVerify: true,
	}

	mux := &http.ServeMux{}
	mux.HandleFunc("/wss", onWebsocket)

	svr = nbhttp.NewServerTLS(nbhttp.Config{
		Network:                 "tcp",
		Addrs:                   []string{"localhost:8888"},
		ReleaseWebsocketPayload: true,
	}, mux, nil, tlsConfig)
	svr.OnOpen(func(c *nbio.Conn) {
		c.SetReadBuffer(1024 * 1024 * 2)
		c.SetWriteBuffer(1024 * 1024 * 2)
	})
	err = svr.Start()
	if err != nil {
		fmt.Printf("nbio.Start failed: %v\n", err)
		return
	}
	defer svr.Stop()

	go func() {
		ticker := time.NewTicker(time.Second)
		for i := 0; true; i++ {
			<-ticker.C
			q := atomic.SwapInt64(&qps, 0)
			f := atomic.SwapInt64(&flow, 0)
			qpsTotal += q
			flowTotal += f
			log.Printf("[%03d] [qps: %v, qpsTotal: %v] [flow: %v M, flowTotal: %v M]", i, q, qpsTotal, f/1024/1024, flowTotal/1024/1024)
		}
	}()

	interrupt := make(chan os.Signal, 1)
	signal.Notify(interrupt, os.Interrupt)
	<-interrupt
	log.Println("exit")
}

var rsaCertPEM = []byte(`-----BEGIN CERTIFICATE-----
MIIDazCCAlOgAwIBAgIUJeohtgk8nnt8ofratXJg7kUJsI4wDQYJKoZIhvcNAQEL
BQAwRTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM
GEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0yMDEyMDcwODIyNThaFw0zMDEy
MDUwODIyNThaMEUxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEw
HwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwggEiMA0GCSqGSIb3DQEB
AQUAA4IBDwAwggEKAoIBAQCy+ZrIvwwiZv4bPmvKx/637ltZLwfgh3ouiEaTchGu
IQltthkqINHxFBqqJg44TUGHWthlrq6moQuKnWNjIsEc6wSD1df43NWBLgdxbPP0
x4tAH9pIJU7TQqbznjDBhzRbUjVXBIcn7bNknY2+5t784pPF9H1v7h8GqTWpNH9l
cz/v+snoqm9HC+qlsFLa4A3X9l5v05F1uoBfUALlP6bWyjHAfctpiJkoB9Yw1TJa
gpq7E50kfttwfKNkkAZIbib10HugkMoQJAs2EsGkje98druIl8IXmuvBIF6nZHuM
lt3UIZjS9RwPPLXhRHt1P0mR7BoBcOjiHgtSEs7Wk+j7AgMBAAGjUzBRMB0GA1Ud
DgQWBBQdheJv73XSOhgMQtkwdYPnfO02+TAfBgNVHSMEGDAWgBQdheJv73XSOhgM
QtkwdYPnfO02+TAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBf
SKVNMdmBpD9m53kCrguo9iKQqmhnI0WLkpdWszc/vBgtpOE5ENOfHGAufHZve871
2fzTXrgR0TF6UZWsQOqCm5Oh3URsCdXWewVMKgJ3DCii6QJ0MnhSFt6+xZE9C6Hi
WhcywgdR8t/JXKDam6miohW8Rum/IZo5HK9Jz/R9icKDGumcqoaPj/ONvY4EUwgB
irKKB7YgFogBmCtgi30beLVkXgk0GEcAf19lHHtX2Pv/lh3m34li1C9eBm1ca3kk
M2tcQtm1G89NROEjcG92cg+GX3GiWIjbI0jD1wnVy2LCOXMgOVbKfGfVKISFt0b1
DNn00G8C6ttLoGU2snyk
-----END CERTIFICATE-----
`)

var rsaKeyPEM = []byte(`-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEAsvmayL8MImb+Gz5rysf+t+5bWS8H4Id6LohGk3IRriEJbbYZ
KiDR8RQaqiYOOE1Bh1rYZa6upqELip1jYyLBHOsEg9XX+NzVgS4HcWzz9MeLQB/a
SCVO00Km854wwYc0W1I1VwSHJ+2zZJ2Nvube/OKTxfR9b+4fBqk1qTR/ZXM/7/rJ
6KpvRwvqpbBS2uAN1/Zeb9ORdbqAX1AC5T+m1soxwH3LaYiZKAfWMNUyWoKauxOd
JH7bcHyjZJAGSG4m9dB7oJDKECQLNhLBpI3vfHa7iJfCF5rrwSBep2R7jJbd1CGY
0vUcDzy14UR7dT9JkewaAXDo4h4LUhLO1pPo+wIDAQABAoIBAF6yWwekrlL1k7Xu
jTI6J7hCUesaS1yt0iQUzuLtFBXCPS7jjuUPgIXCUWl9wUBhAC8SDjWe+6IGzAiH
xjKKDQuz/iuTVjbDAeTb6exF7b6yZieDswdBVjfJqHR2Wu3LEBTRpo9oQesKhkTS
aFF97rZ3XCD9f/FdWOU5Wr8wm8edFK0zGsZ2N6r57yf1N6ocKlGBLBZ0v1Sc5ShV
1PVAxeephQvwL5DrOgkArnuAzwRXwJQG78L0aldWY2q6xABQZQb5+ml7H/kyytef
i+uGo3jHKepVALHmdpCGr9Yv+yCElup+ekv6cPy8qcmMBqGMISL1i1FEONxLcKWp
GEJi6QECgYEA3ZPGMdUm3f2spdHn3C+/+xskQpz6efiPYpnqFys2TZD7j5OOnpcP
ftNokA5oEgETg9ExJQ8aOCykseDc/abHerYyGw6SQxmDbyBLmkZmp9O3iMv2N8Pb
Nrn9kQKSr6LXZ3gXzlrDvvRoYUlfWuLSxF4b4PYifkA5AfsdiKkj+5sCgYEAzseF
XDTRKHHJnzxZDDdHQcwA0G9agsNj64BGUEjsAGmDiDyqOZnIjDLRt0O2X3oiIE5S
TXySSEiIkxjfErVJMumLaIwqVvlS4pYKdQo1dkM7Jbt8wKRQdleRXOPPN7msoEUk
Ta9ZsftHVUknPqblz9Uthb5h+sRaxIaE1llqDiECgYATS4oHzuL6k9uT+Qpyzymt
qThoIJljQ7TgxjxvVhD9gjGV2CikQM1Vov1JBigj4Toc0XuxGXaUC7cv0kAMSpi2
Y+VLG+K6ux8J70sGHTlVRgeGfxRq2MBfLKUbGplBeDG/zeJs0tSW7VullSkblgL6
nKNa3LQ2QEt2k7KHswryHwKBgENDxk8bY1q7wTHKiNEffk+aFD25q4DUHMH0JWti
fVsY98+upFU+gG2S7oOmREJE0aser0lDl7Zp2fu34IEOdfRY4p+s0O0gB+Vrl5VB
L+j7r9bzaX6lNQN6MvA7ryHahZxRQaD/xLbQHgFRXbHUyvdTyo4yQ1821qwNclLk
HUrhAoGAUtjR3nPFR4TEHlpTSQQovS8QtGTnOi7s7EzzdPWmjHPATrdLhMA0ezPj
Mr+u5TRncZBIzAZtButlh1AHnpN/qO3P0c0Rbdep3XBc/82JWO8qdb5QvAkxga3X
BpA7MNLxiqss+rCbwf3NbWxEMiDQ2zRwVoafVFys7tjmv6t2Xck=
-----END RSA PRIVATE KEY-----
`)

// cleint.go
package main

import (
	"context"
	"crypto/rand"
	"fmt"
	"net/url"
	"os"
	"os/signal"
	"time"

	"github.com/lesismal/nbio"
	"github.com/lesismal/nbio/nbhttp"
	"github.com/lesismal/nbio/nbhttp/websocket"

	"github.com/lesismal/llib/std/crypto/tls"
)

func newUpgrader() *websocket.Upgrader {
	u := websocket.NewUpgrader()
	u.OnMessage(func(c *websocket.Conn, messageType websocket.MessageType, data []byte) {
		// echo
		// time.AfterFunc(1000, func() {
		c.WriteMessage(messageType, data)
		// })
	})

	u.OnClose(func(c *websocket.Conn, err error) {
		fmt.Println("OnClose:", c.RemoteAddr().String(), err)
	})

	return u
}

func main() {
	engine := nbhttp.NewEngineTLS(nbhttp.Config{
		SupportClient:           true,
		ReleaseWebsocketPayload: true,
	})
	engine.OnOpen(func(c *nbio.Conn) {
		c.SetReadBuffer(1024 * 1024 * 2)
		c.SetWriteBuffer(1024 * 1024 * 2)
	})
	err := engine.Start()
	if err != nil {
		fmt.Printf("nbio.Start failed: %v\n", err)
		return
	}

	tlsConfig := &tls.Config{
		InsecureSkipVerify: true,
	}
	clientNum := 10
	conns := make([]*websocket.Conn, clientNum)
	for i := 0; i < clientNum; i++ {
		u := url.URL{Scheme: "wss", Host: "localhost:8888", Path: "/wss"}
		dialer := &websocket.Dialer{
			Engine:          engine,
			Upgrader:        newUpgrader(),
			DialTimeout:     time.Second * 3,
			TLSClientConfig: tlsConfig,
		}
		c, _, err := dialer.Dial(u.String(), nil)
		if err != nil {
			panic(fmt.Errorf("dial: %v", err))
		}
		c.SetReadDeadline(time.Time{})
		conns[i] = c
	}
	data := make([]byte, 1024*1024)
	rand.Read(data)
	for _, c := range conns {
		c.WriteMessage(websocket.BinaryMessage, data)
	}

	interrupt := make(chan os.Signal, 1)
	signal.Notify(interrupt, os.Interrupt)
	<-interrupt
	ctx, cancel := context.WithTimeout(context.Background(), time.Second*5)
	defer cancel()
	engine.Shutdown(ctx)
}

server log

root@ubuntu:~/large# go run ./server/
2021/10/03 11:00:42.966 [INF] Gopher[NB] start listen on: ["localhost:8888"]
2021/10/03 11:00:43 [000] [qps: 0, qpsTotal: 0] [flow: 0 M, flowTotal: 0 M]
2021/10/03 11:00:44 [001] [qps: 0, qpsTotal: 0] [flow: 0 M, flowTotal: 0 M]
2021/10/03 11:00:45 [002] [qps: 0, qpsTotal: 0] [flow: 0 M, flowTotal: 0 M]
2021/10/03 11:00:46 [003] [qps: 0, qpsTotal: 0] [flow: 0 M, flowTotal: 0 M]
2021/10/03 11:00:47 [004] [qps: 0, qpsTotal: 0] [flow: 0 M, flowTotal: 0 M]
2021/10/03 11:00:48 [005] [qps: 0, qpsTotal: 0] [flow: 0 M, flowTotal: 0 M]
OnOpen: 127.0.0.1:35620
OnOpen: 127.0.0.1:35622
OnOpen: 127.0.0.1:35624
OnOpen: 127.0.0.1:35626
OnOpen: 127.0.0.1:35628
OnOpen: 127.0.0.1:35630
OnOpen: 127.0.0.1:35632
OnOpen: 127.0.0.1:35634
OnOpen: 127.0.0.1:35636
OnOpen: 127.0.0.1:35638
2021/10/03 11:00:49 [006] [qps: 29, qpsTotal: 29] [flow: 29 M, flowTotal: 29 M]
2021/10/03 11:00:50 [007] [qps: 273, qpsTotal: 302] [flow: 273 M, flowTotal: 302 M]
2021/10/03 11:00:51 [008] [qps: 280, qpsTotal: 582] [flow: 280 M, flowTotal: 582 M]
2021/10/03 11:00:52 [009] [qps: 271, qpsTotal: 853] [flow: 271 M, flowTotal: 853 M]
2021/10/03 11:00:53 [010] [qps: 276, qpsTotal: 1129] [flow: 276 M, flowTotal: 1129 M]
2021/10/03 11:00:54 [011] [qps: 279, qpsTotal: 1408] [flow: 279 M, flowTotal: 1408 M]
2021/10/03 11:00:55 [012] [qps: 277, qpsTotal: 1685] [flow: 277 M, flowTotal: 1685 M]
2021/10/03 11:00:56 [013] [qps: 269, qpsTotal: 1954] [flow: 269 M, flowTotal: 1954 M]
2021/10/03 11:00:57 [014] [qps: 282, qpsTotal: 2236] [flow: 282 M, flowTotal: 2236 M]
2021/10/03 11:00:58 [015] [qps: 264, qpsTotal: 2500] [flow: 264 M, flowTotal: 2500 M]
2021/10/03 11:00:59 [016] [qps: 269, qpsTotal: 2769] [flow: 269 M, flowTotal: 2769 M]
2021/10/03 11:01:00 [017] [qps: 276, qpsTotal: 3045] [flow: 276 M, flowTotal: 3045 M]
2021/10/03 11:01:01 [018] [qps: 273, qpsTotal: 3318] [flow: 273 M, flowTotal: 3318 M]
2021/10/03 11:01:02 [019] [qps: 257, qpsTotal: 3575] [flow: 257 M, flowTotal: 3575 M]
2021/10/03 11:01:03 [020] [qps: 274, qpsTotal: 3849] [flow: 274 M, flowTotal: 3849 M]
2021/10/03 11:01:04 [021] [qps: 274, qpsTotal: 4123] [flow: 274 M, flowTotal: 4123 M]
2021/10/03 11:01:05 [022] [qps: 267, qpsTotal: 4390] [flow: 267 M, flowTotal: 4390 M]
2021/10/03 11:01:06 [023] [qps: 264, qpsTotal: 4654] [flow: 264 M, flowTotal: 4654 M]
2021/10/03 11:01:07 [024] [qps: 284, qpsTotal: 4938] [flow: 284 M, flowTotal: 4938 M]
2021/10/03 11:01:08 [025] [qps: 261, qpsTotal: 5199] [flow: 261 M, flowTotal: 5199 M]
2021/10/03 11:01:09 [026] [qps: 268, qpsTotal: 5467] [flow: 268 M, flowTotal: 5467 M]
2021/10/03 11:01:10 [027] [qps: 283, qpsTotal: 5750] [flow: 283 M, flowTotal: 5750 M]
2021/10/03 11:01:11 [028] [qps: 278, qpsTotal: 6028] [flow: 278 M, flowTotal: 6028 M]
2021/10/03 11:01:12 [029] [qps: 283, qpsTotal: 6311] [flow: 283 M, flowTotal: 6311 M]
2021/10/03 11:01:13 [030] [qps: 276, qpsTotal: 6587] [flow: 276 M, flowTotal: 6587 M]
2021/10/03 11:01:14 [031] [qps: 267, qpsTotal: 6854] [flow: 267 M, flowTotal: 6854 M]
2021/10/03 11:01:15 [032] [qps: 248, qpsTotal: 7102] [flow: 248 M, flowTotal: 7102 M]
2021/10/03 11:01:16 [033] [qps: 278, qpsTotal: 7380] [flow: 278 M, flowTotal: 7380 M]
2021/10/03 11:01:17 [034] [qps: 279, qpsTotal: 7659] [flow: 279 M, flowTotal: 7659 M]
2021/10/03 11:01:18 [035] [qps: 277, qpsTotal: 7936] [flow: 277 M, flowTotal: 7936 M]
2021/10/03 11:01:19 [036] [qps: 271, qpsTotal: 8207] [flow: 271 M, flowTotal: 8207 M]
2021/10/03 11:01:20 [037] [qps: 273, qpsTotal: 8480] [flow: 273 M, flowTotal: 8480 M]
2021/10/03 11:01:21 [038] [qps: 273, qpsTotal: 8753] [flow: 273 M, flowTotal: 8753 M]
OnClose: 127.0.0.1:35624 connection reset by peer
OnClose: 127.0.0.1:35628 <nil>
OnClose: 127.0.0.1:35632 EOF
OnClose: 127.0.0.1:35620 <nil>
OnClose: 127.0.0.1:35630 <nil>
OnClose: 127.0.0.1:35626 <nil>
OnClose: 127.0.0.1:35636 <nil>
OnClose: 127.0.0.1:35634 <nil>
OnClose: 127.0.0.1:35622 <nil>
OnClose: 127.0.0.1:35638 <nil>
2021/10/03 11:01:22 [039] [qps: 46, qpsTotal: 8799] [flow: 46 M, flowTotal: 8799 M]
OnOpen: 127.0.0.1:35640
OnOpen: 127.0.0.1:35642
OnOpen: 127.0.0.1:35644
OnOpen: 127.0.0.1:35646
OnOpen: 127.0.0.1:35648
OnOpen: 127.0.0.1:35650
OnOpen: 127.0.0.1:35652
OnOpen: 127.0.0.1:35654
OnOpen: 127.0.0.1:35656
OnOpen: 127.0.0.1:35658
2021/10/03 11:01:23 [040] [qps: 237, qpsTotal: 9036] [flow: 237 M, flowTotal: 9036 M]
2021/10/03 11:01:24 [041] [qps: 272, qpsTotal: 9308] [flow: 272 M, flowTotal: 9308 M]
2021/10/03 11:01:25 [042] [qps: 273, qpsTotal: 9581] [flow: 273 M, flowTotal: 9581 M]
2021/10/03 11:01:26 [043] [qps: 284, qpsTotal: 9865] [flow: 284 M, flowTotal: 9865 M]
2021/10/03 11:01:27 [044] [qps: 263, qpsTotal: 10128] [flow: 263 M, flowTotal: 10128 M]
2021/10/03 11:01:28 [045] [qps: 264, qpsTotal: 10392] [flow: 264 M, flowTotal: 10392 M]
2021/10/03 11:01:29 [046] [qps: 271, qpsTotal: 10663] [flow: 271 M, flowTotal: 10663 M]
2021/10/03 11:01:30 [047] [qps: 274, qpsTotal: 10937] [flow: 274 M, flowTotal: 10937 M]
2021/10/03 11:01:31 [048] [qps: 280, qpsTotal: 11217] [flow: 280 M, flowTotal: 11217 M]
OnClose: 127.0.0.1:35658 broken pipe
OnClose: 127.0.0.1:35648 <nil>
OnClose: 127.0.0.1:35642 <nil>
OnClose: 127.0.0.1:35654 EOF
OnClose: 127.0.0.1:35646 <nil>
OnClose: 127.0.0.1:35650 <nil>
OnClose: 127.0.0.1:35644 <nil>
OnClose: 127.0.0.1:35652 broken pipe
OnClose: 127.0.0.1:35640 broken pipe
OnClose: 127.0.0.1:35656 <nil>
2021/10/03 11:01:32 [049] [qps: 76, qpsTotal: 11293] [flow: 76 M, flowTotal: 11293 M]
OnOpen: 127.0.0.1:35660
OnOpen: 127.0.0.1:35662
OnOpen: 127.0.0.1:35664
OnOpen: 127.0.0.1:35666
OnOpen: 127.0.0.1:35668
OnOpen: 127.0.0.1:35670
OnOpen: 127.0.0.1:35672
OnOpen: 127.0.0.1:35674
OnOpen: 127.0.0.1:35676
OnOpen: 127.0.0.1:35678
2021/10/03 11:01:33 [050] [qps: 184, qpsTotal: 11477] [flow: 184 M, flowTotal: 11477 M]
2021/10/03 11:01:34 [051] [qps: 277, qpsTotal: 11754] [flow: 277 M, flowTotal: 11754 M]
2021/10/03 11:01:35 [052] [qps: 271, qpsTotal: 12025] [flow: 271 M, flowTotal: 12025 M]
2021/10/03 11:01:36 [053] [qps: 267, qpsTotal: 12292] [flow: 267 M, flowTotal: 12292 M]
2021/10/03 11:01:37 [054] [qps: 273, qpsTotal: 12565] [flow: 273 M, flowTotal: 12565 M]
2021/10/03 11:01:38 [055] [qps: 273, qpsTotal: 12838] [flow: 273 M, flowTotal: 12838 M]
2021/10/03 11:01:39 [056] [qps: 274, qpsTotal: 13112] [flow: 274 M, flowTotal: 13112 M]
2021/10/03 11:01:40 [057] [qps: 276, qpsTotal: 13388] [flow: 276 M, flowTotal: 13388 M]
2021/10/03 11:01:41 [058] [qps: 269, qpsTotal: 13657] [flow: 269 M, flowTotal: 13657 M]
2021/10/03 11:01:42 [059] [qps: 277, qpsTotal: 13934] [flow: 277 M, flowTotal: 13934 M]
2021/10/03 11:01:43 [060] [qps: 266, qpsTotal: 14200] [flow: 266 M, flowTotal: 14200 M]
2021/10/03 11:01:44 [061] [qps: 267, qpsTotal: 14467] [flow: 267 M, flowTotal: 14467 M]
OnClose: 127.0.0.1:35672 broken pipe
OnClose: 127.0.0.1:35674 <nil>
OnClose: 127.0.0.1:35668 EOF
OnClose: 127.0.0.1:35664 <nil>
OnClose: 127.0.0.1:35660 EOF
OnClose: 127.0.0.1:35676 <nil>
OnClose: 127.0.0.1:35670 <nil>
OnClose: 127.0.0.1:35666 <nil>
OnClose: 127.0.0.1:35662 <nil>
OnClose: 127.0.0.1:35678 <nil>
2021/10/03 11:01:45 [062] [qps: 229, qpsTotal: 14696] [flow: 229 M, flowTotal: 14696 M]
OnOpen: 127.0.0.1:35680
OnOpen: 127.0.0.1:35682
OnOpen: 127.0.0.1:35684
OnOpen: 127.0.0.1:35686
OnOpen: 127.0.0.1:35688
OnOpen: 127.0.0.1:35690
OnOpen: 127.0.0.1:35692
OnOpen: 127.0.0.1:35694
OnOpen: 127.0.0.1:35696
OnOpen: 127.0.0.1:35698
2021/10/03 11:01:46 [063] [qps: 108, qpsTotal: 14804] [flow: 108 M, flowTotal: 14804 M]
2021/10/03 11:01:47 [064] [qps: 274, qpsTotal: 15078] [flow: 274 M, flowTotal: 15078 M]
2021/10/03 11:01:48 [065] [qps: 268, qpsTotal: 15346] [flow: 268 M, flowTotal: 15346 M]
2021/10/03 11:01:49 [066] [qps: 272, qpsTotal: 15618] [flow: 272 M, flowTotal: 15618 M]
2021/10/03 11:01:50 [067] [qps: 272, qpsTotal: 15890] [flow: 272 M, flowTotal: 15890 M]
2021/10/03 11:01:51 [068] [qps: 276, qpsTotal: 16166] [flow: 276 M, flowTotal: 16166 M]
2021/10/03 11:01:52 [069] [qps: 277, qpsTotal: 16443] [flow: 277 M, flowTotal: 16443 M]
2021/10/03 11:01:53 [070] [qps: 269, qpsTotal: 16712] [flow: 269 M, flowTotal: 16712 M]
2021/10/03 11:01:54 [071] [qps: 276, qpsTotal: 16988] [flow: 276 M, flowTotal: 16988 M]
2021/10/03 11:01:55 [072] [qps: 271, qpsTotal: 17259] [flow: 271 M, flowTotal: 17259 M]
2021/10/03 11:01:56 [073] [qps: 271, qpsTotal: 17530] [flow: 271 M, flowTotal: 17530 M]
2021/10/03 11:01:57 [074] [qps: 266, qpsTotal: 17796] [flow: 266 M, flowTotal: 17796 M]
2021/10/03 11:01:58 [075] [qps: 271, qpsTotal: 18067] [flow: 271 M, flowTotal: 18067 M]
2021/10/03 11:01:59 [076] [qps: 263, qpsTotal: 18330] [flow: 263 M, flowTotal: 18330 M]
2021/10/03 11:02:00 [077] [qps: 277, qpsTotal: 18607] [flow: 277 M, flowTotal: 18607 M]
2021/10/03 11:02:01 [078] [qps: 275, qpsTotal: 18882] [flow: 275 M, flowTotal: 18882 M]
2021/10/03 11:02:02 [079] [qps: 284, qpsTotal: 19166] [flow: 284 M, flowTotal: 19166 M]
2021/10/03 11:02:03 [080] [qps: 278, qpsTotal: 19444] [flow: 278 M, flowTotal: 19444 M]
2021/10/03 11:02:04 [081] [qps: 276, qpsTotal: 19720] [flow: 276 M, flowTotal: 19720 M]
2021/10/03 11:02:05 [082] [qps: 281, qpsTotal: 20001] [flow: 281 M, flowTotal: 20001 M]
2021/10/03 11:02:06 [083] [qps: 275, qpsTotal: 20276] [flow: 275 M, flowTotal: 20276 M]
2021/10/03 11:02:07 [084] [qps: 279, qpsTotal: 20555] [flow: 279 M, flowTotal: 20555 M]
2021/10/03 11:02:08 [085] [qps: 270, qpsTotal: 20825] [flow: 270 M, flowTotal: 20825 M]
2021/10/03 11:02:09 [086] [qps: 280, qpsTotal: 21105] [flow: 280 M, flowTotal: 21105 M]
2021/10/03 11:02:10 [087] [qps: 279, qpsTotal: 21384] [flow: 279 M, flowTotal: 21384 M]
2021/10/03 11:02:11 [088] [qps: 284, qpsTotal: 21668] [flow: 284 M, flowTotal: 21668 M]
2021/10/03 11:02:12 [089] [qps: 278, qpsTotal: 21946] [flow: 278 M, flowTotal: 21946 M]
2021/10/03 11:02:13 [090] [qps: 275, qpsTotal: 22221] [flow: 275 M, flowTotal: 22221 M]
2021/10/03 11:02:14 [091] [qps: 287, qpsTotal: 22508] [flow: 287 M, flowTotal: 22508 M]
2021/10/03 11:02:15 [092] [qps: 282, qpsTotal: 22790] [flow: 282 M, flowTotal: 22790 M]
2021/10/03 11:02:16 [093] [qps: 279, qpsTotal: 23069] [flow: 279 M, flowTotal: 23069 M]
2021/10/03 11:02:17 [094] [qps: 280, qpsTotal: 23349] [flow: 280 M, flowTotal: 23349 M]
2021/10/03 11:02:18 [095] [qps: 277, qpsTotal: 23626] [flow: 277 M, flowTotal: 23626 M]
2021/10/03 11:02:19 [096] [qps: 270, qpsTotal: 23896] [flow: 270 M, flowTotal: 23896 M]
2021/10/03 11:02:20 [097] [qps: 285, qpsTotal: 24181] [flow: 285 M, flowTotal: 24181 M]
2021/10/03 11:02:21 [098] [qps: 286, qpsTotal: 24467] [flow: 286 M, flowTotal: 24467 M]
2021/10/03 11:02:22 [099] [qps: 285, qpsTotal: 24752] [flow: 285 M, flowTotal: 24752 M]
2021/10/03 11:02:23 [100] [qps: 286, qpsTotal: 25038] [flow: 286 M, flowTotal: 25038 M]
2021/10/03 11:02:24 [101] [qps: 279, qpsTotal: 25317] [flow: 279 M, flowTotal: 25317 M]
2021/10/03 11:02:25 [102] [qps: 285, qpsTotal: 25602] [flow: 285 M, flowTotal: 25602 M]
2021/10/03 11:02:26 [103] [qps: 282, qpsTotal: 25884] [flow: 282 M, flowTotal: 25884 M]
2021/10/03 11:02:27 [104] [qps: 272, qpsTotal: 26156] [flow: 272 M, flowTotal: 26156 M]
2021/10/03 11:02:28 [105] [qps: 276, qpsTotal: 26432] [flow: 276 M, flowTotal: 26432 M]
2021/10/03 11:02:29 [106] [qps: 283, qpsTotal: 26715] [flow: 283 M, flowTotal: 26715 M]
2021/10/03 11:02:30 [107] [qps: 287, qpsTotal: 27002] [flow: 287 M, flowTotal: 27002 M]
2021/10/03 11:02:31 [108] [qps: 285, qpsTotal: 27287] [flow: 285 M, flowTotal: 27287 M]
2021/10/03 11:02:32 [109] [qps: 277, qpsTotal: 27564] [flow: 277 M, flowTotal: 27564 M]
2021/10/03 11:02:33 [110] [qps: 272, qpsTotal: 27836] [flow: 272 M, flowTotal: 27836 M]
2021/10/03 11:02:34 [111] [qps: 276, qpsTotal: 28112] [flow: 276 M, flowTotal: 28112 M]
2021/10/03 11:02:35 [112] [qps: 262, qpsTotal: 28374] [flow: 262 M, flowTotal: 28374 M]
2021/10/03 11:02:36 [113] [qps: 281, qpsTotal: 28655] [flow: 281 M, flowTotal: 28655 M]
2021/10/03 11:02:37 [114] [qps: 280, qpsTotal: 28935] [flow: 280 M, flowTotal: 28935 M]
2021/10/03 11:02:38 [115] [qps: 280, qpsTotal: 29215] [flow: 280 M, flowTotal: 29215 M]
2021/10/03 11:02:39 [116] [qps: 273, qpsTotal: 29488] [flow: 273 M, flowTotal: 29488 M]
2021/10/03 11:02:40 [117] [qps: 274, qpsTotal: 29762] [flow: 274 M, flowTotal: 29762 M]
2021/10/03 11:02:41 [118] [qps: 259, qpsTotal: 30021] [flow: 259 M, flowTotal: 30021 M]
2021/10/03 11:02:42 [119] [qps: 276, qpsTotal: 30297] [flow: 276 M, flowTotal: 30297 M]
2021/10/03 11:02:43 [120] [qps: 285, qpsTotal: 30582] [flow: 285 M, flowTotal: 30582 M]
2021/10/03 11:02:44 [121] [qps: 282, qpsTotal: 30864] [flow: 282 M, flowTotal: 30864 M]
2021/10/03 11:02:45 [122] [qps: 283, qpsTotal: 31147] [flow: 283 M, flowTotal: 31147 M]
2021/10/03 11:02:46 [123] [qps: 284, qpsTotal: 31431] [flow: 284 M, flowTotal: 31431 M]
2021/10/03 11:02:47 [124] [qps: 280, qpsTotal: 31711] [flow: 280 M, flowTotal: 31711 M]
2021/10/03 11:02:48 [125] [qps: 287, qpsTotal: 31998] [flow: 287 M, flowTotal: 31998 M]
2021/10/03 11:02:49 [126] [qps: 280, qpsTotal: 32278] [flow: 280 M, flowTotal: 32278 M]
2021/10/03 11:02:50 [127] [qps: 281, qpsTotal: 32559] [flow: 281 M, flowTotal: 32559 M]
2021/10/03 11:02:51 [128] [qps: 289, qpsTotal: 32848] [flow: 289 M, flowTotal: 32848 M]
2021/10/03 11:02:52 [129] [qps: 279, qpsTotal: 33127] [flow: 279 M, flowTotal: 33127 M]
2021/10/03 11:02:53 [130] [qps: 281, qpsTotal: 33408] [flow: 281 M, flowTotal: 33408 M]
2021/10/03 11:02:54 [131] [qps: 290, qpsTotal: 33698] [flow: 290 M, flowTotal: 33698 M]
2021/10/03 11:02:55 [132] [qps: 281, qpsTotal: 33979] [flow: 281 M, flowTotal: 33979 M]
2021/10/03 11:02:56 [133] [qps: 286, qpsTotal: 34265] [flow: 286 M, flowTotal: 34265 M]
2021/10/03 11:02:57 [134] [qps: 280, qpsTotal: 34545] [flow: 280 M, flowTotal: 34545 M]
2021/10/03 11:02:58 [135] [qps: 262, qpsTotal: 34807] [flow: 262 M, flowTotal: 34807 M]
2021/10/03 11:02:59 [136] [qps: 285, qpsTotal: 35092] [flow: 285 M, flowTotal: 35092 M]
2021/10/03 11:03:00 [137] [qps: 277, qpsTotal: 35369] [flow: 277 M, flowTotal: 35369 M]
2021/10/03 11:03:01 [138] [qps: 279, qpsTotal: 35648] [flow: 279 M, flowTotal: 35648 M]
2021/10/03 11:03:02 [139] [qps: 282, qpsTotal: 35930] [flow: 282 M, flowTotal: 35930 M]
2021/10/03 11:03:03 [140] [qps: 283, qpsTotal: 36213] [flow: 283 M, flowTotal: 36213 M]
2021/10/03 11:03:04 [141] [qps: 278, qpsTotal: 36491] [flow: 278 M, flowTotal: 36491 M]
2021/10/03 11:03:05 [142] [qps: 281, qpsTotal: 36772] [flow: 281 M, flowTotal: 36772 M]
2021/10/03 11:03:06 [143] [qps: 281, qpsTotal: 37053] [flow: 281 M, flowTotal: 37053 M]
2021/10/03 11:03:07 [144] [qps: 263, qpsTotal: 37316] [flow: 263 M, flowTotal: 37316 M]
2021/10/03 11:03:08 [145] [qps: 278, qpsTotal: 37594] [flow: 278 M, flowTotal: 37594 M]
2021/10/03 11:03:09 [146] [qps: 281, qpsTotal: 37875] [flow: 281 M, flowTotal: 37875 M]
2021/10/03 11:03:10 [147] [qps: 277, qpsTotal: 38152] [flow: 277 M, flowTotal: 38152 M]
2021/10/03 11:03:11 [148] [qps: 286, qpsTotal: 38438] [flow: 286 M, flowTotal: 38438 M]
2021/10/03 11:03:12 [149] [qps: 279, qpsTotal: 38717] [flow: 279 M, flowTotal: 38717 M]
2021/10/03 11:03:13 [150] [qps: 282, qpsTotal: 38999] [flow: 282 M, flowTotal: 38999 M]
2021/10/03 11:03:14 [151] [qps: 284, qpsTotal: 39283] [flow: 284 M, flowTotal: 39283 M]
2021/10/03 11:03:15 [152] [qps: 275, qpsTotal: 39558] [flow: 275 M, flowTotal: 39558 M]
2021/10/03 11:03:16 [153] [qps: 271, qpsTotal: 39829] [flow: 271 M, flowTotal: 39829 M]
2021/10/03 11:03:17 [154] [qps: 283, qpsTotal: 40112] [flow: 283 M, flowTotal: 40112 M]
2021/10/03 11:03:18 [155] [qps: 272, qpsTotal: 40384] [flow: 272 M, flowTotal: 40384 M]
2021/10/03 11:03:19 [156] [qps: 279, qpsTotal: 40663] [flow: 279 M, flowTotal: 40663 M]
2021/10/03 11:03:20 [157] [qps: 280, qpsTotal: 40943] [flow: 280 M, flowTotal: 40943 M]
2021/10/03 11:03:21 [158] [qps: 276, qpsTotal: 41219] [flow: 276 M, flowTotal: 41219 M]
2021/10/03 11:03:22 [159] [qps: 277, qpsTotal: 41496] [flow: 277 M, flowTotal: 41496 M]
2021/10/03 11:03:23 [160] [qps: 278, qpsTotal: 41774] [flow: 278 M, flowTotal: 41774 M]
2021/10/03 11:03:24 [161] [qps: 282, qpsTotal: 42056] [flow: 282 M, flowTotal: 42056 M]
2021/10/03 11:03:25 [162] [qps: 278, qpsTotal: 42334] [flow: 278 M, flowTotal: 42334 M]
2021/10/03 11:03:26 [163] [qps: 287, qpsTotal: 42621] [flow: 287 M, flowTotal: 42621 M]
2021/10/03 11:03:27 [164] [qps: 280, qpsTotal: 42901] [flow: 280 M, flowTotal: 42901 M]
2021/10/03 11:03:28 [165] [qps: 273, qpsTotal: 43174] [flow: 273 M, flowTotal: 43174 M]
2021/10/03 11:03:29 [166] [qps: 275, qpsTotal: 43449] [flow: 275 M, flowTotal: 43449 M]
2021/10/03 11:03:30 [167] [qps: 279, qpsTotal: 43728] [flow: 279 M, flowTotal: 43728 M]
2021/10/03 11:03:31 [168] [qps: 283, qpsTotal: 44011] [flow: 283 M, flowTotal: 44011 M]
2021/10/03 11:03:32 [169] [qps: 278, qpsTotal: 44289] [flow: 278 M, flowTotal: 44289 M]
2021/10/03 11:03:33 [170] [qps: 280, qpsTotal: 44569] [flow: 280 M, flowTotal: 44569 M]
2021/10/03 11:03:34 [171] [qps: 286, qpsTotal: 44855] [flow: 286 M, flowTotal: 44855 M]
2021/10/03 11:03:35 [172] [qps: 279, qpsTotal: 45134] [flow: 279 M, flowTotal: 45134 M]
2021/10/03 11:03:36 [173] [qps: 270, qpsTotal: 45404] [flow: 270 M, flowTotal: 45404 M]
2021/10/03 11:03:37 [174] [qps: 278, qpsTotal: 45682] [flow: 278 M, flowTotal: 45682 M]
2021/10/03 11:03:38 [175] [qps: 283, qpsTotal: 45965] [flow: 283 M, flowTotal: 45965 M]
2021/10/03 11:03:39 [176] [qps: 276, qpsTotal: 46241] [flow: 276 M, flowTotal: 46241 M]
2021/10/03 11:03:40 [177] [qps: 273, qpsTotal: 46514] [flow: 273 M, flowTotal: 46514 M]
2021/10/03 11:03:41 [178] [qps: 285, qpsTotal: 46799] [flow: 285 M, flowTotal: 46799 M]
2021/10/03 11:03:42 [179] [qps: 282, qpsTotal: 47081] [flow: 282 M, flowTotal: 47081 M]
2021/10/03 11:03:43 [180] [qps: 281, qpsTotal: 47362] [flow: 281 M, flowTotal: 47362 M]
2021/10/03 11:03:44 [181] [qps: 275, qpsTotal: 47637] [flow: 275 M, flowTotal: 47637 M]
2021/10/03 11:03:45 [182] [qps: 275, qpsTotal: 47912] [flow: 275 M, flowTotal: 47912 M]
2021/10/03 11:03:46 [183] [qps: 272, qpsTotal: 48184] [flow: 272 M, flowTotal: 48184 M]
2021/10/03 11:03:47 [184] [qps: 280, qpsTotal: 48464] [flow: 280 M, flowTotal: 48464 M]
2021/10/03 11:03:48 [185] [qps: 253, qpsTotal: 48717] [flow: 253 M, flowTotal: 48717 M]
2021/10/03 11:03:49 [186] [qps: 259, qpsTotal: 48976] [flow: 259 M, flowTotal: 48976 M]
2021/10/03 11:03:50 [187] [qps: 267, qpsTotal: 49243] [flow: 267 M, flowTotal: 49243 M]
2021/10/03 11:03:51 [188] [qps: 284, qpsTotal: 49527] [flow: 284 M, flowTotal: 49527 M]
2021/10/03 11:03:52 [189] [qps: 278, qpsTotal: 49805] [flow: 278 M, flowTotal: 49805 M]
2021/10/03 11:03:53 [190] [qps: 282, qpsTotal: 50087] [flow: 282 M, flowTotal: 50087 M]
2021/10/03 11:03:54 [191] [qps: 284, qpsTotal: 50371] [flow: 284 M, flowTotal: 50371 M]
2021/10/03 11:03:55 [192] [qps: 277, qpsTotal: 50648] [flow: 277 M, flowTotal: 50648 M]
2021/10/03 11:03:56 [193] [qps: 275, qpsTotal: 50923] [flow: 275 M, flowTotal: 50923 M]
2021/10/03 11:03:57 [194] [qps: 281, qpsTotal: 51204] [flow: 281 M, flowTotal: 51204 M]
2021/10/03 11:03:58 [195] [qps: 289, qpsTotal: 51493] [flow: 289 M, flowTotal: 51493 M]
2021/10/03 11:03:59 [196] [qps: 271, qpsTotal: 51764] [flow: 271 M, flowTotal: 51764 M]
2021/10/03 11:04:00 [197] [qps: 274, qpsTotal: 52038] [flow: 274 M, flowTotal: 52038 M]
2021/10/03 11:04:01 [198] [qps: 274, qpsTotal: 52312] [flow: 274 M, flowTotal: 52312 M]
2021/10/03 11:04:02 [199] [qps: 283, qpsTotal: 52595] [flow: 283 M, flowTotal: 52595 M]
2021/10/03 11:04:03 [200] [qps: 278, qpsTotal: 52873] [flow: 278 M, flowTotal: 52873 M]
2021/10/03 11:04:04 [201] [qps: 278, qpsTotal: 53151] [flow: 278 M, flowTotal: 53151 M]
2021/10/03 11:04:05 [202] [qps: 280, qpsTotal: 53431] [flow: 280 M, flowTotal: 53431 M]
2021/10/03 11:04:06 [203] [qps: 280, qpsTotal: 53711] [flow: 280 M, flowTotal: 53711 M]
2021/10/03 11:04:07 [204] [qps: 281, qpsTotal: 53992] [flow: 281 M, flowTotal: 53992 M]
2021/10/03 11:04:08 [205] [qps: 286, qpsTotal: 54278] [flow: 286 M, flowTotal: 54278 M]
2021/10/03 11:04:09 [206] [qps: 279, qpsTotal: 54557] [flow: 279 M, flowTotal: 54557 M]
2021/10/03 11:04:10 [207] [qps: 271, qpsTotal: 54828] [flow: 271 M, flowTotal: 54828 M]
2021/10/03 11:04:11 [208] [qps: 280, qpsTotal: 55108] [flow: 280 M, flowTotal: 55108 M]
2021/10/03 11:04:12 [209] [qps: 283, qpsTotal: 55391] [flow: 283 M, flowTotal: 55391 M]
2021/10/03 11:04:13 [210] [qps: 279, qpsTotal: 55670] [flow: 279 M, flowTotal: 55670 M]
2021/10/03 11:04:14 [211] [qps: 283, qpsTotal: 55953] [flow: 283 M, flowTotal: 55953 M]
2021/10/03 11:04:15 [212] [qps: 279, qpsTotal: 56232] [flow: 279 M, flowTotal: 56232 M]
2021/10/03 11:04:16 [213] [qps: 277, qpsTotal: 56509] [flow: 277 M, flowTotal: 56509 M]
2021/10/03 11:04:17 [214] [qps: 281, qpsTotal: 56790] [flow: 281 M, flowTotal: 56790 M]
2021/10/03 11:04:18 [215] [qps: 279, qpsTotal: 57069] [flow: 279 M, flowTotal: 57069 M]
2021/10/03 11:04:19 [216] [qps: 283, qpsTotal: 57352] [flow: 283 M, flowTotal: 57352 M]
2021/10/03 11:04:20 [217] [qps: 265, qpsTotal: 57617] [flow: 265 M, flowTotal: 57617 M]
2021/10/03 11:04:21 [218] [qps: 287, qpsTotal: 57904] [flow: 287 M, flowTotal: 57904 M]
2021/10/03 11:04:22 [219] [qps: 280, qpsTotal: 58184] [flow: 280 M, flowTotal: 58184 M]
2021/10/03 11:04:23 [220] [qps: 277, qpsTotal: 58461] [flow: 277 M, flowTotal: 58461 M]
2021/10/03 11:04:24 [221] [qps: 277, qpsTotal: 58738] [flow: 277 M, flowTotal: 58738 M]
2021/10/03 11:04:25 [222] [qps: 281, qpsTotal: 59019] [flow: 281 M, flowTotal: 59019 M]
2021/10/03 11:04:26 [223] [qps: 265, qpsTotal: 59284] [flow: 265 M, flowTotal: 59284 M]
OnClose: 127.0.0.1:35684 <nil>
OnClose: 127.0.0.1:35690 EOF
OnClose: 127.0.0.1:35688 broken pipe
OnClose: 127.0.0.1:35686 <nil>
OnClose: 127.0.0.1:35680 broken pipe
OnClose: 127.0.0.1:35694 <nil>
OnClose: 127.0.0.1:35692 <nil>
OnClose: 127.0.0.1:35696 <nil>
OnClose: 127.0.0.1:35698 broken pipe
OnClose: 127.0.0.1:35682 <nil>
2021/10/03 11:04:27 [224] [qps: 168, qpsTotal: 59452] [flow: 168 M, flowTotal: 59452 M]
2021/10/03 11:04:28 [225] [qps: 0, qpsTotal: 59452] [flow: 0 M, flowTotal: 59452 M]
2021/10/03 11:04:29 [226] [qps: 0, qpsTotal: 59452] [flow: 0 M, flowTotal: 59452 M]
2021/10/03 11:04:30 [227] [qps: 0, qpsTotal: 59452] [flow: 0 M, flowTotal: 59452 M]
2021/10/03 11:04:31 [228] [qps: 0, qpsTotal: 59452] [flow: 0 M, flowTotal: 59452 M]
2021/10/03 11:04:32 [229] [qps: 0, qpsTotal: 59452] [flow: 0 M, flowTotal: 59452 M]
OnOpen: 127.0.0.1:35700
OnOpen: 127.0.0.1:35702
OnOpen: 127.0.0.1:35704
OnOpen: 127.0.0.1:35706
OnOpen: 127.0.0.1:35708
OnOpen: 127.0.0.1:35710
OnOpen: 127.0.0.1:35712
OnOpen: 127.0.0.1:35714
OnOpen: 127.0.0.1:35716
OnOpen: 127.0.0.1:35718
2021/10/03 11:04:33 [230] [qps: 164, qpsTotal: 59616] [flow: 164 M, flowTotal: 59616 M]
2021/10/03 11:04:34 [231] [qps: 261, qpsTotal: 59877] [flow: 261 M, flowTotal: 59877 M]
2021/10/03 11:04:35 [232] [qps: 272, qpsTotal: 60149] [flow: 272 M, flowTotal: 60149 M]
2021/10/03 11:04:36 [233] [qps: 270, qpsTotal: 60419] [flow: 270 M, flowTotal: 60419 M]
2021/10/03 11:04:37 [234] [qps: 279, qpsTotal: 60698] [flow: 279 M, flowTotal: 60698 M]
2021/10/03 11:04:38 [235] [qps: 268, qpsTotal: 60966] [flow: 268 M, flowTotal: 60966 M]
2021/10/03 11:04:39 [236] [qps: 274, qpsTotal: 61240] [flow: 274 M, flowTotal: 61240 M]
2021/10/03 11:04:40 [237] [qps: 281, qpsTotal: 61521] [flow: 281 M, flowTotal: 61521 M]
2021/10/03 11:04:41 [238] [qps: 275, qpsTotal: 61796] [flow: 275 M, flowTotal: 61796 M]
2021/10/03 11:04:42 [239] [qps: 274, qpsTotal: 62070] [flow: 274 M, flowTotal: 62070 M]
2021/10/03 11:04:43 [240] [qps: 274, qpsTotal: 62344] [flow: 274 M, flowTotal: 62344 M]
2021/10/03 11:04:44 [241] [qps: 280, qpsTotal: 62624] [flow: 280 M, flowTotal: 62624 M]
2021/10/03 11:04:45 [242] [qps: 270, qpsTotal: 62894] [flow: 270 M, flowTotal: 62894 M]
2021/10/03 11:04:46 [243] [qps: 272, qpsTotal: 63166] [flow: 272 M, flowTotal: 63166 M]
2021/10/03 11:04:47 [244] [qps: 272, qpsTotal: 63438] [flow: 272 M, flowTotal: 63438 M]
2021/10/03 11:04:48 [245] [qps: 271, qpsTotal: 63709] [flow: 271 M, flowTotal: 63709 M]
2021/10/03 11:04:49 [246] [qps: 267, qpsTotal: 63976] [flow: 267 M, flowTotal: 63976 M]
OnClose: 127.0.0.1:35706 <nil>
OnClose: 127.0.0.1:35712 <nil>
OnClose: 127.0.0.1:35710 <nil>
OnClose: 127.0.0.1:35714 EOF
OnClose: 127.0.0.1:35708 <nil>
OnClose: 127.0.0.1:35700 <nil>
OnClose: 127.0.0.1:35704 <nil>
OnClose: 127.0.0.1:35716 <nil>
OnClose: 127.0.0.1:35702 broken pipe
OnClose: 127.0.0.1:35718 <nil>
2021/10/03 11:04:50 [247] [qps: 181, qpsTotal: 64157] [flow: 181 M, flowTotal: 64157 M]
^C2021/10/03 11:04:51 exit
2021/10/03 11:04:51.663 [ERR] Poller[NB_LISTENER_0] Accept failed: accept tcp 127.0.0.1:8888: use of closed network connection, exit...
2021/10/03 11:04:51.664 [INF] Gopher[NB] stop

lesismal · 2021-10-03T11:17:44Z

I need your samples, then I'll look into it.

acgreek · 2021-10-04T19:17:31Z

This example here is similar to what I need and avoids the memory usage explosion. With this, I see the memory usage stop at a ~fix point and not grow beyond that regardless of stop/starting clients.

#117

acgreek added the enhancement New feature or request label Oct 1, 2021

acgreek self-assigned this Oct 1, 2021

acgreek added bug Something isn't working and removed enhancement New feature or request labels Oct 2, 2021

acgreek changed the title ~~optimize memory usage for large websocket messages~~ memory leak Oct 2, 2021

acgreek added documentation Improvements or additions to documentation and removed bug Something isn't working labels Oct 4, 2021

acgreek changed the title ~~memory leak~~ memory usage explosion with provided examples Oct 4, 2021

acgreek closed this as completed Oct 4, 2021

lesismal mentioned this issue Oct 25, 2021

使用中遇到的问题 #137

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

memory usage explosion with provided examples #113

memory usage explosion with provided examples #113

acgreek commented Oct 1, 2021

acgreek commented Oct 2, 2021

lesismal commented Oct 2, 2021 •

edited

Loading

lesismal commented Oct 2, 2021 •

edited

Loading

lesismal commented Oct 2, 2021

acgreek commented Oct 2, 2021 •

edited

Loading

acgreek commented Oct 2, 2021

lesismal commented Oct 2, 2021

acgreek commented Oct 2, 2021

acgreek commented Oct 2, 2021

lesismal commented Oct 3, 2021 •

edited

Loading

lesismal commented Oct 3, 2021

lesismal commented Oct 3, 2021 •

edited

Loading

acgreek commented Oct 4, 2021

memory usage explosion with provided examples #113

memory usage explosion with provided examples #113

Comments

acgreek commented Oct 1, 2021

acgreek commented Oct 2, 2021

lesismal commented Oct 2, 2021 • edited Loading

lesismal commented Oct 2, 2021 • edited Loading

lesismal commented Oct 2, 2021

acgreek commented Oct 2, 2021 • edited Loading

acgreek commented Oct 2, 2021

lesismal commented Oct 2, 2021

acgreek commented Oct 2, 2021

acgreek commented Oct 2, 2021

lesismal commented Oct 3, 2021 • edited Loading

lesismal commented Oct 3, 2021

ENV

Server Cost

Client Cost

Example Code

server log

lesismal commented Oct 3, 2021 • edited Loading

acgreek commented Oct 4, 2021

lesismal commented Oct 2, 2021 •

edited

Loading

lesismal commented Oct 2, 2021 •

edited

Loading

acgreek commented Oct 2, 2021 •

edited

Loading

lesismal commented Oct 3, 2021 •

edited

Loading

lesismal commented Oct 3, 2021 •

edited

Loading