Too many CLOSE_WAIT after access via http api

[suutaku]

Version information:

JohnSuus-MacBook-Pro-4:build john$ ipfs version --all go-ipfs version: 0.4.21-dev-7e9eb72bf Repo version: 7 System version: amd64/darwin Golang version: go1.12.4

Type:

Bug

Description:

Running ipfs daemon on Mac OS X 10.13.6.
Access 127.0.0.1:5001/api/v0/xxxxxx once per second. after about 10 minute, running
sudo losf -nP -iTCP:5001
I got 300+ TCP CLOSE_WAIT status.
It's some kind of FD leaks?

I had follow @Stebalien 's suggestion to modify code
go-ipfs/core/corehttp/corehttp.go

by added
server.SetKeepAlivesEnabled(false)
after

server := &http.Server{ 
 	Handler: handler, 
 } 

but it finally invalid.

and I also set a nginx reverse proxy out in front, the number of CLOSE_WAIT was keep increasing.

can you take a look agin?
now my server is down due this issue....

[suutaku]

update

I found CLOSE_WAIT only happens on invalid access. for example, client side reach the timeout, terminate the connection and have not receive response from daemon.
I tested command of IPFS HTTP API.
by add progress=true flag to /pin/add command, no CLOSE_WAIT happen even the hash is invalid.
but commands like pin/rm have no flag like progress , so it still CLOSE_WAIT when client reach the timeout.

[Stebalien]

Wait, are you seeing TIME_WAIT or CLOSE_WAIT?

[suutaku]

Wait, are you seeing TIME_WAIT or CLOSE_WAIT?

I'm sorry! typing miss. It's CLOSE_WAIT !!!!!!!

[suutaku]

@Stebalien
after check the source code. if client reach the timeout, IPFS hang on serverHandler{c.server}.ServeHTTP(w, w.req) at net/http/server.go:1884 (I added some debug logs, maybe that code is not line 1884).
I think is IPFS handleFunc's bug?

Update

I add a ticker to check if connection was CLOSE_WAIT status close it. but only close connection, serverHandler{c.server}.ServeHTTP(w, w.req) still not return.

[Stebalien]

Could you run curl http://127.0.0.1:5001/debug/pprof/goroutine?debug=2 > goroutines and upload the result? That'll tell me where it's getting stuck.

[suutaku]

http://www.cotnetwork.com/goroutines
ipfs version 0.4.21-dev

[Stebalien]

Were you reproducing the issue when you ran curl http://127.0.0.1:5001/debug/pprof/goroutine?debug=2 > goroutines? I'm not seeing anything stuck in ServeHTTP.

[suutaku]

client (curl) was quit. but ServeHTTP for client request still running (for a long time). At this time, TCP was CLOSE_WAIT on server side.

[Stebalien]

I see. So the bug is specifically with ipfs pin add and ipfs pin rm. It looks like we aren't feeding the context all the way through so we're not completely canceling the operation.

[suutaku]

thank you. I'm going to test it.

[suutaku]

problem solved.
thank you agin!