go http concurrency and Goroutines do not release memory!!!

[andymacau853]

1. What version of Go are you using (go version)?
   go version go1.6.2 linux/amd64
2. What operating system and processor architecture are you using (go env)?
   GOARCH="amd64"
   GOBIN=""
   GOEXE=""
   GOHOSTARCH="amd64"
   GOHOSTOS="linux"
   GOOS="linux"
   GOPATH="/home/xxxxxx/Desktop/maas"
   GORACE=""
   GOROOT="/usr/local/go"
   GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
   GO15VENDOREXPERIMENT="1"
   CC="gcc"
   GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
   CXX="g++"
   CGO_ENABLED="1"

Linux Development 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

1. What did you do?
   Do many http.request to website in order to keep monitoring http or some services.
2. What did you expect to see?
   Go 1.6 should auto release or GC all go Goroutines if finished jobs...
3. What did you see instead?
   i use command to filer: ps -e -o 'pid,comm,args,pcpu,rsz,vsz,stime,user,uid' | grep test
   20310 test /tmp/go-build396582964/comm 12.0 17604 196984 17:52 root 0
   20648 grep grep --color=auto test 0.0 964 112644 17:52 john 1000
   [johnkou@Development Desktop]$ ps -e -o 'pid,comm,args,pcpu,rsz,vsz,stime,user,uid' | grep test
   20310 test /tmp/go-build396582964/comm 12.6 16972 196984 17:52 root 0
   20673 grep grep --color=auto test 0.0 960 112644 17:52 john 1000
   [johnkou@Development Desktop]$ ps -e -o 'pid,comm,args,pcpu,rsz,vsz,stime,user,uid' | grep test
   20310 test /tmp/go-build396582964/comm 11.0 24036 203036 17:52 root 0
   21681 grep grep --color=auto test 0.0 964 112644 17:53 john 1000
   [johnkou@Development Desktop]$ ps -e -o 'pid,comm,args,pcpu,rsz,vsz,stime,user,uid' | grep test
   20310 test /tmp/go-build396582964/comm 11.0 24284 203036 17:52 root 0
   21727 grep grep --color=auto test 0.0 964 112644 17:53 john 1000
   [johnkou@Development Desktop]$ ps -e -o 'pid,comm,args,pcpu,rsz,vsz,stime,user,uid' | grep test
   20310 test /tmp/go-build396582964/comm 11.0 24540 204092 17:52 root 0
   21792 grep grep --color=auto test 0.0 964 112644 17:53 john 1000
   [johnkou@Development Desktop]$ ps -e -o 'pid,comm,args,pcpu,rsz,vsz,stime,user,uid' | grep test
   20310 test /tmp/go-build396582964/comm 11.1 31056 210436 17:52 root 0
   22886 grep grep --color=auto test 0.0 964 112644 17:54 john 1000
   [johnkou@Development Desktop]$ ps -e -o 'pid,comm,args,pcpu,rsz,vsz,stime,user,uid' | grep test
   20310 test /tmp/go-build396582964/comm 11.1 30824 210436 17:52 root 0
   22899 grep grep --color=auto test 0.0 964 112644 17:54 john 1000
   [johnkou@Development Desktop]$ ps -e -o 'pid,comm,args,pcpu,rsz,vsz,stime,user,uid' | grep test
   20310 test /tmp/go-build396582964/comm 11.6 42492 222576 17:52 root 0
   25171 grep grep --color=auto test 0.0 964 112644 17:56 john 1000

I have already used debug.FreeOSMemory() or runtime.GC() again, but .... wow memory usage is very high and more and more!!!! It means that if many go routine (go func()), it will not release memory!!!????

Could you see my code: https://play.golang.org/p/GxeL81eeJ2
If you want to test, please add more website url in order to expect the result.
Thanks and have a nice day.

[nussjustin]

The problem is that you are running an unbounded number of concurrent requests, each with it's own client, all competeting for bounded resources, causing them to pile up, taking more and more memory and slowing down the requests even more.

(I used the "net/http/pprof" package, explained here, to look at the goroutine blocking profile)

You should use a bounded number of goroutines for requests. The easiest way is to spawn some worker goroutines and send the URLs via a channel.

Also you shouldn't create a new http.Client for each request. Instead use a single http.Client for all requests. The http.Client is designed to be used by many Goroutines simultaneously and keeps it's own pool of connections which then can be reused for new connections (as long as you use the same http.Client).

Here is a adjusted version of your code that bounds the number of concurrent requests, keeping the memory usage to ~ 3MB (in my case). I also added some comments and change a bit of the logic to be more idiomatic.

Also note that tools like top and ps can report much higher memory usage then what is actually used. If you want more accurate values you can use runtime.MemStats as done in the code I linked.

The next time please use the golang-nuts mailing list or stack overflow before creating a new issue.

[andymacau853]

Ok, i will try your suggestion and understand it, thx for your support!

[andymacau853]

I have tried your code, but the memory always more and more by RSZ and VSZ, i think go 1.6 still need to optimize GC???

[bradfitz]

Please move this discussion to https://golang.org/wiki/Questions