Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Daemon panic: concurrent write to websocket connection #6197

Closed
Kycklingar opened this issue Apr 9, 2019 · 38 comments · Fixed by multiformats/go-multistream#50
Closed

Daemon panic: concurrent write to websocket connection #6197

Kycklingar opened this issue Apr 9, 2019 · 38 comments · Fixed by multiformats/go-multistream#50
Labels
kind/bug A bug in existing code (including security flaws)

Comments

@Kycklingar
Copy link

Version information:

go-ipfs version: 0.4.20-rc1-495f12c02
Repo version: 7
System version: amd64/linux
Golang version: go1.12.2

Type:

panic

Description:

Daemon is ready
panic: concurrent write to websocket connection

goroutine 77927234 [running]:
github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc005d79740, 0xc0112c2101, 0x0, 0x0, 0x0, 0xc00264cd10, 0xc00264cd10)
	/home/kyck/go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:591 +0x77a
github.com/gorilla/websocket.(*messageWriter).Close(0xc005d79740, 0x155ea80, 0xc00264cd28)
	/home/kyck/go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:709 +0x56
github.com/gorilla/websocket.(*Conn).prepWrite(0xc002691e40, 0x2, 0x10000c00ae47800, 0x0)
	/home/kyck/go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:459 +0x20d
github.com/gorilla/websocket.(*Conn).NextWriter(0xc002691e40, 0x2, 0x8f30e9, 0xc00025e350, 0x10, 0x92658d)
	/home/kyck/go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:494 +0x39
github.com/gorilla/websocket.(*Conn).WriteMessage(0xc002691e40, 0x2, 0xc0045b7000, 0x14, 0x1000, 0x10, 0xc012dfce00)
	/home/kyck/go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:755 +0x73
github.com/libp2p/go-ws-transport.(*Conn).Write(0xc005d79530, 0xc0045b7000, 0x14, 0x1000, 0xc00264ceb8, 0x8f3978, 0x10)
	/home/kyck/go/pkg/mod/github.com/libp2p/go-ws-transport@v0.0.2/conn.go:74 +0x59
bufio.(*Writer).Flush(0xc016ea16c0, 0xc016ea16c0, 0xc011bbef60)
	/usr/local/go/src/bufio/bufio.go:590 +0x7b
github.com/multiformats/go-multistream.delimWriteBuffered(0x7feef2b2b208, 0xc005d79680, 0xc011bbef60, 0x12, 0x12, 0x1, 0x0)
	/home/kyck/go/pkg/mod/github.com/multiformats/go-multistream@v0.0.1/multistream.go:64 +0xb9
github.com/multiformats/go-multistream.handshake.func1(0xc0072fdc20, 0x7feef2b2b150, 0xc005d79680)
	/home/kyck/go/pkg/mod/github.com/multiformats/go-multistream@v0.0.1/client.go:49 +0xab
created by github.com/multiformats/go-multistream.handshake
	/home/kyck/go/pkg/mod/github.com/multiformats/go-multistream@v0.0.1/client.go:48 +0x84
@Stebalien
Copy link
Member

Dammit! I thought that was fixed.

@Stebalien Stebalien added the kind/bug A bug in existing code (including security flaws) label Apr 9, 2019
@Stebalien
Copy link
Member

Are you seeing any other stack traces? I'm having trouble reproducing this.


I'm pretty sure this isn't a concurrent write/close issue. It looks like a concurrent write issue when negotiating the security transport but I'm not sure how this is possible.

@inetic
Copy link

inetic commented Apr 10, 2019

For what it's worth, we're seeing this. But that's IPFS v0.4.19 and on Android.

@Stebalien
Copy link
Member

Thanks. Unfortunately, I'm still going to need to do some work to figure out what's actually causing this.

@Kycklingar
Copy link
Author

Kycklingar commented Apr 12, 2019

Are you seeing any other stack traces? I'm having trouble reproducing this.

Nope, that's all I got. It's only ever happened twice, both on the same day.

@songjiayang
Copy link

@Kycklingar Same with me, no panic now.

@Stebalien
Copy link
Member

Interesting... I wonder if this can only be triggered by a non-ipfs websocket client (someone may have been scanning).

@songjiayang
Copy link

@Stebalien seem look like it , but ipfs node should not panic.

@Stebalien
Copy link
Member

I agree. I'm just trying to reproduce it so I can fix it.

Stebalien added a commit to libp2p/go-conn-security-multistream that referenced this issue Apr 26, 2019
Make sure to wait until we're done writing before returning. The close should
interrupt this process but we should still wait.

might be related to ipfs/kubo#6197
Stebalien added a commit to libp2p/go-libp2p-transport-upgrader that referenced this issue Apr 26, 2019
Stebalien added a commit to libp2p/go-libp2p-transport-upgrader that referenced this issue Apr 26, 2019
@Stebalien
Copy link
Member

@inetic can you reproduce this reliably? Could you try building #6263?

@Stebalien Stebalien reopened this May 6, 2019
Stebalien added a commit to libp2p/go-libp2p-secio that referenced this issue Jun 15, 2019
That makes sure we aren't still writing/reading after we relinquish control.

fixes #40

may be related to ipfs/kubo#6197?
@hsn10
Copy link

hsn10 commented Nov 11, 2019

I have this crash daily, running with --enable-pubsub-experiment --enable-namesys-pubsub --enable-mplex-experiment

go-ipfs version: 0.4.22-
panic: concurrent write to websocket connection

goroutine 42561526 [running]:
github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc00dad2540, 0x1, 0x0, 0x0, 0x0, 0x1, 0xc00c5fb900)
        pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:591 +0x781
github.com/gorilla/websocket.(*messageWriter).Close(0xc00dad2540, 0x11f8649, 0x3)
        pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:709 +0x5d
github.com/gorilla/websocket.(*Conn).prepWrite(0xc00740adc0, 0x2, 0x8aa4bd8, 0x3f)
        pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:459 +0x214
github.com/gorilla/websocket.(*Conn).NextWriter(0xc00740adc0, 0x2, 0xc01d5dafc0, 0x8aa4bd8, 0x0, 0x1000000014e1fc0)
        pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:494 +0x40
github.com/gorilla/websocket.(*Conn).WriteMessage(0xc00740adc0, 0x2, 0xc01d5dafc0, 0x20, 0x40, 0x10, 0xc018931de0)
        pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:755 +0x7a
github.com/libp2p/go-ws-transport.(*Conn).Write(0xc00dd0ff50, 0xc01d5dafc0, 0x20, 0x40, 0x10946c0, 0x11d1be0, 0x2ab4620)
        pkg/mod/github.com/libp2p/go-ws-transport@v0.0.4/conn.go:74 +0x60
bytes.(*Buffer).WriteTo(0xc00db3a6c0, 0x79393c8, 0xc00dad2450, 0x2ab4620, 0xc00db3a6c0, 0x8ba401)
        /usr/lib/go/src/bytes/buffer.go:242 +0xbf
io.copyBuffer(0x79393c8, 0xc00dad2450, 0x14c4920, 0xc00db3a6c0, 0x0, 0x0, 0x0, 0xc018bd7370, 0x438d5d, 0x1382308)
        /usr/lib/go/src/io/io.go:384 +0x355
io.Copy(...)
        /usr/lib/go/src/io/io.go:364
github.com/multiformats/go-multistream.SelectProtoOrFail.func1(0x1200771, 0xa, 0x7d5a580, 0xc00dad2450, 0xc00ec41d40)
        pkg/mod/github.com/multiformats/go-multistream@v0.0.4/client.go:27 +0x16b
created by github.com/multiformats/go-multistream.SelectProtoOrFail
        pkg/mod/github.com/multiformats/go-multistream@v0.0.4/client.go:23 +0x9c

@MatthewSteeples
Copy link

I also see this crash just running ipfs daemon --enable-pubsub-experiment --enable-namesys-pubsub

go-ipfs version: 0.4.22-
Repo version: 7
System version: arm/linux
Golang version: go1.12.7

Running on a Raspberry Pi 3B+

@Stebalien Stebalien reopened this Nov 11, 2019
@Stebalien
Copy link
Member

@MatthewSteeples could you try running go-ipfs master? We believe this may be fixed.

@bonedaddy
Copy link
Contributor

bonedaddy commented Nov 11, 2019

This issue is also happening with general usage of libp2p websockets. I'm getting this issue outside of go-ipfs as well. libp2p/go-ws-transport#56 is my bug report on go-ws-transport

This is a bug within go-ws-transport and makes websockets completely unusable. I have a custom node that's crashing every 30 minutes due to bug.

OP opened report using go-ws-transport v0.0.2 and it's still present in the latest master branch

@djdv
Copy link
Contributor

djdv commented Nov 15, 2019

Follow up:
(still go-ipfs 410e7e8, but with libp2p/go-ws-transport@e2dded5 vendored)
Started the daemon, tried accessing hashes of mine through the gateway, eventually this happened.

Edit:
gist with more logging after running for more time.

https://gist.github.com/djdv/1d6cd8c552d2d2983cfe11f7918058d2

@vyzo
Copy link
Contributor

vyzo commented Nov 15, 2019

I just pushed a debug/issue-6197 branch which has the patch too.

@UltraNote
Copy link

UltraNote commented Nov 18, 2019

go-ipfs version: 0.4.22 and go-ipfs version: 0.5.0-dev

ipfs_panic

@zquestz
Copy link

zquestz commented Nov 19, 2019

Also seeing this on my node:

panic: concurrent write to websocket connection goroutine 154743888 [running]: github.com/gorilla/websocket.(*messageWriter).flushFrame(0xc02f8aeb40, 0x20613ed80877501, 0x0, 0x0, 0x0, 0xc01822a4d0, 0xdc2fc7) /go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:591 +0x77a github.com/gorilla/websocket.(*messageWriter).Close(0xc02f8aeb40, 0x40, 0xc01822a548) /go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:709 +0x56 github.com/gorilla/websocket.(*Conn).prepWrite(0xc00a9cdce0, 0x2, 0x0, 0x0) /go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:459 +0x20d github.com/gorilla/websocket.(*Conn).NextWriter(0xc00a9cdce0, 0x2, 0x7f0fdc9430e8, 0x20, 0x200, 0x100) /go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:494 +0x39 github.com/gorilla/websocket.(*Conn).WriteMessage(0xc00a9cdce0, 0x2, 0xc030ee7900, 0x22, 0x40, 0x10, 0xc01adfb0d0) /go/pkg/mod/github.com/gorilla/websocket@v1.4.0/conn.go:755 +0x73 github.com/libp2p/go-ws-transport.(*Conn).Write(0xc02f8ae840, 0xc030ee7900, 0x22, 0x40, 0x163dfa0, 0x1795ea0, 0x7f10199f5cc0) /go/pkg/mod/github.com/libp2p/go-ws-transport@v0.0.4/conn.go:74 +0x59 bytes.(*Buffer).WriteTo(0xc02c3e8150, 0x7f101805ca88, 0xc02f8aeab0, 0x7f10199f5cc0, 0xc02c3e8150, 0xe25e01) /usr/local/go/src/bytes/buffer.go:242 +0xb8 io.copyBuffer(0x7f101805ca88, 0xc02f8aeab0, 0x1a98d00, 0xc02c3e8150, 0x0, 0x0, 0x0, 0xc01ad51500, 0x9a59d6, 0x1946a18) /usr/local/go/src/io/io.go:384 +0x34e io.Copy(...) /usr/local/go/src/io/io.go:364 github.com/multiformats/go-multistream.SelectProtoOrFail.func1(0x17c705d, 0xc, 0x7f101805c9f0, 0xc02f8aeab0, 0xc001bb30e0) /go/pkg/mod/github.com/multiformats/go-multistream@v0.0.4/client.go:27 +0x164 created by github.com/multiformats/go-multistream.SelectProtoOrFail /go/pkg/mod/github.com/multiformats/go-multistream@v0.0.4/client.go:23 +0x95

@vyzo
Copy link
Contributor

vyzo commented Nov 19, 2019

Can you try running with the debug/issue-6197 branch? It should eliminate the panic and also print stack traces to help us debug the issue.

@zquestz
Copy link

zquestz commented Nov 19, 2019 via email

@jpeletier
Copy link

@zquestz Here you go: epiclabs/go-ipfs:issue-6197 https://hub.docker.com/repository/docker/epiclabs/go-ipfs

@jpeletier
Copy link

@vyzo Here is a stack trace of issue-6197 that printed out after some time running:

WebUI: http://127.0.0.1:5001/webui
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready
goroutine 159924 [running]:
runtime/debug.Stack(0xc0007e9d08, 0x9e3b61, 0x1a60e88)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
	/usr/local/go/src/runtime/debug/stack.go:16 +0x22
github.com/libp2p/go-ws-transport.(*Conn).Write.func1(0xc0019091c0)
	/home/jm/go/pkg/mod/github.com/libp2p/go-ws-transport@v0.1.3-0.20191115171038-e2dded5874d3/conn_native.go:82 +0x3d
github.com/libp2p/go-ws-transport.(*Conn).Write(0xc0019091c0, 0xc001fb5500, 0x34, 0x40, 0x34, 0x0, 0x0)
	/home/jm/go/pkg/mod/github.com/libp2p/go-ws-transport@v0.1.3-0.20191115171038-e2dded5874d3/conn_native.go:90 +0x18b
github.com/libp2p/go-libp2p-secio.(*etmWriter).WriteMsg(0xc0019093c0, 0xc00349af50, 0x10, 0x10, 0x0, 0x0)
	/home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-secio@v0.2.0/rw.go:64 +0x2ee
github.com/libp2p/go-libp2p-secio.readWriteMsg.func1(0xc000e33620, 0x7f8991ec8418, 0xc001cb54a0, 0xc00349af50, 0x10, 0x10)
	/home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-secio@v0.2.0/rw.go:252 +0x4f
created by github.com/libp2p/go-libp2p-secio.readWriteMsg
	/home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-secio@v0.2.0/rw.go:251 +0xab
goroutine 159804 [running]:
runtime/debug.Stack(0x9f6391, 0xc0004c98d8, 0x2)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
	/usr/local/go/src/runtime/debug/stack.go:16 +0x22
github.com/libp2p/go-ws-transport.(*Conn).Close(0xc0019091c0, 0x0, 0x0)
	/home/jm/go/pkg/mod/github.com/libp2p/go-ws-transport@v0.1.3-0.20191115171038-e2dded5874d3/conn_native.go:98 +0x1cd
github.com/libp2p/go-libp2p-secio.(*secureSession).runHandshake(0xc0007ed800, 0x1beb560, 0xc0012f1620, 0x0, 0x0)
	/home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-secio@v0.2.0/protocol.go:131 +0x29c
github.com/libp2p/go-libp2p-secio.newSecureSession(0x1beb560, 0xc0012f0c00, 0xc000042210, 0x22, 0x1bf3240, 0xc00093ed20, 0x7f8991e6c508, 0xc002663da0, 0xc002127950, 0x22, ...)
	/home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-secio@v0.2.0/protocol.go:95 +0x1c5
github.com/libp2p/go-libp2p-secio.(*Transport).SecureOutbound(0xc0001f0360, 0x1beb560, 0xc0012f0c00, 0x7f8991e6c508, 0xc002663da0, 0xc002127950, 0x22, 0xc0001f0360, 0x0, 0x0, ...)
	/home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-secio@v0.2.0/transport.go:40 +0x9b
github.com/libp2p/go-conn-security-multistream.(*SSMuxer).SecureOutbound(0xc0001f2190, 0x1beb560, 0xc0012f0c00, 0x7f8991e6c508, 0xc002663da0, 0xc002127950, 0x22, 0x17a5100, 0x1818560, 0xc0004c9a00, ...)
	/home/jm/go/pkg/mod/github.com/libp2p/go-conn-security-multistream@v0.1.0/ssms.go:56 +0xf2
github.com/libp2p/go-libp2p-transport-upgrader.(*Upgrader).setupSecurity(0xc000214040, 0x1beb560, 0xc0012f0c00, 0x7f8991e6c508, 0xc002663da0, 0xc002127950, 0x22, 0x1bcf500, 0xc00093c6e0, 0x1c011e0, ...)
	/home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-transport-upgrader@v0.1.1/upgrader.go:108 +0x10c
github.com/libp2p/go-libp2p-transport-upgrader.(*Upgrader).upgrade(0xc000214040, 0x1beb560, 0xc0012f0c00, 0x1bee720, 0xc0002d2038, 0x1bfd8e0, 0xc002663da0, 0xc002127950, 0x22, 0x0, ...)
	/home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-transport-upgrader@v0.1.1/upgrader.go:86 +0x29e
github.com/libp2p/go-libp2p-transport-upgrader.(*Upgrader).UpgradeOutbound(0xc000214040, 0x1beb560, 0xc0012f0c00, 0x1bee720, 0xc0002d2038, 0x1bfd8e0, 0xc002663da0, 0xc002127950, 0x22, 0x0, ...)
	/home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-transport-upgrader@v0.1.1/upgrader.go:57 +0xcf
github.com/libp2p/go-ws-transport.(*WebsocketTransport).Dial(0xc0002d2038, 0x1beb560, 0xc0012f0c00, 0x1c011e0, 0xc0020cb280, 0xc002127950, 0x22, 0xc002262580, 0xc0028f9450, 0xdf84755bf, ...)
	/home/jm/go/pkg/mod/github.com/libp2p/go-ws-transport@v0.1.3-0.20191115171038-e2dded5874d3/websocket.go:63 +0xed
github.com/libp2p/go-libp2p-swarm.(*Swarm).dialAddr(0xc00091f080, 0x1beb560, 0xc0012f0c00, 0xc002127950, 0x22, 0x1c011e0, 0xc0020cb280, 0x2790e60, 0x9e330a, 0xc0012f0c00, ...)
	/home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.2/swarm_dial.go:462 +0x219
github.com/libp2p/go-libp2p-swarm.(*dialLimiter).executeDial(0xc000167e50, 0xc002414a80)
	/home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.2/limiter.go:218 +0x1a9
created by github.com/libp2p/go-libp2p-swarm.(*dialLimiter).addCheckFdLimit
	/home/jm/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.2.2/limiter.go:168 +0x5fc

@lazyweirdo
Copy link

Same issue here, go-ipfs 0.4.22 , swarm, api and read only gateway open ...

@asiniscalchi
Copy link

asiniscalchi commented Dec 16, 2019

same here ...
gp-ipfs 0.4.22 and 0.4.18

@Stebalien
Copy link
Member

This should be fixed by #6804.

Stebalien added a commit that referenced this issue Jan 17, 2020
Stebalien added a commit that referenced this issue Jan 17, 2020
Stebalien added a commit that referenced this issue Jan 17, 2020
Stebalien added a commit that referenced this issue Jan 17, 2020
Stebalien added a commit that referenced this issue Jan 17, 2020
Stebalien added a commit that referenced this issue Jan 17, 2020
Stebalien added a commit that referenced this issue Jan 18, 2020
Stebalien added a commit to multiformats/go-multistream that referenced this issue Feb 4, 2020
This was causing a double-write.

fixes libp2p/go-libp2p-pnet#31
probably fixes ipfs/kubo#6802
fixes ipfs/kubo#6197
ralendor pushed a commit to ralendor/go-ipfs that referenced this issue Jun 6, 2020
ralendor pushed a commit to ralendor/go-ipfs that referenced this issue Jun 8, 2020
ralendor pushed a commit to ralendor/go-ipfs that referenced this issue Jun 8, 2020
marten-seemann pushed a commit to libp2p/go-libp2p that referenced this issue Apr 26, 2022
Make sure to wait until we're done writing before returning. The close should
interrupt this process but we should still wait.

might be related to ipfs/kubo#6197
marten-seemann pushed a commit to libp2p/go-libp2p that referenced this issue Apr 26, 2022
hannahhoward pushed a commit to filecoin-project/kubo-api-client that referenced this issue Jun 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug A bug in existing code (including security flaws)
Projects
None yet
Development

Successfully merging a pull request may close this issue.