fix: non-blocking peerlog logging #7232
Conversation
Stebalien
force-pushed
the
fix/less-verbose
branch
from
4580d78
to
0b0c6c9
Compare
Avoid ever blocking new connections in the peer logger. Instead: 1. Send all new peers to a highly buffered channel. 2. Emit "dropped event" errors whenever we detect that we're dropping events and falling behind. 3. Don't log protocols, they're too large. 4. Don't log disconnects, we don't need them.
Stebalien
force-pushed
the
fix/less-verbose
branch
from
0b0c6c9
to
bdbb79d
Compare
|
Context: this is the plugin we use to count peers in the network. Unfortunately, something sometimes goes wrong with logging and it starts blocking. When that happens, we end up halting everything because we can't log. This logs from a separate goroutine when we can so we don't completely block everything else. |
Stebalien
force-pushed
the
fix/less-verbose
branch
from
7502e35
to
20ad9f2
Compare
| "peer", conn.RemotePeer().Pretty(), | ||
| ) | ||
| } | ||
| notifee.DisconnectedF = func(net network.Network, conn network.Conn) { |
There was a problem hiding this comment.
We're intentionally removing this event.
| } | ||
| node.PeerHost.Network().Notify(¬ifee) | ||
|
|
||
| sub, err := node.PeerHost.EventBus().Subscribe( | ||
| new(event.EvtPeerIdentificationCompleted), | ||
| eventbus.BufSize(1024), |
There was a problem hiding this comment.
There's a default buffer size of 16 and that should be enough for us now.
| "identified", | ||
| "peer", e.Peer.Pretty(), | ||
| "agent", agent, | ||
| "protocols", protocols, |
There was a problem hiding this comment.
We're removing the protocols (too much data).
|
odd that this is the place where logging gets backed up. it doesn't seem like that much more than what other parts of the system are logging. |
|
I believe the real issue is writing to stdout. We usually only enable error logging which is very low-throughput. |
Stebalien
force-pushed
the
fix/less-verbose
branch
from
20ad9f2
to
a10a14f
Compare
Avoid ever blocking new connections in the peer logger. Instead: