Add new metrics to help operators figure out whether there any stuck packets in a channel

[adizere]

Closes: #2175

Description

TODOs:

• add mechanism workaround to handle PosOverflow errors in parse.
  • ref in code
• clarify and fix this comment Add new metrics requested by operators #2182 (comment)
• clarify and fix this Add new metrics requested by operators #2182 (comment)
• fix for reporting cf. Report on slow/stalled channel traffic #2175

---

PR author checklist:

• Added changelog entry, using unclog.
• Added tests: integration (for Hermes) or unit/mock tests (for modules).
• Linked to GitHub issue.
• Updated code comments and documentation (e.g., docs/).

Reviewer checklist:

• Reviewed Files changed in the GitHub PR explorer.
• Manually tested (in case integration/unit/mock tests are absent).

[romac]

@adizere Anything I can help with here?

[adizere]

@adizere Anything I can help with here?

Actually @ljoss17 is leading this effort, he has some interesting WIP in his local copy of this branch.

[adizere]

I ran some tests and noticed we might have a possibly duplicate metric:

# HELP send_packet_count Number of SendPacket relayed.
# TYPE send_packet_count counter
send_packet_count{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer"} 1101
send_packet_count{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer"} 1610

which records data similarly to:

# HELP ibc_receive_packets Number of receive packets relayed per channel
# TYPE ibc_receive_packets counter
ibc_receive_packets{src_chain="ibc-0",src_channel="channel-0",src_port="transfer"} 1101
ibc_receive_packets{src_chain="ibc-1",src_channel="channel-0",src_port="transfer"} 1610

Looking in the code, it seems they are not the same, however:

1. send_packet_count is updated from the supervisor, using events directly received from the websocket.
   • this means that this metrics counts all send packet events that a chain emits (counting the activity of all relayers)
   • the metrics is also updated from packet clearing logic
2. ibc_receive_packets is updated from the packet worker
   • so this metrics counts the send packet events that the current Hermes instance has relayed (not from all relayers active on that chain)

I think we introduced send_packet_count specifically for reporting on slow channels (#2175), but now that we have the oldest_sequence metric, if that counts accurately, then we can probably remove send_packet_count. Or should we keep it for some other reason?

[ljoss17]

I ran some tests and noticed we might have a possibly duplicate metric:

# HELP send_packet_count Number of SendPacket relayed.
# TYPE send_packet_count counter
send_packet_count{chain="ibc-0",channel="channel-0",counterparty="ibc-1",port="transfer"} 1101
send_packet_count{chain="ibc-1",channel="channel-0",counterparty="ibc-0",port="transfer"} 1610

which records data similarly to:

# HELP ibc_receive_packets Number of receive packets relayed per channel
# TYPE ibc_receive_packets counter
ibc_receive_packets{src_chain="ibc-0",src_channel="channel-0",src_port="transfer"} 1101
ibc_receive_packets{src_chain="ibc-1",src_channel="channel-0",src_port="transfer"} 1610

Looking in the code, it seems they are not the same, however:

1. send_packet_count is updated from the supervisor, using events directly received from the websocket.

   • this means that this metrics counts all send packet events that a chain emits (counting the activity of all relayers)
   • the metrics is also updated from packet clearing logic

2. ibc_receive_packets is updated from the packet worker

   • so this metrics counts the send packet events that the current Hermes instance has relayed (not from all relayers active on that chain)

I think we introduced send_packet_count specifically for reporting on slow channels (#2175), but now that we have the oldest_sequence metric, if that counts accurately, then we can probably remove send_packet_count. Or should we keep it for some other reason?

The send_packet_count could be useful if there are pending packets. The oldest_sequence will only give the oldest pending packet, while the send_packet_count combined with the acknowledgement_count will give how many packets are pending.
If this information doesn't seem very useful, then I don't see any reason to keep the send_packet and acknowledgement_count.