Improve resilience to nodes going down

[romac]

From @ancazamfir's comment:

A few thoughts on Hermes resilience to nodes going down:

• since it is possible that events have been missed, workers should "reset", e.g. unipath should clear packets, client recheck updates, etc
• it is also possible that events that would have spawned new workers were missed
• we need to make an analysis on handling websocket vs rpc vs grpc failures. These services can be configured to use different nodes but we know that for the passive relayer the events are vital. So maybe just stopping all workers involved and rescanning state at reconnect time would be ok (similar to remove+add chain)

Testing

• the test with dev-env is ok to test some basic things but not enough.
• this test simulates chains stopping and restarting (e.g. chain upgrade with reset height). Normally this should happen with a higher version number (so for hermes these are new chains, e.g. ibc-0 node will be upgraded and restarted as ibc-1)
  • tracked individually in Chain upgrade requirements for Hermes (meta issue) #1209
• ?? if the gaiad restart happens fast enough we are left with some workers from the old instance for channels/ clients that don't exist anymore.
• instead we should use gaia manager to setup a test environment for these scenarios (we would need two full nodes, one to bring down and used for events/queries etc by hermes start and one to be used by another hermes instance that makes IBC state changes):
  • concretely: we need a CI where a chain is restarted and where we assert that hermes can recover after the restart
    • we'll track this in Test scenarios for multi-network IBC #897

Other issues I noticed while debugging #1026:

• logs are very inconsistent in looks, see below, we should enforce a consistent span on all logs.
  • DEBUG ibc_relayer::worker::client: client 'ibc-1 -> ibc-0:07-tendermint-0'
  • DEBUG ibc_relayer::foreign_client: [ibc-0 -> ibc-1:07-tendermint-0]
  • INFO ibc_relayer::worker: [channel-0/transfer:ibc-0->ibc-1]
• packet workers retry and exit, client workers do not
• channel worker is not yet integrated with the telemetry
• channel worker does not have an outer retry loop (it retries for individual commands)

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[adizere]

Some of these issues were handled (e.g., #1205, #1138).

[adizere]

Closing as all issues have either been handled or are tracked separately in more appropriate meta-issues.