fix(net): Avoid potential concurrency bugs in outbound handshakes

[teor2345]

Motivation

This PR fixes a bunch of potential concurrency bugs in Zebra's outbound handshake code.

Part of #6763.

This is not a blocker for the first stable release.

Complex Code or Requirements

This is concurrent code, but the final code should be a lot easier to reason about than the previous code, because it's all running concurrently in independent tasks.

Solution

• Stop sending peer errors on the PeerSet channel
  • this fixes a potential deadlock or panic when the channel gets full
• Move locking out of the cralwer select!
  • this avoids using the address book lock in a select! statement, which is very hard to reason about
• Move report_failed() out of the CandidateSet
  • another move that makes it easier to reason about concurrency, because the code is now in the same file with no other locking
• Make CandidateSet Send
  • just changes to generic bounds, it seems the compiler has got smarter about this since I last tried (or I have!)
• Make all CandidateSet operations concurrent
  • this code has caused hang and deadlock bugs in the past, now it is fully concurrent, so there are no weird dependencies
• Reduce the gap between handshakes and peer set updates
  • this is another concurrency risk - we used to delay updating the peer set or address book until the next time the crawler task looped around, now we just do it concurrently in the handshake task
• Exit the crawler task on shutdown
  • when channels are disconnected, exit the task, rather than panicking or hanging

Testing

Our existing unit and integration tests have good coverage of this code, and "is this task concurrent" is hard to test.

Review

@arya2 and I discussed part of this design about a week ago.

I can split this into 2-3 PRs if it would be easier, or do a video review.

Reviewer Checklist

• Will the PR name make sense to users?
  • Does it need extra CHANGELOG info? (new features, breaking changes, large changes)
• Are the PR labels correct?
• Does the code do what the ticket and PR says?
  • Does it change concurrent code, unsafe code, or consensus rules?
• How do you know it works? Does it have tests?

Follow Up Work

Try diagnosing any remaining bugs in #6763.

[codecov]

Codecov Report

Merging #6869 (1e643e1) into main (8bd5a98) will decrease coverage by 0.06%.
The diff coverage is 76.92%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6869      +/-   ##
==========================================
- Coverage   77.73%   77.68%   -0.06%     
==========================================
  Files         310      310              
  Lines       41416    41441      +25     
==========================================
- Hits        32196    32194       -2     
- Misses       9220     9247      +27     

[arya2]

This is marvelous, thank you.