Broadcast VC requests in parallel and fix subscription error

[michaelsproul]

Issue Addressed

Closes:

• VC fallback broadcast runs in series #5232

Proposed Changes

Use join_all to run futures in parallel.

This seemed a bit cleaner than using JoinSet, and is similarly efficient. Each loop iteration should be quick because .status().await just checks an rwlock, it doesn't actually make any requests to the BN.

This PR also fixes a bug whereby we wouldn't mark subscriptions as sent if any node failed. This meant that in the case of 1 offline node, subscriptions would continue getting spammed at the BN right up to the duty slot, which would result in the infamous warning:

WARN Not enough time for a discovery search

This is fixed in 2 ways:

• We check if we are too close to the duty slot in should_send_subscription_at
• We mark subscriptions as sent as long as one node receives them (we also just log a warning in this case rather than an error).

Along the way I also added a 500ms timeout for subscription requests, which previously had no timeout! I'm open to tweaking this timeout or putting it in another PR.

Additional Info

This improvement should help the performance of clusters of nodes when one of them goes AWOL due to an issue like:

• Expensive fork-choice queued attestation mutation #6206

[realbigsean]

LGTM

[michaelsproul]

This isn't working as well as expected (we still seem to have requests getting massively delayed), so I'm trying some improvements

[michaelsproul]

I think I've fixed the bug in 1126795

Rolling out now

[jimmygchen]

Random fact:

The probability of getting the first 9 characters of a Git commit hash as digits (0-9) is approximately 1.455%

[michaelsproul]

@mergify queue

[mergify]

queue

✅ The pull request has been merged automatically

The pull request has been merged automatically at a68f34a