Attempts to connet to a non-running server spam it with connect attempts

[smurfix]

This error does not make sense, on the face of it – either you time out, implying that there was no answer, or the connection was rejected, implying that the server told you there's no listener.

OSError: Timed out trying to connect to 'tcp://127.0.0.1:60590' after 10 s: in <distributed.comm.tcp.TCPConnector object at 0x7fce383a1ef0>: ConnectionRefusedError: [Errno 111] Connection refused

What actually happens is that distributed spams the server with connect requests for ten seconds, all of which get rejected. This is not a good idea. At all. Please either use a reasonable back-off or simply believe the server when it says ECONNREFUSED.

[mrocklin]

Thank you for the issue @smurfix

Please either use a reasonable back-off

This seems reasonable to me. Is this something that you would be interested in contributing?

or simply believe the server when it says ECONNREFUSED.

Sometimes the server comes online during the connection process. This happens frequently in Kubernetes situations where the address genuinely may not have existed a short while ago.

[smurfix]

Exponential backoff is easy … PR created.