Cluster.JoinAsync / Cluster.JoinSeedNodesAsync

[Horusiath]

Motivation

So far we have only fire-and-forget version of Cluster.Join. If user wanted to be sure, that the current node actually joined the cluster, he/she had to define his/her logic in Cluster.OnMemberUp callback. Example:

var cluster = Cluster.Get(system);
cluster.Join(seedAddress);
cluster.OnMemberUp(() => {
    // some initialization logic
});

Proposal

This PR introduces two methods:

• Task Cluster.JoinAsync(Address, CancellationToken = default(CancellationToken)
• Task Cluster.JoinSeedNodesAsync(IEnumerable<Address>, CancellationToken = default(CancellationToken)

With them, we can asynchronously await until the current node will successfully join the cluster:

var cluster = Cluster.Get(system);
await cluster.JoinAsync(seedAddress);
// initialization logic

Concerns

One thing, I'm curious about (probably @Aaronontheweb will be able to answer) - in current implementation JoinAsync can be cancelled, but it cannot fail. How can we detect that nodes had failed to join the cluster? Will Cluster.OnMemberRemoved be helpful here? Could node be removed before it actually reaches member UP status?

[Horusiath]

@Aaronontheweb question to you: does this RegisterOnMemberRemoved has any sense from the perspective of joining to cluster? I've taken a look at cluster protocol description, and theoretically it's possible, however I've never run into such case.

[Aaronontheweb]

How can we detect that nodes had failed to join the cluster? Will Cluster.OnMemberRemoved be helpful here? Could node be removed before it actually reaches member UP status?

Going to noodle on that one; the task has to fail in order for this to be useful.

[Horusiath]

I've added a test case for failure scenario (simply joining to some random address, which isn't used by any cluster node). It passes - this way we have covered all cases: successful/failed async join and cancellation.

[Aaronontheweb]

Need to add a pair of additional tests to ensure that this method is idempotent. I'm confident that based on its implementation that it should be but it's best to have specs that can prove it.

[Aaronontheweb]

@Horusiath need a spec here that tests for "what happens if this method is called when we're already connected to the cluster?"

[Horusiath]

It will immediately complete with success (I've seen this in the impl), but I guess you're right. We probably should have tests for that.

[Aaronontheweb]

@Horusiath need a spec here that tests for "what happens if this method is called when we're already connected to the cluster?"

[Horusiath]

I've added additional checks:

1. Assert that JoinAsync/JoinSeedNodesAsync will return successfully completed task on second call (if first call succeeded ofc).
2. Assert that JoinAsync/JoinSeedNodesAsync will fail after cluster shutdown has been called.