Fix testFailOverOnFollower

[dnhatn]

We introduced the remote_cluster_client node role in #53924 and assigned follow-tasks to nodes with that role in #54146. Primary shards of follower indices must also be assigned to nodes with that role because they connect to the remote cluster in the bootstrap step (i.e., remote restore).

I think we need a broader fix for this issue. This commit only adjusts the tests to reduce noise on CI.

Closes #58534

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/CCR)

[dnhatn]

@elasticmachine update branch

[jasontedor]

Primary shards of follower indices must also be assigned to nodes with that role because they connect to the remote cluster in the bootstrap step (i.e., remote restore).

I have a fix for this.

[jasontedor]

@dnhatn I think we want something like this:

public class CcrPrimaryFollowerRemoteClusterClientAllocationDecider extends AllocationDecider {

    static final String LABEL = "ccr_primary_follower_remote_cluster_client";

    @Override
    public Decision canAllocate(final ShardRouting shardRouting, final RoutingNode node, final RoutingAllocation allocation) {
        final IndexMetadata indexMetadata = allocation.metadata().index(shardRouting.index());
        final Map<String, String> ccrCustomData = indexMetadata.getCustomData(Ccr.CCR_CUSTOM_METADATA_KEY);
        if (ccrCustomData == null) {
            return allocation.decision(Decision.YES, LABEL, "shard is not a follower and is not under the purview of this decider");
        }
        if (shardRouting.primary() == false) {
            return allocation.decision(Decision.YES, LABEL, "shard is a replica of a follower and is not under the purview of this decider");
        }
        if (node.node().isRemoteClusterClient() == false) {
            return allocation.decision(
                Decision.NO,
                LABEL,
                "node does not have the " + DiscoveryNodeRole.REMOTE_CLUSTER_CLIENT_ROLE.roleName() + " role"
            );
        }
        return allocation.decision(
            Decision.YES,
            LABEL,
            "shard is a primary follower and node has the " + DiscoveryNodeRole.REMOTE_CLUSTER_CLIENT_ROLE + " role"
        );
    }

}

and to make Ccr implement ClusterPlugin and return this decider in the ClusterPlugin#createAllocationDeciders method.

Do you think you can make this into a formal PR and add tests? Note that the master node must also have the remote cluster client role, something that I had not realized until today (because we reach out to the cluster holding the leader index when creating the follower).

[dnhatn]

Do you think you can make this into a formal PR and add tests?

Sure, happy to do it.

[jasontedor]

Thank you so very much @dnhatn. ❤️

[dnhatn]

Closing in favor of #59375.