[BUG] RemoteShardsBalancer cannot rebalance shards as expected

[bugmakerrrrrr]

Describe the bug

When rebalancing searchable snapshot indices, we first calculate the average number of primary shards that each node should has. In current implementation, the formula is as follows.

(totalNumberOfRemotePrimaryShards + totalNumberOfUnassignedShards)/totalNumberOfRoutingNodes

OpenSearch/server/src/main/java/org/opensearch/cluster/routing/allocation/allocator/RemoteShardsBalancer.java

Lines 247 to 251 in e8ee6db

	final Map<String, Integer> nodePrimaryShardCount = calculateNodePrimaryShardCount(remoteRoutingNodes);
	int totalPrimaryShardCount = nodePrimaryShardCount.values().stream().reduce(0, Integer::sum);
	totalPrimaryShardCount += routingNodes.unassigned().getNumPrimaries();
	int avgPrimaryPerNode = (totalPrimaryShardCount + routingNodes.size() - 1) / routingNodes.size();

If a cluster has both search nodes and dedicated data nodes, this average number of shards calculated using the current formula may be lower than the actual number.. I think that the formula needs to be adjusted as follows:

(totalNumberOfRemotePrimaryShards + totalNumberOfUnassignedRemoteShards)/totalNumberOfSearchNodes

Related component

Search:Searchable Snapshots

To Reproduce

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

RemoteShardsBalancer should rebalance shards as expected

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[bugmakerrrrrr]

@kotwanikunal @andrross you might be interested in this :)