Add a wait for no intializing shards parameter to cluster health

[bleskes]

Sometime it's handy to know everything has stabilized in the cluster when we run rest tests on it. A common problem is that we run the test on varying number of nodes. This means we sometime have assigned replicas other times there is a single node and thus the replicas will stay unassigned. This means we can't use wait for green or wait for yellow.

It will be handy to add a new options to the cluster health requests allowing to wait for all shard initialization to settle down (similar to the current option to wait for all relocations to stop). The new option may even supersede the relocation option (as it is a super set).

[Hogusong]

I'd like to look into this as my first PR. is there a good place to start looking in the code to address this?

[bleskes]

@Hogusong thanks for offering. You can look at the TransportClusterHealthAction code and how it handles the waitForNoRelocatingShards option.

[Hogusong]

@bleskes
So I'm looking at the TransportClusterHealthRequest and understand that I have to create a new member variable "waitForNoInitializingShards". However, I don't fully understand how to wait for all shard initializations to settle down. I see that waitForNoRelocatingShards is a boolean that's set on class creation but would like a little more information on how to structure "waitForNoInitializatingShards"

[jasontedor]

However, I don't fully understand how to wait for all shard initializations to settle down.

@Hogusong You have to check the routing table in the cluster state and possibly wait for changes. I'm happy to give an even bigger hint, but I err on the side of less rather than more so as to not take away the joy of solving the problem.

[matarrese]

Hello, I don't know if @Hogusong is still working on this issue, so I decided to bring it up and send you a pull request. It is not intended to be ready to be tested, as it is failing some tests. I am just asking if it's possible to have a review and understand if it goes in the right direction. If other people working on this, please feel free to close it.