All master-eligible nodes may be upgraded before all master-ineligible nodes

[barkbay]

To reproduce:

1. Create the following deployment:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-sample
spec:
  version: 7.16.1
  updateStrategy:
    changeBudget:
      maxUnavailable: 2
  nodeSets:
  - name: content
    config:
      node.roles: ["data_content"]
      node.store.allow_mmap: false
    count: 2
  - name: master
    config:
      node.roles: ["master"]
      node.store.allow_mmap: false
    count: 1

2. Create an index with the following settings:

{
  "settings": {
    "index": {
      "number_of_shards": 2,
      "number_of_replicas": 1
    }
  }
}

3. Upgrade the deployment (to 7.16.2 for example)

The master node instance is upgraded/deleted before one of the two data node, which is not able to join the cluster anymore:

"stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-sample-es-master-0][10.48.178.44:9300][internal:cluster/coordination/join]",
"Caused by: java.lang.IllegalStateException: node version [7.16.1] may not join a cluster comprising only nodes of version [7.16.2] or greater",

The reason is that the predicate do_not_delete_last_master_if_data_nodes_are_not_upgraded only relies on the label "elasticsearch.k8s.elastic.co/node-data" to assess if a node is not a master node:

cloud-on-k8s/pkg/controller/elasticsearch/driver/upgrade_predicates.go

Line 380 in f2fcf2e

if label.IsDataNode(pod) {

[pebrc]

This sounds like something we want to fix asap and include in 2.0?