Add support for the desired nodes API

[barkbay]

This PR is a draft implementation to add support for the Elasticsearch desired API. I want to raise it now to allow anyone interested in this feature to try it out. There is an ongoing discussion regarding how the CPU capacity should be provided to Elasticsearch (refer to the related ECK issue or this Elasticsearch PR) which is not yet reflected in this draft.

Required resources

For the operator to be able to call the desired node API the following resources are expected in nodeSets[*].podTemplate.spec.containers["elasticsearch"]:

• memory: Both the limit and the request must be set to a same value.
• cpu:
  • limit is returned if set
  • if limit is not set then request is returned
• storage: Multiple storage data path is not supported. The storage path must be backed by a volume claim. The storage capacity is read from the PersistentVolumeClaim status, or from the PersistentVolumeClaim specification if it is not bound yet.

What if resources are missing?

If the capacity of at least one resource (cpu,memory or storage) cannot be calculated the reconciliation is not stopped, the desired node API is however not called, it is also cleared. The Elasticsearch status subresource helps to understand why, here are a few examples:

    {
      "lastTransitionTime": "2022-05-10T12:54:14Z",
      "message": "Cannot get compute and storage resources from Elasticsearch resource generation 2: cannot compute resources for NodeSet \"ml\": no CPU request or limit set",
      "status": "False",
      "type": "ResourcesAwareManagement"
    }

    {
      "lastTransitionTime": "2022-05-10T12:59:29Z",
      "message": "Cannot get compute and storage resources from Elasticsearch resource generation 2: cannot compute resources for NodeSet \"hot\": Memory limit is set but value is 0",
      "status": "False",
      "type": "ResourcesAwareManagement"
    }

If the state of the desired node API can be calculated for a generation:

    {
      "lastTransitionTime": "2022-05-10T13:03:44Z",
      "message": "Successfully calculated compute and storage resources from Elasticsearch resource generation 4",
      "status": "True",
      "type": "ResourcesAwareManagement"
    }

Testing

Minimum Elasticsearch version to test this PR is 8.1.0

[pebrc]

This looks really clean 👌. I left a few minor comments. In my experiments what I found confusing is the interaction between our resource defaults and resource limits/requests I set explicitly but that is nothing to do with your work here.

[barkbay]

Commit f511711 relies on elastic/elasticsearch#86434 to set the CPU resources, and makes Elasticsearch 8.3.0 the min. supported version.

[barkbay]

This PR is ready for review, however you'll need elastic/elasticsearch#86434 to be merged in 8.3.0-SNAPSHOT to test it or run the new E2E check. I'll post a comment here once it is the case.

[fcofdez]

I've just merged elastic/elasticsearch#86434. It should be available in the next 8.3.0-SNAPSHOT build

[barkbay]

Jenkins test this please

[barkbay]

elastic/elasticsearch#86434 is now available in the last 8.3.0-SNAPSHOT Elasticsearch image (docker.elastic.co/elasticsearch/elasticsearch@sha256:2e822d038c0b0f14ac906ca9077c594fdc06458cab0566966b8a95c9ddce6f9b).

[pebrc]

LGTM 👍 as I said before its very clean and easy to read. I did some tests with the latest snapshot build of Elasticsearch that contains the processor ranges change. But my tests where not super thorough I just checked that API was called and that it contained desired nodes information. I am not sure how to verify whether Elasticsearch can take advantage of that information.

[pebrc]

I would like to better understand why we have to replace these variables for the desired nodes API. #5528 (comment) it seems that you are worried some of the optimisations that ES does will be negatively affected if there are still variables in the settings. Can you maybe add a comment?

[barkbay]

I added a comment, the main motivation is to provide a value to the node.name parameter.

[thbkrkr]

Left some very minor comments.
LGTM!