Allow disabling of elastic user.

[naemono]

resolves #7719

Primary changes

This change allows a user to disable the elastic user from being created upon Elasticsearch creation. A use case for this would be when an organization/user would prefer to manage all users/roles via SSO (SAML/IDP/LDAP/etc).

This adds a new field to the Elasticsearch CRD:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-sample
spec:
  auth:
    disableElasticUser: true

This also updates the documentation to reference the new change in multiple sections where the elastic user is mentioned.

Additional changes

Since one of the primary uses of the elastic user is to allow our ECK diagnostics to run, a new user with minimal privileges has been created to allow this functionality: elastic-internal-diagnostics.

Testing done

1. Simple elasticsearch cluster creation
2. Full stack creation using ES/Kibana/Fleet server/Agents with file-realm user added; Kibana login/functionality worked without issues, along with Fleet server and all agents worked perfectly.
3. ECK diagnostics run using this version of ECK.

Open issues

• The eck-diagnostics call to /api/exception_lists/items/_find?list_id=endpoint_host_isolation_exceptions&namespace_type=agnostic is still non-functional. I'm working with the relevant team to try and find the right permissions to adjust.

TODO:

• Documentation updates
• e2e testing; unit testing seems sufficient for this, as we check for the non-existence of the elastic secret.
• Fix unit tests
• Fix e2e smoke tests

[kvalliyurnatt]

LGTM from a code perspective, have not tested it myself.

[pebrc]

Looks good! I was trying to think if the disableElasticUser attribute is the right name but could not come up with something more succinct 👍

[pebrc]

Are you still investigating this or is this the least privileges we can give?

[naemono]

I think I'm going to take another shot at this, as one of the items referenced in the readme (/api/exception_lists/items/_find?list_id=endpoint_host_isolation_exceptions&namespace_type=agnostic) is still under investigation. The rest of the pieces here are ready for review. This could change, and I'll note it here if it does.

[naemono]

@pebrc update: I've found that some of these endpoints require platinum licenses, which is why I'm getting unauthorized errors when attempting to call them, even with privileges that should allow me to call them, as I'm running a dev version of ECK make go-run which doesn't support enterprise features.

I'm going to try and test this a bit differently and update.

[naemono]

Ok @pebrc with the current roles, which are as limited as I can get them, I can successfully run eck-diagnostics with no issues. This should be ready for final reviews. An upcoming PR to eck-diagnostics will be upcoming once this is merged.

[pebrc]

LGTM code wise. Did a quick test disabling/enabling the user. I also got the errors on the api/exception_lists in the Kibana diagnostics, I am approving nonetheless assuming you will update the permissions as needed.

[naemono]

LGTM code wise. Did a quick test disabling/enabling the user. I also got the errors on the api/exception_lists in the Kibana diagnostics, I am approving nonetheless assuming you will update the permissions as needed.

I'm taking one more look at this...

[naemono]

LGTM code wise. Did a quick test disabling/enabling the user. I also got the errors on the api/exception_lists in the Kibana diagnostics, I am approving nonetheless assuming you will update the permissions as needed.

@pebrc just ran the diagnostics using the elastic superuser, and I get a 404 on this endpoint as well, so this is expected, or at least in line with current behavior:

15:08:52.552 [main] INFO  co.elastic.support.diagnostics.commands.BaseQuery - kibana_security_endpoint_host_isolation_1   /api/exception_lists/items/_find?list_id=endpoint_host_isolation_exceptions&namespace_type=agnostic&page=1&per_page=100  failed. Bypassing
15:08:52.552 [main] INFO  co.elastic.support.diagnostics.commands.BaseQuery - See archived diagnostics.log for more detail. Status: 404  Reason: Endpoint does not exist.
15:08:52.578 [main] INFO  co.elastic.support.diagnostics.commands.BaseQuery - kibana_security_endpoint_event_filters_1   /api/exception_lists/items/_find?list_id=endpoint_event_filters&namespace_type=agnostic&page=1&per_page=100  failed. Bypassing
15:08:52.578 [main] INFO  co.elastic.support.diagnostics.commands.BaseQuery - See archived diagnostics.log for more detail. Status: 404  Reason: Endpoint does not exist.