Introduce Kibana config field in stack config policy

[kvalliyurnatt]

What does this change?

Introduce Kibana config Field in StackConfigPolicy. This field can be used to manage multiple kibana instances using stackConfigPolicy. The only configuration that can be configured for kibana using the StackConfigPolicy is the kibanaConfig which are the configurations that go into the kibana.yml file.

• Add unit tests
• Test locally in dev
• Introduce secure settings fields individually under ES and Kibana
• Try to configure an auth policy

Testing

I Have tested with all four auth config policies that we intend to support , JWT, OIDC, SAML and LDAP. I was able to successfully set these up with the StackConfigPolicy and also login into Kibana(make API calls against Elasticsearch in case of JWT) successfully.

However there are limitations here. With JWT and LDAP it is straightforward to configure multiple elasticsearch clusters with the same auth config. With OIDC we generate client secret and application ID and while doing so in the identity provider we enter a redirect url which is generally KIBANA_ENDPOINT_URL/api/security/oidc/callback now to be able to configure multiple KIbana instances to work with this, there are some identity providers like Google that allow you enter more than on redirect url. So it is possible to configure OIDC for multiple elasticsearch clusters as long as the provider being used allows entering multiple urls for the redirect url. However for SAML I could not find a similar option and to me it seems like we might not be able to configure multiple instances using the same SAML config as the SAML config is generated using a redirect url that points to a specific kibana instance.

JWT

I tested configuring JWT auth policy and it worked as expected. I add the jwks secret, elasticsearch config, role mapping and shared secret to the stackconfigpolicy, and then used the JWT and shared secret in headers to call Elasticsearch

curl -s -k -X GET -H "Authorization: Bearer $JWT" -H "ES-Client-Authentication: sharedsecret $SHAREDSECRET" https://localhost:9200/_security/_authenticate -H 'Content-Type: application/json'

Output

{"username":"elastic-agent","roles":["remote_monitoring_collector"],"full_name":null,"email":null,"metadata":{"jwt_claim_jti":"<removed>","jwt_token_type":"id_token","jwt_claim_iss":"https://es.credentials.controller.k8s.elastic.co","jwt_claim_aud":["elasticsearch"],"jwt_claim_sub":"elastic-agent"},"enabled":true,"authentication_realm":{"name":"jwt1","type":"jwt"},"lookup_realm":{"name":"jwt1","type":"jwt"},"authentication_type":"realm"}

I used the elastic-agent tokens so that I don't have to create a new role. But I found this was causing issues to Kibana due to the secure settings being applied to both Kibana and Elasticsearch. Working on a fix for this

OIDC

StackConfigPolicy

apiVersion: stackconfigpolicy.k8s.elastic.co/v1alpha1
kind: StackConfigPolicy
metadata:
  name: test-stack-config-policy
  # namespace: elastic-system or test-namespace
spec:
  resourceSelector:
    matchLabels:
      env: my-label
  elasticsearch:
    secureSettings:
    - secretName: oidc-secret
    securityRoleMappings:
      oidc_kibana:
        roles: [ "superuser" ]
        rules:
          all:
            - field: { realm.name: "oidc1" }
            - field: { username: "<username>" }
        enabled: true
    config:
       logger.org.elasticsearch.discovery: DEBUG
       xpack:
         security:
           authc:
             token.enabled: true
             realms:
               oidc:
                 oidc1:
                   order: 2
                   rp.client_id: "<Application Client ID>"
                   rp.response_type: "code"
                   rp.requested_scopes: ["openid", "email"]
                   rp.redirect_uri: "https://kibana.eck-ocp.elastic.dev/api/security/oidc/callback"
                   op.issuer: "https://accounts.google.com"
                   op.authorization_endpoint: "https://accounts.google.com/o/oauth2/v2/auth"
                   op.token_endpoint: "https://oauth2.googleapis.com/token"
                   op.userinfo_endpoint: "https://openidconnect.googleapis.com/v1/userinfo"
                   op.jwkset_path: "https://www.googleapis.com/oauth2/v3/certs"
                   claims.principal: email
                   claim_patterns.principal: "^([^@]+)@elastic\\.co$"
    secretMounts:
    - secretName: "test1234567891234567"
      mountPath: "test"
  kibana:
    config:
      server:
        publicBaseUrl: "https://kibana.eck-ocp.elastic.dev"
      xpack.security.authc.providers:
        oidc.oidc1:
          order: 0
          realm: oidc1
          description: "Log in with GCP"
        basic.basic1:
          order: 1

SAML with Okta

StackConfigPolicy

apiVersion: stackconfigpolicy.k8s.elastic.co/v1alpha1
kind: StackConfigPolicy
metadata:
  name: test-stack-config-policy
  # namespace: elastic-system or test-namespace
spec:
  resourceSelector:
    matchLabels:
      env: my-label
  elasticsearch:
    securityRoleMappings:
      cloud_saml_kibana:
        roles: [ "superuser" ]
        rules:
          all:
            - field: { realm.name: "test-saml" }
            - field: { username: "<email address>" }
        enabled: true
    config:
      logger.org.elasticsearch.discovery: DEBUG
      xpack.security.authc.realms:
        saml.test-saml:
          order: 2
          nameid_format: "urn:oasis:names:tc:SAML:2.0:nameid-format:transient"
          attributes.principal: "nameid:persistent"
          attributes.groups: "groups"
          idp.metadata.path: "<okta idp url>"
          idp.entity_id: "<okta entity id url>"
          sp.entity_id: "https://kibana.eck-ocp.elastic.dev"
          sp.acs: "https://kibana.eck-ocp.elastic.dev/api/security/saml/callback"
          sp.logout: "https://kibana.eck-ocp.elastic.dev/logout"
  kibana:
    config:
      server:
        publicBaseUrl: "https://kibana.eck-ocp.elastic.dev"
      xpack.security.authc.providers:
        saml.saml1:
          order: 0
          realm: test-saml
          description: "Log in with Okta"
          icon: "https://www.okta.com/themes/custom/okta_www_theme/images/logo.svg"
        basic.basic1:
          order: 1

LDAP

I set up a LDAP server using the openLDAP helm chart on my dev kubernetes cluster and then configured two elasticsearch clusters via stack config policy to interact with it. I was able to sucessfully login via kibana using a LDAP user.
stackConfigPolicy

apiVersion: stackconfigpolicy.k8s.elastic.co/v1alpha1
kind: StackConfigPolicy
metadata:
  name: test-stack-config-policy
  # namespace: elastic-system or test-namespace
spec:
  resourceSelector:
    matchLabels:
      env: my-label
  elasticsearch:
    secureSettings:
    - secretName: ldap-secret
    securityRoleMappings:
      inline_ldap_role:
        roles: [ "superuser" ]
        rules:
          all:
            - field: { realm.name: "ldap1" }
            - field: { dn: "*,ou=users,dc=example,dc=org" }
        enabled: true
    config:
      xpack.security.authc.realms:
        ldap:
          ldap1:
            order: 0
            url: "ldap://openldap.default.svc.cluster.local:1389"
            bind_dn: "cn=admin,dc=example,dc=org"
            user_search:
              base_dn: "dc=example,dc=org"
              filter: "(cn={0})"
            group_search:
              base_dn: "dc=example,dc=org"
            unmapped_groups_as_roles: false

[barkbay]

\"test-stack-config-policy\" is invalid: spec.elasticsearch: Required value: Elasticsearch settings are mandatory and must not be empty"

I guess we should add something along those lines in the validation logic:

	if policy.Spec.Kibana.Config != nil {
		settingsCount += len(policy.Spec.Kibana.Config.Data)
	}

[naemono]

Haven't gotten fully through this, or tested locally, but some initial things I saw.

[naemono]

Still yet to test this locally, but have gotten through looking at all of the changes...

[naemono]

I know this isn't your code, but do you see any reason this shouldn't be re-written as:

if _, secret := range secrets.Items {

[kvalliyurnatt]

I think it is because the linter will complain when we try send the memory address of the secret here

err := c.Delete(ctx, &secret)
		if err != nil && !apierrors.IsNotFound(err) {
			return err
		}

referencing memory of temporary loop variables

[pebrc]

Code looks good to me. I have not gotten round to actually testing it though.

[kvalliyurnatt]

buildkite test this -f p=gke

[pebrc]

LGTM. Nice work! I did some very basic tests only. We already discussed that we need some documentation updates. Are you also going to adjust/extend our e2e tests for the stack config policy feature?

[kvalliyurnatt]

LGTM. Nice work! I did some very basic tests only. We already discussed that we need some documentation updates. Are you also going to adjust/extend our e2e tests for the stack config policy feature?

Yeah I can look into adding/ adjusting existing e2e tests, can that be a separate PR or should be part of this ?

[pebrc]

LGTM. Nice work! I did some very basic tests only. We already discussed that we need some documentation updates. Are you also going to adjust/extend our e2e tests for the stack config policy feature?

Yeah I can look into adding/ adjusting existing e2e tests, can that be a separate PR or should be part of this ?

Can be separate

[barkbay]

LGTM, great job 👍

[barkbay]

Should we return an error here?

[kvalliyurnatt]

This function is called from secureSettingsVolume in the common keystore package, which I think could be called from the apm, beat logstash etc, that is why I just decided to log and return.