Documentation for running ECK, Elasticsearch, and Kibana on GKE Autopilot

docs/advanced-topics/advanced-topics.asciidoc

@@ -10,6 +10,7 @@ endif::[]
 [partintro]
 --
 - <<{p}-openshift>>
+- <<{p}-autopilot>>
 - <<{p}-custom-images>>
 - <<{p}-service-meshes>>
 - <<{p}-traffic-splitting>>
@@ -20,6 +21,7 @@ endif::[]
 --
 
 include::openshift.asciidoc[leveloffset=+1]
+include::gke-autopilot.asciidoc[leveloffset=+1]
 include::custom-images.asciidoc[leveloffset=+1]
 include::service-meshes.asciidoc[leveloffset=+1]
 include::traffic-splitting.asciidoc[leveloffset=+1]

docs/advanced-topics/gke-autopilot.asciidoc

@@ -0,0 +1,281 @@
+:page_id: autopilot 
+ifdef::env-github[]
+****
+link:https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-{page_id}.html[View this document on the Elastic website]
+****
+endif::[]
+[id="{p}-{page_id}"]
+= Deploy ECK on GKE Autopilot
+
+This page shows how to run ECK on GKE Autopilot.
+
+* <<{p}-autopilot-before-you-begin>>
+* <<{p}-autopilot-deploy-the-operator>>
+* <<{p}-autopilot-deploy-elasticsearch>>
+* <<{p}-autopilot-deploy-kibana>>
+
+[float]
+[id="{p}-autopilot-before-you-begin"]
+== Before you begin
+
+. Ensure you are running a supported version of Kubernetes within Autopilot.
++
+Setting the kernel setting `vm.max_map_count` is critical for production workloads as described in the <<{p}-virtual-memory>> section. If you are intending to set `vm.max_map_count` you must be running at least version 1.25 when on the `regular` channel or using the `rapid` channel, which currently runs version 1.26.
+
+[id="{p}-autopilot-setting-virtual-memory"]
+== Ensuring virtual memory kernel settings
+
+If you are intending to run production workloads on GKE Autopilot then `vm.max_map_count` should be set. The only way to set this kernal setting on the Autopilot hosts is with a `Daemonset` as described in the <<{p}-virtual-memory>> section.
+
+CAUTION: Only use the provided `Daemonset` exactly as specified or it could be rejected by the Autopilot control plane.
+
+[id="{p}-autopilot-deploy-the-operator"]
+== Deploying the ECK Operator
+
+. Apply the all-in-one template, as described in the link:k8s-quickstart.html[quickstart section].
++
+[source,shell,subs="attributes"]
+----
+kubectl create -f https://download.elastic.co/downloads/eck/{eck_version}/crds.yaml
+kubectl apply -f https://download.elastic.co/downloads/eck/{eck_version}/operator.yaml
+----
+
+. As an alternative, you can use the Helm chart to install ECK as described in the link:k8s-install-helm.html[installation docs].
++
+[source,shell,subs="attributes"]
+----
+helm repo add elastic https://helm.elastic.co && helm repo update
+helm install elastic-operator elastic/eck-operator -n elastic-system --create-namespace --version {eck_version}
+----
+
+. Create a namespace to hold the Elastic resources ({eck_resources_list}):
++
+[source,shell]
+----
+kubectl create namespace elastic
+----
+
+[id="{p}-autopilot-deploy-elasticsearch"]
+== Deploy an Elasticsearch instance
+
+Create an Elasticsearch cluster. If you are using the `Daemonset` described in the <<{p}-virtual-memory>> section to set `max_map_count` ensure that the `initContainer` below is also used to ensure the setting prior to starting Elasticsearch.
+
+[source,shell,subs="attributes,+macros"]
+----
+cat $$<<$$EOF | kubectl apply -n elastic -f -
+apiVersion: elasticsearch.k8s.elastic.co/{eck_crd_version}
+kind: Elasticsearch
+metadata:
+  name: elasticsearch-sample
+spec:
+  version: {version}
+  nodeSets:
+  - name: default
+    count: 1
+    # Only uncomment the below section if you are not using the Daemonset to set max_map_count.
+    # config:
+    #  node.store.allow_mmap: false
+    podTemplate:
+      spec:
+        # This init container ensures that the `max_map_count` setting has been applied before starting Elasticsearch.
+        # This is not required, but is encouraged when using the previous Daemonset to set max_map_count.
+        # Do not use this if setting config.node.store.allow_mmap: false
+        initContainers:
+        - name: max-map-count-check
+          command: ['sh', '-c', "while true; do mmc=$(cat /proc/sys/vm/max_map_count); if [ ${mmc} -eq 262144 ]; then exit 0; fi; sleep 1; done"]
+EOF
+----
+
+[id="{p}-autopilot-deploy-kibana"]
+== Deploy a Kibana instance
+
+Use the following code to create a Kibana instance:
+
+[source,shell,subs="attributes,+macros"]
+----
+cat $$<<$$EOF | kubectl apply -n elastic -f -
+apiVersion: kibana.k8s.elastic.co/{eck_crd_version}
+kind: Kibana
+metadata:
+  name: kibana-sample
+spec:
+  version: {version}
+  count: 1
+  elasticsearchRef:
+    name: "elasticsearch-sample"
+  podTemplate:
+    spec:
+      containers:
+      - name: kibana
+        resources:
+          limits:
+            memory: 1Gi
+            cpu: 1
+EOF
+----
+
+[id="{p}-autopilot-deploy-agent"]
+== Deploy a standalone Elastic Agent
+
+Use the following code to deploy Elastic Agent as a DaemonSet in standalone mode with system integration enabled:
+
+[source,shell,subs="attributes,+macros"]
+----
+cat $$<<$$EOF | kubectl apply -n elastic -f -
+apiVersion: agent.k8s.elastic.co/v1alpha1
+kind: Agent
+metadata:
+  name: elastic-agent
+spec:
+  version: {version}
+  elasticsearchRefs:
+    name: "elasticsearch-sample"
+  daemonSet:
+    podTemplate:
+      spec:
+        volumes:
+        - name: "agent-data"
+          ephemeral:
+            volumeClaimTemplate:
+              spec:
+                accessModes: [ "ReadWriteOnce" ]
+                storageClassName: "e2e-default"
+                resources:
+                  requests:
+                    storage: 10Gi
+        containers:
+          - name: agent
+            # These requests come from
+            # docs/manifests/elastic-agent-managed-gke-autopilot.yaml
+            # in https://github.com/elastic/elastic-agent
+            resources:
+              requests:
+                # cpu: 200m # Keep this commented. We dont set the cpu limit to avoid scheduling problems of agent in autopilot scenarios
+                memory: 700Mi
+              limits:
+                # cpu: 200m # Keep this commented. We dont set the cpu limit to avoid scheduling problems of agent in autopilot scenarios
+                memory: 700Mi
+                ephemeral-storage: "500Mi"
+        serviceAccountName: elastic-agent
+        automountServiceAccountToken: true
+        securityContext:
+          runAsUser: 0
+  config:
+    id: 488e0b80-3634-11eb-8208-57893829af4e
+    revision: 2
+    agent:
+      monitoring:
+        enabled: true
+        use_output: default
+        logs: true
+        metrics: true
+    inputs:
+    - id: 4917ade0-3634-11eb-8208-57893829af4e
+      name: system-1
+      revision: 1
+      type: system/metrics
+      use_output: default
+      meta:
+        package:
+          name: system
+          version: {version}
+      data_stream:
+        namespace: default
+      streams:
+      - id: system/metrics-system.cpu
+        data_stream:
+          dataset: system.cpu
+          type: metrics
+        metricsets:
+        - cpu
+        cpu.metrics:
+        - percentages
+        - normalized_percentages
+        period: 10s
+      - id: system/metrics-system.diskio
+        data_stream:
+          dataset: system.diskio
+          type: metrics
+        metricsets:
+        - diskio
+        diskio.include_devices: null
+        period: 10s
+      - id: system/metrics-system.filesystem
+        data_stream:
+          dataset: system.filesystem
+          type: metrics
+        metricsets:
+        - filesystem
+        period: 1m
+        processors:
+        - drop_event.when.regexp:
+            system.filesystem.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
+      - id: system/metrics-system.fsstat
+        data_stream:
+          dataset: system.fsstat
+          type: metrics
+        metricsets:
+        - fsstat
+        period: 1m
+        processors:
+        - drop_event.when.regexp:
+            system.fsstat.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
+      - id: system/metrics-system.load
+        data_stream:
+          dataset: system.load
+          type: metrics
+        metricsets:
+        - load
+        period: 10s
+      - id: system/metrics-system.memory
+        data_stream:
+          dataset: system.memory
+          type: metrics
+        metricsets:
+        - memory
+        period: 10s
+      - id: system/metrics-system.network
+        data_stream:
+          dataset: system.network
+          type: metrics
+        metricsets:
+        - network
+        period: 10s
+        network.interfaces: null
+      - id: system/metrics-system.process
+        data_stream:
+          dataset: system.process
+          type: metrics
+        metricsets:
+        - process
+        period: 10s
+        process.include_top_n.by_cpu: 5
+        process.include_top_n.by_memory: 5
+        process.cmdline.cache.enabled: true
+        process.cgroups.enabled: false
+        process.include_cpu_ticks: false
+        processes:
+        - .*
+      - id: system/metrics-system.process_summary
+        data_stream:
+          dataset: system.process_summary
+          type: metrics
+        metricsets:
+        - process_summary
+        period: 10s
+      - id: system/metrics-system.socket_summary
+        data_stream:
+          dataset: system.socket_summary
+          type: metrics
+        metricsets:
+        - socket_summary
+        period: 10s
+      - id: system/metrics-system.uptime
+        data_stream:
+          dataset: system.uptime
+          type: metrics
+        metricsets:
+        - uptime
+        period: 10s
+EOF
+----

docs/orchestrating-elastic-stack-applications/elasticsearch/virtual-memory.asciidoc

@@ -11,7 +11,11 @@ endif::[]
 By default, Elasticsearch uses memory mapping (`mmap`) to efficiently access indices.
 Usually, default values for virtual address space on Linux distributions are too low for Elasticsearch to work properly, which may result in out-of-memory exceptions. This is why link:k8s-quickstart.html[the quickstart example] disables `mmap` through the `node.store.allow_mmap: false` setting. For production workloads, it is strongly recommended to increase the kernel setting `vm.max_map_count` to `262144` and leave `node.store.allow_mmap` unset.
 
-The kernel setting `vm.max_map_count=262144` can be set on the host either directly or by a dedicated init container, which must be privileged. To add an init container that changes the host kernel setting before your Elasticsearch pod starts, you can use the following example Elasticsearch spec:
+The kernel setting `vm.max_map_count=262144` can be set on the host directly, by a dedicated init container which must be privileged, or a dedicated Daemonset.
+
+== Using an Init Container to set virtual memory
+
+To add an init container that changes the host kernel setting before your Elasticsearch pod starts, you can use the following example Elasticsearch spec:
 [source,yaml,subs="attributes,+macros"]
 ----
 cat $$<<$$EOF | kubectl apply -f -
@@ -37,6 +41,69 @@ EOF
 
 Note that this requires the ability to run privileged containers, which is likely not the case on many secure clusters.
 
+== Using a Daemonset to set virtual memory
+
+To use a Daemonset that changes the host kernel setting on all nodes in your cluster and run an Elasticsearch instance that waits for the kernel setting to be in place:
+
+[source,yaml,subs="attributes,+macros"]
+----
+cat $$<<$$EOF | kubectl apply -n elastic -f -
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: max-map-count-setter
+  labels:
+    k8s-app: max-map-count-setter
+spec:
+  selector:
+    matchLabels:
+      name: max-map-count-setter
+  template:
+    metadata:
+      labels:
+        name: max-map-count-setter
+    spec:
+      initContainers:
+        - name: max-map-count-setter
+          image: docker.io/bash:5.2.15
+          resources:
+            limits:
+              cpu: 100m
+              memory: 32Mi
+          securityContext:
+            privileged: true
+            runAsUser: 0
+          command: ['/usr/local/bin/bash', '-e', '-c', 'echo 262144 > /proc/sys/vm/max_map_count']
+      containers:
+        - name: sleep
+          image: docker.io/bash:5.2.15
+          command: ['sleep', 'infinity']
+---
+apiVersion: elasticsearch.k8s.elastic.co/{eck_crd_version}
+kind: Elasticsearch
+metadata:
+  name: elasticsearch-sample
+spec:
+  version: {version}
+  nodeSets:
+  - name: default
+    count: 1
+    # Only uncomment the below section if you are not using the previous Daemonset to set max_map_count.
+    # config:
+    #  node.store.allow_mmap: false
+    podTemplate:
+      spec:
+        # This init container ensures that the `max_map_count` setting has been applied before starting Elasticsearch.
+        # This is not required, but is encouraged when using the previous Daemonset to set max_map_count.
+        # Do not use this if setting config.node.store.allow_mmap: false
+        initContainers:
+        - name: max-map-count-check
+          command: ['sh', '-c', "while true; do mmc=$(cat /proc/sys/vm/max_map_count); if [ ${mmc} -eq 262144 ]; then exit 0; fi; sleep 1; done"]
+---
+EOF
+----
+
 For more information, check the Elasticsearch documentation on
 link:https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html[Virtual memory].