From dc88752f42fce32e45335600ca2d7075ac64193b Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Wed, 5 Jun 2024 15:46:12 +0100 Subject: [PATCH 01/17] part update --- .../monitor-and-alert/with-grafana-cloud.md | 79 +++---------------- 1 file changed, 12 insertions(+), 67 deletions(-) diff --git a/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md b/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md index 28aa4922bbd5..63ff7b1bd332 100644 --- a/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md +++ b/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md @@ -20,81 +20,26 @@ This topic will walk you through using Grafana Cloud to monitor a Loki installat - Helm 3 or above. See [Installing Helm](https://helm.sh/docs/intro/install/). - A Grafana Cloud account and stack (including Cloud Grafana, Cloud Metrics, and Cloud Logs). -- [Grafana Kubernetes Monitoring using Agent Flow](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/config-k8s-agent-flow/) configured for the Kubernetes cluster. - A running Loki deployment installed in that Kubernetes cluster via the Helm chart. -**Prequisites for Monitoring Loki:** -You must setup the Grafana Kubernetes Integration following the instructions in [Grafana Kubernetes Monitoring using Agent Flow](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/config-k8s-agent-flow/) as this will install necessary components for collecting metrics about your Kubernetes cluster and sending them to Grafana Cloud. Many of the dashboards installed as a part of the Loki integration rely on these metrics. +** Grafana Cloud Connection Credentials:** -Walking through this installation will create two Grafana Agent configurations, one for metrics and one for logs, that will add the external label `cluster: cloud`. In order for the Dashboards in the self-hosted Grafana Loki integration to work, the cluster name needs to match your Helm installation name. If you installed Loki using the command `helm install best-loki-cluster grafana/loki`, you would need to change the `cluster` value in both Grafana Agent configurations from `cloud` to `best-loki-cluster` when setting up the Grafana Kubernetes integration. +The meta-monitoring stack sends metrics, logs and traces to Grafana Cloud. To do this, connection Credentials need to be collected from Grafana Cloud. To do this, follow the steps below: -**To set up the Loki integration in Grafana Cloud:** +1. Create a new Cloud Access Policy in Grafana Cloud. This policy should have the following permissions: + - Logs: Write + - Metrics: Write + - Traces: Write + To do this sign into [Grafana Cloud](https://grafana.com/auth/sign-in/) and select `Access Policies` located under `security` in the left-hand menu. Click on `Create access policy`; name it and select the permissions as described above. Then click `Create`. -1. Get valid Push credentials for your Cloud Metrics and Cloud Logs instances. -1. Create a secret in the same namespace as Loki to store your Cloud Logs credentials. +1. Once the policy is created, click on the policy and then click on `Add token`. Name the token, select an expiration date and click `Create`. Copy the token to a secure location as it will not be displayed again. - ```bash - cat <<'EOF' | NAMESPACE=loki /bin/sh -c 'kubectl apply -n $NAMESPACE -f -' - apiVersion: v1 - data: - password: - username: - kind: Secret - metadata: - name: grafana-cloud-logs-credentials - type: Opaque - EOF - ``` +1. Finaly collect the `Username / Instance ID` and `URL` for the following components in the Grafana Cloud stack: + - **Logs (Loki):** Select `Send Logs`, copy down: `User` and `URL`. From the *Using Grafana with Logs* section. + - **Metrics (Prometheus):** Select `Send Metrics`, copy down: `User` and `URL`. From the *Using a self-hosted Grafana instance with Grafana Cloud Metrics* section. + - **Traces (OTLP):** Select `Configure`, copy down: `User` and `URL`. From the *Using Grafana with Traces* section. -1. Create a secret to store your Cloud Metrics credentials. - ```bash - cat <<'EOF' | NAMESPACE=loki /bin/sh -c 'kubectl apply -n $NAMESPACE -f -' - apiVersion: v1 - data: - password: - username: - kind: Secret - metadata: - name: grafana-cloud-metrics-credentials - type: Opaque - EOF - ``` -1. Enable monitoring metrics and logs for the Loki installation to be sent your cloud database instances by adding the following to your Helm `values.yaml` file: - ```yaml - --- - monitoring: - dashboards: - enabled: false - rules: - enabled: false - selfMonitoring: - logsInstance: - clients: - - url: - basicAuth: - username: - name: grafana-cloud-logs-credentials - key: username - password: - name: grafana-cloud-logs-credentials - key: password - serviceMonitor: - metricsInstance: - remoteWrite: - - url: - basicAuth: - username: - name: grafana-cloud-metrics-credentials - key: username - password: - name: grafana-cloud-metrics-credentials - key: password - ``` - -1. Install the self-hosted Grafana Loki integration by going to your hosted Grafana instance, selecting **Connections** from the Home menu, then search for and install the **Self-hosted Grafana Loki** integration. - -1. Once the self-hosted Grafana Loki integration is installed, click the **View Dashboards** button to see the installed dashboards. From 951cbd37049b19c9a550f5aa92405df5744d3be3 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Fri, 7 Jun 2024 15:31:24 +0100 Subject: [PATCH 02/17] feat: Updated Helm loki monitor with new meta-monitor helm --- .../monitor-and-alert/with-grafana-cloud.md | 230 +++++++++++- .../with-local-monitoring.md | 338 ++++++++---------- 2 files changed, 366 insertions(+), 202 deletions(-) diff --git a/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md b/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md index 63ff7b1bd332..4fdd3df258d9 100644 --- a/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md +++ b/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md @@ -1,7 +1,7 @@ --- -title: Configure monitoring and alerting of Loki using Grafana Cloud +title: Monitor Loki with Grafana Cloud menuTitle: Monitor Loki with Grafana Cloud -description: Configuring monitoring and alerts for Loki using Grafana Cloud. +description: Configuring monitoring for Loki using Grafana Cloud. aliases: - ../../../../installation/helm/monitor-and-alert/with-grafana-cloud weight: 200 @@ -12,34 +12,242 @@ keywords: - grafana cloud --- -# Configure monitoring and alerting of Loki using Grafana Cloud +# Monitor Loki with Grafana Cloud This topic will walk you through using Grafana Cloud to monitor a Loki installation that is installed with the Helm chart. This approach leverages many of the chart's _self monitoring_ features, but instead of sending logs back to Loki itself, it sends them to a Grafana Cloud Logs instance. This approach also does not require the installation of the Prometheus Operator and instead sends metrics to a Grafana Cloud Metrics instance. Using Grafana Cloud to monitor Loki has the added benefit of being able to troubleshoot problems with Loki when the Helm installed Loki is down, as the logs will still be available in the Grafana Cloud Logs instance. -**Before you begin:** +## Before you begig - Helm 3 or above. See [Installing Helm](https://helm.sh/docs/intro/install/). - A Grafana Cloud account and stack (including Cloud Grafana, Cloud Metrics, and Cloud Logs). - A running Loki deployment installed in that Kubernetes cluster via the Helm chart. +## Configure the meta namespace -** Grafana Cloud Connection Credentials:** +The meta-monitoring stack will be installed in a separate namespace called `meta`. To create this namespace, run the following command: + + ```bash + kubectl create namespace meta + ``` + +## Grafana Cloud Connection Credentials The meta-monitoring stack sends metrics, logs and traces to Grafana Cloud. To do this, connection Credentials need to be collected from Grafana Cloud. To do this, follow the steps below: 1. Create a new Cloud Access Policy in Grafana Cloud. This policy should have the following permissions: - - Logs: Write - - Metrics: Write - - Traces: Write - To do this sign into [Grafana Cloud](https://grafana.com/auth/sign-in/) and select `Access Policies` located under `security` in the left-hand menu. Click on `Create access policy`; name it and select the permissions as described above. Then click `Create`. + - Logs: Write + - Metrics: Write + - Traces: Write + + To do this sign into [Grafana Cloud](https://grafana.com/auth/sign-in/) and select `Access Policies` located under `security` in the left-hand menu. Click on `Create access policy`; name it and select the permissions as described above. Then click `Create`. 1. Once the policy is created, click on the policy and then click on `Add token`. Name the token, select an expiration date and click `Create`. Copy the token to a secure location as it will not be displayed again. -1. Finaly collect the `Username / Instance ID` and `URL` for the following components in the Grafana Cloud stack: +2. Next, collect the `Username / Instance ID` and `URL` for the following components in the Grafana Cloud stack: - **Logs (Loki):** Select `Send Logs`, copy down: `User` and `URL`. From the *Using Grafana with Logs* section. - **Metrics (Prometheus):** Select `Send Metrics`, copy down: `User` and `URL`. From the *Using a self-hosted Grafana instance with Grafana Cloud Metrics* section. - - **Traces (OTLP):** Select `Configure`, copy down: `User` and `URL`. From the *Using Grafana with Traces* section. + - **Traces (OTLP):** Select `Configure`, copy down: `Instance ID` and `Endpoint`. From the *OTLP Endpoint* section. + +3. Finally, generate the secrets for each metric type within your K8's cluster: + ```bash + kubectl create secret generic logs -n meta \ + --from-literal=username= \ + --from-literal= \ + --from-literal=endpoint='https:///loki/api/v1/push' + + kubectl create secret generic metrics -n meta \ + --from-literal=username= \ + --from-literal=password= \ + --from-literal=endpoint='https:///api/prom/push' + + kubectl create secret generic traces -n meta \ + --from-literal=username= \ + --from-literal=password= \ + --from-literal=endpoint='https:///otlp' + ``` + +## Configuration and Installation + +To install the meta-monitoring helm chart, a `values.yaml` file will need to be created. This file at a minimum should contain the following: + * The namespace to monitor + * Enablement of cloud monitoring + +This example `values.yaml` file provides the minimum configuration to monitor the `loki` namespace: + +```yaml + namespacesToMonitor: + - default + + cloud: + logs: + enabled: true + secret: "logs" + metrics: + enabled: true + secret: "metrics" + traces: + enabled: true + secret: "traces" +``` +For further configuration options, refer to the [reference file](https://github.com/grafana/meta-monitoring-chart/blob/main/charts/meta-monitoring/values.yaml). + +To install the meta-monitoring helm chart, run the following command: + +```bash +helm repo add grafana https://grafana.github.io/helm-charts +helm repo update +helm install meta-monitoring grafana/meta-monitoring -n meta -f values.yaml +``` +or when upgrading the configuration: +```bash +helm upgrade meta-monitoring grafana/meta-monitoring -n meta -f values.yaml +``` + +To verify the installation, run the following command: + +```bash +kubectl get pods -n meta +``` +It should return the following pods: +```bash +NAME READY STATUS RESTARTS AGE +meta-alloy-0 2/2 Running 0 23h +meta-alloy-1 2/2 Running 0 23h +meta-alloy-2 2/2 Running 0 23h +``` + + +## Enable Loki Tracing + +By default, Loki does not have tracing enabled. To enable tracing, modify the Loki configuration by editing the `values.yaml` file and adding the following configuration: + +Set the `tracing.enabled` configuration to `true`: +```yaml +loki: + tracing: + enabled: true +``` + +Next, instrument each of the Loki components to send traces to the meta-monitoring stack. Add the following configuration to each of the Loki components: + +```yaml +ingester: + replicas: 3 + extraEnv: + - name: JAEGER_ENDPOINT + value: "http://mmc-alloy-external.default.svc.cluster.local:14268/api/traces" + # This sets the Jaeger endpoint where traces will be sent. + # The endpoint points to the mmc-alloy service in the default namespace at port 14268. + + - name: JAEGER_AGENT_TAGS + value: 'cluster="prod",namespace="default"' + # This specifies additional tags to attach to each span. + # Here, the cluster is labeled as "prod" and the namespace as "default". + + - name: JAEGER_SAMPLER_TYPE + value: "ratelimiting" + # This sets the sampling strategy for traces. + # "ratelimiting" means that traces will be sampled at a fixed rate. + + - name: JAEGER_SAMPLER_PARAM + value: "1.0" + # This sets the parameter for the sampler. + # For ratelimiting, "1.0" typically means one trace per second. +``` + +Since the meta-monitoring stack is installed in the `meta` namespace, the Loki components will need to be able to communicate with the meta-monitoring stack. To do this, create a new `externalname` service in the `default` namespace that points to the `meta` namespace by running the following command: + +```bash +kubectl create service externalname mmc-alloy-external --external-name meta-alloy.meta.svc.cluster.local -n default +``` + +Finally, upgrade the Loki installation with the new configuration: + +```bash +helm upgrade --values values.yaml loki grafana/loki +``` + +## Import the Loki Dashboards to Grafana Cloud + +The meta-monitoring stack includes a set of dashboards that can be imported into Grafana Cloud. These can be located within the [meta-monitoring repository](https://github.com/grafana/meta-monitoring-chart/tree/main/charts/meta-monitoring/src/dashboards). + + +## Installing Rules + +The meta-monitoring stack includes a set of rules that can be installed to monitor the Loki installation. These rules can be located within the [meta-monitoring repository](https://github.com/grafana/meta-monitoring-chart/). To install the rules: + +1. Clone the repository: + ```bash + git clone https://github.com/grafana/meta-monitoring-chart/ + ``` +1. Install `mimirtool` based on the instructions located [here](https://grafana.com/docs/mimir/latest/manage/tools/mimirtool/) +1. Create a new access policy token in Grafana Cloud with the following permissions: + - Rules: Write + - Rules: Read +1. Create a token for the access policy and copy it to a secure location. +1. Install the rules: + ```bash + mimirtool rules load --address= --id= --key= *.yaml + ``` +1. Verify that the rules have been installed: + ```bash + mimirtool rules list --address= --id= --key= + ``` + It should return a list of rules that have been installed. + ```bash + + loki-rules: + - name: loki_rules + rules: + - record: cluster_job:loki_request_duration_seconds:99quantile + expr: histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket[5m])) by (le, cluster, job)) + - record: cluster_job:loki_request_duration_seconds:50quantile + expr: histogram_quantile(0.50, sum(rate(loki_request_duration_seconds_bucket[5m])) by (le, cluster, job)) + - record: cluster_job:loki_request_duration_seconds:avg + expr: sum(rate(loki_request_duration_seconds_sum[5m])) by (cluster, job) / sum(rate(loki_request_duration_seconds_count[5m])) by (cluster, job) + - record: cluster_job:loki_request_duration_seconds_bucket:sum_rate + expr: sum(rate(loki_request_duration_seconds_bucket[5m])) by (le, cluster, job) + - record: cluster_job:loki_request_duration_seconds_sum:sum_rate + expr: sum(rate(loki_request_duration_seconds_sum[5m])) by (cluster, job) + - record: cluster_job:loki_request_duration_seconds_count:sum_rate + expr: sum(rate(loki_request_duration_seconds_count[5m])) by (cluster, job) + - record: cluster_job_route:loki_request_duration_seconds:99quantile + expr: histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket[5m])) by (le, cluster, job, route)) + - record: cluster_job_route:loki_request_duration_seconds:50quantile + expr: histogram_quantile(0.50, sum(rate(loki_request_duration_seconds_bucket[5m])) by (le, cluster, job, route)) + - record: cluster_job_route:loki_request_duration_seconds:avg + expr: sum(rate(loki_request_duration_seconds_sum[5m])) by (cluster, job, route) / sum(rate(loki_request_duration_seconds_count[5m])) by (cluster, job, route) + - record: cluster_job_route:loki_request_duration_seconds_bucket:sum_rate + expr: sum(rate(loki_request_duration_seconds_bucket[5m])) by (le, cluster, job, route) + - record: cluster_job_route:loki_request_duration_seconds_sum:sum_rate + expr: sum(rate(loki_request_duration_seconds_sum[5m])) by (cluster, job, route) + - record: cluster_job_route:loki_request_duration_seconds_count:sum_rate + expr: sum(rate(loki_request_duration_seconds_count[5m])) by (cluster, job, route) + - record: cluster_namespace_job_route:loki_request_duration_seconds:99quantile + expr: histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket[5m])) by (le, cluster, namespace, job, route)) + - record: cluster_namespace_job_route:loki_request_duration_seconds:50quantile + expr: histogram_quantile(0.50, sum(rate(loki_request_duration_seconds_bucket[5m])) by (le, cluster, namespace, job, route)) + - record: cluster_namespace_job_route:loki_request_duration_seconds:avg + expr: sum(rate(loki_request_duration_seconds_sum[5m])) by (cluster, namespace, job, route) / sum(rate(loki_request_duration_seconds_count[5m])) by (cluster, namespace, job, route) + - record: cluster_namespace_job_route:loki_request_duration_seconds_bucket:sum_rate + expr: sum(rate(loki_request_duration_seconds_bucket[5m])) by (le, cluster, namespace, job, route) + - record: cluster_namespace_job_route:loki_request_duration_seconds_sum:sum_rate + expr: sum(rate(loki_request_duration_seconds_sum[5m])) by (cluster, namespace, job, route) + - record: cluster_namespace_job_route:loki_request_duration_seconds_count:sum_rate + expr: sum(rate(loki_request_duration_seconds_count[5m])) by (cluster, namespace, job, route) + ``` +## Kube-state-metrics +Metrics about Kubernetes objects are scraped from [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics). This needs to be installed in the cluster. The `kubeStateMetrics.endpoint` entry in meta-monitoring `values.yaml` should be set to it's address (without the `/metrics` part in the URL): +```yaml +kubeStateMetrics: + # Scrape https://github.com/kubernetes/kube-state-metrics by default + enabled: true + # This endpoint is created when the helm chart from + # https://artifacthub.io/packages/helm/prometheus-community/kube-state-metrics/ + # is used. Change this if kube-state-metrics is installed somewhere else. + endpoint: kube-state-metrics.kube-state-metrics.svc.cluster.local:8080 +``` diff --git a/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md b/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md index dfa491fe966f..b127a712007c 100644 --- a/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md +++ b/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md @@ -1,7 +1,7 @@ --- -title: Configure monitoring and alerting -menuTitle: Configure monitoring and alerting -description: Configuring monitoring and alerts using the Helm chart. +title: Monitor Loki using a local LGTM stack +menuTitle: Monitor Loki using a local LGTM stack +description: Monitor Loki using a local LGTM stack aliases: - ../../../../installation/helm/monitor-and-alert/with-local-monitoring/ weight: 100 @@ -11,203 +11,159 @@ keywords: - alerting --- -# Configure monitoring and alerting +# Monitor Loki using a local LGTM stack -By default this Helm Chart configures meta-monitoring of metrics (service monitoring) and logs (self monitoring). This topic will walk you through configuring monitoring using a monitoring solution local to the same cluster where Loki is installed. +This topic will walk you through deploying the local meta-monitoring helm to monitor a Loki installation that is installed with the Helm chart. This approach leverages many of the chart's _self monitoring_ features, but instead of sending logs back to Loki itself, it sends them to a small LGTM stack running within the meta namespace. -The `ServiceMonitor` resource works with either the Prometheus Operator or the Grafana Agent Operator, and defines how Loki's metrics should be scraped. Scraping this Loki cluster using the scrape config defined in the `SerivceMonitor` resource is required for the included dashboards to work. A `MetricsInstance` can be configured to write the metrics to a remote Prometheus instance such as Grafana Cloud Metrics. +## Before you begin -_Self monitoring_ is enabled by default. This will deploy a `GrafanaAgent`, `LogsInstance`, and `PodLogs` resource which will instruct the Grafana Agent Operator (installed separately) on how to scrape this Loki cluster's logs and send them back to itself. Scraping this Loki cluster using the scrape config defined in the `PodLogs` resource is required for the included dashboards to work. +- Helm 3 or above. See [Installing Helm](https://helm.sh/docs/intro/install/). +- A running Kubernetes cluster with a running Loki deployment. -Rules and alerts are automatically deployed. +## Configure the meta namespace -**Before you begin:** +The meta-monitoring stack will be installed in a separate namespace called `meta`. To create this namespace, run the following command: -- Helm 3 or above. See [Installing Helm](https://helm.sh/docs/intro/install/). -- A running Kubernetes cluster with a running Loki deployment. -- A running Grafana instance. -- A running Prometheus Operator installed using the `kube-prometheus-stack` Helm chart. + ```bash + kubectl create namespace meta + ``` + + +## Configuration and Installation + +The meta-monitoring stack is installed using the `meta-monitoring` Helm chart. The local mode deploys a small LGTM stack that includes Alloy, Grafana, Mimir, Loki, and Tempo. To configure the meta-monitoring stack, create a `values.yaml` file with the following content: + +```yaml +namespacesToMonitor: +- default + +cloud: + logs: + enabled: false + metrics: + enabled: false + traces: + enabled: false + +local: + grafana: + enabled: true + logs: + enabled: true + metrics: + enabled: true + traces: + enabled: true + minio: + enabled: true +``` + +For further configuration options, refer to the [reference file](https://github.com/grafana/meta-monitoring-chart/blob/main/charts/meta-monitoring/values.yaml). + +Local mode by default will also enable Minio, which will act as the object storage for the LGTM stack. To provide access to Minio a generic needs to be created. To create the generic secret, run the following command: + +```bash +kubectl create secret generic minio -n meta \ + --from-literal= \ + --from-literal= +``` +**Note:** Username and password must have a minimum of 8 characters. + +To install the meta-monitoring stack, run the following command: + +```bash +helm repo add grafana https://grafana.github.io/helm-charts +helm repo update +helm install meta-monitoring grafana/meta-monitoring -n meta -f values.yaml +``` + +or when upgrading the configuration: +```bash +helm upgrade meta-monitoring grafana/meta-monitoring -n meta -f values.yaml +``` + +To verify the installation, run the following command: + +```bash +kubectl get pods -n meta +``` +It should return the following pods: +```bash +grafana-59d664f55f-dtfqr 1/1 Running 2 (2m7s ago) 137m +loki-backend-0 2/2 Running 2 (2m7s ago) 137m +loki-backend-1 2/2 Running 4 (2m7s ago) 137m +loki-backend-2 2/2 Running 3 (2m7s ago) 137m +loki-read-6f775d8c5-6t749 1/1 Running 1 (2m7s ago) 137m +loki-read-6f775d8c5-kdd8m 1/1 Running 1 (2m7s ago) 137m +loki-read-6f775d8c5-tsw2r 1/1 Running 1 (2m7s ago) 137m +loki-write-0 1/1 Running 1 (2m7s ago) 137m +loki-write-1 1/1 Running 1 (2m7s ago) 137m +loki-write-2 1/1 Running 1 (2m7s ago) 137m +meta-alloy-0 2/2 Running 2 (2m7s ago) 137m +meta-alloy-1 2/2 Running 2 (2m7s ago) 137m +... +``` +## Enable Loki Tracing + +By default, Loki does not have tracing enabled. To enable tracing, modify the Loki configuration by editing the `values.yaml` file and adding the following configuration: + +Set the `tracing.enabled` configuration to `true`: +```yaml +loki: + tracing: + enabled: true +``` -**Prometheus Operator Prequisites** +Next, instrument each of the Loki components to send traces to the meta-monitoring stack. Add the following configuration to each of the Loki components: -The dashboards require certain metric labels to display Kubernetes metrics. The best way to accomplish this is to install the `kube-prometheus-stack` Helm chart with the following values file, replacing `CLUSTER_NAME` with the name of your cluster. The cluster name is what you specify during the helm installation, so a cluster installed with the command `helm install loki-cluster grafana/loki` would be called `loki-cluster`. +```yaml +ingester: + replicas: 3 + extraEnv: + - name: JAEGER_ENDPOINT + value: "http://mmc-alloy-external.default.svc.cluster.local:14268/api/traces" + # This sets the Jaeger endpoint where traces will be sent. + # The endpoint points to the mmc-alloy service in the default namespace at port 14268. + + - name: JAEGER_AGENT_TAGS + value: 'cluster="prod",namespace="default"' + # This specifies additional tags to attach to each span. + # Here, the cluster is labeled as "prod" and the namespace as "default". + + - name: JAEGER_SAMPLER_TYPE + value: "ratelimiting" + # This sets the sampling strategy for traces. + # "ratelimiting" means that traces will be sampled at a fixed rate. + + - name: JAEGER_SAMPLER_PARAM + value: "1.0" + # This sets the parameter for the sampler. + # For ratelimiting, "1.0" typically means one trace per second. +``` + +## Kube-state-metrics + +Metrics about Kubernetes objects are scraped from [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics). This needs to be installed in the cluster. The `kubeStateMetrics.endpoint` entry in meta-monitoring `values.yaml` should be set to it's address (without the `/metrics` part in the URL): ```yaml -kubelet: - serviceMonitor: - cAdvisorRelabelings: - - action: replace - replacement: - targetLabel: cluster - - targetLabel: metrics_path - sourceLabels: - - "__metrics_path__" - - targetLabel: "instance" - sourceLabels: - - "node" - -defaultRules: - additionalRuleLabels: - cluster: - -"kube-state-metrics": - prometheus: - monitor: - relabelings: - - action: replace - replacement: - targetLabel: cluster - - targetLabel: "instance" - sourceLabels: - - "__meta_kubernetes_pod_node_name" - -"prometheus-node-exporter": - prometheus: - monitor: - relabelings: - - action: replace - replacement: - targetLabel: cluster - - targetLabel: "instance" - sourceLabels: - - "__meta_kubernetes_pod_node_name" - -prometheus: - monitor: - relabelings: - - action: replace - replacement: - targetLabel: cluster +kubeStateMetrics: + # Scrape https://github.com/kubernetes/kube-state-metrics by default + enabled: true + # This endpoint is created when the helm chart from + # https://artifacthub.io/packages/helm/prometheus-community/kube-state-metrics/ + # is used. Change this if kube-state-metrics is installed somewhere else. + endpoint: kube-state-metrics.kube-state-metrics.svc.cluster.local:8080 ``` -The `kube-prometheus-stack` installs `ServiceMonitor` and `PrometheusRule` resources for monitoring Kubernetes, and it depends on the `kube-state-metrics` and `prometheus-node-exporter` helm charts which also install `ServiceMonitor` resources for collecting `kubelet` and `node-exporter` metrics. The above values file adds the necessary additional labels required for these metrics to work with the included dashboards. - -If you are using this helm chart in an environment which does not allow for the installation of `kube-prometheus-stack` or custom CRDs, you should run `helm template` on the `kube-prometheus-stack` helm chart with the above values file, and review all generated `ServiceMonitor` and `PrometheusRule` resources. These resources may have to be modified with the correct ports and selectors to find the various services such as `kubelet` and `node-exporter` in your environment. - -**To install the dashboards:** - -1. Dashboards are enabled by default. Set `monitoring.dashboards.namespace` to the namespace of the Grafana instance if it is in a different namespace than this Loki cluster. -1. Dashbards must be mounted to your Grafana container. The dashboards are in `ConfigMap`s named `loki-dashboards-1` and `loki-dashboards-2` for Loki, and `enterprise-logs-dashboards-1` and `enterprise-logs-dashboards-2` for GEL. Mount them to `/var/lib/grafana/dashboards/loki-1` and `/var/lib/grafana/dashboards/loki-2` in your Grafana container. -1. Create a dashboard provisioning file called `dashboards.yaml` in `/etc/grafana/provisioning/dashboards` of your Grafana container with the following contents (_note_: you may need to edit the `orgId`): - - ```yaml - --- - apiVersion: 1 - providers: - - disableDeletion: true - editable: false - folder: Loki - name: loki-1 - options: - path: /var/lib/grafana/dashboards/loki-1 - orgId: 1 - type: file - - disableDeletion: true - editable: false - folder: Loki - name: loki-2 - options: - path: /var/lib/grafana/dashboards/loki-2 - orgId: 1 - type: file - ``` - -**To add add additional Prometheus rules:** - -1. Modify the configuration file `values.yaml`: - - ```yaml - monitoring: - rules: - additionalGroups: - - name: loki-rules - rules: - - record: job:loki_request_duration_seconds_bucket:sum_rate - expr: sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, job) - - record: job_route:loki_request_duration_seconds_bucket:sum_rate - expr: sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, job, route) - - record: node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate - expr: sum(rate(container_cpu_usage_seconds_total[1m])) by (node, namespace, pod, container) - ``` - -**To disable monitoring:** - -1. Modify the configuration file `values.yaml`: - - ```yaml - selfMonitoring: - enabled: false - - serviceMonitor: - enabled: false - ``` - -**To use a remote Prometheus and Loki instance such as Grafana Cloud** - -1. Create a `secrets.yaml` file with credentials to access the Grafana Cloud services: - - ```yaml - --- - apiVersion: v1 - kind: Secret - metadata: - name: primary-credentials-metrics - namespace: default - stringData: - username: "" - password: "" - --- - apiVersion: v1 - kind: Secret - metadata: - name: primary-credentials-logs - namespace: default - stringData: - username: "" - password: "" - ``` - -2. Add the secret to Kubernetes with `kubectl create -f secret.yaml`. - -3. Add a `remoteWrite` section to `serviceMonitor` in `values.yaml`: - - ```yaml - monitoring: - ... - serviceMonitor: - enabled: true - ... - metricsInstance: - remoteWrite: - - url: - basicAuth: - username: - name: primary-credentials-metrics - key: username - password: - name: primary-credentials-metrics - key: password - ``` - -4. Add a client to `monitoring.selfMonitoring.logsInstance.clients`: - - ```yaml - monitoring: - --- - selfMonitoring: - enabled: true - logsInstance: - clients: - - url: - basicAuth: - username: - name: primary-credentials-logs - key: username - password: - name: primary-credentials-logs - key: password - lokiCanary: - enabled: false - ``` - -5. Install the `Loki meta-motoring` connection on Grafana Cloud. +## Accessing the meta-monitoring stack + +To access the meta-monitoring stack, you can use port-forwarding to access the Grafana dashboard. To do this, run the following command: + +```bash +kubectl port-forward -n meta svc/grafana 3000:3000 +``` + +## Dashboards and Rules + +The local meta-monitoring stack comes with a set of pre-configured dashboards and alerting rules. These can be accessed via +[http://localhost:3000](http://localhost:3000) using the default credentials `admin` and `admin`. \ No newline at end of file From 9f8ff80cfe235ffc127c74cfc4588eac1f49ac15 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Fri, 7 Jun 2024 15:45:36 +0100 Subject: [PATCH 03/17] feat: Added link to repo --- .../setup/install/helm/monitor-and-alert/with-grafana-cloud.md | 2 ++ .../install/helm/monitor-and-alert/with-local-monitoring.md | 2 ++ 2 files changed, 4 insertions(+) diff --git a/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md b/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md index 4fdd3df258d9..125762d93667 100644 --- a/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md +++ b/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md @@ -16,6 +16,8 @@ keywords: This topic will walk you through using Grafana Cloud to monitor a Loki installation that is installed with the Helm chart. This approach leverages many of the chart's _self monitoring_ features, but instead of sending logs back to Loki itself, it sends them to a Grafana Cloud Logs instance. This approach also does not require the installation of the Prometheus Operator and instead sends metrics to a Grafana Cloud Metrics instance. Using Grafana Cloud to monitor Loki has the added benefit of being able to troubleshoot problems with Loki when the Helm installed Loki is down, as the logs will still be available in the Grafana Cloud Logs instance. +These instructions are based off the [meta-monitoring-chart repository](https://github.com/grafana/meta-monitoring-chart/tree/main). + ## Before you begig - Helm 3 or above. See [Installing Helm](https://helm.sh/docs/intro/install/). diff --git a/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md b/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md index b127a712007c..6a0530a04b32 100644 --- a/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md +++ b/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md @@ -15,6 +15,8 @@ keywords: This topic will walk you through deploying the local meta-monitoring helm to monitor a Loki installation that is installed with the Helm chart. This approach leverages many of the chart's _self monitoring_ features, but instead of sending logs back to Loki itself, it sends them to a small LGTM stack running within the meta namespace. +These instructions are based off the [meta-monitoring-chart repository](https://github.com/grafana/meta-monitoring-chart/tree/main). + ## Before you begin - Helm 3 or above. See [Installing Helm](https://helm.sh/docs/intro/install/). From 1865bcde783bf4bafa933294266847bdcb16ef76 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Tue, 11 Jun 2024 14:36:54 +0100 Subject: [PATCH 04/17] Update docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md Co-authored-by: J Stickler --- .../setup/install/helm/monitor-and-alert/with-grafana-cloud.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md b/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md index 125762d93667..6483088782f3 100644 --- a/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md +++ b/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md @@ -18,7 +18,7 @@ This topic will walk you through using Grafana Cloud to monitor a Loki installat These instructions are based off the [meta-monitoring-chart repository](https://github.com/grafana/meta-monitoring-chart/tree/main). -## Before you begig +## Before you begin - Helm 3 or above. See [Installing Helm](https://helm.sh/docs/intro/install/). - A Grafana Cloud account and stack (including Cloud Grafana, Cloud Metrics, and Cloud Logs). From 0d94877bee6b9c9add0a61448b1121f1d32d7153 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Tue, 11 Jun 2024 14:37:09 +0100 Subject: [PATCH 05/17] Update docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md Co-authored-by: J Stickler --- .../setup/install/helm/monitor-and-alert/with-grafana-cloud.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md b/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md index 6483088782f3..f05a2a25e4e7 100644 --- a/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md +++ b/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md @@ -34,7 +34,7 @@ The meta-monitoring stack will be installed in a separate namespace called `meta ## Grafana Cloud Connection Credentials -The meta-monitoring stack sends metrics, logs and traces to Grafana Cloud. To do this, connection Credentials need to be collected from Grafana Cloud. To do this, follow the steps below: +The meta-monitoring stack sends metrics, logs, and traces to Grafana Cloud. This requires that you know your connection credentials to Grafana Cloud. To obtain connection credentials, follow the steps below: 1. Create a new Cloud Access Policy in Grafana Cloud. This policy should have the following permissions: - Logs: Write From 2e9e3796167b23b77ee9905132a7658fd21bcf50 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Tue, 11 Jun 2024 14:37:32 +0100 Subject: [PATCH 06/17] Update docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md Co-authored-by: J Stickler --- .../helm/monitor-and-alert/with-grafana-cloud.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md b/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md index f05a2a25e4e7..4922512c312f 100644 --- a/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md +++ b/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md @@ -36,12 +36,16 @@ The meta-monitoring stack will be installed in a separate namespace called `meta The meta-monitoring stack sends metrics, logs, and traces to Grafana Cloud. This requires that you know your connection credentials to Grafana Cloud. To obtain connection credentials, follow the steps below: -1. Create a new Cloud Access Policy in Grafana Cloud. This policy should have the following permissions: +1. Create a new Cloud Access Policy in Grafana Cloud. + 1. Sign into [Grafana Cloud](https://grafana.com/auth/sign-in/). + 1. In the main menu, select **Administration > Users and Access > Cloud Access Policies**. + 1. Click **Create access policy**. + 1. Give the policy a **Name** and select the following permissions: - Logs: Write - Metrics: Write - Traces: Write - - To do this sign into [Grafana Cloud](https://grafana.com/auth/sign-in/) and select `Access Policies` located under `security` in the left-hand menu. Click on `Create access policy`; name it and select the permissions as described above. Then click `Create`. + 1. Click **Create**. + 1. Once the policy is created, click on the policy and then click on `Add token`. Name the token, select an expiration date and click `Create`. Copy the token to a secure location as it will not be displayed again. From 675cf6ca02eb2a3cc4b02842d896104a9a20a8ef Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Tue, 11 Jun 2024 14:37:41 +0100 Subject: [PATCH 07/17] Update docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md Co-authored-by: J Stickler --- .../install/helm/monitor-and-alert/with-grafana-cloud.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md b/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md index 4922512c312f..d095aaed76c0 100644 --- a/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md +++ b/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md @@ -47,7 +47,9 @@ The meta-monitoring stack sends metrics, logs, and traces to Grafana Cloud. This 1. Click **Create**. -1. Once the policy is created, click on the policy and then click on `Add token`. Name the token, select an expiration date and click `Create`. Copy the token to a secure location as it will not be displayed again. +1. Once the policy is created, select the policy and click **Add token**. +1. Name the token, select an expiration date, then click **Create**. +1. Copy the token to a secure location as it will not be displayed again. 2. Next, collect the `Username / Instance ID` and `URL` for the following components in the Grafana Cloud stack: - **Logs (Loki):** Select `Send Logs`, copy down: `User` and `URL`. From the *Using Grafana with Logs* section. From b889859ff9ce82ca2d5e114f8ac58383a95fc059 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Tue, 11 Jun 2024 14:38:01 +0100 Subject: [PATCH 08/17] Update docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md Co-authored-by: J Stickler --- .../install/helm/monitor-and-alert/with-local-monitoring.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md b/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md index 6a0530a04b32..4e65ebf08d98 100644 --- a/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md +++ b/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md @@ -69,7 +69,9 @@ kubectl create secret generic minio -n meta \ --from-literal= \ --from-literal= ``` -**Note:** Username and password must have a minimum of 8 characters. +{{< admonition type="note" >}} +Username and password must have a minimum of 8 characters. +{{< /admonition >}} To install the meta-monitoring stack, run the following command: From f30f4a17b97342c941d90559ea82ad1302386578 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Tue, 11 Jun 2024 14:38:11 +0100 Subject: [PATCH 09/17] Update docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md Co-authored-by: J Stickler --- .../install/helm/monitor-and-alert/with-local-monitoring.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md b/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md index 4e65ebf08d98..438ca4664db5 100644 --- a/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md +++ b/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md @@ -73,7 +73,7 @@ kubectl create secret generic minio -n meta \ Username and password must have a minimum of 8 characters. {{< /admonition >}} -To install the meta-monitoring stack, run the following command: +To install the meta-monitoring stack, run the following commands: ```bash helm repo add grafana https://grafana.github.io/helm-charts From b09c737cd2f324e6acd6fa0302dce8b4e36b2e8b Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Tue, 11 Jun 2024 14:38:20 +0100 Subject: [PATCH 10/17] Update docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md Co-authored-by: J Stickler --- .../install/helm/monitor-and-alert/with-local-monitoring.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md b/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md index 438ca4664db5..1efa09ff9af8 100644 --- a/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md +++ b/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md @@ -118,7 +118,7 @@ loki: enabled: true ``` -Next, instrument each of the Loki components to send traces to the meta-monitoring stack. Add the following configuration to each of the Loki components: +Next, instrument each of the Loki components to send traces to the meta-monitoring stack. Add the `extraEnv` configuration to each of the Loki components: ```yaml ingester: From 436d0b33499107b852f9adcd2f397ff7aba24f3c Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Tue, 11 Jun 2024 14:38:29 +0100 Subject: [PATCH 11/17] Update docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md Co-authored-by: J Stickler --- .../install/helm/monitor-and-alert/with-local-monitoring.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md b/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md index 1efa09ff9af8..ede50071dc30 100644 --- a/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md +++ b/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md @@ -147,7 +147,7 @@ ingester: ## Kube-state-metrics -Metrics about Kubernetes objects are scraped from [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics). This needs to be installed in the cluster. The `kubeStateMetrics.endpoint` entry in meta-monitoring `values.yaml` should be set to it's address (without the `/metrics` part in the URL): +Metrics about Kubernetes objects are scraped from [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics). This needs to be installed in the cluster. The `kubeStateMetrics.endpoint` entry in the meta-monitoring `values.yaml` should be set to its address (without the `/metrics` part in the URL): ```yaml kubeStateMetrics: From c6e5a4a11df561a7810f9b557646ef8a7abe7861 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Tue, 11 Jun 2024 15:04:15 +0100 Subject: [PATCH 12/17] Updated Cloud desciption and fixed wording in MinIO section --- .../setup/install/helm/monitor-and-alert/with-grafana-cloud.md | 2 +- .../install/helm/monitor-and-alert/with-local-monitoring.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md b/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md index d095aaed76c0..f878c71b288f 100644 --- a/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md +++ b/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md @@ -14,7 +14,7 @@ keywords: # Monitor Loki with Grafana Cloud -This topic will walk you through using Grafana Cloud to monitor a Loki installation that is installed with the Helm chart. This approach leverages many of the chart's _self monitoring_ features, but instead of sending logs back to Loki itself, it sends them to a Grafana Cloud Logs instance. This approach also does not require the installation of the Prometheus Operator and instead sends metrics to a Grafana Cloud Metrics instance. Using Grafana Cloud to monitor Loki has the added benefit of being able to troubleshoot problems with Loki when the Helm installed Loki is down, as the logs will still be available in the Grafana Cloud Logs instance. +This guide will walk you through using Grafana Cloud to monitor a Loki installation set up with the Helm chart. This method takes advantage of many of the chart's self-monitoring features, sending metrics, logs, and traces from the Loki deployment to Grafana Cloud. Monitoring Loki with Grafana Cloud offers the added benefit of troubleshooting Loki issues even when the Helm-installed Loki is down, as the telemetry data will remain available in the Grafana Cloud instance. These instructions are based off the [meta-monitoring-chart repository](https://github.com/grafana/meta-monitoring-chart/tree/main). diff --git a/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md b/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md index ede50071dc30..85201b109535 100644 --- a/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md +++ b/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md @@ -62,7 +62,7 @@ local: For further configuration options, refer to the [reference file](https://github.com/grafana/meta-monitoring-chart/blob/main/charts/meta-monitoring/values.yaml). -Local mode by default will also enable Minio, which will act as the object storage for the LGTM stack. To provide access to Minio a generic needs to be created. To create the generic secret, run the following command: +Local mode by default will also enable Minio, which will act as the object storage for the LGTM stack. To provide access to Minio a generic secret needs to be created. To create the generic secret, run the following command: ```bash kubectl create secret generic minio -n meta \ From 2e21355fb79a34684dffdde6d78dc27404e8e5ed Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Wed, 12 Jun 2024 09:46:45 +0100 Subject: [PATCH 13/17] Apply suggestions from code review Committing Suggestions made by Julie Co-authored-by: J Stickler --- .../monitor-and-alert/with-grafana-cloud.md | 39 +++++++++++-------- .../with-local-monitoring.md | 7 ++-- 2 files changed, 25 insertions(+), 21 deletions(-) diff --git a/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md b/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md index f878c71b288f..2eff6a0f2520 100644 --- a/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md +++ b/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md @@ -38,12 +38,12 @@ The meta-monitoring stack sends metrics, logs, and traces to Grafana Cloud. This 1. Create a new Cloud Access Policy in Grafana Cloud. 1. Sign into [Grafana Cloud](https://grafana.com/auth/sign-in/). - 1. In the main menu, select **Administration > Users and Access > Cloud Access Policies**. + 1. In the main menu, select **Security > Access Policies**. 1. Click **Create access policy**. 1. Give the policy a **Name** and select the following permissions: - - Logs: Write - - Metrics: Write - - Traces: Write + - Metrics: Write + - Logs: Write + - Traces: Write 1. Click **Create**. @@ -51,12 +51,17 @@ The meta-monitoring stack sends metrics, logs, and traces to Grafana Cloud. This 1. Name the token, select an expiration date, then click **Create**. 1. Copy the token to a secure location as it will not be displayed again. -2. Next, collect the `Username / Instance ID` and `URL` for the following components in the Grafana Cloud stack: - - **Logs (Loki):** Select `Send Logs`, copy down: `User` and `URL`. From the *Using Grafana with Logs* section. - - **Metrics (Prometheus):** Select `Send Metrics`, copy down: `User` and `URL`. From the *Using a self-hosted Grafana instance with Grafana Cloud Metrics* section. - - **Traces (OTLP):** Select `Configure`, copy down: `Instance ID` and `Endpoint`. From the *OTLP Endpoint* section. - -3. Finally, generate the secrets for each metric type within your K8's cluster: +1. Navigate to the Grafana Cloud Portal **Overview** page. +1. Click the **Details** button for your Prometheus or Mimir instance. + 1. From the **Using a self-hosted Grafana instance with Grafana Cloud Metrics** section, collect the instance **Name** and **URL**. + 1. Navigate back to the **Overview** page. +1. Click the **Details** button for your Loki instance. + 1. From the **Using Grafana with Logs** section, collect the instance **Name** and **URL**. + 1. Navigate back to the **Overview** page. +1. Click the **Details** button for your Tempo instance. + 1. From the **Using Grafana with Tempo** section, collect the instance **Name** and **URL**. + +3. Finally, generate the secrets to store your credentials for each metric type within your Kubernetes cluster: ```bash kubectl create secret generic logs -n meta \ --from-literal=username= \ @@ -76,7 +81,7 @@ The meta-monitoring stack sends metrics, logs, and traces to Grafana Cloud. This ## Configuration and Installation -To install the meta-monitoring helm chart, a `values.yaml` file will need to be created. This file at a minimum should contain the following: +To install the meta-monitoring helm chart, you must create a `values.yaml` file. At a minimum this file should contain the following: * The namespace to monitor * Enablement of cloud monitoring @@ -97,9 +102,9 @@ This example `values.yaml` file provides the minimum configuration to monitor th enabled: true secret: "traces" ``` -For further configuration options, refer to the [reference file](https://github.com/grafana/meta-monitoring-chart/blob/main/charts/meta-monitoring/values.yaml). +For further configuration options, refer to the [sample values.yaml file](https://github.com/grafana/meta-monitoring-chart/blob/main/charts/meta-monitoring/values.yaml). -To install the meta-monitoring helm chart, run the following command: +To install the meta-monitoring helm chart, run the following commands: ```bash helm repo add grafana https://grafana.github.io/helm-charts @@ -136,7 +141,7 @@ loki: enabled: true ``` -Next, instrument each of the Loki components to send traces to the meta-monitoring stack. Add the following configuration to each of the Loki components: +Next, instrument each of the Loki components to send traces to the meta-monitoring stack. Add the `extraEnv` configuration to each of the Loki components: ```yaml ingester: @@ -177,12 +182,12 @@ helm upgrade --values values.yaml loki grafana/loki ## Import the Loki Dashboards to Grafana Cloud -The meta-monitoring stack includes a set of dashboards that can be imported into Grafana Cloud. These can be located within the [meta-monitoring repository](https://github.com/grafana/meta-monitoring-chart/tree/main/charts/meta-monitoring/src/dashboards). +The meta-monitoring stack includes a set of dashboards that can be imported into Grafana Cloud. These can be found in the [meta-monitoring repository](https://github.com/grafana/meta-monitoring-chart/tree/main/charts/meta-monitoring/src/dashboards). ## Installing Rules -The meta-monitoring stack includes a set of rules that can be installed to monitor the Loki installation. These rules can be located within the [meta-monitoring repository](https://github.com/grafana/meta-monitoring-chart/). To install the rules: +The meta-monitoring stack includes a set of rules that can be installed to monitor the Loki installation. These rules can be found in the [meta-monitoring repository](https://github.com/grafana/meta-monitoring-chart/). To install the rules: 1. Clone the repository: ```bash @@ -246,7 +251,7 @@ The meta-monitoring stack includes a set of rules that can be installed to monit ``` ## Kube-state-metrics -Metrics about Kubernetes objects are scraped from [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics). This needs to be installed in the cluster. The `kubeStateMetrics.endpoint` entry in meta-monitoring `values.yaml` should be set to it's address (without the `/metrics` part in the URL): +Metrics about Kubernetes objects are scraped from [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics). This needs to be installed in the cluster. The `kubeStateMetrics.endpoint` entry in the meta-monitoring `values.yaml` should be set to its address (without the `/metrics` part in the URL): ```yaml kubeStateMetrics: diff --git a/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md b/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md index 85201b109535..5f7860fdd8f4 100644 --- a/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md +++ b/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md @@ -13,9 +13,8 @@ keywords: # Monitor Loki using a local LGTM stack -This topic will walk you through deploying the local meta-monitoring helm to monitor a Loki installation that is installed with the Helm chart. This approach leverages many of the chart's _self monitoring_ features, but instead of sending logs back to Loki itself, it sends them to a small LGTM stack running within the meta namespace. +This topic will walk you through using the meta-monitoring Helm chart to deploy a local stack to monitor your production Loki installation. This approach leverages many of the chart's _self monitoring_ features, but instead of sending logs back to Loki itself, it sends them to a small Loki, Grafana, Tempo, Mimir (LGTM) stack running within the `meta` namespace. -These instructions are based off the [meta-monitoring-chart repository](https://github.com/grafana/meta-monitoring-chart/tree/main). ## Before you begin @@ -60,9 +59,9 @@ local: enabled: true ``` -For further configuration options, refer to the [reference file](https://github.com/grafana/meta-monitoring-chart/blob/main/charts/meta-monitoring/values.yaml). +For further configuration options, refer to the [sample values.yaml file](https://github.com/grafana/meta-monitoring-chart/blob/main/charts/meta-monitoring/values.yaml). -Local mode by default will also enable Minio, which will act as the object storage for the LGTM stack. To provide access to Minio a generic secret needs to be created. To create the generic secret, run the following command: +Local mode by default will also enable Minio, which will act as the object storage for the LGTM stack. To provide access to Minio, you need to create a generic secret. To create the generic secret, run the following command: ```bash kubectl create secret generic minio -n meta \ From e68cc392fa5bcc85e47dd4c830353f187d0f0396 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Wed, 12 Jun 2024 10:04:44 +0100 Subject: [PATCH 14/17] Added definition for LGTM --- .../install/helm/monitor-and-alert/with-local-monitoring.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md b/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md index 5f7860fdd8f4..bea8f7877e4b 100644 --- a/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md +++ b/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md @@ -11,7 +11,7 @@ keywords: - alerting --- -# Monitor Loki using a local LGTM stack +# Monitor Loki using a local LGTM (Loki, Grafana, Tempo and Mimir) stack This topic will walk you through using the meta-monitoring Helm chart to deploy a local stack to monitor your production Loki installation. This approach leverages many of the chart's _self monitoring_ features, but instead of sending logs back to Loki itself, it sends them to a small Loki, Grafana, Tempo, Mimir (LGTM) stack running within the `meta` namespace. From 60c548921f8d1d7c26b2459a528f8524de1eaaf4 Mon Sep 17 00:00:00 2001 From: Jayclifford345 Date: Wed, 12 Jun 2024 10:50:53 +0100 Subject: [PATCH 15/17] Fixed title issue --- .../install/helm/monitor-and-alert/with-local-monitoring.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md b/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md index bea8f7877e4b..810ba9e9d188 100644 --- a/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md +++ b/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md @@ -1,7 +1,7 @@ --- -title: Monitor Loki using a local LGTM stack +title: Monitor Loki using a local LGTM (Loki, Grafana, Tempo and Mimir) stack menuTitle: Monitor Loki using a local LGTM stack -description: Monitor Loki using a local LGTM stack +description: Monitor Loki using a local LGTM (Loki, Grafana, Tempo and Mimir) stack aliases: - ../../../../installation/helm/monitor-and-alert/with-local-monitoring/ weight: 100 From f26e21c40521a2559d43e86a4b3f0908737f600f Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Wed, 12 Jun 2024 15:26:40 +0100 Subject: [PATCH 16/17] Apply suggestions from code review Co-authored-by: J Stickler --- .../install/helm/monitor-and-alert/with-grafana-cloud.md | 6 +++--- .../install/helm/monitor-and-alert/with-local-monitoring.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md b/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md index 2eff6a0f2520..64c032dc9635 100644 --- a/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md +++ b/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md @@ -14,7 +14,7 @@ keywords: # Monitor Loki with Grafana Cloud -This guide will walk you through using Grafana Cloud to monitor a Loki installation set up with the Helm chart. This method takes advantage of many of the chart's self-monitoring features, sending metrics, logs, and traces from the Loki deployment to Grafana Cloud. Monitoring Loki with Grafana Cloud offers the added benefit of troubleshooting Loki issues even when the Helm-installed Loki is down, as the telemetry data will remain available in the Grafana Cloud instance. +This guide will walk you through using Grafana Cloud to monitor a Loki installation set up with the `meta-monitoring` Helm chart. This method takes advantage of many of the chart's self-monitoring features, sending metrics, logs, and traces from the Loki deployment to Grafana Cloud. Monitoring Loki with Grafana Cloud offers the added benefit of troubleshooting Loki issues even when the Helm-installed Loki is down, as the telemetry data will remain available in the Grafana Cloud instance. These instructions are based off the [meta-monitoring-chart repository](https://github.com/grafana/meta-monitoring-chart/tree/main). @@ -104,7 +104,7 @@ This example `values.yaml` file provides the minimum configuration to monitor th ``` For further configuration options, refer to the [sample values.yaml file](https://github.com/grafana/meta-monitoring-chart/blob/main/charts/meta-monitoring/values.yaml). -To install the meta-monitoring helm chart, run the following commands: +To install the `meta-monitoring` Helm chart, run the following commands: ```bash helm repo add grafana https://grafana.github.io/helm-charts @@ -249,7 +249,7 @@ The meta-monitoring stack includes a set of rules that can be installed to monit - record: cluster_namespace_job_route:loki_request_duration_seconds_count:sum_rate expr: sum(rate(loki_request_duration_seconds_count[5m])) by (cluster, namespace, job, route) ``` -## Kube-state-metrics +## Install kube-state-metrics Metrics about Kubernetes objects are scraped from [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics). This needs to be installed in the cluster. The `kubeStateMetrics.endpoint` entry in the meta-monitoring `values.yaml` should be set to its address (without the `/metrics` part in the URL): diff --git a/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md b/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md index 810ba9e9d188..1c1fb6d244f3 100644 --- a/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md +++ b/docs/sources/setup/install/helm/monitor-and-alert/with-local-monitoring.md @@ -144,7 +144,7 @@ ingester: # For ratelimiting, "1.0" typically means one trace per second. ``` -## Kube-state-metrics +## Install kube-state-metrics Metrics about Kubernetes objects are scraped from [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics). This needs to be installed in the cluster. The `kubeStateMetrics.endpoint` entry in the meta-monitoring `values.yaml` should be set to its address (without the `/metrics` part in the URL): From 702d877183f09d104cc570b1873a27da01035af9 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Wed, 12 Jun 2024 15:27:15 +0100 Subject: [PATCH 17/17] Apply suggestions from code review Co-authored-by: J Stickler --- .../setup/install/helm/monitor-and-alert/with-grafana-cloud.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md b/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md index 64c032dc9635..73303816db14 100644 --- a/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md +++ b/docs/sources/setup/install/helm/monitor-and-alert/with-grafana-cloud.md @@ -81,7 +81,7 @@ The meta-monitoring stack sends metrics, logs, and traces to Grafana Cloud. This ## Configuration and Installation -To install the meta-monitoring helm chart, you must create a `values.yaml` file. At a minimum this file should contain the following: +To install the `meta-monitoring` Helm chart, you must create a `values.yaml` file. At a minimum this file should contain the following: * The namespace to monitor * Enablement of cloud monitoring