Fleet Usage telemetry extension

[juliaElastic]

Summary

Closes https://github.com/elastic/ingest-dev/issues/1261

Added a snippet to the telemetry that I added for each requirement. Please review and let me know if any changes are needed.
Also asked a few questions below. @jlind23 @kpollich

6. is blocked by elasticsearch change to give kibana_system the missing privilege to read logs-elastic_agent* indices.

Took inspiration for task versioning from https://github.com/elastic/kibana/pull/144494/files#diff-0c7c49bf5c55c45c19e9c42d5428e99e52c3a39dd6703633f427724d36108186

• 1. Elastic Agent versions
  Versions of all the Elastic Agent running: agent.version field on .fleet-agents documents

"agent_versions": [
    "8.6.0"
  ],

• 2. Fleet server configuration
  Think we can query for .fleet-policies where some input has type: 'fleet-server' for this, as well as use the Fleet Server Hosts settings that we define via saved objects in Fleet

  "fleet_server_config": {
    "policies": [
      {
        "input_config": {
          "server": {
            "limits.max_agents": 10000
          },
          "server.runtime": "gc_percent:20"
        }
      }
    ]
  }

• 3. Number of policies
  Count of .fleet-policies index

To confirm, did we mean agent policies here?

 "agent_policies": {
    "count": 7,

• 4. Output type contained in those policies
  Collecting this from ts logic, querying from .fleet-policies index. The alternative would be to write a painless script (because the outputs are an object with dynamic keys, we can't do an aggregation directly).

"agent_policies": {
    "output_types": [
      "elasticsearch"
    ]
  }

Did we mean to just collect the types here, or any other info? e.g. output urls

• 5. Average number of checkin failures
  We only have the most recent checkin status and timestamp on .fleet-agents.

Do we mean here to publish the total last checkin failure count? E.g. 3 if 3 agents are in failure checkin status currently.
Or do we mean to publish specific info for all agents (last_checkin_status, last_checkin time, last_checkin_message)?
Are the only statuses error and degraded that we want to send?

  "agent_last_checkin_status": {
    "error": 0,
    "degraded": 0
  },

• 6. Top 3 most common errors in the Elastic Agent logs

Do we mean here elastic-agent logs only, or fleet-server logs as well (maybe separately)?

I found an alternative way to query the message field using sampler and categorize text aggregation:

GET logs-elastic_agent*/_search
{
    "size": 0,
    "query": {
        "bool": {
            "must": [
                {
                    "term": {
                        "log.level": "error"
                    }
                },
                {
                    "range": {
                        "@timestamp": {
                            "gte": "now-1h"
                        }
                    }
                }
            ]
        }
    },
    "aggregations": {
        "message_sample": {
            "sampler": {
                "shard_size": 200
            },
            "aggs": {
                "categories": {
                    "categorize_text": {
                        "field": "message",
                        "size": 10
                    }
                }
            }
        }
    }
}

Example response:

"aggregations": {
    "message_sample": {
      "doc_count": 112,
      "categories": {
        "buckets": [
          {
            "doc_count": 73,
            "key": "failed to unenroll offline agents",
            "regex": ".*?failed.+?to.+?unenroll.+?offline.+?agents.*?",
            "max_matching_length": 36
          },
          {
            "doc_count": 7,
            "key": """stderr panic close of closed channel n ngoroutine running Stop ngithub.com/elastic/beats/v7/libbeat/cmd/instance Beat launch.func5 \n\t/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go n

• 7. Number of checkin failure over the past period of time

I think this is almost the same as #5. The difference would be to report new failures happened only in the last hour, or report all agents in failure state. (which would be an increasing number if the agent stays in failed state).
Do we want these 2 separate telemetry fields?

EDIT: removed the last1hr query, instead added a new field to report agents enrolled per policy (top 10). See comments below.

  "agent_checkin_status": {
    "error": 3,
    "degraded": 0
  },
  "agents_per_policy": [2, 1000],

• 8. Number of Elastic Agent and number of fleet server

This is already there in the existing telemetry:

  "agents": {
    "total_enrolled": 0,
    "healthy": 0,
    "unhealthy": 0,
    "offline": 0,
    "total_all_statuses": 1,
    "updating": 0
  },
  "fleet_server": {
    "total_enrolled": 0,
    "healthy": 0,
    "unhealthy": 0,
    "offline": 0,
    "updating": 0,
    "total_all_statuses": 0,
    "num_host_urls": 1
  },

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

[juliaElastic]

@elasticmachine merge upstream

[juliaElastic]

@elasticmachine merge upstream

[nchaulet]

@juliaElastic @jlind23 I am wondering if it's safe to collect the whole fleet server input (probably the same question for host_urls values too) as telemetry data, does it could have sensitive info here?
Maybe we could sanitize the info we really found interesting here.

[nchaulet]

it will be a great improvements to write an integration tests for these collectors(I think it's doable with a jest integration test).
Maybe this can be done in a follow up issue if it's too much work.

[juliaElastic]

I'll look into adding integration tests.

I think I might create a separate pr for the agent logs collector too, since that depends on ES change.

[juliaElastic]

Added integration test.
I wonder how could we improve the integration test framework, it seems rather slow to start up an ES instance for every test. And it is slow to test changes locally too.
Raised a separate issue for improvements: #145988

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[juliaElastic]

abortController helps to terminate queries if the 1m task timeout is reached. Didn't pass it to all es queries yet as it requires changes in many files.

Currently the task takes about 300ms to run on average, will keep an eye on it after adding the agent logs queries.

[kpollich]

Thanks for being mindful about performance here. As I understand it, things like background tasks, telemetry collection, etc will eventually run on Kibana's separate "service" process, so heavy operations like this will eventually be optimized such that they aren't taking time or resources away from the main Kibana server/UI. Still, it's good to make sure we're not running a bunch of long running queries here.

[juliaElastic]

@elasticmachine merge upstream

[jlind23]

@nchaulet might include some sensitive data indeed. Hosts and port value can definitely be removed without issues. I guess for the others it is going to be a case by case decision.

[juliaElastic]

might include some sensitive data indeed. Hosts and port value can definitely be removed without issues. I guess for the others it is going to be a case by case decision.

Updated to remove host port information from fleet server config, and also removed fleet server hosts (as it didn't contain anything useful other than host port). I suppose it doesn't add much value to send config ids to telemetry.

[kpollich]

To confirm, did we mean agent policies here?

Yes this looks correct.

Do we mean here to publish the total last checkin failure count? E.g. 3 if 3 agents are in failure checkin status currently.
Or do we mean to publish specific info for all agents (last_checkin_status, last_checkin time, last_checkin_message)?

I think just total checkin failure counts is acceptable here. The granular timestamps/metrics are probably just noise for telemetry purposes.

Are the only statuses error and degraded that we want to send?

Yes let's only "alert" on non-healthy checkins.

Do we mean here elastic-agent logs only, or fleet-server logs as well (maybe separately)?

Not sure on @jlind23's original ask here. I think agent logs are most useful here, but I'm sure fleet server logs could be helpful as well. We'd need to update the ES permissions to include fleet_server-logs-* as well for this, right?

I think this is almost the same as #5. The difference would be to report new failures happened only in the last hour, or report all agents in failure state. (which would be an increasing number if the agent stays in failed state).
Do we want these 2 separate telemetry fields?

This does seem redundant, and I'd rather stick to the previously defined checkin failure field and not risk duplication.

[juliaElastic]

Not sure on @jlind23's original ask here. I think agent logs are most useful here, but I'm sure fleet server logs could be helpful as well. We'd need to update the ES permissions to include fleet_server-logs-* as well for this, right?

I found that logs-elastic_agent.fleet_server* pattern contains fleet server logs.

[joshdover]

A few notes about extracting data from the logs:

• We likely need to add privileges for kibana_system to read these data streams
• We need to be particularly performance conscious and conservative on these searches as it's quite likely to be a lot of data
• We should limit the time range to the past 24 hrs
• We should use bool: filter instead of bool: must. Filter provides better performance because it's results are cacheable by aggregations and queries.
• We may get better performance with the random_sampler agg instead of sampler since sampler at the cost of less interesting results. random_sampler is in tech preview, but I think it's ok if we use it for telemetry anyways.

[juliaElastic]

A few notes about extracting data from the logs:

• We likely need to add privileges for kibana_system to read these data streams

Yes, this is already in a pr: elastic/elasticsearch#91701

• We need to be particularly performance conscious and conservative on these searches as it's quite likely to be a lot of data

Yes, that is why I added the APM span and task timeout, to keep the performance in check.

• We should limit the time range to the past 24 hrs
• We should use bool: filter instead of bool: must. Filter provides better performance because it's results are cacheable by aggregations and queries.

I'll make these changes, BTW I planned to query the last 1h since we are publishing the telemetry hourly.

• We may get better performance with the random_sampler agg instead of sampler since sampler at the cost of less interesting results. random_sampler is in tech preview, but I think it's ok if we use it for telemetry anyways.

I think I tried this and didn't work for the message field, I'll check again.
Getting this error on random_sampler agg:

 "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Fielddata is disabled on [message] in [.ds-logs-elastic_agent-default-2022.11.21-000001]. Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [message] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
      },

[nchaulet]

I think there is still some sensitive info here we could have some path in ssl what about just whitelisting what we think is interesting here? maybe just timeouts and limits?

[juliaElastic]

Refactored with whitelisting.

[nchaulet]

LGTM! thanks for adding that integration test it really help to understand what is collected! and I agree we probably do not need to have a ES started for all the tests, maybe we can move some of these test to the FTR config

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 241a5e9

Failed CI Steps

• FTR Configs #31

Test Failures

• [job] [logs] FTR Configs #31 / machine learning - anomaly detection create jobs from lens "after each" hook for "can create a single metric job from vis with single layer"
• [job] [logs] FTR Configs #31 / machine learning - anomaly detection create jobs from lens with wizard can create multi metric job from vis with single layer

Metrics [docs]

Unknown metric groups

ESLint disabled in files

id	before	after	diff
osquery	1	2	+1

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
fleet	59	65	+6
osquery	109	115	+6
securitySolution	442	448	+6
total			+20

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
fleet	67	73	+6
osquery	110	117	+7
securitySolution	519	525	+6
total			+21

History

• 💚 Build #89976 succeeded 1fa218b
• 💚 Build #89956 succeeded 99f5a61
• 💚 Build #89576 succeeded 43788cb
• 💔 Build #89557 failed 6c6de57
• 💚 Build #89271 succeeded 4c5f29e
• 💚 Build #89185 succeeded d95598a

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @juliaElastic

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.6

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

[jlind23]

@juliaElastic @kpollich sorry for the last minute comment but i started looking at the stats and one that would be better to have is number of agent per versions.
We now have number of agent and what are the versions but we do not know how much Elastic Agent per versions are running.
@juliaElastic would that be feasible?

[juliaElastic]

@jlind23 added agents per version telemetry:
https://stack-telemetry.elastic.dev/goto/6da82a10-79f1-11ed-a03e-a77716e362c6

[jlind23]

Thanks @juliaElastic this a great piece of work 👍🏼