[Search service] Add search IDs to requests made to Elasticsearch

[jpcarey]

Describe the feature: Elasticsearch now supports providing a search id by setting X-Opaque-Id http header. Having a search ID set could be useful in tracking down the source of problematic queries (if this could be logged at both kibana & elasticsearch)

elastic/elasticsearch#27764

Extra terms for seach-ability:

• Mapping query to saved object source via id

[danopia]

I have a reverse proxy in front of Kibana which knows user identities and is capable of attaching X-Opaque-Id to Kibana's /elasticsearch/ requests to identify where the search request came from. Unfortunately, Kibana didn't seem to carry them through to the backend, and in research I ended up on this issue.

If anyone else is in the same boat—making your own opaque IDs in front of Kibana—dropping this into kibana.yml lets it pass through 😄 (env-var didn't seem to work)

elasticsearch.requestHeadersWhitelist: [ x-opaque-id ]

[sergii-sakharov]

Worked this issue around by introducing "dummy filters" with object ids embedded in them for each Kibana object.
The reverse proxy between Kibana and Elasticsearch is logging all of the _msearch requests and thus allows for precise tracking. As a bonus we're ingesting those logs with a pipeline that parses epoch timestamps and object ids out so that its possible to have a dashboard/search that monitors

1. how long requests took
2. all object ids involved
3. who done it
4. what time range and time offset was used
5. what was the actual query sent to ES

Can probably setup a watcher on top of all of this to send out best practices to potential culprits )

The worst thing in all of this are those "dummy filters" as they have to be re-generated and re-uploaded into Kibana regularly with a script

Here's an example of the generated filter for a visualization
{ "query": { "bool": { "must_not": [ { "term": {
"_index": "visualizationId:MessageBus_ActiveMessageCount"
} } ] } } }

With a custom label: metadata (don't remove)

[lanerjo]

+1
This IMO should be a high priority. Kibana should be able to generate a unique ID per request so that it is possible to trace down problematic queries through logs much easier.

This should be implemented as part of the service as a feature, we shouldn't have to depend on a third party application or some "hack" to implement a feature such as this.

[stacey-gammon]

What information should be associated with that search id?

Some things to keep in mind:

• A query might not originate from something with a saved object id. For instance, a user does "create new visualize -> creates an expensive query".
• A query might not be reproducible with a URL. For instance, a user starts to type into the filter bar something that causes KQL to make a very expensive call to get the unique field values for auto-complete. Refreshing the page however will cause you to lose that filter because it hasn't yet been applied. This could also be the case with different permissions - one user hits more indices than another user for the same URL.

Maybe it would help if we included a lot of information, such as:

• An optional associated saved object id
• The url
• An optional description a developer could pass with a query, like "KQL: auto complete"
• Plugin id?

[mshustov]

@stacey-gammon I started working on #101587 Could you elaborate on a few moments:

An optional associated saved object id

Not all Kibana entities have SO, as you said. Maybe we should pass Kibana entity identifier? With this id we should be able to find an associated SO.

The url

Do you mean the URL to a current Kibana entity (a visualization, for example)? the url to a SO? the current page URL?

[stacey-gammon]

Not all Kibana entities have SO, as you said. Maybe we should pass Kibana entity identifier? With this id we should be able to find an associated SO.

What is the Kibana Entity Identifier? You mean the type, like "Lens"?

Do you mean the URL to a current Kibana entity (a visualization, for example)? the url to a SO? the current page URL?

I meant the current page URL. For example, if an expensive query is coming from the security network page - there is no saved object.

[mshustov]

What is the Kibana Entity Identifier? You mean the type, like "Lens"?

yes, a dashboard panel, canvas workpad, etc.

if an expensive query is coming from the security network page - there is no saved object.

Let's consider a dashboard page with several panels. It's hard to identify what panel causes a specific query. So, I'd say we are interested in panel URL that a user can open to the browser and modify the panel.

[stacey-gammon]

Let's consider a dashboard page with several panels. It's hard to identify what panel causes a specific query. So, I'd say we are interested in panel URL that a user can open to the browser and modify the panel.

Yea, that's a good example. That might be tough though because the dashboard doesn't send the query, the embeddable does. The embeddable knows it's in a dashboard, but doesn't know where that dashboard is embedded.

[lukasolson]

Closing in favor of #97934.