[O11y][Salesforce] Update _id field in Salesforce dashboard

[rajvi-patel-22]

Description

Below two Salesforce dashboards panels uses _id field:

Dashboard	Panel	Data stream
[Logs Salesforce] Setup Audit Trail Dashboard	Changes made in the setup	Setup audit trail
[Logs Salesforce] Login Dashboard	Login events table	Login rest

These visualizaions are working as expected till 8.4.0 and 8.5.0. But in Kibana 8.6.0 and later versions it shows below error.

According to documention _id fields are not supported in aggregations:

The _id field is restricted from use in aggregations, sorting, and scripting. In case sorting or aggregating on the _id field is required, it is advised to duplicate the content of the _id field into another field that has doc_values enabled.

Related Issues

• [Lens] meta fields are not supported in visualizations kibana#155344

[rajvi-patel-22]

There are two possible solutions to resolve this issue:

Solution 1:

As mentioned in the documentation of _id, the content of _id field can be copied in another field using ingest pipeline and the new field can be used in visualizations.

The _id field is restricted from use in aggregations, sorting, and scripting. In case sorting or aggregating on the _id field is required, it is advised to duplicate the content of the _id field into another field that has doc_values enabled.

Solution 2 [Recommended]:

The current visualizations have below limitations:

• It can only show maximum 10000 events [Reference]
• Search visualization has better performance than lens table visualization. [Reference]

The panels are showing list of events. Hence, the saved search can be used to display all the events. The only drawback in creating search visualization is column name. In search panels, column names are not configurable.

Please find below screenshots of originial visualization and suggested panel:

Original visualization:

Suggested search panel:

[rajvi-patel-22]

@agithomas, The possible solutions of meta data field (_id) issue are mentioned in the above comment. Please let us know how we should proceed ahead.

[rajvi-patel-22]

Two panels that use _id meta field belongs to two different data streams:

Dashboard	Panel	Data stream
[Logs Salesforce] Setup Audit Trail Dashboard	Changes made in the setup	Setup audit trail
[Logs Salesforce] Login Dashboard	Login events table	Login rest

The _id generation happens after the ingest pipeline is applied. Login Rest data stream has the fingerprint processor which assigns value to the _id field in pipeline itself [Reference]. So we can copy the value of _id field to custom field in ingest pipeline.

- fingerprint:
    fields:
    - json.REQUEST_ID
    - json.SESSION_KEY
    - json.TIMESTAMP_DERIVED
    - json.USER_ID
    target_field: _id
    ignore_missing: true

But in Setup audit trail data stream, the fingerprint processor is not used. So _id field will be generated automatically once the document has been parsed from the ingest pipeline and because of that we can not copy the value from _id field to the custom field in ingest pipeline (because _id field will be empty at that moment ). However, there is a event.id field in Setup audit trail data stream which represents the unique id extracted from the incoming events [Reference]. The event.id field can be used in the panel instead of _id as per the analysis.

@agithomas, @SubhrataK Can you please provide your suggestions? Whether we should use event.id or not ?

[agithomas]

Setup audit trail data stream which represents the unique id extracted from the incoming events [Reference]. The event.id field can be used in the panel instead of _id as per the analysis.

It is not uncommon to have _id and event.id depending on same json field value . Reference

If json.Id (stored as event.id) is unique enough, why _id was computed using fingerprint processor originally using other fields?

[rajvi-patel-22]

If json.Id (stored as event.id) is unique enough, why _id was computed using fingerprint processor originally using other fields?

@agithomas, In Login Rest data stream, It was required to create fingerprint hence _id was computed using 4 fields ( json.REQUEST_ID, json.SESSION_KEY, json.TIMESTAMP_DERIVED, json.USER_ID). But in Setup audit trail data stream, we didn't require fingerprint that creates _id field. So, fingerprint processor was not used over there. Hence _id is not accessible from the pipeline.

However, we can add fingerprint processor on json.Id field and store the value in _id field which can be copied into salesforce.login.document_id field. In this case we will need to add two more processors but in our proposed solution we won't need to add any extra processors.

[agithomas]

However, we can add fingerprint processor on json.Id field and store the value in _id field which can be copied into salesforce.login.document_id field. In this case we will need to add two more processors but in our proposed solution we won't need to add any extra processors.

Thanks for sharing the details.

I do not find any concerns here going with the above mentioned approach.

Please create the PR and share the link here . Please include @lalit-satapathy as additional reviewer.

[rajvi-patel-22]

PR is approved and merged, hence closing this issue: #6041