[ML] Fixing categorization token highlighting for multi-line messages

[jgowdyelastic]

Fixes issue introduced in elastic/elasticsearch#73828

Multi-lined messages are now no longer lost after the end of the last matched token.

Issue as described by @droberts195
The problem arises when there’s a token that ends at the end of the first line of the message.
Because the first_non_blank_line char filter deletes everything after it, that token is reported as ending at the very end of the original message, even though it’s short.
Then, in the highlighting, the UI replaces the last token on the first line plus all the other lines with the single short token.
Thus making it look like the second and subsequent lines never existed

Solution is to base our end_offset on the token length, rather than the supplied end_offset from the analyze endpoint

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[droberts195]

LGTM

[droberts195]

Fixes issue introduced in elastic/elasticsearch#73828

Just to clarify, elastic/elasticsearch#73828 means people using the default categorization analyzer would have seen the problem from 7.14, but it's always been a bug that if you had a char_filter that deleted the end of the original message and had a token that was right at the end of the truncated message then this problem will have occurred. The difference is that in 7.13 and earlier the default categorization analyzer didn't contain a char_filter, so only advanced users who defined their own categorization analyzer would see this.

I am not suggesting that we backport this to 7.13, but I do think it should be release noted as a bug fix, just in case someone ever observes the effect on an older version.

[peteharverson]

LGTM

[jgowdyelastic]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 2e65999
• Storybooks Preview

Metrics [docs]

✅ unchanged

History

• 💚 Build #133338 succeeded 0048bc8

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jgowdyelastic

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.