[Security Solution][Detections] Modify threshold rule synthetic signal generation to use data from last hit in bucket #82444
Conversation
madirey
added
release_note:fix
v8.0.0
Feature:Detection Rules
madirey
changed the title
madirey
changed the title
💚 Build SucceededMetrics [docs]
History
To update your PR or re-run it, just comment with: |
| const source = { | ||
| '@timestamp': new Date().toISOString(), | ||
| '@timestamp': get(timestampOverride ?? '@timestamp', hit._source), |
There was a problem hiding this comment.
Does this set the threshold signal date time to be the one from the hit?
I ask because previously there has been some debate previously about what a signal's @timestamp should be as some people prefer the source signal timestamp instead of the current date time that the signal is created but the decision so far has been that the @timestamp for a signal should be the current date time stamp and then if a rule has an original_timestamp it would be stored underneath signal.original_time
Threshold is unique in that is synthetic but the @timestamp should still be the timestamp of when the signal is created for consistency. That doesn't mean that we can't change them all as part of
There was a problem hiding this comment.
@FrankHassanabad I thought that too initially, but the signal timestamp is created later. This timestamp is used to populate signal.original_time (it will be the original time of the last event in the timespan...
There was a problem hiding this comment.
Oh I see, thanks. Yeah, that makes sense. I get it.
…l generation to use data from last hit in bucket (elastic#82444) * Fix threshold rule synthetic signal generation * Use top_hits aggregation * Add timestampOverride * Account for when threshold.field is not supplied * Ensure we're getting the last event when threshold.field is not provided * Add missing import
…l generation to use data from last hit in bucket (#82444) (#83213) * Fix threshold rule synthetic signal generation * Use top_hits aggregation * Add timestampOverride * Account for when threshold.field is not supplied * Ensure we're getting the last event when threshold.field is not provided * Add missing import
Summary
Fixes #77253 by using the
top_hitsaggregation to return the most recent hit for each bucket which exceeded the given threshold (or by using the returned search results for threshold rules that are not bucketed by any field). This hit is used to populate the synthetic signal(s) that is generated.This ensures that our signal contains only valid field values, as opposed to potential wildcards or CIDR IP ranges that were used for the original search.
For example,
192.168.0.0/16(the CIDR ranged used by the rule to search) becomes192.168.0.16(the host IP from the most recent event captured on the interval).Checklist
For maintainers