test: Add new tests for Drain pattern matching #12655
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
benclive
force-pushed
the
add-tests-for-drain-patterns
branch
from
3d57d66
to
02faf08
Compare
cyriltovena
reviewed
Apr 19, 2024
pkg/pattern/drain/drain_test.go
Outdated
| patterns: []string{ | ||
| "<_> capacity <_>", | ||
| "<_> capacity changes <_>", | ||
| "{\"id\":\"D4Oh1ivB6cdLWa08\",\"level\":\"debug\",\"max-pool\":4,\"min-pool\":0,\"msg\":\"check capacity\",\"pending-builds\":0,\"running-builds\":0,\"server-buffer\":0,\"server-capacity\":0,\"server-count\":0,\"time\":\"2024-04-16T14:48:52Z\"}", |
There was a problem hiding this comment.
You can use string with ` to avoid the pain of escaping.
cyriltovena
reviewed
Apr 19, 2024
pkg/pattern/drain/drain_test.go
Outdated
Comment on lines
12
to
33
| // At training, if at the depth of LogClusterDepth there is a cluster with | ||
| // similarity coefficient greater that SimTh, then the log message is added | ||
| // to that cluster. Otherwise, a new cluster is created. | ||
| // | ||
| // LogClusterDepth should be equal to the number of constant tokens from | ||
| // the beginning of the message that likely determine the message contents. | ||
| // | ||
| // > In this step, Drain traverses from a 1-st layer node, which | ||
| // > is searched in step 2, to a leaf node. This step is based on | ||
| // > the assumption that tokens in the beginning positions of a log | ||
| // > message are more likely to be constants. Specifically, Drain | ||
| // > selects the next internal node by the tokens in the beginning | ||
| // > positions of the log message | ||
| LogClusterDepth: 8, | ||
| // SimTh is basically a ratio of matching/total in the cluster. | ||
| // Cluster tokens: "foo <*> bar fred" | ||
| // Log line: "foo bar baz qux" | ||
| // * * * x | ||
| // Similarity of these sequences is 0.75 (the distance) | ||
| // Both SimTh and MaxClusterDepth impact branching factor: the greater | ||
| // MaxClusterDepth and SimTh, the less the chance that there will be | ||
| // "similar" clusters, but the greater the footprint. |
There was a problem hiding this comment.
Super cool thank you for that !
There was a problem hiding this comment.
I can't take credit for this! I copied the descriptions and parameters over from
Line 30 in 491d251
But now I think it makes more sense to unify it under the DefaultConfig function inside the drain package, at least until it becomes clear that we'd need different configs for different sets of logs.
cyriltovena
reviewed
Apr 19, 2024
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it: