Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add to Transform Filter to support remove_entry_if_equal and remove_entry_if_not_equal #273

Merged
merged 2 commits into from
Aug 9, 2022
Merged

Add to Transform Filter to support remove_entry_if_equal and remove_entry_if_not_equal #273

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
166023d
support remove_entry_if_equal and remove_entry_if_not_equal
KalmanMeth Aug 1, 2022
85bf981
updated api.md
KalmanMeth Aug 2, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -361,6 +361,8 @@ pipeline:
Using `remove_entry_if_doesnt_exist` in the rule reverses the logic and will not remove the above example entry
Using `remove_field` in the rule `type` instead, results in outputting the entry after
removal of only the `SrcPort` key and value
Using `remove_entry_if_equal` will remove the entry if the specified field exists and is equal to the specified value.
Using `remove_entry_if_not_equal` will remove the entry if the specified field exists and is not equal to the specified value.

### Transform Network

Expand Down
8 changes: 8 additions & 0 deletions docs/api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,14 @@ Following is the supported API format for filter transformations:
remove_field: removes the field from the entry
remove_entry_if_exists: removes the entry if the field exists
remove_entry_if_doesnt_exist: removes the entry if the field doesnt exist
remove_entry_if_equal: removes the entry if the field value equals specified value
remove_entry_if_not_equal: removes the entry if the field value does not equal specified value
value: (enum) specified value of input field:
remove_field: removes the field from the entry
remove_entry_if_exists: removes the entry if the field exists
remove_entry_if_doesnt_exist: removes the entry if the field doesnt exist
remove_entry_if_equal: removes the entry if the field value equals specified value
remove_entry_if_not_equal: removes the entry if the field value does not equal specified value
</pre>
## Transform Network API
Following is the supported API format for network transformations:
Expand Down
7 changes: 5 additions & 2 deletions pkg/api/transform_filter.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,16 @@ type TransformFilterOperationEnum struct {
RemoveField string `yaml:"remove_field" json:"remove_field" doc:"removes the field from the entry"`
RemoveEntryIfExists string `yaml:"remove_entry_if_exists" json:"remove_entry_if_exists" doc:"removes the entry if the field exists"`
RemoveEntryIfDoesntExist string `yaml:"remove_entry_if_doesnt_exist" json:"remove_entry_if_doesnt_exist" doc:"removes the entry if the field doesnt exist"`
RemoveEntryIfEqual string `yaml:"remove_entry_if_equal" json:"remove_entry_if_equal" doc:"removes the entry if the field value equals specified value"`
RemoveEntryIfNotEqual string `yaml:"remove_entry_if_not_equal" json:"remove_entry_if_not_equal" doc:"removes the entry if the field value does not equal specified value"`
}

func TransformFilterOperationName(operation string) string {
return GetEnumName(TransformFilterOperationEnum{}, operation)
}

type TransformFilterRule struct {
Input string `yaml:"input,omitempty" json:"input,omitempty" doc:"entry input field"`
Type string `yaml:"type,omitempty" json:"type,omitempty" enum:"TransformFilterOperationEnum" doc:"one of the following:"`
Input string `yaml:"input,omitempty" json:"input,omitempty" doc:"entry input field"`
Type string `yaml:"type,omitempty" json:"type,omitempty" enum:"TransformFilterOperationEnum" doc:"one of the following:"`
Value interface{} `yaml:"value,omitempty" json:"value,omitempty" enum:"TransformFilterOperationEnum" doc:"specified value of input field:"`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that the enum:"TransformFilterOperationEnum" tag of Value is redundant. If so, then docs/api.md should be updated as well.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Value is needed to specify the equal / not_equal to match the key supplied in Input to decide whether to remove flow entries.

}
12 changes: 12 additions & 0 deletions pkg/pipeline/transform/transform_filter.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,18 @@ func (f *Filter) Transform(input []config.GenericMap) []config.GenericMap {
if _, ok := entry[rule.Input]; !ok {
addToOutput = false
}
case api.TransformFilterOperationName("RemoveEntryIfEqual"):
if val, ok := entry[rule.Input]; ok {
if val == rule.Value {
addToOutput = false
}
}
case api.TransformFilterOperationName("RemoveEntryIfNotEqual"):
if val, ok := entry[rule.Input]; ok {
if val != rule.Value {
addToOutput = false
}
}
default:
log.Panicf("unknown type %s for transform.Filter rule: %v", rule.Type, rule)
}
Expand Down
71 changes: 71 additions & 0 deletions pkg/pipeline/transform/transform_filter_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,38 @@ parameters:
- input: doesntSrcPort
type: remove_entry_if_doesnt_exist
`
const testConfigTransformFilterRemoveEntryIfEqual = `---
log-level: debug
pipeline:
- name: filter1
parameters:
- name: filter1
transform:
type: filter
filter:
rules:
- input: message
type: remove_entry_if_equal
value: "test message"
- input: value
type: remove_entry_if_equal
value: 8.0
`

const testConfigTransformFilterRemoveEntryIfNotEqual = `---
log-level: debug
pipeline:
- name: filter1
parameters:
- name: filter1
transform:
type: filter
filter:
rules:
- input: message
type: remove_entry_if_not_equal
value: "test message"
`

func getFilterExpectedOutput() config.GenericMap {
return config.GenericMap{
Expand Down Expand Up @@ -113,6 +145,45 @@ func TestNewTransformFilterRemoveEntryIfDoesntExists(t *testing.T) {
output := transformFilter.Transform([]config.GenericMap{input})
require.Equal(t, output, []config.GenericMap{})
}

func TestNewTransformFilterRemoveEntryIfEqual(t *testing.T) {
newTransform := InitNewTransformFilter(t, testConfigTransformFilterRemoveEntryIfEqual)
transformFilter := newTransform.(*Filter)
require.Len(t, transformFilter.Rules, 2)

input := test.GetIngestMockEntry(false)

output := transformFilter.Transform([]config.GenericMap{input})
require.Equal(t, 0, len(output))

input["message"] = "dummy message"
output = transformFilter.Transform([]config.GenericMap{input})
require.Equal(t, 1, len(output))
require.Contains(t, output[0], "message")
require.Equal(t, output[0]["message"], "dummy message")

input["value"] = 8.0
output = transformFilter.Transform([]config.GenericMap{input})
require.Equal(t, 0, len(output))
}

func TestNewTransformFilterRemoveEntryIfNotEqual(t *testing.T) {
newTransform := InitNewTransformFilter(t, testConfigTransformFilterRemoveEntryIfNotEqual)
transformFilter := newTransform.(*Filter)
require.Len(t, transformFilter.Rules, 1)

input := test.GetIngestMockEntry(false)

output := transformFilter.Transform([]config.GenericMap{input})
require.Equal(t, 1, len(output))
require.Contains(t, output[0], "message")
require.Equal(t, output[0]["message"], "test message")

input["message"] = "dummy message"
output = transformFilter.Transform([]config.GenericMap{input})
require.Equal(t, 0, len(output))
}

func InitNewTransformFilter(t *testing.T, configFile string) Transformer {
v, cfg := test.InitConfig(t, configFile)
require.NotNil(t, v)
Expand Down