New processor: truncate_fields #11297

kvch · 2019-03-18T20:59:47Z

This PR introduces a new processor truncate_fields. To keep raw messages use this processor with copy_fields.

`truncate_fields`

This processor truncates configured fields. Example configuration is below:

processors:
- truncate_fields:
    fields:
      - message
    max_bytes: 1024
    fail_on_error: false
    ignore_missing: true

Keep raw events

This preserves the orignal field and truncates it, if it's too long.

processors:
- copy_fields:
    fields:
        - from: message
          to: event.original
    fail_on_error: false
    ignore_missing: true
- truncate_fields:
    fields:
      - event.original
    max_bytes: 1024
    fail_on_error: false
    ignore_missing: true

Depends on #11303

libbeat/processors/actions/copy_fields.go

libbeat/processors/actions/truncate_fields.go

ruflin · 2019-03-19T08:46:44Z

@kvch Any chance to split this up into at least 2 PR's? One per processor?

kvch · 2019-03-19T10:04:38Z

@ruflin sure

michalpristas

besides what andrew mentioned LGTM

kvch · 2019-03-19T10:40:49Z

@ruflin done. other PR: #11297

ruflin

Thanks for opening a separate PR. Is the plan to rebase this one as soon as the other one is merged. If yes, please ignore my reviews.

CHANGELOG.next.asciidoc

auditbeat/auditbeat.reference.yml

kvch · 2019-03-25T14:28:58Z

This should be ready for another round of review.

libbeat/processors/actions/truncate_fields.go

andrewkroh · 2019-03-25T19:22:57Z

libbeat/processors/actions/truncate_fields.go

+}
+
+func (f *truncateFields) addTruncatedString(field, value string, event *beat.Event) (*beat.Event, error) {
+	truncated, err := f.truncate(f, []byte(value))


Converting from string -> []byte -> string going to do two separate copies of the string data, but since it's only truncating I think this could be avoided with some refactoring. It would be good if someone could double-check my assumption about the two copies.

It is indeed copying twice. But after thinking a bit more about the conversion and copying data I am not sure if it makes sense to eliminate them. First I need to copy and convert the string, as it is inmutable type, in order to modify it. Then I need to create a new truncated copy, so the previous long array can be freed. Then as I need to return a string, a new copy is needed to get the inmutable type again.
Without converting the string to []byte, I can only do slicing which does not touch the underlying data, thus the long line stays in memory as long there is a reference to it. I think it would take away the advantage of decreasing the size of the data in memory.
WDYT?

I also think that we would mostly truncate []byte fields (e.g. message) and rarely on string. Thus, this code path would be less frequently run.

Also, I can reduce the number of copies by not passing around the arrays. Will do that.

libbeat/processors/actions/truncate_fields.go

ph · 2019-03-29T19:50:31Z

libbeat/processors/actions/checks.go

+		return nil
+	}
+
+}


unneeded space above?
maybe adding a unit test?

ph · 2019-03-29T19:54:08Z

libbeat/processors/actions/truncate_fields.go

+	_, err = event.PutValue(field, truncated)
+	if err != nil {
+		return event, fmt.Errorf("could not add truncated byte slice value for key: %s, Error: %+v", field, err)
+	}


We probably want to add the log.flags flag?

ph · 2019-03-29T19:58:42Z

libbeat/processors/actions/truncate_fields_test.go

+		"truncate characters of too long byte line": {
+			MaxChars: 10,
+			Input: common.MapStr{
+				"message": []byte("ez egy túl hosszú sor"), // this is a too long line (hungarian)


101 hungarian :)

This PR introduces a new processor `truncate_fields`. To keep raw messages use this processor with `copy_fields`. ### `truncate_fields` This processor truncates configured fields. Example configuration is below: ```yaml processors: - truncate_fields: fields: - message max_bytes: 1024 fail_on_error: false ignore_missing: true ``` ### Keep raw events This preserves the orignal field and truncates it, if it's too long. ```yaml processors: - copy_fields: fields: - from: message to: event.original fail_on_error: false ignore_missing: true - truncate_fields: fields: - event.original max_bytes: 1024 fail_on_error: false ignore_missing: true ```

kvch added review libbeat needs_docs labels Mar 18, 2019

kvch requested review from a team as code owners March 18, 2019 20:59

kvch requested review from ph and ruflin and removed request for a team March 18, 2019 21:09

kvch force-pushed the feature-add-copy-truncate-processors branch from 2642cbe to b0a8760 Compare March 18, 2019 21:41

andrewkroh reviewed Mar 19, 2019

View reviewed changes

libbeat/processors/actions/copy_fields.go Outdated Show resolved Hide resolved

libbeat/processors/actions/truncate_fields.go Outdated Show resolved Hide resolved

kvch force-pushed the feature-add-copy-truncate-processors branch from b0a8760 to 835e02a Compare March 19, 2019 10:15

kvch mentioned this pull request Mar 19, 2019

New processor: copy_fields #11303

Merged

michalpristas approved these changes Mar 19, 2019

View reviewed changes

kvch force-pushed the feature-add-copy-truncate-processors branch from 835e02a to 69b3201 Compare March 19, 2019 10:38

kvch changed the title ~~New processors: copy_fields, truncate_fields~~ New processor: truncate_fields Mar 19, 2019

kvch force-pushed the feature-add-copy-truncate-processors branch 3 times, most recently from 8edd7af to 375a21d Compare March 19, 2019 12:50

ruflin reviewed Mar 20, 2019

View reviewed changes

CHANGELOG.next.asciidoc Show resolved Hide resolved

auditbeat/auditbeat.reference.yml Outdated Show resolved Hide resolved

kvch force-pushed the feature-add-copy-truncate-processors branch from 375a21d to 6624d24 Compare March 22, 2019 14:06

kvch added the in progress Pull request is currently in progress. label Mar 22, 2019

kvch force-pushed the feature-add-copy-truncate-processors branch from 07078ee to 75bd386 Compare March 25, 2019 14:27

kvch removed the in progress Pull request is currently in progress. label Mar 25, 2019

andrewkroh reviewed Mar 25, 2019

View reviewed changes

ph suggested changes Mar 29, 2019

View reviewed changes

kvch added 6 commits April 1, 2019 12:33

new processor: truncate_fields

c2f0071

refactor unit test running

ea27393

add E2E test

cad13a3

rename option to max_characters

4179120

add more info to String

40d5258

refactoring to eliminate unnecessary copy && add flag

991a4da

kvch force-pushed the feature-add-copy-truncate-processors branch from 36acbb0 to 991a4da Compare April 1, 2019 10:51

kvch added 3 commits April 1, 2019 15:31

rm unnecessary newlines

7af7abb

add unit test to check all processor configuration checks

ad203ff

fix comment

626472c

kvch mentioned this pull request Apr 2, 2019

Keep original messages in case of Filebeat modules #8448

Closed

ph approved these changes Apr 2, 2019

View reviewed changes

kvch merged commit 555a89a into elastic:master Apr 2, 2019

kvch mentioned this pull request Apr 5, 2019

Filebeat modules: keep raw message #8083

Closed

ph mentioned this pull request Apr 8, 2019

Make it possible to chain Beat processors in the script processor #11680

Merged

kvch mentioned this pull request Apr 9, 2019

Add configuration option to keep raw JSON messages #9172

Closed

andrewkroh mentioned this pull request Jun 21, 2019

Missing docs for copy_fields and truncate_fields processors #12633

Closed

dedemorton removed the needs_docs label Nov 15, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New processor: truncate_fields #11297

New processor: truncate_fields #11297

kvch commented Mar 18, 2019 •

edited

Loading

ruflin commented Mar 19, 2019

kvch commented Mar 19, 2019

michalpristas left a comment

kvch commented Mar 19, 2019

ruflin left a comment

kvch commented Mar 25, 2019

andrewkroh Mar 25, 2019

kvch Mar 29, 2019

kvch Mar 29, 2019

kvch Mar 29, 2019

ph Mar 29, 2019

ph Mar 29, 2019

ph Mar 29, 2019

New processor: truncate_fields #11297

New processor: truncate_fields #11297

Conversation

kvch commented Mar 18, 2019 • edited Loading

truncate_fields

Keep raw events

ruflin commented Mar 19, 2019

kvch commented Mar 19, 2019

michalpristas left a comment

Choose a reason for hiding this comment

kvch commented Mar 19, 2019

ruflin left a comment

Choose a reason for hiding this comment

kvch commented Mar 25, 2019

andrewkroh Mar 25, 2019

Choose a reason for hiding this comment

kvch Mar 29, 2019

Choose a reason for hiding this comment

kvch Mar 29, 2019

Choose a reason for hiding this comment

kvch Mar 29, 2019

Choose a reason for hiding this comment

ph Mar 29, 2019

Choose a reason for hiding this comment

ph Mar 29, 2019

Choose a reason for hiding this comment

ph Mar 29, 2019

Choose a reason for hiding this comment

kvch commented Mar 18, 2019 •

edited

Loading

`truncate_fields`