Add line number counter for multiline #2279
Conversation
ruflin
added
Filebeat
filebeat/harvester/reader/json.go
Outdated
| } | ||
| reader.Content, reader.Fields = r.decodeJSON(reader.Content) | ||
| return reader, nil | ||
| var fields = common.MapStr{} |
There was a problem hiding this comment.
I think this should be var fields common.MapStr since you don't need to allocate a MapStr because decodeJSON will be returning one.
|
#2290 should be reviewed and merged first as it will require additional changes in this PR. |
filebeat/harvester/log.go
Outdated
| @@ -99,6 +99,13 @@ func (h *Harvester) Harvest() { | |||
| event.Text = &text | |||
| event.JSONFields = message.Fields | |||
| event.EventMetadata = h.config.EventMetadata | |||
| if message.Fields != nil { | |||
| if event.EventMetadata.Fields != nil { | |||
| event.EventMetadata.Fields.Update(message.Fields) | |||
There was a problem hiding this comment.
is this overwriting EventMetadata.Fields or does it create a new common.MapStr internally? EventMetadata directly uses config.EventMetadata.
There was a problem hiding this comment.
This will get more obvious when is merged: #2290 Meta data addition and merging should not be a task of the harvester.
ruflin
force-pushed
the
multiline-lines
branch
3 times, most recently
from
6469e3f
to
509dd52
Compare
ruflin
force-pushed
the
multiline-lines
branch
3 times, most recently
from
76a3a7b
to
517ff24
Compare
ruflin
force-pushed
the
multiline-lines
branch
from
517ff24
to
bbd195e
Compare
| @@ -225,6 +227,8 @@ func (mlr *Multiline) clear() { | |||
| // finalize writes the existing content into the returned message and resets all reader variables. | |||
| func (mlr *Multiline) finalize() Message { | |||
|
|
|||
| lines := common.MapStr{"lines": mlr.numLines} | |||
| mlr.message.AddFields(common.MapStr{"multiline": lines}) | |||
There was a problem hiding this comment.
What will happen if someone defines json or multiline as fields in the config file, together with fields_under_root?
There was a problem hiding this comment.
Good point. Our current behaviour here is that is that fields will overwrite it. For json the behaviour does not change here. So in case someone has
fields:
json: "Hello world"
already now, this would overwrite the fields from the json reader. The same applies multiline. So I think the behaviour didn't change even though I was never really thinking of the json overwriting by fields before. Seems like it wasn't an issue so far. The good thing is that we have it even mentioned in our docs: https://www.elastic.co/guide/en/beats/filebeat/1.3/configuration-filebeat-options.html#fields-under-root
ruflin
force-pushed
the
multiline-lines
branch
2 times, most recently
from
da69d9e
to
5fc39ec
Compare
ruflin
added
in progress
Each reader can add fields to the message object. The reader itself should always add data under its own namespace to prevent conflicts. All these fields are then added to the Data object. This will allow each reader in the future to add its own data if needed. The JSON reader was simplified in the way that data by default is written under the `json` namespace. Now no special fields have to be passed for JSON and the processing can still happen on the event level. This PR was extracted from elastic#2279
* Refactor fields handling with readers Each reader can add fields to the message object. The reader itself should always add data under its own namespace to prevent conflicts. All these fields are then added to the Data object. This will allow each reader in the future to add its own data if needed. The JSON reader was simplified in the way that data by default is written under the `json` namespace. Now no special fields have to be passed for JSON and the processing can still happen on the event level. This PR was extracted from #2279 * Add comment and switch field
* Refactor fields handling with readers Each reader can add fields to the message object. The reader itself should always add data under its own namespace to prevent conflicts. All these fields are then added to the Data object. This will allow each reader in the future to add its own data if needed. The JSON reader was simplified in the way that data by default is written under the `json` namespace. Now no special fields have to be passed for JSON and the processing can still happen on the event level. This PR was extracted from elastic#2279 * Add comment and switch field
ruflin
force-pushed
the
multiline-lines
branch
from
5fc39ec
to
4e16c13
Compare
ruflin
force-pushed
the
multiline-lines
branch
2 times, most recently
from
c1a10fe
to
34b1234
Compare
ruflin
added
review
and removed
in progress
ruflin
force-pushed
the
multiline-lines
branch
2 times, most recently
from
ba8d0d3
to
54b882b
Compare
|
@ruflin Can you please rebase this PR? |
ruflin
force-pushed
the
multiline-lines
branch
2 times, most recently
from
5d18acc
to
1865dcf
Compare
This allows to indicate if an event was multiline or not. The number of lines will be put under the multiline namespace and looks as following:
```
{
...
"message": "[2015] hello world\n First Line\n Second Line",
"multiline": {
"lines": 3
},
...
}
```
See elastic#957
|
Closing this PR as no request for this feature has shown up recently. |
This allows to indicate if an event was multiline or not. The number of lines will be put under the multiline namespace and looks as following:
See elastic#957