Add JSON options to autodiscover hints

[ChrsMark]

Closes #12634.

cc: @odacremolbap, @exekias

Manual testing

Configure filebeat autodiscover

filebeat:
  autodiscover.providers:
    - type: docker
      hints.enabled: true

Having filebeat running, launch containers with the proper labels each time.
Make sure that Filebeat has permissions to read from /var/lib/docker/containers/, if not a quick option could be to run with sudo like sudo -E ./filebeat -e -d "*" --strict.perms=false. Note that for testing this a Linux host is needed so as Filebeat to be able to locate and read log files properly.

Test that keys are not copied under top level in the output document

docker run -l co.elastic.logs/json.keys_under_root=false -it busybox echo '{"foo":"bar", "error_msg": "error opening file"}'

You should see in the event the following:

"json": {
      "foo": "bar",
      "error_msg": "error opening file"
    },

Test keys are copied under top level in the output document

docker run -l co.elastic.logs/json.keys_under_root=true -it busybox echo '{"foo":"bar", "error_msg": "error opening file"}'

You should see the foo and error_msg in top level.

Test that error is reported

docker run -l co.elastic.logs/json.keys_under_root=true -l co.elastic.logs/json.add_error_key=true -it busybox echo '{"foo":"bar", "error_msg": "error opening file" and some buggy stuff}'

Event should be enriched with:

"error": {
      "type": "json",
      "message": "Error decoding JSON: invalid character 's' after object key:value pair"
    },

[jsoriano]

Great addition!

[exekias]

Looking good!

[exekias]

sorry didn't notice failing tests, this will require a make update, and probably a rebase

[exekias]

Also, outside of the scope of this PR (maybe a followup). What would you think about simplifying these parameters for the user? I'm thinking something like this would make things better:

co.elastic.logs/json.enabled=true

This somehow would be an alias for:

co.elastic.logs/json.message_key=message

which is probably what most users want

[ChrsMark]

Also, outside of the scope of this PR (maybe a followup). What would you think about simplifying these parameters for the user? I'm thinking something like this would make things better:

co.elastic.logs/json.enabled=true

This somehow would be an alias for:

co.elastic.logs/json.message_key=message

which is probably what most users want

🤔 could you elaborate more?

[exekias]

Sure! sorry for my short explanation 🤕

I think that current config keys are not very straight forward. Having that in most cases what users want is probably to just decode the message field as a JSON, I think we should add a more intuitive setting name, that will just do that. This could be co.elastic.logs/json.enabled=true, which should enable JSON decoding from the message field.

This would be the same behavior as using co.elastic.logs/json.message_key=message today

[exekias]

@ChrsMark had a chat and we decided it probably makes more sense to have the enabled keyword in the original JSON settings if possible, I will create an issue for that

[ChrsMark]

jenkins, test this again please

[ChrsMark]

Merging this since Travis went green and failing tests of Jenkins look irrelevant to this patch.

[ycombinator]

Assigning myself for manually testing this PR for 7.6.0 release per test-plan label.

[ycombinator]

Tested manually. Works as expected!

[ycombinator]

Removed test-plan-ok label for now. Will re-test with BC (previously I tested by building from the 7.x branch).

[ycombinator]

Tested with BC1 and this works as expected!