Force ECS options in filebeat pipelines when filebeat version is over 7.0 and ES version is 6.7.X

[jsoriano]

When using the user_agent processor to ingest data from Filebeat 7.0
into Elasticsearch 6.X conflicts appear with ECS user_agent fields, this can
be solved by setting ecs: true when pipelines are being loaded into
Elasticsearch 6.7.0.

For minor versions where ecs option is not available, pipelines will fail
to load.

Fix #10655

[andrewkroh]

Thanks for tackling this problem. And happy one year anniversary!

[jsoriano]

@andrewkroh suggestions added. Thanks for the review!

[ruflin]

@jsoriano If someone upgrades Elasticsearch from 6.7 to 7.0 and keeps Filebeat 6.7, this means the user has to overwrite the ingest pipelines?

[ruflin]

@jsoriano One more question: I can't see a backport label on this one. Should this also go into 7.0?

[jsoriano]

@jsoriano If someone upgrades Elasticsearch from 6.7 to 7.0 and keeps Filebeat 6.7, this means the user has to overwrite the ingest pipelines?

@ruflin I think this is only a problem if Filebeat 7.0 is used.

@jsoriano One more question: I can't see a backport label on this one. Should this also go into 7.0?

It was backported on #10930

[ruflin]

FB version: Right, an we recommend to upgrade ES first so we should be good. Probably we should still leave a not about it somewhere.

Backport: 👍

[andrewkroh]

@jsoriano I believe this PR is also necessary in the 6.7 branch to allow all modules that use user_agent to work with Elasticsearch <=6.6. Additionally, we need to remove the ecs: false from the modules and let this PR dynamically add the ecs tag as necessary based on the version.

@tsg said he saw, "processor [user_agent] doesn't support one or more provided configuration parameters [ecs]" when using Filebeat 6.7 with ES 6.6.