Import user.group changes from ECS

[webmat]

The dependency on elastic/ecs#308 is resolved (see also elastic/ecs#304 for full context). The change is merged. This ECS change lets use nest both the group name and id at user.group.id and user.group.name.

This PR imports these changes in Beats, and will unblock #9963 and #10192.

Mapping Conflict Warning

If you have any index template from Beats 7.0 pre-release in your Elasticsearch, this change introduces a mapping conflict.

Before:

{ "user": { "group": "root" } }

After this PR:

{ "user": { "group": { "id": 0, "name": "root" } } }

You will have to overwrite your index templates, delete you indices and your Kibana index templates.

[webmat]

make check fails for legit reason: this PR tweaks some ECS fields, and make check detects they haven't made it to x-pack/auditbeat.

However running make update in x-pack/auditbeat/ or auditbeat/ doesn't update anything more than when I run it at the top of the Beats repo.

@cwurm or @andrewkroh what's the proper incantation, to update field definitions for x-pack/auditbeat?

[andrewkroh]

However running make update in x-pack/auditbeat/ or auditbeat/ doesn't update anything more than when I run it at the top of the Beats repo.

There is nothing in x-pack/auditbeat to update w.r.t. common fields. The auditbeat/include/fields.go content is inherited by x-pack/auditbeat.

[andrewkroh]

Did you have DEV_OS=darwin set when you updated? Looks like some thing that should be changed have been changed.

[webmat]

My problem is that I had GOOS and GOARCH set manually, I think. I just resubmitted after generating without those, and it got rid of the config file weirdness.

[webmat]

Figured out the make update problem mentioned above. First legit test run just started, so we need to wait for that.

But otherwise, this is now ready for official review.

[webmat]

@andrewkroh Part of this (or separate PR) would surely be to import the generated Go files?

Not sure how/where to integrate those. Can you show me how? Is it applicable here, or is it not yet being used in Beats?

[andrewkroh]

We should try to always keep the fields yml and the generated code in sync. It should just be a matter of running

govendor fetch github.com/elastic/ecs/code/go/ecs from withing elastic/beats/.

Assuming that you have govendor installed.

[webmat]

@andrewkroh Alright, I'll run this here, thanks for the command.

Is this encapsulated in a make/mage command, and is govendor be part of the development dependencies?

[webmat]

@andrewkroh Ran the command. The old keyword field is no longer there, but the group field set is not being nested at user.group (it should). Is this a part I should do manually? How have the other reusable field sets been imported?

[webmat]

Jenkins failure limited to Metricbeat, and appears to be due to network flakiness (tar segfault when getting a Docker image, seems like)

[andrewkroh]

Is this a part I should do manually? How have the other reusable field sets been imported?

It's the same problem we have with the generated fields.yml not including them. The reusable fields are not incorporated into generated code so don't worry about adding them. For now if someone wants to use the code it's easy enough to compose your own struct that embeds the reusable type.

[webmat]

This is ready for final review. I would like to get this in Thursday, to unblock #9963 and #10192.

@andrewkroh Gotcha, thanks for confirming!

[webmat]

Just noticed I hadn't entered a changelog for a change that affects all beats and causes a mapping conflict. 🤦‍♂️

Added the changelog now, and will wait until tomorrow to merge :-)

[ruflin]

LGTM. You can ignore Jenkins.

[ruflin]

jenkins, test this

[webmat]

@cwurm This has been merged 😃

[cwurm]

Thanks a lot @webmat, I‘ve been eagerly awaiting it! :)

[ruflin]

Not 100% sure but it seems this PR broke CI. Not sure why this was merged with a red build?

[ruflin]

Seems like it was caused by a change in ES: #10336 We should still if possible not merge red PR's.