Add host inventory metrics to ec2 metricset #20171

kaiyan-sheng · 2020-07-22T23:32:37Z

What does this PR do?

This PR is to add proposed host common fields into ec2 metricset:

host.id
host.name
host.cpu.pct
host.network.in.bytes
host.network.in.packets
host.network.out.bytes
host.network.out.packets
host.disk.read.bytes
host.disk.write.bytes

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Start Metricbeat AWS module: ./metricbeat modules enable aws
Edit modules.d/aws.yml to only include ec2 metricset:

- module: aws
  period: 5m
  credential_profile_name: elastic-beats
  metricsets:
    - ec2

Change add_host_metadata processor config in metricbeat.yml file:

processors:
  - add_host_metadata:
      replace_fields: false

Start metricbeat
You should see metrics from ec2 metricset and includes fields listed above.
(host.name only equals to EC2 instance name when EC2 instance name is available.)

Event Example

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "aws": {
        "ec2": {
            "cpu": {
                "credit_balance": 144,
                "credit_usage": 0.05477,
                "surplus_credit_balance": 0,
                "surplus_credits_charged": 0,
                "total": {
                    "pct": 1.166484417714406
                }
            },
            "diskio": {
                "read": {
                    "bytes": 0,
                    "bytes_per_sec": 0,
                    "count": 0,
                    "count_per_sec": 0
                },
                "write": {
                    "bytes": 0,
                    "bytes_per_sec": 0,
                    "count": 0,
                    "count_per_sec": 0
                }
            },
            "instance": {
                "core": {
                    "count": 1
                },
                "image": {
                    "id": "ami-04bc3da8f14823e88"
                },
                "monitoring": {
                    "state": "disabled"
                },
                "private": {
                    "dns_name": "ip-172-31-9-119.us-west-1.compute.internal",
                    "ip": "172.31.9.119"
                },
                "public": {
                    "dns_name": "ec2-13-52-163-56.us-west-1.compute.amazonaws.com",
                    "ip": "13.52.163.56"
                },
                "state": {
                    "code": 16,
                    "name": "running"
                },
                "threads_per_core": 1
            },
            "network": {
                "in": {
                    "bytes": 6644.4,
                    "bytes_per_sec": 22.148,
                    "packets": 44.6,
                    "packets_per_sec": 0.14866666666666667
                },
                "out": {
                    "bytes": 6016.6,
                    "bytes_per_sec": 20.055333333333333,
                    "packets": 35.4,
                    "packets_per_sec": 0.118
                }
            },
            "status": {
                "check_failed": 0,
                "check_failed_instance": 0,
                "check_failed_system": 0
            }
        },
        "tags": {
            "Name": "mysql-test",
            "created-by": "ks"
        }
    },
    "cloud": {
        "account": {
            "id": "428152502467",
            "name": "elastic-beats"
        },
        "availability_zone": "us-west-1b",
        "instance": {
            "id": "i-0516ddaca5c1d231f",
            "name": "mysql-test"
        },
        "machine": {
            "type": "t2.micro"
        },
        "provider": "aws",
        "region": "us-west-1"
    },
    "event": {
        "dataset": "aws.ec2",
        "duration": 115000,
        "module": "aws"
    },
    "host": {
        "cpu": {
            "pct": 1.166484417714406
        },
        "disk": {
            "read": {
                "bytes": 0
            }
        },
        "diskio": {
            "write": {
                "bytes": 0
            }
        },
        "id": "i-0516ddaca5c1d231f",
        "name": "mysql-test",
        "network": {
            "in": {
                "bytes": 6644.4,
                "packets": 44.6
            },
            "out": {
                "bytes": 6016.6,
                "packets": 35.4
            }
        }
    },
    "metricset": {
        "name": "ec2",
        "period": 10000
    },
    "service": {
        "type": "aws"
    }
}

elasticmachine · 2020-07-22T23:39:24Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request #20171 updated]
Start Time: 2020-08-18T16:35:20.865+0000
Duration: 69 min 42 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3541
Skipped	794
Total	4335

elasticmachine · 2020-07-23T22:09:48Z

Pinging @elastic/integrations-platforms (Team:Platforms)

jsoriano

This is looking good.

x-pack/metricbeat/module/aws/ec2/ec2.go

jsoriano · 2020-07-31T11:07:51Z

x-pack/metricbeat/module/aws/ec2/_meta/data.json

+                "bytes": 6644.4,
+                "packets": 44.6


Is this a counter? Why is it a decimal number?

It's a gauge actually and it's an average value among 5 minutes:

CHANGELOG.next.asciidoc

jsoriano

There are some issues with documented fields, for the rest it is looking good!

metricbeat/docs/fields.asciidoc

x-pack/metricbeat/module/aws/_meta/fields.yml

Issues fixed

jsoriano

LGTM if CI gets green.

sorantis · 2020-08-20T09:51:51Z

@kaiyan-sheng, @jsoriano, in the following configuration

processors:
  - add_host_metadata:
      replace_fields: false

what does the replace_fields flag do?

jsoriano · 2020-08-20T11:43:48Z

@kaiyan-sheng, @jsoriano, in the following configuration
processors:
  - add_host_metadata:
      replace_fields: false
what does the replace_fields flag do?

It can be used to control if add_host_metadata should replace existing host fields. It defaults to true, so this processor overwrites existing host fields, and can be set to false to avoid this overwrite, that is what we are going to want with inventory fields added by modules. We added this option to support inventory fields, while avoiding the introduction of a breaking change by now.

Full discussion and change to add this flag can be found here: #20490

sorantis · 2020-08-20T14:11:21Z

ok, my expectation was that this field would either map or remove the respective ec2.* fields in favor of host.* fields.
Because the resulting document contains duplicate metrics (e.g.: ec2.cpu.total.pct = host.cpu.pct), which increases the doc size by 10-11%

* Add host inventory metrics to ec2 metricset * move host fields into metricbeat/_meta/fields.common.yml (cherry picked from commit 54d2268)

kaiyan-sheng · 2020-08-20T15:46:40Z

@sorantis Currently we are duplicating the fields so yes we have both ec2.* fields and host.* fields. But I'm planning to create a PR to remove the duplicated ec2.* fields and use alias to keep backward compatibility. For example:

- name: aws.ec2
  type: group
  fields:
    - name: cpu.total.pct
      type: alias
      path: host.cpu.pct
      migration: true
    - name: network.in.bytes
      type: alias
      path: host.network.in.bytes
      migration: true

sorantis · 2020-08-20T16:10:48Z

Great! Thanks @kaiyan-sheng .

* Add host inventory metrics to ec2 metricset * move host fields into metricbeat/_meta/fields.common.yml

Add host inventory metrics to ec2 metricset

9278d0f

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 22, 2020

add changelog

5ed97cf

kaiyan-sheng self-assigned this Jul 22, 2020

kaiyan-sheng added needs_backport PR is waiting to be backported to other branches. Team:Platforms Label for the Integrations - Platforms team labels Jul 22, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 22, 2020

kaiyan-sheng mentioned this pull request Jul 22, 2020

Common Fields for Host Inventory Schema #19757

Closed

fix unit test

1f300df

kaiyan-sheng marked this pull request as ready for review July 23, 2020 22:09

kaiyan-sheng added 3 commits July 28, 2020 11:15

use instance ID for host.name when no name is given in ec2 tag

a92375f

update data.json file

45aaa2d

fix unit test

4817708

kaiyan-sheng added the review label Jul 30, 2020

jsoriano previously requested changes Jul 31, 2020

View reviewed changes

Change typo in host.disk.write.bytes and add host fields to fields.yml

194084e

kaiyan-sheng mentioned this pull request Aug 3, 2020

Investigate instance id collected from add_cloud_metadata processor #19758

Closed

3 tasks

kaiyan-sheng added 3 commits August 4, 2020 12:02

Merge remote-tracking branch 'upstream/master' into add_host_data

b04f90c

Change host.cpu.pct from percent to from 0 to 1

f949375

fix unit test for ec2

a04d1e2

kaiyan-sheng mentioned this pull request Aug 10, 2020

Add replace_fields config option in add_host_metadata for replacing host fields #20490

Merged

6 tasks

Merge remote-tracking branch 'upstream/master' into add_host_data

408134d

kaiyan-sheng mentioned this pull request Aug 14, 2020

Cherry-pick #20490 to 7.x: Add replace_fields config option in add_host_metadata for replacing host fields #20617

Merged

6 tasks

jsoriano reviewed Aug 18, 2020

View reviewed changes

metricbeat/docs/fields.asciidoc Outdated Show resolved Hide resolved

x-pack/metricbeat/module/aws/_meta/fields.yml Outdated Show resolved Hide resolved

kaiyan-sheng added 2 commits August 18, 2020 08:41

Merge remote-tracking branch 'upstream/master' into add_host_data

7a6292b

Move new host fields into libbeat/_meta/fields.common.yml

eb04b02

move host fields into metricbeat/_meta/fields.common.yml

d39ce30

jsoriano approved these changes Aug 18, 2020

View reviewed changes

kaiyan-sheng merged commit 54d2268 into elastic:master Aug 18, 2020

kaiyan-sheng deleted the add_host_data branch August 18, 2020 18:20

kaiyan-sheng mentioned this pull request Aug 18, 2020

Cherry-pick #20171 to 7.x: Add host inventory metrics to ec2 metricset #20673

Merged

6 tasks

kaiyan-sheng added v7.10.0 and removed needs_backport PR is waiting to be backported to other branches. labels Aug 18, 2020

kaiyan-sheng added a commit that referenced this pull request Aug 20, 2020

Add host inventory metrics to ec2 metricset (#20171) (#20673)

0996442

* Add host inventory metrics to ec2 metricset * move host fields into metricbeat/_meta/fields.common.yml (cherry picked from commit 54d2268)

melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020

Add host inventory metrics to ec2 metricset (elastic#20171)

225a8af

* Add host inventory metrics to ec2 metricset * move host fields into metricbeat/_meta/fields.common.yml

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add host inventory metrics to ec2 metricset #20171

Add host inventory metrics to ec2 metricset #20171

kaiyan-sheng commented Jul 22, 2020 •

edited

Loading

elasticmachine commented Jul 22, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

elasticmachine commented Jul 23, 2020

jsoriano left a comment

jsoriano Jul 31, 2020

kaiyan-sheng Jul 31, 2020

jsoriano left a comment

jsoriano left a comment •

edited

Loading

sorantis commented Aug 20, 2020

jsoriano commented Aug 20, 2020

sorantis commented Aug 20, 2020 •

edited

Loading

kaiyan-sheng commented Aug 20, 2020

sorantis commented Aug 20, 2020

Add host inventory metrics to ec2 metricset #20171

Add host inventory metrics to ec2 metricset #20171

Conversation

kaiyan-sheng commented Jul 22, 2020 • edited Loading

What does this PR do?

Checklist

How to test this PR locally

Event Example

elasticmachine commented Jul 22, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

elasticmachine commented Jul 23, 2020

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano Jul 31, 2020

Choose a reason for hiding this comment

kaiyan-sheng Jul 31, 2020

Choose a reason for hiding this comment

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano left a comment • edited Loading

Choose a reason for hiding this comment

sorantis commented Aug 20, 2020

jsoriano commented Aug 20, 2020

sorantis commented Aug 20, 2020 • edited Loading

kaiyan-sheng commented Aug 20, 2020

sorantis commented Aug 20, 2020

kaiyan-sheng commented Jul 22, 2020 •

edited

Loading

elasticmachine commented Jul 22, 2020 •

edited by jenkins-beats-ci bot

Loading

jsoriano left a comment •

edited

Loading

sorantis commented Aug 20, 2020 •

edited

Loading