Skip to content

Commit

Permalink
Create-metricset docs slight update (elastic#8375)
Browse files Browse the repository at this point in the history
  • Loading branch information
sayden committed Feb 14, 2020
1 parent 4795676 commit 2f58eb9
Show file tree
Hide file tree
Showing 2 changed files with 19 additions and 27 deletions.
10 changes: 5 additions & 5 deletions docs/devguide/create-metricset.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,14 @@ implementation of existing metricsets for inspiration.

To create a new metricset:

. Run the following command inside your beat directory:
. Run the following command inside the metricbeat beat directory:
+
[source,bash]
----
make create-metricset
----
+
You'll be prompted to enter a module and metricset name. Only use characters `[a-z]`
You need Python to run this command, then, you'll be prompted to enter a module and metricset name. Remember that a module represents the service you want to retrieve metrics from (like Redis) and a metricset is a specific set of grouped metrics (like `info` on Redis). Only use characters `[a-z]`
and, if required, underscores (`_`). No other characters are allowed.
+
When you run `make create-metricset`, it creates all the basic files for your metricset, along with the required module
Expand All @@ -37,12 +37,12 @@ make
----
+
The first command, `make collect`, updates all generated files with the most recent files, data, and meta information from the metricset. The second command,
`make`, compiles your source code and provides you with a binary called `{beat}` in the beat folder. You can run the
`make`, compiles your source code and provides you with a binary called metricbeat in the same folder. You can run the
binary in debug mode with the following command:
+
[source,bash]
----
./{beat} -e -d "*"
./metricbeat -e -d "*"
----

After running the make commands, you'll find the metricset, along with its generated files, under `module/{module}/{metricset}`. This directory
Expand Down Expand Up @@ -92,7 +92,7 @@ func init() {
[float]
===== Definition

The MetricSet type defines all fields of the metricset. As a minimum it must inherit the `mb.BaseMetricSet` fields,
The MetricSet type defines all fields of the metricset. As a minimum it must be composed of the `mb.BaseMetricSet` fields,
but can be extended with additional entries. These variables can be used to persist data or configuration between
multiple fetch calls.

Expand Down
36 changes: 14 additions & 22 deletions docs/devguide/metricset-details.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -87,27 +87,21 @@ functionality of the metricset and `Fetch` method from the data mapping.
[float]
==== fields.yml

The `fields.yml` file is used for different purposes:

* Creates the Elasticsearch template
* Creates the Kibana index pattern configuration
* Creates the Exported Fields documentation for the metricset

To make sure the Elasticsearch template is correct, it's important to keep this file up-to-date
with all the changes. There is a `fields.yml` file under `module/{module}/_meta/fields.yml` that contains
the general top level structure for all metricsets. Normally you only need to modify the description in this file.

Here an example for the `fields.yml` file from the MySQL module.
You can find up to 3 different types of files named `fields.yml` in the beats repository for each metricbeat module:

* `metricbeat/fields.yml`: Contains the definitions to create the Elasticsearch template, the Kibana index pattern configuration and the exported fields documentation for metricsets. To make sure the Elasticsearch template is correct, it's important to keep this file up-to-date with all the changes. Generally, you shouldn't touch this file manually because it's generated by some commands in the build environment.
* `metricbeat/module/{module}/_meta/fields.yml`: Contains the general top level structure for all metricsets in a module.
Normally you only need to modify the description in this file. Here is an example for the `fields.yml` file from the MySQL module.
+
[source,yaml]
----
include::../../metricbeat/module/mysql/_meta/fields.yml[]
----

There is another `fields.yml` file under `module/{module}/{metricset}/_meta/fields.yml` that contains all fields retrieved
by the metricset. As field types, each field must have a core data type
+
* `metricbeat/module/{module}/{metricset}/_meta/fields.yml`: Contains all fields definitions retrieved by the metricset.
As field types, each field must have a core data type
https://www.elastic.co/guide/en/elasticsearch/reference/master/mapping-types.html#_core_datatypes[supported by elasticsearch]. Here's a very basic example that shows one group from the MySQL `status` metricset:

+
[source,yaml]
----
- name: status
Expand All @@ -117,8 +111,7 @@ https://www.elastic.co/guide/en/elasticsearch/reference/master/mapping-types.htm
fields:
- name: aborted
type: group
description: >
Aborted status fields.
description: Aborted status fields.
fields:
- name: clients
type: integer
Expand All @@ -130,8 +123,7 @@ https://www.elastic.co/guide/en/elasticsearch/reference/master/mapping-types.htm
description: >
The number of failed attempts to connect to the MySQL server.
----

As you can see, if there are nested fields, you must use the type `group`.
+

// TODO: Add link to general fields.yml developer guide

Expand All @@ -148,14 +140,14 @@ We recommend that you use all three when you create a metricset. Unit tests are
written in Go and have no dependencies. Integration tests are also written
in Go but require the service from which the module collects metrics to also be running.
System tests for Metricbeat also require the service to be running in most cases and are
written in Python based on our small Python test framework.
written in Python {python_major_version} based on our small Python test framework.
We use `virtualenv` to deal with Python dependencies.
You can simply run the command `make python-env` and then `. build/python-env/bin/activate` .

You should use a combination of the three test types to test your metricsets because
each method has advantages and disadvantages. To get started with your own tests, it's best
to look at the existing tests. You'll find the unit and integration tests
in the `_test.go` files under existing modules and metricsets.
in the `_test.go` files under existing modules and metricsets.
Integration tests usually take the form of `TestFetch` and `TestData`.
The system tests are under `tests/systems`.

Expand Down Expand Up @@ -240,7 +232,7 @@ func GetEnvPort() string { <2>
----
<1> Add any additional config options your metricset needs here.
<2> The endpoint used by the metricset needs to be configurable for manual and automated testing.
<2> The endpoint used by the metricset needs to be configurable for manual and automated testing.
Environment variables should be defined in the module under `_meta/env` and included in the `docker-compose.yml` file.

The `TestFetch` integration test will return a single event from your metricset, which you can use to test the validity of the data.
Expand Down

0 comments on commit 2f58eb9

Please sign in to comment.