Add a HAProxy filebeat module. #8014

Tethik · 2018-08-18T14:50:30Z

I created a new module for parsing HAProxy http logs via filebeat. Originally developed on version 6.3 of elasticsearch/filebeat.

Feedback would be appreciated, please let me know if there's anything missing.

elasticmachine · 2018-08-18T14:50:32Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

jsoriano

Really nice work here, thanks for this! I have added some comments.

jsoriano · 2018-08-20T08:05:35Z

filebeat/module/haproxy/_meta/config.yml

@@ -0,0 +1,8 @@
+- module: haproxy
+  # All logs
+  {fileset}:


This should be replaced by the name of the fileset, I guess http in this case.

jsoriano · 2018-08-20T08:08:46Z

filebeat/docs/fields.asciidoc

+
+--
+
+*`haproxy.http.status_code`*::


Could you follow ECS in fields naming if possible? for example http response fields could be at the root level under http.response....

jsoriano · 2018-08-20T08:12:00Z

filebeat/module/haproxy/http/_meta/fields.yml

+
+    type: long
+  - name: srvconn
+    description: srv_conn is the total number of concurrent connections still active on the server when the session was logged.


These field names are a bit cryptic, maybe we could have a connections object with different counts, so these fields are named connections.active, connections.frontend, connections.backend, connections.server, wdyt?

I copied the names of the fields directly from the HAProxy documentation
https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#8.2.3

I don't mind changing them though.

Would you move them up from the fileset though? e.g. haproxy.connections.* as opposed to haproxy.http.connections.*

I think we could leave them directly on haproxy.connections, so if at some moment we have also a TCP fileset they can be aggregated more easily.

Sounds good. What about the rest of the fields that are shared between the TCP and HTTP logs? e.g. client_ip, frontend_name or the different queue times? Should we put these directly on the parent (as haproxy.client_ip, haproxy.frontend_name) or try to group them more?

Good point, I also think it would also make sense to move these fields to the module level.

jsoriano · 2018-08-20T08:14:27Z

filebeat/module/haproxy/_meta/config.yml

+
+    # Set custom paths for the log files. If left empty,
+    # Filebeat will choose the paths depending on your OS.
+    #var.paths:


Add here also a reference to var.input

jsoriano · 2018-08-20T08:16:00Z

filebeat/docs/modules/haproxy.asciidoc

+include::../include/configuring-intro.asciidoc[]
+
+The module is by default configured to run via syslog on port 9001. However
+it can also be configured to read from a file path. See the following example.


Nice addition to be able to use syslog and file.

Thanks. I started out with a normal file log but setting it up in HAProxy requires a bit of effort. Going via syslog was easier as HAProxy has that support built in. I intend to document the HAProxy configuration necessary for using the module too.

jsoriano · 2018-08-20T08:16:14Z

filebeat/docs/modules/haproxy.asciidoc

+=== Example dashboard
+
+This module comes with a sample dashboard showing geolocation, distribution of requests between backends and frontends,
+and status codes over time. For example:


Thanks for the dashboard! 😃

jsoriano · 2018-08-20T08:25:27Z

filebeat/docs/modules/haproxy.asciidoc

+
+
+[float]
+==== `{fileset}` log fileset settings


Replace {fileset} also here

Tethik · 2018-08-20T12:44:04Z

Pushed some changes now, but I still need to update the dashboard and test the changes. Hope I can do so later tonight.

jsoriano · 2018-08-23T08:59:03Z

filebeat/module/haproxy/http/test/haproxy.log-expected.json

@@ -0,0 +1,61 @@
+{


This file should contain only the list of events (each event being the value of _source)

jsoriano · 2018-08-23T09:00:17Z

filebeat/module/haproxy/http/test/haproxy.log-expected.json

+                    "lon": 13.6333,
+                    "lat": 52.3167
+                  }
+                },


Be careful with exposing personal data in examples/tests 🙂

Minor slip up, but not really sensitive. I'll see if I can scrub it.

jsoriano · 2018-08-23T09:00:57Z

filebeat/modules.d/haproxy.yml.disabled

@@ -0,0 +1,8 @@
+- module: haproxy
+  # All logs
+  {fileset}:


http also here

jsoriano · 2018-08-23T09:03:33Z

filebeat/module/haproxy/http/manifest.yml

+  - name: syslog_port
+    default: 9001
+  - name: input
+    default: syslog


Being syslog the default is going to be tricky for the test_modules test as it instantiates filebeat with the default configurations.
@kvch @ph is there a way to provide a different config than the default for the test_modules?

filebeat/tests/system/config/filebeat.yml.j2 is the template used during running system tests. This needs to be extended with syslog input.

The current system test framework also needs to be extended to be able to send logs over TCP and UDP to Filebeat. Maybe running netcat with proper config does the trick of feeding Filebeat with input.

Thanks, I'll take a look to this.

Tethik · 2018-08-23T10:36:50Z

Still working on the dashboard fyi, quite busy these coming days though so it might take a while.

jsoriano · 2018-08-27T08:13:55Z

filebeat/filebeat.reference.yml

+#------------------------------- haproxy Module ------------------------------
+- module: haproxy
+  # All logs
+  {fileset}:


Another placeholder here, remember to run also make update after modifying these files.

Tethik · 2018-08-27T22:40:18Z

Dashboard remade and tests scrubbed.

Here's also a sample haproxy config that should work to send logs to this module.
https://gist.github.com/Tethik/b458036e734f5ec78bb362bba53e9d1c

jsoriano · 2018-08-29T11:42:39Z

@Tethik thanks for your work on this, I'm going to take a look on how we can extend our tests for syslog-based modules, I'll keep you updated.

Tethik · 2018-08-29T12:11:48Z

Glad to be able to contribute. I hope to add some more modules in the near future too 🎁

jsoriano · 2018-08-31T09:40:27Z

@Tethik we have merged #8140, that will force the use of file input for tests, please update your branch with master to check your test files.

…fore update.

Tethik · 2018-08-31T20:50:17Z

Done, I rebased to the latest master. Not sure how I'm supposed to run the tests though.

jsoriano · 2018-09-03T11:09:58Z

To run the tests locally, you can run INTEGRATION_TESTS=1 make system-tests from the filebeat directory.
This takes a while as it runs all system tests, if you want to run only the tests with the haproxy log files, you will need first a running elasticsearch, you can run it with make start-environment, and then:

make python-env
source build/python-env/bin/activate
make filebeat.test
INTEGRATION_TESTS=1 TEST_MODULES=haproxy BEAT_STRICT_PERMS=false ES_HOST=<easticsearch ip> nosetest tests/system/test_module.py

You can check the logs for more info in build/system-tests.

jsoriano · 2018-09-03T11:12:44Z

filebeat/module/haproxy/http/test/haproxy.log

@@ -0,0 +1 @@
+Jul 30 09:03:52 localhost haproxy[32450]: 1.2.3.4:38862 [30/Jul/2018:09:03:52.726] incoming~ docs_microservice/docs 0/0/1/0/2 304 168 - - ---- 6/6/0/0/0 0/0 {docs.example.internal||} {|||} "GET /component---src-pages-index-js-4b15624544f97cf0bb8f.js HTTP/1.1"


This file may need a new line at the end to ensure that this is read on tests.

Pushed the newline. No change though

There was a change 🙂 now the test build failed due to expected object not found. I have added a comment with the missing fields.

Ah sorry, been trying to run the tests locally but the environment is not cooperating with me. (I get different failures from the other unrelated tests too)

jsoriano · 2018-09-03T15:00:26Z

filebeat/module/haproxy/http/test/haproxy.log-expected.json

+        "haproxy.connections.frontend": 6,
+        "@timestamp": "2018-07-30T09:03:52.726Z",
+        "message": "Jul 30 09:03:52 localhost haproxy[32450]: 1.2.3.4:38862 [30/Jul/2018:09:03:52.726] incoming~ docs_microservice/docs 0/0/1/0/2 304 168 - - ---- 6/6/0/0/0 0/0 {docs.example.internal||} {|||} \"GET /component---src-pages-index-js-4b15624544f97cf0bb8f.js HTTP/1.1\""
+    }


These fields are missing:

"input.type": "log", "prospector.type": "log", "fileset.module": "haproxy", "fileset.name": "http", "haproxy.geoip.region_iso_code": "US-WA", "offset": 0,

jsoriano · 2018-09-03T16:04:33Z

jenkins, test this please

jsoriano · 2018-09-03T17:27:59Z

jenkins, test this again please

Change was introduced in elastic#8014, it broke docs build

jsoriano · 2018-09-04T14:27:35Z

To be backported to 6.x along with #8215

Change was introduced in #8014, it broke docs build

(cherry picked from commit e5ed867)

Change was introduced in elastic#8014, it broke docs build (cherry picked from commit e7631ff)

(cherry picked from commit e5ed867)

Change was introduced in elastic#8014, it broke docs build (cherry picked from commit e7631ff)

* Add a HAProxy filebeat module. (#8014) (cherry picked from commits e5ed867, e7631ff and dc94a44)

ruflin added module review Filebeat Filebeat labels Aug 20, 2018

ruflin requested a review from jsoriano August 20, 2018 06:29

jsoriano requested changes Aug 20, 2018

View reviewed changes

jsoriano reviewed Aug 23, 2018

View reviewed changes

jsoriano reviewed Aug 27, 2018

View reviewed changes

Tethik force-pushed the feature/haproxy-filebeat branch from 2c1d135 to 52966c6 Compare August 27, 2018 21:05

jsoriano mentioned this pull request Aug 29, 2018

Force input to file in filebeat module tests #8140

Merged

Tethik added 11 commits August 31, 2018 21:41

Add a HAProxy filebeat module.

483fdc9

Update default module config.

dd33f84

Correct fileset in HAProxy docs.

9d2039c

Add syslog port too.

f38ce1f

update haproxy http pipeline, fix documentation slightly of fields be…

260e038

…fore update.

move most haproxy http fields into the parent haproxy.* group.

8411c2f

fixed broken indentation in yaml files. Bah.

f8783d7

fix default haproxy config fileset.

532916b

Redo config commits. Must have accidentally overwritten them again.

14165d0

update filebeat.reference.yml

32ac3bb

remade dashboard for new fields. exported it according to docs.

41c94a1

Tethik added 2 commits August 31, 2018 21:48

update image.

ae7af09

add new test with flattened result json.

8f99bb2

Tethik force-pushed the feature/haproxy-filebeat branch from 5f25df1 to 8f99bb2 Compare August 31, 2018 19:49

jsoriano reviewed Sep 3, 2018

View reviewed changes

Add newline to haproxy.log testfile.

fa147de

jsoriano reviewed Sep 3, 2018

View reviewed changes

add missing fields.

3d95e85

jsoriano approved these changes Sep 3, 2018

View reviewed changes

jsoriano merged commit e5ed867 into elastic:master Sep 4, 2018

jsoriano added a commit to jsoriano/beats that referenced this pull request Sep 4, 2018

Add missing description to haproxy field name

0b48eb4

Change was introduced in elastic#8014, it broke docs build

jsoriano mentioned this pull request Sep 4, 2018

Add missing description to haproxy field name #8215

Merged

exekias added the needs_backport PR is waiting to be backported to other branches. label Sep 4, 2018

jsoriano mentioned this pull request Sep 4, 2018

Add travis job to test docs build #8217

Merged

jsoriano added a commit that referenced this pull request Sep 4, 2018

Add missing description to haproxy field name (#8215)

e7631ff

Change was introduced in #8014, it broke docs build

jsoriano pushed a commit to jsoriano/beats that referenced this pull request Sep 4, 2018

Add a HAProxy filebeat module. (elastic#8014)

3beb0fb

(cherry picked from commit e5ed867)

jsoriano added a commit to jsoriano/beats that referenced this pull request Sep 4, 2018

Add missing description to haproxy field name (elastic#8215)

425076f

Change was introduced in elastic#8014, it broke docs build (cherry picked from commit e7631ff)

jsoriano mentioned this pull request Sep 4, 2018

Cherry-pick #8014 to 6.x: Add a HAProxy filebeat module. #8220

Closed

jsoriano added v6.5.0 and removed needs_backport PR is waiting to be backported to other branches. labels Sep 4, 2018

jsoriano mentioned this pull request Sep 4, 2018

Cherry-pick #8014 to 6.x: Add a HAProxy filebeat module. #8221

Merged

jsoriano pushed a commit to jsoriano/beats that referenced this pull request Sep 4, 2018

Add a HAProxy filebeat module. (elastic#8014)

375e4c9

(cherry picked from commit e5ed867)

jsoriano added a commit to jsoriano/beats that referenced this pull request Sep 4, 2018

Add missing description to haproxy field name (elastic#8215)

21909ac

Change was introduced in elastic#8014, it broke docs build (cherry picked from commit e7631ff)

jsoriano added a commit that referenced this pull request Sep 6, 2018

Cherry-pick #8014 to 6.x: Add a HAProxy filebeat module. (#8221)

d3975d7

* Add a HAProxy filebeat module. (#8014) (cherry picked from commits e5ed867, e7631ff and dc94a44)

		@@ -0,0 +1 @@
		Jul 30 09:03:52 localhost haproxy[32450]: 1.2.3.4:38862 [30/Jul/2018:09:03:52.726] incoming~ docs_microservice/docs 0/0/1/0/2 304 168 - - ---- 6/6/0/0/0 0/0 {docs.example.internal\|\|} {\|\|\|} "GET /component---src-pages-index-js-4b15624544f97cf0bb8f.js HTTP/1.1"


		--

		`haproxy.http.status_code`::

Add a HAProxy filebeat module. #8014

Add a HAProxy filebeat module. #8014

Conversation

Tethik commented Aug 18, 2018

elasticmachine commented Aug 18, 2018

jsoriano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Tethik commented Aug 20, 2018

Choose a reason for hiding this comment

jsoriano Aug 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Tethik commented Aug 23, 2018

Choose a reason for hiding this comment

Tethik commented Aug 27, 2018

jsoriano commented Aug 29, 2018

Tethik commented Aug 29, 2018

jsoriano commented Aug 31, 2018

Tethik commented Aug 31, 2018

jsoriano commented Sep 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano commented Sep 3, 2018

jsoriano commented Sep 3, 2018

jsoriano commented Sep 4, 2018

jsoriano Aug 23, 2018 •

edited

Loading