Add PostgresSQL module for Filebeat #4763

kvch · 2017-07-26T19:52:30Z

I have been working on a PostgreSQL module for Filebeat. Unfortunately, I got stuck at viewing logs in Kibana or maybe at indexing logs in Elasticsearch.

The fields for PostgreSQL I have defined are visible in the index. When I test the the pipeline I have created using the Simulate API with the sample logs in the commit, the fields are parsed properly. However, when I read logs of PostgreSQL using Filebeat, it does not seem to work when viewed in Kibana.

This is what I see:

I would expect the logs to be properly parsed, but I don't think it is what I see. To be honest, I am not sure what to expect. Could you give me some pointers on how to go on?

Also I have collected a few TODOs, but please let me know if I missed something.

TODO

urso · 2017-07-27T18:00:23Z

filebeat/module/postgresql/log/ingest/pipeline.json

+        "ignore_missing": true,
+        "patterns": [
+          "%{LOCALDATETIME:postgresql.log.timestamp} %{WORD:postgresql.log.timeformat} \[%{NUMBER:postgresql.log.thread_id}\] %{USERNAME:postgresql.log.username}@%{HOSTNAME:postgresql.log.host} %{WORD:postgresql.log.level}:  ?%{GREEDYDATA:postgresql.log.message}",
+          "%{LOCALDATETIME:postgresql.log.timestamp} %{WORD:postgresql.log.timeformat} \[%{NUMBER:postgresql.log.thread_id}\] \[%{USERNAME:postgresql.username}\]@\[%{HOSTNAME:postgresql.log.host}\] %{WORD:postgresql.log.level}:  ?%{GREEDYDATA:postgresql.log.message}",


s/postgresql.username/postgreseql.log.username/ ?

urso · 2017-07-27T18:01:22Z

filebeat/module/postgresql/log/ingest/pipeline.json

+        "patterns": [
+          "%{LOCALDATETIME:postgresql.log.timestamp} %{WORD:postgresql.log.timeformat} \[%{NUMBER:postgresql.log.thread_id}\] %{USERNAME:postgresql.log.username}@%{HOSTNAME:postgresql.log.host} %{WORD:postgresql.log.level}:  ?%{GREEDYDATA:postgresql.log.message}",
+          "%{LOCALDATETIME:postgresql.log.timestamp} %{WORD:postgresql.log.timeformat} \[%{NUMBER:postgresql.log.thread_id}\] \[%{USERNAME:postgresql.username}\]@\[%{HOSTNAME:postgresql.log.host}\] %{WORD:postgresql.log.level}:  ?%{GREEDYDATA:postgresql.log.message}",
+          "%{LOCALDATETIME:postgresql.log.timestamp} %{WORD:postgresql.log.timeformat} \[%{NUMBER:postgresql.log.thread_id}\] %{WORD:postgresql.log.level}:  ?%{GREEDYDATA:postgresql.log.message}"


I wonder if you need to use \\ here. grok is basically build on regular expressions. Plus this file being json might might affect the \ on read.

kvch · 2017-07-28T10:50:28Z

My initial problem was solved. Thank you for your help. :) I updated the TODO list based on the current state of the PR.

tsg · 2017-07-31T06:41:14Z

This is green enough, the remaining failure is Metricbeat on Windows.

tsg

LGTM. I think only the Changelog is missing.

tsg · 2017-07-31T06:48:19Z

@kvch It would be nice to also have a fileset for slow logs, like we have for MySQL. But totally fine to do that in a second PR.

kvch · 2017-07-31T21:55:11Z

@tsg I am trying to add slowlog parsing. It is not in a different fileset, because PostgreSQL puts slowlogs in the same file as the other logs by default.

Again the Grok parser has failed me. :D I have worked out a regex for multiline query statements using Kibana Dev Tools, but it does not seem to work with Filebeat.

kvch · 2017-07-31T22:20:20Z

The problem is fixed. Multiline configuration was missing from the module.

tsg · 2017-08-01T12:42:33Z

@kvch Something else I noticed, I think you didn't yet add -expected.json files. See for example, the one from the Nginx module. There's test.log, which contains a few sample log lines, and test.log-expected.json which contain the result after passing it through Elasticsearch. If you add such an *-expected.json file, the system tests will automatically check the results. See the responsible system test here.

kvch · 2017-08-01T12:51:47Z

@tsg I will add it. Thanks.

kvch · 2017-08-02T17:19:38Z

I have added a new commit containing a test refactor.

This module includes a sample dashboard and two sample searches for Kibana.

Previously, only 10 logs could be in *.log and *-expected.json files, as it is the default page size of ES. However, to be more future proof the test was extended, so it can validate more than 10 logs and events.

tsg · 2017-08-08T06:50:56Z

Remaining test failure is not related. Merging.

jpountz

The links to some images look wrong?

jpountz · 2017-08-08T07:57:35Z

filebeat/docs/modules/postgresql.asciidoc

+
+The second one shows the slowlogs of PostgreSQL.
+
+image::./images/filebeat-postgresql-postgresql-overview.png[]


it should be images/filebeat-postgresql-slowlog-overview.png I think

Should be fixed with: #4843

jpountz · 2017-08-08T07:57:41Z

filebeat/module/postgresql/_meta/docs.asciidoc

+
+The second one shows the slowlogs of PostgreSQL.
+
+image::./images/filebeat-postgresql-postgresql-overview.png[]


it should be images/filebeat-postgresql-slowlog-overview.png I think

liuqian1990 · 2017-12-27T08:00:17Z

2017-12-27 11:19:07 GMT [30072]-[14] session=5a43112b.7578 user=postgres db=postgres app=[unknown] client=10.0.2.73 LOG: duration: 1.229 ms statement:
SELECT COUNT(mode) AS count, mode
FROM pg_locks GROUP BY mode
UNION SELECT COUNT(*) AS count, 'waiting' AS mode
FROM pg_locks
WHERE granted is false;

2017-12-27 11:19:07 GMT [30072]-[15] session=5a43112b.7578 user=postgres db=postgres app=[unknown] client=10.0.2.73 LOG: duration: 0.334 ms parse : SELECT pg_database_size($1) AS size;
2017-12-27 11:19:07 GMT [30072]-[16] session=5a43112b.7578 user=postgres db=postgres app=[unknown] client=10.0.2.73 LOG: duration: 0.206 ms bind : SELECT pg_database_size($1) AS size;
2017-12-27 11:19:07 GMT [30072]-[17] session=5a43112b.7578 user=postgres db=postgres app=[unknown] client=10.0.2.73 DETAIL: parameters: $1 = 'postgres'
2017-12-27 11:19:07 GMT [30070]-[18] session=5a43112b.7576 user=postgres db=postgres app=[unknown] client=10.0.2.73 LOG: duration: 52.563 ms execute : SELECT pg_database_size($1) AS size;
2017-12-27 11:19:07 GMT [30070]-[19] session=5a43112b.7576 user=postgres db=postgres app=[unknown] client=10.0.2.73 DETAIL: parameters: $1 = 'postgres'
2017-12-27 11:19:07 GMT [30070]-[20] session=5a43112b.7576 user=postgres db=postgres app=[unknown] client=10.0.2.73 LOG: duration: 0.112 ms parse : SELECT pg_database_size($1) AS size;
2017-12-27 11:19:07 GMT [30070]-[21] session=5a43112b.7576 user=postgres db=postgres app=[unknown] client=10.0.2.73 LOG: duration: 0.165 ms bind : SELECT pg_database_size($1) AS size;

How to resolve these logs, set up can not be resolved

liuqian1990 · 2017-12-27T08:01:09Z

Provided Grok expressions do not match field value: [2017-12-27 13:54:07 GMT [30071]-[3741] session=5a43112b.7577 user=postgres db=postgres app=[unknown] client=10.0.2.73 LOG: duration: 0.180 ms bind : SELECT pg_database_size($1) AS size;]

ruflin · 2017-12-28T02:15:42Z

@liuqian1990 Could you please open a new topic on discuss for further discussions as this is a closed PR? https://discuss.elastic.co/c/beats/filebeat Please provide there your config and the filebeat version you are using + the logs as you did above.

kvch added Filebeat Filebeat in progress Pull request is currently in progress. review labels Jul 26, 2017

kvch force-pushed the feature/filebeat-postgresql-module branch 2 times, most recently from 528c9a5 to 5f93a94 Compare July 26, 2017 20:04

urso reviewed Jul 27, 2017

View reviewed changes

kvch force-pushed the feature/filebeat-postgresql-module branch 2 times, most recently from 2335916 to ceba36c Compare July 28, 2017 10:32

kvch force-pushed the feature/filebeat-postgresql-module branch 5 times, most recently from bf388ab to 69234ae Compare July 28, 2017 19:49

tsg approved these changes Jul 31, 2017

View reviewed changes

kvch force-pushed the feature/filebeat-postgresql-module branch from 69234ae to 2781c17 Compare July 31, 2017 20:19

kvch force-pushed the feature/filebeat-postgresql-module branch 2 times, most recently from 4805b3b to 00bb4a3 Compare July 31, 2017 22:19

kvch force-pushed the feature/filebeat-postgresql-module branch 2 times, most recently from c66cae4 to e1128f9 Compare August 2, 2017 16:39

kvch removed the in progress Pull request is currently in progress. label Aug 2, 2017

filebeat: add PostgreSQL module

f199dd3

This module includes a sample dashboard and two sample searches for Kibana.

kvch added 2 commits August 3, 2017 12:28

filebeat: add PostgreSQL slowlog

bd3c901

system-tests: make it possible to test more than 10 logs

c36915d

Previously, only 10 logs could be in *.log and *-expected.json files, as it is the default page size of ES. However, to be more future proof the test was extended, so it can validate more than 10 logs and events.

kvch force-pushed the feature/filebeat-postgresql-module branch from e1128f9 to c36915d Compare August 3, 2017 10:28

tsg merged commit 2caf75d into elastic:master Aug 8, 2017

jpountz reviewed Aug 8, 2017

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add PostgresSQL module for Filebeat #4763

Add PostgresSQL module for Filebeat #4763

kvch commented Jul 26, 2017 •

edited

Loading

urso Jul 27, 2017

urso Jul 27, 2017

kvch commented Jul 28, 2017

tsg commented Jul 31, 2017

tsg left a comment

tsg commented Jul 31, 2017

kvch commented Jul 31, 2017

kvch commented Jul 31, 2017

tsg commented Aug 1, 2017

kvch commented Aug 1, 2017

kvch commented Aug 2, 2017

tsg commented Aug 8, 2017

jpountz left a comment

jpountz Aug 8, 2017

tsg Aug 8, 2017

jpountz Aug 8, 2017

liuqian1990 commented Dec 27, 2017

liuqian1990 commented Dec 27, 2017

ruflin commented Dec 28, 2017


		The second one shows the slowlogs of PostgreSQL.

		image::./images/filebeat-postgresql-postgresql-overview.png[]

Add PostgresSQL module for Filebeat #4763

Add PostgresSQL module for Filebeat #4763

Conversation

kvch commented Jul 26, 2017 • edited Loading

urso Jul 27, 2017

Choose a reason for hiding this comment

urso Jul 27, 2017

Choose a reason for hiding this comment

kvch commented Jul 28, 2017

tsg commented Jul 31, 2017

tsg left a comment

Choose a reason for hiding this comment

tsg commented Jul 31, 2017

kvch commented Jul 31, 2017

kvch commented Jul 31, 2017

tsg commented Aug 1, 2017

kvch commented Aug 1, 2017

kvch commented Aug 2, 2017

tsg commented Aug 8, 2017

jpountz left a comment

Choose a reason for hiding this comment

jpountz Aug 8, 2017

Choose a reason for hiding this comment

tsg Aug 8, 2017

Choose a reason for hiding this comment

jpountz Aug 8, 2017

Choose a reason for hiding this comment

liuqian1990 commented Dec 27, 2017

How to resolve these logs, set up can not be resolved

liuqian1990 commented Dec 27, 2017

ruflin commented Dec 28, 2017

kvch commented Jul 26, 2017 •

edited

Loading