-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix grok expression for traefik #6136
Conversation
Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually? |
@e8kor Thanks for the PR. Could you add a log file for the 1.4 events for the tests: https://github.com/elastic/beats/tree/master/filebeat/module/traefik/access/tests Perhaps we should name the files accordingly. I don't remember if there is an easy way to generated the expected outcome. @kvch might know? |
@e8kor Please also add an entry to the CHANGELOG file. |
@ruflin and expected log output, current test.log is completely outdated |
Here is example of proper log entries of traefik, minimal and maximal:
and examples of parsed output maximal: {
"traefik": {
"access": {
"response_code": 304,
"agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36",
"method": "GET",
"user_name": "user",
"http_version": "1.1",
"backend_url": "http://172.19.0.3:5601",
"url": "/ui/favicons/favicon.ico",
"request_count": 271,
"duration": 3,
"referrer": "http://example.com/login",
"remote_ip": "85.181.35.98",
"frontend_name": "Host-host1",
"body_sent": {
"bytes": 0
},
"time": "02/Oct/2017:20:22:08 +0000"
}
}
} minimal: {
"traefik": {
"access": {
"duration": 0,
"remote_ip": "85.181.35.98",
"method": "GET",
"http_version": "1.1",
"time": "02/Oct/2017:20:22:08 +0000",
"url": "/ui/favicons/favicon.ico"
}
}
} I have refactored grok expression to have single one instead of two patterns. |
@kvch revert then ? I'm using new one on production and so far so good, I don't know how pipelines behaves in this case, is pipeline choose first matched or its doing processing for each, if you can guide me throw this cases I can come with better case. Have you compered previous expressions with new one ? |
I think one expression will probably more efficient on the ES side in case the second pattern is hit. But for readability 2 patterns are probably better. Like this we have one for each "version" of traefik. @e8kor Did you try to add an additional log entry with expected output to the tests directory? |
@ruflin what is expected from me now to finish pull request? Sorry for long delay |
@e8kor any updates? |
Also strange, but I don't have frontend name in Kibana |
It was hard to understand how tests should look like and because of lack of explanations I haven't completed pullrequest. |
@ruflin Could you please help him with tests :)? |
@e8kor You need to add example logs to Please, rebase your branch. If you need further help, let me know. :) |
it's time of 1.7.x traefik, filebeat still does not have this fixed :( |
@ruflin
https://docs.traefik.io/configuration/logs/#clf-common-log-format |
@ruflin Howdy, yup they look like this
It's also said in traefik docs that CLF is used: But personally I would prefer to see the filebeat kibana dashboard with native traefik json format. Since using filebeat modules in clusters such as k8s is pain. Passing the volumes there and back, unneeded complexity and security flaws... |
@iwex @dennybaa Thanks for the logs. @sayden FYI ^ @dennybaa I would expect the modules to work very well in combination with autodiscovery: https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-autodiscover.html When you write JSON output you mean the message packed into JSON by docker or actually structured logs as JSON? |
@ruflin traefik also has an option to write logs in json format. I'll find example and attach somewhere here :) |
@ruflin Huge amount of data:
|
@ruflin thank you for the information.
Please any working configuration would be preferable not just the link (and specifically for the metrics beat which is not the one we are talking about, right ?). While https://www.elastic.co/guide/en/beats/filebeat/master/configuration-autodiscover.html#_kubernetes docs are hardly sufficient for a successful setup. What filebeat can successfully do so far is just grabbing docker stdout/stderr logs without autodiscover module templates kicking in :( Unfortunately it does not feel even like beta now (filebeat 6.4.2). |
@dennybaa We probably better take this discussion to discuss instead of here in the issue. Could you open a topic there? https://discuss.elastic.co/c/beats/filebeat |
@ruflin Well yup my bad, sorry for ranting. |
Finally fixed in #8768 |
This pull request fixes Grok expression for parsing access logs from traefik.
Related issue: #6111
Grok expressions will work for traefik version 1.4.+ and 1.5.+ and it respects options fields in logs according to logger format.