Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert Filebeat postgresql.log to ECS #9308

Merged
merged 11 commits into from
Dec 20, 2018
Merged

Conversation

webmat
Copy link
Contributor

@webmat webmat commented Nov 30, 2018

Caveats

  • Module used to keep original message in message, and cleaned up message in postgresql.log.message. Keeping original message will be addressed elsewhere, and this PR now replaces message with the cleaned up message.
  • This module previously named the PID as postgresql.log.thread_id, which is incorrect. PG cannot log thread ID, it's not an option. This PR names the PID as process.pid
  • The PG logs contain timestamps without a timezone, as well as a separate timezone field. This module doesn't leverage the timezone field and parses the timestamp as UTC. Properly implementing support would require a module option to enable/disable timezone usage. I consider this out of scope for this field name transition.

Renames

  • postgresql.log.level => log.level
  • postgresql.log.user => user.name
  • postgresql.log.thread_id => process.pid
  • postgresql.log.timezone => event.timezone
  • postgresql.log.message => message

TODO

  • Coerce PID to int
  • Coerce duration to float
  • Save cleaned up message in message field
  • Copy postgresql.log.duration * 1000000 to event.duration
  • Migrate the timezone field as well
  • Make the new core_id field an int as well
  • Alias renamed fields to their ECS counterpart
  • Document field migrations in ecs-migration.yml
  • Changelog

@webmat webmat self-assigned this Nov 30, 2018
@webmat webmat added in progress Pull request is currently in progress. module Filebeat Filebeat ecs labels Nov 30, 2018
@ruflin ruflin mentioned this pull request Nov 30, 2018
@ruflin
Copy link
Member

ruflin commented Dec 3, 2018

+1 on both Caveats

@webmat webmat requested a review from a team as a code owner December 19, 2018 21:22
@webmat webmat changed the title WIP Convert Filebeat postgresql.log to ECS Convert Filebeat postgresql.log to ECS Dec 19, 2018
@webmat webmat added review and removed in progress Pull request is currently in progress. labels Dec 19, 2018
@webmat
Copy link
Contributor Author

webmat commented Dec 19, 2018

Ready for another look. There's one new caveat about the timezone field.

@webmat
Copy link
Contributor Author

webmat commented Dec 20, 2018

jenkins, test this please

dev-tools/ecs-migration.yml Outdated Show resolved Hide resolved
dev-tools/ecs-migration.yml Show resolved Hide resolved
@webmat webmat merged commit e2287ec into elastic:master Dec 20, 2018
@webmat webmat deleted the ecs-postgresql-fb branch December 20, 2018 19:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants