Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow users to convert timezone in elasticsearch module filesets #9761

Merged
merged 10 commits into from
Dec 28, 2018

Conversation

ycombinator
Copy link
Contributor

@ycombinator ycombinator commented Dec 21, 2018

This PR updates the following filesets in the elasticsearch Filebeat module to accept a var.convert_timezone configuration setting:

  • server
  • audit
  • slowlog
  • deprecation

The gc fileset was not modified because Elasticsearch GC logs already emit timestamps with timezone information in them.

Fixes partially #9756. Related: #9797.

@ycombinator ycombinator added bug in progress Pull request is currently in progress. module Filebeat Filebeat needs_backport PR is waiting to be backported to other branches. v7.0.0 Feature:Stack Monitoring v6.7.0 labels Dec 21, 2018
@ycombinator ycombinator requested a review from a team as a code owner December 21, 2018 17:23
@elasticmachine
Copy link
Collaborator

Pinging @elastic/stack-monitoring

@ruflin
Copy link
Member

ruflin commented Dec 27, 2018

I think this needs a make update probably on the top level to also update the x-pack filebeat.

@ycombinator
Copy link
Contributor Author

CI failures are related. Looking into them now.

@ycombinator
Copy link
Contributor Author

jenkins, test this

@ycombinator
Copy link
Contributor Author

@ruflin CI is green now, if you want to review the PR.

"input.type": "log",
"log.offset": 0,
"message": "[2018-06-19T05:16:15,549] [rest] [authentication_failed] origin_address=[147.107.128.77], principal=[i030648], uri=[/_xpack/security/_authenticate]",
"@timestamp": "2018-06-19T05:16:15.549Z",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm surprised by the end line space diff as this file is auto generated there should not be a difference. Did you touch this file manually perhaps?

Copy link
Contributor Author

@ycombinator ycombinator Dec 28, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did manually touch it, to update the @timestamp values in each expected document.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not generate it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I can regenerate them. It was a small change that I could make it global search replace so I didn't bother, but I'll regenerate them now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regenerated in d9a2bac.

Copy link
Member

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code LGTM. Can you fix the changelog?

@@ -202,6 +202,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha1...v7.0.0-alpha2[Check the
- Rename many `icinga.*` fields to map to ECS. {pull}9294[9294]
- Rename many `postgresql.log.*` fields to map to ECS. {pull}9303[9303]
- Rename many `kafka.log.*` fields to map to ECS. {pull}9297[9297]
- Add `convert_timezone` option to Elasticsearch module to convert dates to UTC. {issue}9756[9756] {pull}9761[9761]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this changelog is in the wrong place as alpha2 is already shipped. Best directly add it to .next changelog.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved in d9f5f1f.

@ycombinator
Copy link
Contributor Author

jenkins, test this

@ycombinator ycombinator merged commit 3559e58 into elastic:master Dec 28, 2018
@ycombinator ycombinator removed the needs_backport PR is waiting to be backported to other branches. label Dec 28, 2018
ycombinator added a commit to ycombinator/beats that referenced this pull request Jan 8, 2019
…stic#9761)

This PR updates the following filesets in the `elasticsearch` Filebeat module to accept a `var.convert_timezone` configuration setting:

* [x] server
* [x] audit
* [x] slowlog
* [x] deprecation

The `gc` fileset was not modified because Elasticsearch GC logs already emit timestamps with timezone information in them.

Fixes partially elastic#9756. Related: elastic#9797.

(cherry picked from commit 3559e58)
ycombinator added a commit that referenced this pull request Jan 8, 2019
…earch module filesets (#9830)

Cherry-pick of PR #9761 to 6.x branch. Original message: 

This PR updates the following filesets in the `elasticsearch` Filebeat module to accept a `var.convert_timezone` configuration setting:

* [x] server
* [x] audit
* [x] slowlog
* [x] deprecation

The `gc` fileset was not modified because Elasticsearch GC logs already emit timestamps with timezone information in them.

Fixes partially #9756. Related: #9797.
@ycombinator ycombinator added review v6.6.1 needs_backport PR is waiting to be backported to other branches. and removed in progress Pull request is currently in progress. labels Jan 17, 2019
@ycombinator ycombinator added v6.6.0 and removed needs_backport PR is waiting to be backported to other branches. labels Jan 29, 2019
ycombinator added a commit that referenced this pull request Jan 30, 2019
… module filesets (#10407)

Cherry-pick of PR #9797 to 6.6 branch. Original message: 

This PR updates the following filesets in the `logstash` Filebeat module to accept a `var.convert_timezone` configuration setting:

* [x] log
* [x] slowlog

Fixes partially #9756. Related: #9761
ycombinator added a commit to ycombinator/beats that referenced this pull request Jan 30, 2019
…stic#9761)

This PR updates the following filesets in the `elasticsearch` Filebeat module to accept a `var.convert_timezone` configuration setting:

* [x] server
* [x] audit
* [x] slowlog
* [x] deprecation

The `gc` fileset was not modified because Elasticsearch GC logs already emit timestamps with timezone information in them.

Fixes partially elastic#9756. Related: elastic#9797.

(cherry picked from commit 3559e58)
ycombinator added a commit that referenced this pull request Jan 30, 2019
…earch module filesets (#10403)

* Allow users to convert timezone in elasticsearch module filesets (#9761)

This PR updates the following filesets in the `elasticsearch` Filebeat module to accept a `var.convert_timezone` configuration setting:

* [x] server
* [x] audit
* [x] slowlog
* [x] deprecation

The `gc` fileset was not modified because Elasticsearch GC logs already emit timestamps with timezone information in them.

Fixes partially #9756. Related: #9797.

(cherry picked from commit 3559e58)

* Regenerating generated files

* Fixing duplicated header in CHANGELOG

* Updating generated file
@ycombinator ycombinator deleted the fb-es-tz branch December 25, 2019 11:14
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
…ogstash module filesets (elastic#10407)

Cherry-pick of PR elastic#9797 to 6.6 branch. Original message: 

This PR updates the following filesets in the `logstash` Filebeat module to accept a `var.convert_timezone` configuration setting:

* [x] log
* [x] slowlog

Fixes partially elastic#9756. Related: elastic#9761
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
…lasticsearch module filesets (elastic#10403)

* Allow users to convert timezone in elasticsearch module filesets (elastic#9761)

This PR updates the following filesets in the `elasticsearch` Filebeat module to accept a `var.convert_timezone` configuration setting:

* [x] server
* [x] audit
* [x] slowlog
* [x] deprecation

The `gc` fileset was not modified because Elasticsearch GC logs already emit timestamps with timezone information in them.

Fixes partially elastic#9756. Related: elastic#9797.

(cherry picked from commit 43d976e)

* Regenerating generated files

* Fixing duplicated header in CHANGELOG

* Updating generated file
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants