Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert Filebeat iis.error to ECS and add IPv6 zone support for IIS access logs #9955

Merged
merged 10 commits into from
Jan 11, 2019

Conversation

webmat
Copy link
Contributor

@webmat webmat commented Jan 8, 2019

Caveats

  • Now filtering zone from both IP fields. Using and keeping the .address field to keep the raw address around, and .ip field is only set with the IP value without the zone.
  • Realized that the access logs were still vulnerable to the Microsoft IPv6 zone literal problem. Fixed it in this PR.

Renames

  • read_timestamp => event.created
  • iis.error.remote_ip => source.address
  • iis.error.remote_port => source.port
  • iis.error.server_ip => destination.address
  • iis.error.server_port => destination.port
  • iis.error.http_version => http.version
  • iis.error.method => http.request.method
  • iis.error.url => url.original
  • iis.error.response_code => http.response.status_code
  • iis.error.geoip.* => source.geo.*

Alias adjustments in iis.access

  • iis.access.remote_ip => source.address (instead of .ip)
  • iis.access.server_ip => destination.address (instead of .ip)

TODO

  • Add IPv6 zone support to access logs as well (like Handle IPv6 zone id in IIS filebeat ingest pipeline #9869 did for error logs)
  • Coerce some int fields
  • read_timestamp => event.created
  • Alias renamed fields to their ECS counterpart
  • Document field migrations in ecs-migration.yml
  • Revisit iis.access to fix the potential zone problem there too, using the .address fields
    • Fix the aliases to .address
  • Changelog

@webmat webmat requested a review from a team as a code owner January 8, 2019 19:24
@webmat webmat self-assigned this Jan 8, 2019
@webmat webmat added in progress Pull request is currently in progress. module Filebeat Filebeat ecs labels Jan 8, 2019
@webmat webmat requested a review from a team as a code owner January 8, 2019 20:05
@webmat webmat changed the title WIP Convert Filebeat iis.error to ECS Convert Filebeat iis.error to ECS Jan 8, 2019
@webmat webmat added review and removed in progress Pull request is currently in progress. labels Jan 8, 2019
@ruflin ruflin mentioned this pull request Jan 8, 2019
@webmat
Copy link
Contributor Author

webmat commented Jan 8, 2019

jenkins, test this

@webmat
Copy link
Contributor Author

webmat commented Jan 9, 2019

@ruflin ready for a review. Only Jenkins failures are flaky tests unrelated to Filebeat

@webmat webmat requested a review from ruflin January 9, 2019 20:17
@ruflin
Copy link
Member

ruflin commented Jan 10, 2019

I filed an issue here for the flaky test: #9987

@webmat webmat changed the title Convert Filebeat iis.error to ECS Convert Filebeat iis.error to ECS and add IPv6 zone support for IIS access logs Jan 10, 2019
@webmat
Copy link
Contributor Author

webmat commented Jan 10, 2019

Rebased to fix a conflict.

This reminded me that in migrating iis.error to ECS, I also updated iis.access again, to tackle two things:

  • Support for Microsoft IPv6 zones (like Handle IPv6 zone id in IIS filebeat ingest pipeline #9869 did for error logs)
  • Use the dual .address + .ip fields. I did not initially use .address, but it was a great fit for the IP with zone, prior to copying the cleaned up IP address to the .ip field.

I've also updated the PR title to reflect this. @ruflin Ok with killing two birds with one PR? ;-)

Copy link
Member

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

I resolved the changelog conflict.

@jsoriano
Copy link
Member

Thanks for adding the IPv6 support to the access fileset too, this part would need to be backported as has been done with #9869.

@webmat webmat added the needs_backport PR is waiting to be backported to other branches. label Jan 11, 2019
@webmat
Copy link
Contributor Author

webmat commented Jan 11, 2019

@jsoriano Yes, noted. I'll do that later today.

The backport will cover the support for zone in the IP address, but will not introduce the .address field, to be consistent.

Shall I create a brand new PR just for this on 6.x, and then backport to 6.6 as well, for 6.6.1? Or should I hold off for 6.6? I'm unsure about the exact timing of the procedure :-)

@urso urso removed the request for review from a team January 11, 2019 15:47
@webmat webmat force-pushed the ecs-iis-error-fb branch 2 times, most recently from c2ac4cd to 9a1c851 Compare January 11, 2019 18:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants