Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SASL&mTLS authentication support for Kafka in Promtail #4663

Merged

Conversation

taisho6339
Copy link
Contributor

@taisho6339 taisho6339 commented Nov 5, 2021

What this PR does / why we need it:

Currently, Promtail doesn't support Kafka authentication so we can only connect with no auth.
Let me suggest changes to add some authentication for Kafka.
This PR adds the following auth method.

  • mTLS
  • SASL/PLAIN
  • SASL/SCRAM
  • SASL/PLAIN over TLS
  • SASL/SCRAM over TLS

In addition,

  • Add documentation to configure authentication
  • Add testing tools to check auth with Kafka locally
  • Add go module to implement SASL/SCRAM

Most of the changes are due to additional modules and testing tools.

Which issue(s) this PR fixes:
Fixes #4662

Special notes for your reviewer:

Checklist

  • Documentation added
  • Tests updated
  • Add an entry in the CHANGELOG.md about the changes.

@taisho6339
Copy link
Contributor Author

taisho6339 commented Nov 5, 2021

TODO

  • Add TLS support
  • Add SASL/PLAIN, SASL/SCRAM support
  • Add unit tests for these changes
  • Add documentations
  • Add auth tool for kafka to test
  • Test actual integration with Kafka

@taisho6339 taisho6339 force-pushed the feature/add-authentication-for-kafka branch from 9574df7 to cacf104 Compare November 6, 2021 09:07
@taisho6339 taisho6339 changed the title [WIP] Add authentication config for Kafka in Promtail [WIP] Add SASL&mTLS authentication support for Kafka in Promtail Nov 6, 2021
@taisho6339 taisho6339 force-pushed the feature/add-authentication-for-kafka branch from cacf104 to 0bad5ed Compare November 7, 2021 11:05
@taisho6339 taisho6339 force-pushed the feature/add-authentication-for-kafka branch from 34212a1 to 488fae3 Compare November 7, 2021 13:13
@taisho6339 taisho6339 force-pushed the feature/add-authentication-for-kafka branch from c66d3c5 to d2f4a58 Compare November 7, 2021 13:20
@taisho6339 taisho6339 changed the title [WIP] Add SASL&mTLS authentication support for Kafka in Promtail Add SASL&mTLS authentication support for Kafka in Promtail Nov 7, 2021
@taisho6339 taisho6339 marked this pull request as ready for review November 7, 2021 13:24
@taisho6339
Copy link
Contributor Author

taisho6339 commented Nov 7, 2021

"shell check" in CI is failed but I didn't touch the files in this PR.
Should I fix in another PR?
I don't wanna increase unrelated changes here

Fixed

Copy link
Contributor

@cyriltovena cyriltovena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice contribution @taisho6339. Went through the code once and it looks awesome. I'll run it locally this week and circle back.

@taisho6339
Copy link
Contributor Author

Thank you so much!

Copy link
Contributor

@cyriltovena cyriltovena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't able to connect with SSL locally, because of : java.security.cert.CertificateException: No name matching host.docker.internal found

Otherwise LGTM

@cyriltovena cyriltovena merged commit 1557dab into grafana:main Nov 9, 2021
@taisho6339
Copy link
Contributor Author

Maybe we need to add "host.docker.internal" as SAN field in create certs script.
Anyway thank you for your review!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Promtail] Kafka Authentication Support
2 participants