Skip to content

Latest commit

 

History

History
100 lines (68 loc) · 2.64 KB

kafka-topics.md

File metadata and controls

100 lines (68 loc) · 2.64 KB
Raw

Kafka topics

List topics.

kafka-topics --bootstrap-server localhost:9092 --list

Create topic.

kafka-topics --bootstrap-server localhost:9092 --create --topic new_topic

Read topic config.

kafka-topics --bootstrap-server localhost:9092 --describe --topic new_topic

Add more partitions to topic

Partitions can be added but canot be deleted. Operations is irreversable.

kafka-topics --bootstrap-server localhost:9092 --alter --topic new_topic --partitions 4

Kafka Config

List all configs for topic

kafka-configs --bootstrap-server localhost:9092 --describe --all --topic new_topic

Set retention time in msecs retention.ms.

kafka-configs --bootstrap-server localhost:9092 --alter --topic new_topic --add-config retention.ms=10000

Remove retention time config option

kafka-configs --bootstrap-server localhost:9092 --alter --topic new_topic --delete-config retention.ms

Kafka topics with SSL

Create config file connect.properties with SSL options

security.protocol=SSL
ssl.keystore.location=keystore.jks
ssl.keystore.password=secret
ssl.key.password=secret
ssl.truststore.location=truststore.jks
ssl.truststore.password=secret
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1

Create client, server keystores and truststore using KeyStore Explorer or Openssl and convert it to jks

  1. Root CA key pair (key and certificate) adding CA extention
  2. Server key pair signed with Root CA (Sing new key pair) and added extention TLS Web Server Authentication (1.3.6.1.5.5.7.3.1)
  3. Client key pair signed with Root CA and added extention TLS Web Client Authentication (1.3.6.1.5.5.7.3.2) Export Root CA to trustsore, save server and client key pairs to keystores

Use connect.properties as command config with kafka-topics and kafka-configs

kafka-topics --command-config connect.properties --bootstrap-server "SSL://localhost:9093" --list
kafka-configs --command-config connect.properties --bootstrap-server "SSL://localhost:9093" --describe --all --topic new_topic

Kafka consumer groups

List

kafka-consumer-groups --bootstrap-server "localhost:9092" --list

Describe

kafka-consumer-groups --bootstrap-server "localhost:9092" --group worker-group --describe

Reset Offsets to latest offset. Can be used with dry run option --dry-run. Option --execute confirms operation.

kafka-consumer-groups --bootstrap-server "localhost:9092" --topic worker-queue --group worker-group --reset-offsets --to-latest --execute