Kafka Connector

Contents

• Overview
• Summary
• Source Files
• Configuration
  • Shared Configuration
  • Connector-Specific Configuration
• Diagnostics
• Metrics

Overview

• KafkaConnector is intended for consuming from a single Kafka topic.

• For Kafka mirroring use cases, please refer to KafkaMirrorMakerConnector instead.

Summary

Supports Datastream Updates	Yes
Checkpoint Type	Kafka consumer offsets are regularly committed to Kafka (see commitIntervalMs)

Source Files

1. com.linkedin.datastream.connectors.kafka.KafkaConnector

2. com.linkedin.datastream.connectors.kafka.AbstractKafkaConnector

Configuration

• All connector config properties must be prefixed with brooklin.server.connector.<connectorName>.

• connectorName is an arbitrary user-supplied name specified in Brooklin configuration (brooklin.server.connectorNames)

Shared Configuration

• Configuration properties shared among all Kafka connectors in Brooklin are documented in Kafka Connectors Shared Logic.

• This connector has the following defaults for the configuration property(ies) listed below.

  Property	Description	Default
  consumer.*	Kafka consumer configuration properties	enable.auto.commit: false auto.offset.reset: none security.protocol: SSL OR PLAINTEXT depending on Kafka connection string specified in datastream’s source.connectionString property

Connector-Specific Configuration

Configuration properties specific to KafkaConnector are listed below.

Property	Description	Default
whiteListedClusters	A comma-separated list of Kafka brokers, e.g. host1:port1,host2:port2 Any datastream using this connector must specify one of these Kafka brokers in its source.connectionString	(None)

Diagnostics

All Kafka connectors share the common diagnostic endpoints documented at Kafka Connectors Shared Logic.

Metrics

All Kafka connectors share the common metrics documented at Kafka Connectors Shared Logic.