This repository has been archived.
The Pulsar IO Kafka Connector code can now be found at apache/pulsar/pulsar-io/kafka.
Source documentation: https://pulsar.apache.org/docs/io-kafka-source/
Sink documentation: https://pulsar.apache.org/docs/io-kafka-sink/
pulsar-io-kafka is a Pulsar IO Connector for copying data between Kafka and Pulsar.
NOTE: This connector is an enhanced version of
pulsar-io-kafka
connector to support schema.
This provides a step-by-step example how to use this Kafka source connector to copy json data from a Kafka topic to a Pulsar topic and save the data in AVRO format.
-
Git clone
pulsar-io-kafka
. Assume PULSAR_IO_KAFKA_HOME is the home directory for your clonedpulsar-io-kafka
repo for the remaining steps.$ git clone https://github.com/streamnative/pulsar-io-kafka
-
Build the connector in
${PULSAR_IO_KAFKA_HOME}
directory.mvn clean install -DskipTests
After successfully built the connector, a NAR package is generated under target directory. The NAR package is the one you used for
$ ls target/pulsar-io-kafka-2.5.0-SNAPSHOT.nar target/pulsar-io-kafka-2.5.0-SNAPSHOT.nar
An example yaml config is available here
This example yaml config is used for copying json data from Kafka topic input-topic to Pulsar topic output-topic and save the messages in AVRO format.
-
Download Pulsar 2.4.0 release from Pulsar website and follow the instructions to start a standalone Pulsar. Assume PULSAR_HOME is the home directory for your Pulsar installation for the remaining steps.
Example command to start a standalone Pulsar.
cd ${PULSAR_HOME} bin/pulsar standalone
-
Download Kafka release from Kafka website and follow the instructions to start a Kafka server. Assume KAFKA_HOME is the home directory for your Kafka installation for the remaining steps.
Example commands to start a Kafka server.
cd ${KAFKA_HOME} bin/zookeeper-server-start.sh config/zookeeper.properties bin/kafka-server-start.sh config/server.properties
-
Create a Kafka topic.
Make sure the Kafka topic name is the one you configured in
pulsar-io-kafka.yaml
.bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --create --topic input-topic --replication-factor 1 --partitions 4
-
Copy the pulsar-io-kafka conenctor to
${PULSAR_HOME}/connectors
directory.cd ${PULSAR_HOME} mkdir -p connectors cp ${PULSAR_IO_KAFKA_HOME}/target/pulsar-io-kafka-2.5.0-SNAPSHOT.nar connectors/
-
Localrun the pulsar-io-kafka connector.
NOTE:
--destination-topic-name
is used by pulsar io runtime but not by thispulsar-io-kafka
connector. We can not omit this setting at this momement. So you can given any unused topic name for now.cd ${PULSAR_HOME} bin/pulsar-admin sources localrun -a connectors/pulsar-io-kafka-2.5.0-SNAPSHOT.nar --tenant public --namespace default --name test-kafka-source --source-config-file ${PULSAR_IO_KAFKA_HOME}/conf/pulsar-io-kafka.yaml --destination-topic-name test-kafka-source-topic
Once the connector is running, keep this terminal open during the remaining steps.
-
Now we can use a json kafka producer and an avro pulsar consumer to verify if the connector is working as expected.
Start a json Kafka producer to produce 100 messages.
cd ${PULSAR_IO_KAFKA_HOME} bin/kafka-json-producer.sh localhost:9092 input-topic 100
Start an avro Pulsar consumer to consume the 100 messages (in avro format).
cd ${PULSAR_IO_KAFKA_HOME} bin/pulsar-avro-consumer.sh pulsar://localhost:6650 output-topic sub
You will see similar output in the terminal you run
pulsar-avro-consumer.sh
:Receive message : key = user-99, value = User(name=user-99, age=990, address=address-99)