Copyright (c) 2012-2022, Magnus Edenhill. 2023 Confluent Inc..
https://github.com/confluentinc/librdkafka
librdkafka is a C library implementation of the Apache Kafka protocol, providing Producer, Consumer and Admin clients. It was designed with message delivery reliability and high performance in mind, current figures exceed 1 million msgs/second for the producer and 3 million msgs/second for the consumer.
librdkafka is licensed under the 2-clause BSD license.
KAFKA is a registered trademark of The Apache Software Foundation and has been licensed for use by librdkafka. librdkafka has no affiliation with and is not endorsed by The Apache Software Foundation.
- Full Exactly-Once-Semantics (EOS) support
- High-level producer, including Idempotent and Transactional producers
- High-level balanced KafkaConsumer (requires broker >= 0.9)
- Simple (legacy) consumer
- Admin client
- Compression: snappy, gzip, lz4, zstd
- SSL support
- SASL (GSSAPI/Kerberos/SSPI, PLAIN, SCRAM, OAUTHBEARER) support
- Full list of supported KIPs
- Broker version support: >=0.8 (see Broker version compatibility)
- Guaranteed API stability for C & C++ APIs (ABI safety guaranteed for C)
- Statistics metrics
- Debian package: librdkafka1 and librdkafka-dev in Debian and Ubuntu
- RPM package: librdkafka and librdkafka-devel
- Gentoo package: dev-libs/librdkafka
- Portable: runs on Linux, MacOS X, Windows, Solaris, FreeBSD, AIX, ...
- Public API in C header and C++ header.
- Introduction and manual in INTRODUCTION.md.
- Configuration properties in CONFIGURATION.md.
- Statistics metrics in STATISTICS.md.
- Frequently asked questions.
- Step-by-step tutorial Getting Started with Apache Kafka and C/C++.
NOTE: The master
branch is actively developed, use latest release for production use.
On Mac OSX, install librdkafka with homebrew:
$ brew install librdkafka
On Debian and Ubuntu, install librdkafka from the Confluent APT repositories, see instructions here and then install librdkafka:
$ apt install librdkafka-dev
On RedHat, CentOS, Fedora, install librdkafka from the Confluent YUM repositories, instructions here and then install librdkafka:
$ yum install librdkafka-devel
On Windows, reference librdkafka.redist NuGet package in your Visual Studio project.
For other platforms, follow the source building instructions below.
You can download and install librdkafka using the vcpkg dependency manager:
# Install vcpkg if not already installed
$ git clone https://github.com/Microsoft/vcpkg.git
$ cd vcpkg
$ ./bootstrap-vcpkg.sh
$ ./vcpkg integrate install
# Install librdkafka
$ vcpkg install librdkafka
The librdkafka package in vcpkg is kept up to date by Microsoft team members and community contributors. If the version is out of date, please create an issue or pull request on the vcpkg repository.
The GNU toolchain
GNU make
pthreads
zlib-dev (optional, for gzip compression support)
libssl-dev (optional, for SSL and SASL SCRAM support)
libsasl2-dev (optional, for SASL GSSAPI support)
libzstd-dev (optional, for ZStd compression support)
libcurl-dev (optional, for SASL OAUTHBEARER OIDC support)
NOTE: Static linking of ZStd (requires zstd >= 1.2.1) in the producer
enables encoding the original size in the compression frame header,
which will speed up the consumer.
Use STATIC_LIB_libzstd=/path/to/libzstd.a ./configure --enable-static
to enable static ZStd linking.
MacOSX example:
STATIC_LIB_libzstd=$(brew ls -v zstd | grep libzstd.a$) ./configure --enable-static
./configure
# Or, to automatically install dependencies using the system's package manager:
# ./configure --install-deps
# Or, build dependencies from source:
# ./configure --install-deps --source-deps-only
make
sudo make install
NOTE: See README.win32 for instructions how to build on Windows with Microsoft Visual Studio.
NOTE: See CMake instructions for experimental CMake build (unsupported).
See getting Started with Apache Kafka and C/C++ for a basic tutorial.
-
Refer to the examples directory for code using:
- Producers: basic producers, idempotent producers, transactional producers.
- Consumers: basic consumers, reading batches of messages.
- Performance and latency testing tools.
-
Refer to the examples GitHub repo for code connecting to a cloud streaming data service based on Apache Kafka
-
Link your program with
-lrdkafka
(C) or-lrdkafka++
(C++).
Commercial support is available from Confluent Inc
Only the latest official release is supported for community members.
File bug reports and feature requests using GitHub Issues.
Questions and discussions are welcome on the Discussions forum, and on the Confluent Community slack #clients channel.
- C#/.NET: confluent-kafka-dotnet (based on rdkafka-dotnet)
- C++: cppkafka
- C++: modern-cpp-kafka
- Common Lisp: cl-rdkafka
- D (C-like): librdkafka
- D (C++-like): librdkafkad
- Erlang: erlkaf
- Go: confluent-kafka-go
- Haskell (kafka, conduit, avro, schema registry): hw-kafka
- Kotlin Native: Kafka-Kotlin-Native
- Lua: luardkafka
- Node.js: node-rdkafka
- OCaml: ocaml-kafka
- Perl: Net::Kafka
- PHP: php-rdkafka
- PHP: php-simple-kafka-client
- Python: confluent-kafka-python
- Python: PyKafka
- Ruby: Hermann
- Ruby: rdkafka-ruby
- Rust: rust-rdkafka
- Tcl: KafkaTcl
- Shell: kafkacat - Apache Kafka command line tool
- Swift: Perfect-Kafka
See Powered by librdkafka for an incomplete list of librdkafka users.