librdkafka installation¶
syslogng_kafka depends on the confluent-kafka lib:
confluent-kafka requires librdkafka >= 0.11.0 which is currently not installed with mainstream Linux distribution.
librdkafka is a C library implementation of the Apache Kafka protocol, containing both Producer and Consumer support. It was designed with message delivery reliability and high performance in mind, current figures exceed 1 million msgs/second for the producer and 3 million msgs/second for the consumer.
DEB packages via apt¶
Confluent maintains apt repositories that provide packages for Debian-based Linux distributions such as Debian and Ubuntu and the installation is documented here
First install Confluent’s public key, which is used to sign the packages in the apt repository:
Below are extracts from that page if you are in a hurry:
$ wget -qO - http://packages.confluent.io/deb/3.3/archive.key | sudo apt-key add -
Add the repository to your /etc/apt/sources.list:
$ sudo add-apt-repository "deb [arch=amd64] http://packages.confluent.io/deb/3.3 stable main"
Pin down the version of librdkafka to 0.11.x
$ sudo vim /etc/apt/preferences.d/confluent-librdkafka
Then add the content below:
Package: librdkafka1
Pin: origin "packages.confluent.io"
Pin: version 0.11.*
Pin-Priority: 550
Package: librdkafka-dev
Pin: origin "packages.confluent.io"
Pin: version 0.11.*
Pin-Priority: 550
Package: librdkafka++1
Pin: origin "packages.confluent.io"
Pin: version 0.11.*
Pin-Priority: 550
Then update and install:
$ sudo apt-get update
$ sudo apt-get install librdkafka1 librdkafka-dev
Note, we need to install the -dev package so that pip can compile confluent-kafka
From source¶
Alternatively, you can install librdkafka from source using the script included in this repository:
$ sudo bash tools/bootstrap-librdkafka.sh ${LIBRDKAFKA_VERSION} /usr/local
$ sudo ldconfig -vvv
This actual script is used by the Travis CI integration tests