librdkafka installation

syslogng_kafka depends on the confluent-kafka lib:

confluent-kafka requires librdkafka >= 0.11.0 which is currently not installed with mainstream Linux distribution.

librdkafka is a C library implementation of the Apache Kafka protocol, containing both Producer and Consumer support. It was designed with message delivery reliability and high performance in mind, current figures exceed 1 million msgs/second for the producer and 3 million msgs/second for the consumer.

DEB packages via apt

Confluent maintains apt repositories that provide packages for Debian-based Linux distributions such as Debian and Ubuntu and the installation is documented here

First install Confluent’s public key, which is used to sign the packages in the apt repository:

Below are extracts from that page if you are in a hurry:

$ wget -qO - http://packages.confluent.io/deb/3.3/archive.key | sudo apt-key add -

Add the repository to your /etc/apt/sources.list:

$ sudo add-apt-repository "deb [arch=amd64] http://packages.confluent.io/deb/3.3 stable main"

Pin down the version of librdkafka to 0.11.x

$ sudo vim /etc/apt/preferences.d/confluent-librdkafka

Then add the content below:

Package: librdkafka1
Pin: origin "packages.confluent.io"
Pin: version 0.11.*
Pin-Priority: 550

Package: librdkafka-dev
Pin: origin "packages.confluent.io"
Pin: version 0.11.*
Pin-Priority: 550

Package: librdkafka++1
Pin: origin "packages.confluent.io"
Pin: version 0.11.*
Pin-Priority: 550

Then update and install:

$ sudo apt-get update
$ sudo apt-get install librdkafka1 librdkafka-dev

Note, we need to install the -dev package so that pip can compile confluent-kafka

From source

Alternatively, you can install librdkafka from source using the script included in this repository:

$ sudo bash tools/bootstrap-librdkafka.sh ${LIBRDKAFKA_VERSION} /usr/local
$ sudo ldconfig -vvv

This actual script is used by the Travis CI integration tests