syslog-ng Kafka destination¶
Stable release¶
To install syslog-ng Kafka driver, run this command in your terminal:
$ pip install syslogng_kafka
This is the preferred method to install syslog-ng Kafka driver, as it will always install the most recent stable release.
If you don’t have pip installed, this Python installation guide can guide you through the process.
From sources¶
The sources for syslog-ng Kafka driver can be downloaded from the Github repo.
You can either clone the public repository:
$ git clone git://github.com/anguenot/syslogng_kafka
Or download the tarball:
$ curl -OL https://github.com/anguenot/syslogng_kafka/tarball/master
Once you have a copy of the source, you can install it with:
$ pip install -e .
Configure¶
First, let’s make sure that your syslog-ng instance can accept messages.
Start by editing the main configuration file:
$ sudo vim /etc/syslog-ng/syslog-ng.conf
Below is an example opening TCP and UDP port 514 on all interfaces:
[...]
source s_src {
system();
internal();
tcp(ip(0.0.0.0) port(514));
udp(ip(0.0.0.0) port(514));
};
[...]
Configure the syslog-ng Apache Kafka destination:
$ vim /etc/syslog-ng/conf.d/kafka.conf
Sample driver configuration with every possible options. See below for documentation:
destination syslog_to_kafka {
python(
class("syslogng_kafka.kafkadriver.KafkaDestination")
on-error("fallback-to-string")
options(
hosts("localhost:9092,localhost:9182")
topic("syslog")
partition("10")
msg_key("src_ip")
programs("firewall,nat")
broker_version("0.8.2.1")
verbose("True")
display_stats("True")
producer_config("{'client.id': 'sylog-ng-01', 'retry.backoff.ms': 100, 'message.send.max.retries': 5, 'queue.buffering.max.kbytes': 50240, 'default.topic.config': {'request.required.acks': 1, 'request.timeout.ms': 5000, 'message.timeout.ms': 300000}, 'queue.buffering.max.messages': 100000, 'queue.buffering.max.ms': 1000, 'statistics.interval.ms': 15000, 'socket.timeout.ms': 60000, 'retry.backoff.ms':100,}")
)
);
};
log {
source(s_src);
destination(syslog_to_kafka);
};
The available options are:
- hosts: Kafka bootstrap.servers. One or multiple coma separated
- topic: Topic to produce message to
- partition (optional): Partition to produce to, elses uses the configured partitioner.
- msg_key (optional): Message key
- programs (optional): filter messages by syslog program. One or multiple coma separeted
- broker_version (optional): default is ‘0.9.0.1’
- *verbose (optional): if wether or not to print messages in logs. False by default
- *display_stats (optional): if wether or not to print broker statistics in logs. False by default
- producer_config (optional): The supported configuration values are dictated by the underlying librdkafka C library. For the full range of configuration properties please consult librdkafka’s documentation: https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md
** DO NOT USE value-pairs as indicated in syslog-ng documentation as you will get huge memory leaks...**
Restart the syslog-ng service:
$ service syslog-ng restart
To start the service in the foreground and see errors:
$ syslog-ng -F
Ensure your syslog-ng server is ready to get messages:
$ netstat -tanpu | grep syslog
tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN 11297/syslog-ng
udp 0 0 0.0.0.0:514 0.0.0.0:* 11297/syslog-ng