syslog-ng is an enhanced logging daemon, with a focus on central log collection. It collects logs from many different sources, processes and filters them and forwards them to a destination. This session focuses on how syslog-ng parses important information from incoming messages, how to send this information to “big data” destinations, like HDFS, Kafka, ElasticSearch or MongoDB.
Abstract: syslog-ng is an enhanced logging daemon, with a focus on central log collection. It collects logs from many different sources.
Raw log messages come in a variety of formats:
- lacking any structure most are usually just an almost proper English sentence with some variable parts in it, like user names or IP addresses.
- Fix table-like structure, like Apache access logs.
- A small minority of logs arrive in an already structured form: JSON.
Parsers in syslog-ng make it possible to extract important information from any of these messages and create name-value pairs.Once you have name-value pairs instead of raw log messages, you have many possibilities. On the syslog-ng side, you can use them for filtering, for example, to send an alert if the username is “root”. You can also use them in file names, or messages can be modified to facilitate log rotation or better suit applications processing the logs.
Parsing and preprocessing log messages also allows you to store them more effectively:
- you can send them to the destination (for example, ElasticSearch or MongoDB) in a format that can be easy to process (for example, JSON),
- you can filter irrelevant data, and forward only what is really needed,
- processing is off-loaded to very effective C code.
Finally you will learn about the “big data” destinations that syslog-ng supports, and how they benefit from message parsing:
- Hadoop Distributed File System ( HDFS ),
- Apache Kafka,
- ElasticSearch and Kibana, and
And if syslog-ng cannot already do something that you need, and you are not afraid of writing some code, you can learn about how language bindings of syslog-ng make it possible to add new destinations, not only in C, but also in Java, Lua, Perl, or Python.
Bio: Peter Czanik is community manager at Balabit, developers of syslog-ng. He helps distributions to maintain the syslog-ng package, follows bug trackers, helps users and talks regularly at conferences (FOSDEM, Libre Software Meeting, LOADays, Scale, etc.) In his limited free time he is interested in non-x86 architectures, and works on one of his PPC or ARM machines.