RSYSLOG is a next generation log processing tool. In the frist part, we will explain the new RSYSLOG v8 engine, its motivation and its benefits. Learn, for example, why writing to Elasticsearch is much faster with the new engine. We will describe the tuning parameters vital for making best use of the new features. In the second part we will explain how to write RSYSLOG plugins in any language. Traditionally, writing rsyslog plugins has been considered quite hard, with at least C knowledge necessary. In v8, we have introduced new interfaces which make it possible to write plugins in any language - be it Python, Perl or Java. Even bash will do. In essence, this is a great tool for any admin to add special needs with just a bit of scripting. We will proivde concrete instructions on how to write a plugin, point to read-to-copy samples and tell how to integrate this into rsyslog. NOTE: This is my LinuxTag Berlin 2014 talk.

1. rsyslog v8 improvements
and plugins in ANY language
Rainer Gerhards

2. First things first:
Version check :-)
• Current stable is 8.2.x
• Don't use outdated cruft like 5.8.x (!)
• Packages available at
http://www.rsyslog.com

3. What's in this talk?
• v8 improvements
▫ Infrastructure things
▫ The v8 engine
• Writing plugins in languages other than C
• I will probably “run out of time” - but that's fine, the
slides at the end are optional.

4. The rsyslog doc project
• The doc

5. The rsyslog doc project
• The doc just sucks...
• Spawned a new project to create better one:
https://github.com/rsyslog/rsyslog-doc
• Initiated by James Boylan (a sysadmin)
• Please help
▫ Complain ;-)
▫ open issues
▫ Write some doc...
• We are especially interested to learn what is hard for
beginners!

6. Community involvement
• Rsyslog was traditionally open to contributions and
has a lively community
• Yet we still try to improve things and have made a
couple of moves to encourage further contributions
• Rsyslog & subprojects on github
• Project Admins are David Lang (not affiliated),
Tomas Heinrich (Red Hat) and me
• Subprojects with different commit levels

7. What is new in the v8 engine?
• Output part totally revamped
• Simplified script execution
▫ Even faster execution speed
▫ Less complex code
• Even higher scalability
• Global variable support
• Required changes to output module interface
• Support for plugins in any language

8. The v7 rule engine
rsyslog
core
Queue worker
Queue queue worker Action instance
Queue worker
Single-thread
compartment
 Filter processing
 Message formatting
 Actual output action, like sending msg
 Kept simple & single threaded
 Works well with fast actions
 Has problems with slow ones, e.g.
via HTTP (like Elasticsearch)

9. What was the problem with the v7
engine and slow outputs?
• Traditionally, an action instance is non-reentrant
• Multiple threads are used for
▫ Filtering
▫ Template generation
▫ Execution of independent actions
• But the SAME action could not benefit from the
thread pool
• Not a problem for traditional fast outputs

10. Why is this a problem e.g. for
Elasticsearch?
• Rsyslog-processing is NOT time-dominant
• Transfer via HTTP takes its time
▫ Need to wait on result
▫ Both ES and rsyslog mostly idle in this process
• ES request relatively slow, scaling via multiple
connections
• ES can process many requests concurrently
• With the new engine, we can spawn multiple
concurrent connections from the same
action, and thus keep both ES and rsyslog busy and
de-tangle it from the slow parts!

11. The v8 rule engine
rsyslog
core
Queue worker
Queue worker
Queue worker
Action wrkr inst.
queue
 Now multiple instances per action!
 Queue worker pool automatically
scales outbound connection count
by spawning more worker instances
 Works well with Elasticsearch etc.
 Inherently serial outputs (e.g. local files!)
must serialize themselves
Action wrkr inst.
Action wrkr inst.

12. What if the destination cannot
handle multiple workers?
• A typical example is file output
• Framework will still call into multiple workers
• Worker must use pData mutex to protect itself – this is
not done automatically!
• Basically like in v7, but plugin needs to take care
Action
instance
1
Action
instance
2
wrkr 1
wrkr 2
wrkr 1
wrkr 2
wrkr 3
“real” code
Action-instance mutexes
“real” code

13. Writing plugins
• Traditionally, plugins
▫ Are written in C
▫ Macros hide interface plumbing
▫ Fairly easy to write for the C-literate
▫ Still perceived as “complicated”
• V8 goal
▫ Enable everyone to write plugins (sysadmins!)
▫ Support any language (Python, Perl, ...)
▫ Ability to execute security-sensitive plugin out of
rsyslog security context

14. Types of Plugins
• Output (actions)
▫ Deliver message to some destination system, e.g. file,
ElasticSearch, MongoDB, Solr, ...
▫ Any language supported in v8.2.0+
• Message Modification Plugins (Modules)
▫ Permit on-the-fly modification of message content
(e.g. anonymization, credit card removal)
▫ Any language supported in v8.3.0+
• Input
▫ Accept input messages
▫ Currently being worked on (target 8.3.[3-5]+)

15. Rainer Gerhards
Writing external output plugins
for rsyslog

16. Interface Overview
rsyslog
core
engine
Internal plugin
external plugin
connector
process border
perl plugin
python plugin

17. External Interface Design Goals
• Keep it stupid simple
▫ Must support almost any language
▫ Dumb easy to use even for novice programmer
▫ Do not require explicit threading
• Speed is NOT the most important goal
▫ Don't make it unnecessarily slow
▫ Many real-world log destinations are slow in any case
(like when you connect via http...)
▫ Focus on “enable to build solution”
▫ If necessary, conversion to internal module can be
done later

18. Interface Details: communication
• use pipes
• stdin
▫ one message per line
▫ format can be customized via rsyslog templates
▫ multi-line messags via JSON
• stdout/stderr
▫ Must NOT be written in initial version
▫ Will later convey back state information via plain text
(e.g. “ERR”, “ERRMSG:xxx”, ...)
• Template specifies input format (with JSON
recommended for more complex cases)

19. Interface Details: Threading
• Do NOT care about threading
• Write app according to single-thread paradigm
• rsyslog will spawn multiple instances of your plugin
if there is need to do so
▫ Happens based on config in busy cases
▫ Works well in most cases (e.g. http connects)
▫ Can be disabled if necessary
▫ If your program can run in multiple ter-minal
sessions concurrently, it can also be run as
multiple rsyslog action instances.

20. Startup & Termination
• rsyslog will startup the plugin automatically
• Plugin needs to read stdin until EOF
• Do NOT terminate before EOF is reached
• On EOF, cleanup and terminate
• If the plugin dies, rsyslog restarts a new instance
• Some signals (like sigint) are blocked and should
remain so

21. Skeletons
• The rsyslog project provides sample plugin
skeletons
• Available in ./plugins/external/skeletons
• These contain
▫ the necessary plumbing
▫ often a kind of abstraction layer to make writing
plugins even easier
▫ often performance-enhancement features
• Can simply be copied to create your own plugins,
don't care about the (minimal) plumbing!

22. Call to Action
• If you need to send logs to a destination that is not
yet supported, you can quickly write an external
plugin – in any language you know!
• Writing rsyslog plugins is easy
▫ If there is already a skeleton for your language, copy it
and add your app-specific code
▫ If not ... no problem, the interface is dumb easy
If you can write a script that reads stdin and does
something useful with it, you can also write a
rsyslog plugin!

23. Rainer Gerhards
Writing external output plugins
for RSysLog
IN 2 MINUTES

24. Write the plugin itself
• Choose any language you like
• Implement the pseudocode below
▫ Messages arrive via stdin, one message per line
▫ Read from stdin until EOF
▫ Process each message read as you like
▫ Terminate when EOF is reached
• That's it!
While not EOF(stdin) do {
Read msg from stdin
Process msg
}

25. Make RsysLog call plugin
• Regular filtering applies (as with any action)
• You can specify message format via a template
• Use omprog for the call
module(load=”omprog”) # needed only once in config!
if $rawmsg contains “sometrigger” then
action(type=”omprog”
binary=”/path/to/your/plugin”)

26. Optional: debugging your plugin
• If something doesn't work, it's best to debug outside
of rsyslog
• Do this as you usually debug your programs (e.g. use
your favorite debugger!)
• For example, do
$ echo “testmessage” | /path/to/your/plugin
• Questions about the plugin interface or plugin
integration? Visit
http://kb.monitorware.com/external-plugins-f53.html

27. Questions?
rgerhards@adiscon.com
www.rsyslog.com
https://github.com/rsyslog