Want to make rsyslog talk to some not-yet-supported output? With v8 it's extremly simple.This presentation contains everything you need to know about writing great plugins in ANY language (like perl or pyton).

1. Writing external output plugins
for rsyslog
Rainer Gerhards

2. Output plugins in rsyslog
• Output plugins permit to connect to a variety of log
destinations
• Pre-v8, plugins needed to be internal modules
▫ Loaded into rsyslog process space
▫ Must be written in C (or link to it)
▫ Deeply tied into rsyslog infrastructure
• with v8, we officially support external plugins
▫ Written in any language (Python, Perl, Java, ...)
▫ Have their own process
▫ Relatively de-coupled
Rainer Gerhards, http://blog.gerhards.net

3. Interface Overview
process border
Internal plugin
rsyslog
core
engine
perl plugin
external plugin
connector
python plugin
Rainer Gerhards, http://blog.gerhards.net

4. External Interface Design Goals
• Keep it stupid simple
▫ Must support almost any language
▫ Dumb easy to use even for novice programmer
▫ Do not require explicit threading
• Speed is NOT the most important goal
▫ Don't make it unnecessarily slow
▫ Many real-world log destinations are slow in any case
(like when you connect via http...)
▫ Focus on “enable to build solution”
▫ If necessary, conversion to internal module can be
done later
Rainer Gerhards, http://blog.gerhards.net

5. Interface Details: communication
• use pipes
• stdin
▫ one message per line
▫ format can be customized via rsyslog templates
▫ multi-line messags via JSON
• stdout/stderr
▫ Must NOT be written in initial version
▫ Will later convey back state information via plain text
(e.g. “ERR”, “ERRMSG:xxx”, ...)
• JSON as input format is recommended
Rainer Gerhards, http://blog.gerhards.net

6. Interface Details: Threading
• Do NOT care about threading
• Write app according to single-thread paradigm
• rsyslog will spawn multiple instances of your plugin
if there is need to do so
▫
▫
▫
▫
Happens based on config in busy cases
Works well in most cases (e.g. http connects)
Can be disabled if necessary
If your program can run in multiple ter-minal
sessions concurrently, it can also be run as
multiple rsyslog action instances.
Rainer Gerhards, http://blog.gerhards.net

7. Startup & Termination
•
•
•
•
•
•
rsyslog will startup the plugin automatically
Plugin needs to read stdin until EOF
Do NOT terminate before EOF is reached
On EOF, cleanup and terminate
If the plugin dies, rsyslog restarts a new instance
Some signals (like sigint) are blocked and should
remain so
Rainer Gerhards, http://blog.gerhards.net

8. Skeletons
• The rsyslog project provides sample plugin
skeletons
• Available in ./plugins/external/skeletons
• These contain
▫ the necessary plumbing
▫ often a kind of abstraction layer to make writing
plugins even easier
▫ often performance-enhancement features
• Can simply be copied to create your own plugins,
don't care about the (minimal) plumbing!
Rainer Gerhards, http://blog.gerhards.net

9. Example: Python Skeleton
• Handles all interface plumbing
• Uses “eventHandlers” to call the acutal app coding
• Is used as a basis, for example, for the Solr output
plugin
• Available at
https://github.com/rsyslog/rsyslog/blob/master/plugin
Rainer Gerhards, http://blog.gerhards.net

10. onInit() Handler
• Called when the plugin is initially loaded
• Ready outbound connections, files, etc...
def onInit():
""" Do everything that is needed to initialize processing
(e.g. open files, create handles, connect to systems...)
"""
global outfile
outfile = open("/tmp/logfile", "w")
Rainer Gerhards, http://blog.gerhards.net

11. onReceive()
• Called when new messages arrive
• Receives one or more message via Python list object
--> e.g. place them into one transaction, HTTP
request, ...
def onReceive(msgs):
"""This is the entry point where actual work needs to be done. It receives
a list with all messages pulled from rsyslog. The list is of variable
length, but contains all messages that are currently available. It is
suggest NOT to use any further buffering, as we do not know when the
next message will arrive. It may be in a nanosecond from now, but it
may also be in three hours...
"""
global outfile
for msg in msgs:
outfile.write(msg)
Rainer Gerhards, http://blog.gerhards.net

12. onExit()
• Called immediately before the plugin terminates
• Gurantees that no more messages arrive
• Used to cleanup connections, write some final
records, ...
def onExit():
""" Do everything that is needed to finish processing (e.g.
close files, handles, disconnect from systems...). This is
being called immediately before exiting.
"""
global outfile
outfile.close()
Rainer Gerhards, http://blog.gerhards.net

13. Call to Action
• If you need to send logs to a destination that is not
yet supported, you can quickly write an external
plugin – in any language you know!
• Writing rsyslog plugins is easy
▫ If there is already a skeleton for your language, copy it
and add your app-specific code
▫ If not ... no problem, the interface is dumb easy
If you can write a script that reads stdin and does
something useful with it, you can also write a
rsyslog plugin!
Rainer Gerhards, http://blog.gerhards.net