Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

Rsyslog vs Systemd Journal Presentation

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Wird geladen in …3
×

Hier ansehen

1 von 39 Anzeige

Rsyslog vs Systemd Journal Presentation

Do rsyslog and the journal cooperate? If so, how? This is the presentation from the LinuxTag 2013 conference. It details the rsyslog team's current position on the journal, how it affected rsyslog, what is being done for integration and some notes about how to configure rsyslog to do things that the journal announcement claimed to be impossible.

Do rsyslog and the journal cooperate? If so, how? This is the presentation from the LinuxTag 2013 conference. It details the rsyslog team's current position on the journal, how it affected rsyslog, what is being done for integration and some notes about how to configure rsyslog to do things that the journal announcement claimed to be impossible.

Anzeige
Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Ähnlich wie Rsyslog vs Systemd Journal Presentation (20)

Anzeige

Aktuellste (20)

Anzeige

Rsyslog vs Systemd Journal Presentation

  1. 1. rsyslog vs journal? Rainer Gerhards
  2. 2. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Me & the Talk • Rainer Gerhards ▫ Data center guy ▫ Involved 15+ years in logging ▫ Founded rsyslog in 2003 • The talk ▫ Will rsyslog fight the journal? ▫ Some history on journal-like system ▫ Ways of integration ▫ How to do things the journal announcement claimed as impossible
  3. 3. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Does journal replace syslog? • The initial announcement sounded a bit in that way, or was at least interpreted by most (including me) in that direction. • Looking at how things have evolved ▫ There of course is overlap between both systems ▫ But there are also (large) regions that do not overlap • This is not a new situation, there is some history lesson...
  4. 4. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Windows Event Log! • The Windows Event Log is in many ways similar to systemd journal ▫ Binary database with rollover and fast access time ▫ uses a simple structured format that captures core metadata items (like timestamps, user IDs, …) ▫ uses unique identifiers for different types of log messages ▫ Files are especially secured by OS
  5. 5. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Event Log History • Introduced with Windows NT 3.1 in 1993 • Greatly enhanced in 2007, starting with Windows Vista • Originally single-computer only • Now provides network functionality ▫ EventLog-to-EventLog push and pull subscriptions ▫ Can be used to setup log forwarding in the enterprise
  6. 6. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany So what does history tell us? • If such a system can totally replace syslog, there should be no syslog on Windows at all – and never have been. • Well... there are ample of applications ▫ WinSyslog (initial version by me, 1996) ▫ Kiwi Syslog (Solarwinds) ▫ EventReporter (first ever Windows-to-syslog tool, 1997) ▫ Snare ▫ and many more!
  7. 7. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Obviously, there must be some need to syslog technology... • Face it: syslog is the lingua franca of network event logging. ▫ If you want to process messages from different sources, chances are high you will need it. ▫ Even if not syslog (protocol) is used, you usually need some common denominator  e.g. Linux does not understand native Windows EventLog  Windows neither does understand native journal
  8. 8. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany A key problem solved by syslog • You want to integrate all of your systems into a consolidated log • This either means ▫ A common protocol ▫ A system that is capable of processing multiple protocols and somehow “normalize” them • Syslog is ubiquitous – because a basic client is dumb easy to implement!
  9. 9. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Window as a sender... • Early days: missing network functionality was a problem; brought up the idea of Event Log forwarding • Big customers quickly adopted that for integration into their management system • Today's hot topics: ▫ local filtering and preprocessing ▫ Ability to extract and properly express OS objects ▫ Support all Windows capabilities ▫ Secure protocol choices
  10. 10. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Windows as a receiver... • Windows acts as syslog server • Messages are written to ▫ Local files ▫ Windows Event Log (!) ▫ Some other processing (like alerting) • Typical deployment scenario for SOHO • But some large Windows-only shops also use it for integration of non-Windows sources
  11. 11. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Why I am talking so much about Windows? • As I said, I see strong similarities between journal and Windows Event Log • Except that journal has much more quickly gotten some network functionality • So my best guess is that deployments and end- user needs will evolve into mostly the same directions
  12. 12. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Journal vs. Syslog: low end systems • Usually users of these machines are not at all interested in logging • Journal is very convenient as a troubleshooting tool • Works perfect on personal desktop & notebook • Rsyslog will be needed by some users to integrate e.g. their DSL router's messages into the journal
  13. 13. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Journal vs. Rsyslog: enterprise systems • Impossible to manage without any syslog • Journal integrated as another event source ▫ Journal-centric  As much as possible is done with journal  Integration happens at central head server(s) ▫ Syslog-centric  Journal is used only as much as unavoidable  Each machine runs rsyslog and forwards events ▫ Mode depends on end-user's philosophy
  14. 14. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany How did the journal affect the rsyslog project? • Obviously, we expect less presence on low-end systems • So we re-focussed the project ▫ Previously low-end and enterprise needs were equal peers ▫ Now strong focus on enterprise • The logging world at large got benefit as suddenly everyone was interested in logging – which also helps rsyslog!
  15. 15. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany What have we done to integrate with the journal? • Module omjournal ▫ Provides ability to store messages into the journal ▫ Traditional syslog, text files, ... ▫ Caters for the low-end use case • Module imjournal ▫ Provides ability to pull messages off the journal, just as another event source ▫ Contributed by Red Hat ▫ Caters for the enterprise use case
  16. 16. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany
  17. 17. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Integrating syslog Data into the journal (SOHO env) /* first, we make sure all necessary modules are present: */ module(load="imudp") # input module for UDP syslog module(load="omjournal") # output module for journal /* then, define the actual server that listens to the * router. Note that 514 is the default port for UDP syslog. */ input(type="imudp" port="514" ruleset="writeToJournal") /* inside that ruleset, we just write data to the journal: */ ruleset(name="writeToJournal") { action(type="omjournal") }
  18. 18. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Integrating journal data into syslog module(load="imjournal" PersistStateInterval="100" StateFile="/path/to/file") module(load="mmjsonparse") #load mmjsonparse module for structured logs $template CEETemplate,"%TIMESTAMP% %HOSTNAME% %syslogtag% @cee: %$!all- json%n" #template for messages *.* :mmjsonparse: *.* /var/log/ceelog;CEETemplate • Necessary to obtain extended journal properties • If not needed, regular system log socket can be used
  19. 19. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Why is it simple to integrate the journal? • Rsyslog is actually ▫ A message router ▫ With dynamically loadable inputs and outputs ▫ Highly configurable • So, journal support is as easy as adding some new inputs and outputs! • The rest of the plumbing is already there.
  20. 20. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany rsyslog Architecture journal Network (e.g.TCP) /dev/log file Database Remote system Parsers For- matter Rules & Filters
  21. 21. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany “String” Handling in rsyslog Message String Parser Dictionary of Name/Value Pairs Formatter Output String “special” Outputs Modifi- cation Mod
  22. 22. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Now let's look at some “impossible” things • The original journal paper claimed that syslog is ▫ Seriously broken ▫ Cannot provide some important features • I'll show how to do these “impossible” things ▫ Based on 2011 technology ▫ And on current one (v7.4)
  23. 23. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Log File Manipulation Protection • The traditional approach is to ship logs off the machine, to a central and highly secured system • Keeping them on a system that is “easily compromised” is asking for trouble. • Problem is that local secrets can always be compromised • In rsyslog 7.4, we address these problems via log signatures and encryption...
  24. 24. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Signed Log Records • In 2011, there was no good solution (and journal's solution was also not good) • Things have evolved since them ▫ Journal got “forward secure sealing” ▫ Rsyslog  got a crypto provider interface and a provider for “Keyless Signature Infrastructure” (KSI)  Hash chain for log record is created, and key hashes are chained in a global hash chain, which provides signature & timestamp
  25. 25. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Signing via Hash Chains... • Very rough sample (actually Merkle trees!) • No local secret! • Consider “chain layer” to be operated on a schedule (timer ticks!) Source: http://en.wikipedia.org/wiki/File:Hashlink_timestamping.svg
  26. 26. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Activating Log Signing action(type="omfile" file="/var/log/logfile" sig.provider="gt" sig.keepTreeHashes="on" sig.keepRecordHashes="on") • Parameters except sig.provider are optional • Writes ▫ regular log file ▫ plus signature file (*.gtsig)
  27. 27. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Log File Encryption • Crypto-Provider interface recently added • As well as a libgcrypt-based crypto provider • Symmetric cryptoraphy, all ciphers & modes supported by libgcrypt • Key can come from ▫ Config param (testing only, pls!) ▫ File ▫ Script (interface for advanced key exchange options)
  28. 28. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Activating Log Encryption action(type="omfile" file="/var/log/logfile" cry.provider="gcry" cry.keyprogram=”/path/to/binary”) • Addtl Parameters for ciphers, etc... • Writes ▫ regular log file, encrypted ▫ plus encryption info file (*.encinfo) • Works in conjunction with signatures • In 7.5 extended to rsyslog disk queues!
  29. 29. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Syslog Network Processing • Original Journal accouncement missed improvments and talked only about UDP syslog • We have ▫ TCP & TLS support (RFC5425, 2009) ▫ Mutual authentication & authorization ▫ Multiple hops ▫ Buffered send queues, even with disk buffers ▫ Rsyslog can utilize other protocols as well (RELP, SNMP)!
  30. 30. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Sample: TLS-encrypted for- warding using a buffer queue $DefaultNetstreamDriver gtls # make gtls driver the default # certificate files $DefaultNetstreamDriverCAFile /rsyslog/protected/ca.pem $DefaultNetstreamDriverCertFile /rsyslog/protected/ma-cert.pem $DefaultNetstreamDriverKeyFile /rsyslog/protected/ma-key.pem # authorization $ActionSendStreamDriverAuthMode x509/name $ActionSendStreamDriverPermittedPeer central.example.net $ActionSendStreamDriverMode 1 # run driver in TLS-only mode # Queue $ActionQueueType LinkedList # use own queue $ActionQueueFileName fwq # set file name, enable disk mode *.* @@central.example.net:10514 # forward to remote server
  31. 31. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Indexed Store • Core idea: connect to existing “life stores”, do not push that part into the syslogd ▫ Actually one of the core requirements that started rsyslog (MySQL, ~2004)! • 2011: various relational databases • Today also ▫ Document based databases (MongoDB,...) ▫ Elasticsearch ▫ Journal DB
  32. 32. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Rate-Limiting • 2011 ▫ Repeated message reduction ($RepeatedMsgReduction on) ▫ Output throttling • V7, additionally ▫ Repeated message processing on a per-input basis ▫ Object-based rate limiters (n messages within s seconds)
  33. 33. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Log compression • Log files can be zip compressed by specifying the “ZipLevel” parameter • Of course, accessing compressed log records requires more processing time. • Today, we also have experimental code for log transfer compression (directly built into the protocol/output modules).
  34. 34. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Message authentication and Metadata availablility • Rsyslog uses the same SCM_CREDENTIALS facility that journal does • And in both cases it can be faked – as journal demonstrates when it actually fakes it on the system log socket ;) • The volume of metadata available has been increased starting in 2012 • Total authenticity requires signatures at the original originator level (each app), what currently is impossible in the *nix framework.
  35. 35. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Free-Formedness of Log Records • Traditional syslog messages are much like free- form text • Today, we see the same for typical journal messages • There are a couple of standardization efforts underway to provided structured logging • Project lumberjack (lead by Red Hat) provides JSON-based structured logs
  36. 36. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Unstructured Text Log Duality • If a log format does not support freeform-text, it is not used (at least not more than one can avoid to...) • If it supports freeform-text (among others), that freeform-text will be abused • → unstructured logs won't go away! • We've seen this in Windows Event Log and looks much the same for journal.
  37. 37. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Converting Free-Text Messages via mmnormalize • Uses a “sample rule base” ▫ One sample for each expected message type ▫ Sample contains text (for matching) and property descriptions (like IPv4 Address, char-matches, …) ▫ If sample matches, corresponding properties are extracted ▫ Special parser for iptables • Very fast algorithm (much faster than regex) • Based on liblognorm (which can also be used in other programs to gain this functionality!)
  38. 38. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Timestamp format • No year, no timezone • ... just because distros turn it off. • Remove “$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat” from rsyslog.conf to get rsyslog's default high- precision RFC5424 timestamp • Some tools may have problems with that, but can't be too bad – some Distros use the default format
  39. 39. Rainer Gerhards * http://blog.gerhards.net * LinuxTag 2013, Berlin, Germany Questions? • rgerhards@adiscon.com • http://blog.gerhards.net • http://www.rsyslog.com • http://www.adiscon.com • Associated paper is available on SlideShare.

×