10. Fluentd & Event logs
Before:
App server App server App server
Application Application Application
File File File ... File File File ... File File File ...
File
High latency
must wait for a day
Log server Hard to analyze
complex text parsers
11. Fluentd & Event logs
After:
App server App server App server
Application Application Application
Fluentd Fluentd Fluentd
Realtime!
Fluentd Fluentd
13. # receive events via HTTP # save alerts to a file
<source> <match alert.**>
type http type file
port 8888 path /var/log/fluent/alerts
</source> </match>
# read logs from a file # forward other logs to servers
<source> # (load-balancing + fail-over)
type tail <match **>
path /var/log/httpd.log type forward
format apache <server>
tag apache.access host 192.168.0.11
</source> weight 20
</server>
# save access logs to MongoDB <server>
<match apache.access> host 192.168.0.12
type mongo weight 60
host 127.0.0.1 </server>
</match> </match>
14. Fluentd vs Scribe
⢠Deals with structured logs
⢠Easy to install
> âgem install ďŹuentdâ
> apt-get and yum
http://packages.treasure-data.com/
⢠Easy to customize
⢠add/modify plugins without re-compiling
> âgem search -rd ďŹuent-pluginâ
15. Fluentd vs Flume
⢠Easy to setup
> âsudo ďŹuentd --setup && ďŹuentdâ
⢠Very small footprint
> small engine (3,000 lines) + plugins
⢠JVM-free
⢠Easy to conďŹgure
29. Plugins
⢠Bundled plugins
> ďŹle
writes event logs to ďŹles hourly or daily
> forward
forwards event logs (+fail-over and load balancing)
> exec
passes event logs to/from external commands
> tail
reads event logs from a ďŹle (like `tail -f`)
30. Plugins
⢠3rd party plugins
> scribe
integrates Fluentd with Scribe
> s3
uploads log ďŹles to Amazon S3 hourly or daily
> mongo
writes logs to MongoDB
> hoop
puts log ďŹles on Hadoop HDFS via Hoop
...
31. Plugin developer API
⢠Unit test framework (like âMRUnitâ)
> Fluent::Test::InputTestDriver
> Fluent::Test::OutputTestDriver
> Fluent::Test::BufferedOutputTestDriver
⢠Fluent::TailInput (base class of âtailâ plugin)
> text parser is customizable
def parse_line(line)