1. Fluentd
Structured logging
Pluggable architecture
Reliable forwarding
e Event Collector Service
Sadayuki Furuhashi
Treasure Data, Inc.
@frsyuki

2. • Sadayuki Furuhashi
> twitter: @frsyuki
• Treasure Data, Inc.
> Software Engineer; founder
• Author of MessagePack
• Author of Fluentd

3. What’s Fluentd?
It's like syslogd, but uses JSON for log messages

4. What’s Fluentd?
Application
2012-02-04 01:33:51
myapp.buylog {
Fluentd
“user”: ”me”,
“path”: “/buyItem”,
“price”: 150,
“referer”: “/landing”
Storage }

5. What’s Fluentd?
time
Application tag
2012-02-04 01:33:51
myapp.buylog {
Fluentd
“user”: ”me”,
“path”: “/buyItem”,
“price”: 150,
“referer”: “/landing”
Storage } record

6. What’s Fluentd?
Application
Fluentd lter / bu er / routing
Storage

7. What’s Fluentd?
Application
Fluentd lter / bu er / routing
SaaS Storage Fluentd
Plug-in Plug-in Plug-in

8. What’s Fluentd?
syslogd Scribe Application File Plug-in
tail
Plug-in Plug-in
Fluentd lter / bu er / routing
SaaS Storage Fluentd
Plug-in Plug-in Plug-in

9. What’s Fluentd?
• Client libraries
> Ruby
> Perl Application
> PHP
> Python
> Java Fluentd
> ...
Fluent.open(“myapp”)
Fluent.event(“login”, {“user”=>38})
#=> 2012-02-04 04:56:01 myapp.login {“user”:38}

10. Fluentd & Event logs
Before:
App server App server App server
Application Application Application
File File File ... File File File ... File File File ...
File
High latency
must wait for a day
Log server Hard to analyze
complex text parsers

11. Fluentd & Event logs
After:
App server App server App server
Application Application Application
Fluentd Fluentd Fluentd
Realtime!
Fluentd Fluentd

12. Fluentd & Event logs
Fluentd Fluentd Fluentd
Realtime!
Fluentd Fluentd
Hadoop Amazon Ready to
MongoDB
/ Hive S3 / EMR Analyze!

13. # receive events via HTTP # save alerts to a file
<source> <match alert.**>
type http type file
port 8888 path /var/log/fluent/alerts
</source> </match>
# read logs from a file # forward other logs to servers
<source> # (load-balancing + fail-over)
type tail <match **>
path /var/log/httpd.log type forward
format apache <server>
tag apache.access host 192.168.0.11
</source> weight 20
</server>
# save access logs to MongoDB <server>
<match apache.access> host 192.168.0.12
type mongo weight 60
host 127.0.0.1 </server>
</match> </match>

14. Fluentd vs Scribe
• Deals with structured logs
• Easy to install
> “gem install fluentd”
> apt-get and yum
http://packages.treasure-data.com/
• Easy to customize
• add/modify plugins without re-compiling
> “gem search -rd fluent-plugin”

15. Fluentd vs Flume
• Easy to setup
> “sudo fluentd --setup && fluentd”
• Very small footprint
> small engine (3,000 lines) + plugins
• JVM-free
• Easy to configure

16. Architecture of Fluentd

17. Architecture :: Input

18. Architecture :: Bu er

19. Architecture :: Output

20. Plugins :: out_forward
forward event logs
Fluentd
Heartbeat
out_forward
in_forward
Fluentd Fluentd
✓ load balancing

21. Plugins :: out_forward
forward event logs
Fluentd
Heartbeat
out_forward
φ accrual failure detector
in_forward
Fluentd Fluentd
✓ load balancing

22. Plugins :: out_copy
duplicate event logs
Fluentd
out_copy
out_mongo out_forward
out_ le
MongoDB File Fluentd

23. Plugins :: buf_ le
reliable bu ering
buf_ le
Fluentd le
le
✓ Automatic retry le
✓ 2^N retry interval le
✓ Persistent bu er

24. Plugins :: out_exec
execute external programs
Fluentd
out_exec
TSV → stdin
✓ Python
external
✓ Perl
program
✓ C++

25. Plugins :: out_exec_ lter
execute external programs
out_exec_ lter
stdin external
Fluentd
stdout program
out_exec
TSV → stdin
✓ Python
external
✓ Perl
program
✓ C++

26. Plugins :: in_exec
execute external programs
in_exec out_exec_ lter
external stdout stdin external
Fluentd
program stdout program
out_exec
TSV → stdin
✓ Python
external
✓ Perl
program
✓ C++

27. Plugins :: in_tail
Read event logs from a le
Application
File /var/log/access.log
✓ Apache log parser
in_tail
✓ Syslog parser
✓ Custom parser
Fluentd

28. Plugins :: in_tail
Apache log parser
87.12.1.87 - - [04/Feb/2012:00:20:11 +0900] "GET / HTTP/1.1" 200 98
87.12.1.87 - - [04/Feb/2012:00:20:11 +0900] "GET / HTTP/1.1" 200 98
...
{
“host”: “87.12.1.87”,
“method”: “GET”,
“code”: 200,
“size”: 98,
“path”: “/”
}
...

29. Plugins
• Bundled plugins
> file
writes event logs to files hourly or daily
> forward
forwards event logs (+fail-over and load balancing)
> exec
passes event logs to/from external commands
> tail
reads event logs from a file (like `tail -f`)

30. Plugins
• 3rd party plugins
> scribe
integrates Fluentd with Scribe
> s3
uploads log files to Amazon S3 hourly or daily
> mongo
writes logs to MongoDB
> hoop
puts log files on Hadoop HDFS via Hoop
...

31. Plugin developer API
• Unit test framework (like “MRUnit”)
> Fluent::Test::InputTestDriver
> Fluent::Test::OutputTestDriver
> Fluent::Test::BufferedOutputTestDriver
• Fluent::TailInput (base class of “tail” plugin)
> text parser is customizable
def parse_line(line)

32. Fluentd
• Documents
> http://fluentd.org
• Source code
> http://github.com/fluent
• Twitter
> #fluentd
• Mailing list
> http://groups.google.com/group/fluentd

34. “BIG DATA ANALYTICS PLATFORM”
as a Service

37. Fluentd & Treasure Data
Fluentd Fluentd Fluentd
Realtime!
Fluentd Fluentd
Hadoop
MongoDB
Amazon Ready to
/ Hive S3 / EMR Analyze!

38. Fluentd & Treasure Data
Fluentd Fluentd Fluentd
Realtime!
Fluentd Fluentd
Treasure Data Ready to
Cloud Platform Analyze!

39. Fluentd & Treasure Data
Treasure Data
Cloud Platform
SQL Visualization
SELECT users.age, COUNT(1)
FROM logs
LEFT JOIN users ON
logs.user_id = users.id
GROUP BY users.age
WHERE path = “/buyItem”

41. Contacts :
sales@treasure-data.com