This document discusses the use of Fluentd as a unified logging layer. Fluentd can parse log data into a common format, decouple log collection from data sources, and provide reliability, scalability and extensibility as logs are ingested from various systems and sent to different outputs like search, databases and storage. It provides an example of using Fluentd to parse Apache web server logs and tag them with metadata before outputting. The document advocates that Fluentd can help bring together logs from multiple systems and services to support common needs like analytics, monitoring and incident response.

1. Fluentd:
Towards Unified
Logging Layer
Kiyoto Tamura
@kiyototamura
Treasure Data, Inc.

3. • Fluentd maintainer &
community manager
• data nerd
• math nerd
• nerd
whoami

5. this talk isn’t about
Big Data

6. it’s about
Log Data

7. a motivating anecdote

8. The life of
a data scientist
(me in 2009)

9. http://cacm.acm.org/blogs/blog-cacm/169199-data-science-workflow-overview-
and-challenges/fulltext

10. Acquire Data (or
so you think)
WUT!? Invalid
UTF8?
Fix the encoding
issue…
Yell at the
engineers
Some columns
are missing!?
Run the
script…DIVISION
BY ZERO!!!

11. Hmm…

12. Logging.priority
=> :not_super_high

13. analytics.priority
=> :very_high

14. analytics.needs? :logs
=> true

15. outage.priority
=> :super_high

16. outage.needs? :logs
=> [“no”, “shit”]

17. Metrics and Monitoring
(hint: you need logs)

19. Ops
VPs
Engineers
Managers
PMs
More PMs

21. How can we do better?

22. How to Unify Logging (1)
Common Interface + Decoupling
Mobile Web IoT
Message
Queue
Search
Backend
Analytic DB
Archival
Storage
Unified
Logging Layer
Parse into a
common data format
Decouple from
data sources

23. How to Unify Logging (2)
Reliability & Scalability
Mobile Web IoT
Message
Queue
Search
Backend
Analytic DB
Archival
Storage
Unified
Logging Layer
Need
persistence/buff
ering
Robust retries
and recovery

24. How to Unify Logging (3)
Extensibility
? Web IoT
? Search
Backend
Analytic DB
Archival
Storage
Unified
Logging Layer
Adding a new
in/output must be
easy
Same for filters

25. Fluentd can help us unify logging

26. how it works

28. 127.0.0.1 - - [05/Feb/2012:17:11:55
+0000] "GET / HTTP/1.1" 200 140 "-"
"Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/535.19 (KHTML, like Gecko)
Chrome/18.0.1025.5 Safari/535.19"

29. {
"host": "127.0.0.1",
"user": "-",
"method": "GET",
"path": "/",
"code": "200",
"size": "140",
"referer": "-",
"agent": “Mozilla/5.0 (Windows…"
}

31. Parse as JSON!

32. ?

33. [“05/Feb/2012:17:11:55”,“web.access”,{
"host": "127.0.0.1",
"user": "-",
"method": "GET",
"path": "/",
"code": "200",
"size": "140",
"referer": "-",
"agent": “Mozilla/5.0 (Windows…"
}]
timestamp tag
record

34. ?

35. web.mongodb
web.file
web.hdfs
web.s3
web.mysql

36. Demo: Bring Your Own A/B Testing

37. How A/B Testing Starts
website
<script>…</script>
A/B Testing
SaaS

38. How A/B Testing Evolves
Android iOS
<script>…</script>
A/B Testing
SaaS 1
website
A/B Testing
SaaS 1
A/B Testing
SaaS 1
<script>…</script>
A/B Testing
SaaS 1
event.post()…
<script>…</script>
event.post()…

39. How A/B Testing Evolves
Android iOS
<script>…</script>
A/B Testing
SaaS 1
website
A/B Testing
SaaS 1
A/B Testing
SaaS 1
<script>…</script>
A/B Testing
SaaS 1
event.post()…
<script>…</script>
event.post()…
Looks Familiar?

40. Bring Your Own A/B Testing!
Android website iOS
A/B Testing
SaaS 1
A/B Testing
SaaS 2
Analytic DB
Archival
Storage

41. bit.ly/cf-fluentd

42. {
“install”: “gem install fluentd”,
“website”: “www.fluentd.org”,
“github” : “fluent/fluentd”,
“twitter”: “@fluentd”
}