Fluentd meetup 2015 Summer LT

1. Docker and Fluentd
2015/06/01
Fluentd meetup 2015 Summer
Satoshi Tagomori (@tagomoris)

2. @tagomoris
Treasure Data, Inc.
Fluentd, Norikra, Hadoop, ...

3. Announce:
fluent-plugin-secure-forward
Security issue: v0.2.x or earlier
http://tagomoris.tumblr.com/post/120090873793/vulnerability-and-update-of-secure-forward-plugin
http://tagomoris.hatenablog.com/entry/2015/05/28/182245
Please update to v0.3.2!
v0.3.2:
Updated w/ new config param set / code
Incompatible protocol & configuration :(

5. Which more cute than another?

6. Docker and Fluentd
Docker:
> container platform what you love :D
Fluentd:
> logging platform what you love :D

7. Logging of container architecture
Storage:
> should be outside of container / hosts
Transferring:
> should be over network
Aggregation:
> should be done per container / per service

8. Aggregation Patterns
(1) 1-level aggregation (2) 2-level aggregation

9. Aggregation Patterns
(1) 1-level aggregation (2) 2-level aggregation
pros:
simple configuration
cons:
fixed aggr. address
many connections
high load for aggregator
pros:
less connections
lower load for aggr.
cons:
a bit more container

10. Aggregation Patterns
(1) 1-level aggregation (2) 2-level aggregation
Apps sends logs
over fluent-logger
there're some patterns...
or ...

11. Pattern of 2-level aggr. w/ Fluentd
1. Network transferring
2. Container log & tail

12. (1) Network transferring
Apps sends logs:
> using fluent-logger-*
> over inter-container network
Pros:
> no performance penalty
Cons:
> logs of docker is out of scope
> a bit complex apps for logging
over TCP
app

13. (2) Container logger & tail
Apps write logs to STDOUT:
> docker writes it to log file
> fluentd container read file
Pros:
> simple conf for apps & docker
> logs include container logs
Cons:
> in_tail performance penalty
log file
app
http://www.fluentd.org/guides/recipes/docker-logging

14. Logging drivers
New from docker v1.6!
'Add new Logging driver "fluentd"'
> --log-driver=fluentd
> https://github.com/docker/docker/pull/12876
> New for docker v1.7.0?

15. over TCP
(3) Container logging driver "fluentd"
Apps write logs to STDOUT:
> docker send it to fluentd
directly!
Pros:
> simple conf for apps & docker
> logs include container logs
Cons:
> none?
app
Stay tune!

16. Fluentd docker image
Official image by fluent organization
https://registry.hub.docker.com/u/tagomori/fluentd/
(it should be "fluent/fluentd" ...)
Use it as it is, or build your own container!
https://github.com/fluent/fluentd-docker-image

17. ❤