Presentation given at LinuxCon Japan 2016 by Satoshi "Moris" Tagomori (@tagomoris), Treasure Data. Describes various strategies for aggregating log data in a microservices architecture using containers, e.g. Docker.
5. Topics
• Microservices and logging in various industries
• Difficulties of logging with containers
• Distributed logging architecture
• Patterns of distributed logging architecture
• Case Study: Docker and Fluentd
7. Logging in Various Industries
• Web access logs
• Views/visitors on media
• Views/clicks on Ads
• Commercial transactions (EC, Game, ...)
• Data from devices
• Operation logs on Apps of phones
• Various sensor data
8. Microservices and Logging
• Monolithic service
• a service produces all data
about an user's behavior
• Microservices
• many services produce data
about an user's access
• it's needed to collect logs
from many services to know
what is happening
Users
Service (Application)
Logs
Users
Logs
10. Containers:
"a must" for microservices
• Dividing a service into services
• a service requires less computing resources
(VM -> containers)
• Making services independent from each other
• but it is very difficult :(
• some dependency must be solved even in
development environment
(containers on desktop)
11. Redesign Logging: Why?
• No permanent storages
• No fixed physical/network address
• No fixed mapping between servers and roles
• We should parse/label logs at the source, ship
these logs by pushing to destination ASAP
12. Containers:
immutable & disposable
• No permanent storages
• Where to write logs?
• files in the container
→ gone w/ container instance 😞
• directories shared from hosts
→ hosts are shared by many containers/services
☹
• TODO: ship logs from container to anywhere ASAP
13. Containers:
unfixed addresses
• No fixed physical / network address
• Where should we go to fetch logs?
• Service discovery (e.g., consul)
→ one more component 😞
• rsync? ssh+tail? or ..? Is it installed in containers?
→ one more tool to depend on ☹
• TODO: push logs to anywhere from containers
14. Containers:
instances per roles
• No fixed mapping between servers and roles
• How can we parse / store these logs?
• Central repository about log syntax
→ very hard to maintain 😞
• Label logs by source address
→ many containers/roles in a host ☹
• TODO: label & parse logs at source of logs
17. • Parse/Label (collector)
• Raw logs are not good for processing
• Convert logs to structured data (key-value pairs)
• Split/Sort (aggregator)
• Mixed logs are not good for searching
• Split whole data stream into streams per services
• Store (destination)
• Format logs(records) as destination expects
Collecting and Storing Data
18. Scaling Logging
• Network traffic
• CPU load to parse / format
• Parse logs on each collector (distributed)
• Format logs on aggregator (to be distributed)
• Capability
• Make aggregators redundant
• Controlling delay
• to make sure when we can know what's happening in our
systems
22. Without Source Aggregation
• Pros:
• Simple configuration
• Cons:
• fixed aggregator (endpoint) address
• many network connections
• high load in aggregator
collector
aggregator
23. With Source Aggregation
• Pros:
• less connections
• lower load in aggregator
• less configuration in containers
(by specifying localhost)
• highly flexible configuration
(by deployment only of aggregate containers)
• Cons:
• a bit much resource (+1 container per host)
aggregate
container
aggregator
25. Without Destination Aggregation
• Pros:
• Less nodes
• Simpler configuration
• Cons:
• Storage side change affects collector side
• Worse performance: many small write requests
on storage
26. With Destination Aggregation
• Pros:
• Collector side configuration is
free from storage side changes
• Better performance with fine tune
on destination side aggregator
• Cons:
• More nodes
• A bit complex configuration
aggregator
28. Scaling Up Endpoints
• Pros:
• Simple configuration
in collector nodes
• Cons:
• Limits about scaling up
Load balancer
Backend nodes
29. Scaling Out Endpoints
• Pros:
• Unlimited scaling
by adding aggregator nodes
• Cons:
• Complex configuration
• Client features for round-robin
30. Without
Destination Aggregation
With
Destination Aggregation
Scaling Up
Endpoints
Systems in early stages
Collecting logs over
Internet
or
Using queues
Scaling Out
Endpoints
Impossible :(
Collector nodes must know
all endpoints
↓
Uncontrollable
Collecting logs
in datacenter
33. Why Fluentd?
• Docker Fluentd logging driver
• Docker containers can send logs to Fluentd
directly - less overhead
• Pluggable architecture
• Various destination systems
• Small memory footprint
• Source aggregation requires +1 container per host
• Less additional resource usage ( < 100MB )
34. Destination aggregation + scaling up
• Sending logs directly over TCP by Fluentd logger
library in application code
• Same with patterns of New Relic
• Easy to implement
- good for startups Application code
35. Source aggregation + scaling up
• Kubernetes: Json logger + Fluentd + Elasticsearch
• Applications write logs to STDOUT
• Docker writes logs as JSON in files
• Fluentd
reads logs from file
parse JSON objects
writes logs to Elasticsearch
• EFK stack (like ELK stack)
http://kubernetes.io/docs/getting-started-guides/logging-elasticsearch/
Elasticsearch
Application code
Files (JSON)
36. Source aggregation + scaling up/out
• Docker fluentd logging driver + Fluentd + Kafka
• Applications write logs to STDOUT
• Docker sends logs
to localhost Fluentd
• Fluentd
gets logs over TCP
pushes logs into Kafka
• Highly scalable & less overhead
- very good for huge deployment
Kafka
Application code
37. Application code
Source/Destination aggregation +
scaling out
• Docker fluentd logging driver + Fluentd
• Applications write logs to STDOUT
• Docker sends logs
to localhost Fluentd
• Fluentd
gets logs over TCP
sends logs into Aggregator Fluentd
w/ round-robin load balance
• Highly flexible
- good for complex data processing
requirements
Any other storages
38. What's the Best?
• Writing logs from containers: Some way to do it
• Docker logging driver
• Write logs on files + read/parse it
• Send logs from apps directly
• Make the platform scalable!
• Source aggregation: Fluentd on localhost
• Scalable storage: (Kafka, external services, ...)
• No destination aggregation + Scaling up
• Non-scalable storage: (Filesystems, RDBMSs, ...)
• Destination aggregation + Scaling out
40. Why OSS?
• Logging layer is interface
• transparency
• interoperability
• Keep the platform scalable
• number of nodes
• number of types of source/destination