Presentation at LinuxCon Europe 2016 (Berlin). I introduced the concepts of logging for containers, aggregation patterns, distributted logging, data serialization, Fluentd: internals, architecture, Fluent Bit and it library API.

1. Containers and Logging
Challenges and Solutions
Eduardo Silva (@edsiper)
LinuxCon Europe 2016

2. Logging

3. Logging
●
Monitoring
●
Troubleshooting
●
Statistics
It matters

4. Logging Scenarios
●
Operating System
●
Applications Unit
●
Services
Distribution Channels

5. Containers

6. Containers
Everything is about Isolation

7. Containers & Log Handlers
●
File System
●
Standard I/O interfaces (stdout / stderr)
●
Over Network
Distribution Channels

8. Docker Logging Drivers
●
Json-file
●
Syslog
●
Journald
●
Fluentd
●
...
Outputs
Docker Implement drivers for different formats and
distribution channels:

9. Structured Logs

10. Structured Logs
●
Often based in Key-Value pairs
●
Two minimum keys: time and message
Distribution Channels
Structured logging makes data processing easier

11. Structured Logs
●
JSON: readable format for structured data
●
MessagePack: Binary serialization (json-like)
Common format
Structured logging makes data processing easier

12. Structured Logs
Docker log example
Original Log Message
This is a test message
Structured Log Message
{
"container_id":"bfdd5b9...",
"container_name":"/infallible_mayer",
"source":"stdout",
"log": "This is a test message"
}

13. Microservices

14. Microservices
Monolithic
●
A service produces all
data about users access
Microservices
●
Many services produce
data about users access
●
Log needs to be collected
from many services.

15. Microservices
●
How to deal with different input formats ?
●
Parse plain text is really expensive.
●
Not all containers have permanent storage.
●
Where to write the logs ?
Logging Challenges

16. Distributed Logging

17. Distributed Logging
Architecture

18. Distributed Logging
Workflow
Collector
●
Retrieve raw logs: file system / network.
●
Parse log content.
Aggregator
●
Get data from multiple sources.
●
Convert incoming data into Streams.
Destination
●
Retrieve data streams from Aggregator.
●
Store formatted logs (records) .

19. Scaling Logging
●
Network Traffic
●
CPU Load: parsing and formatting is expensive
●
High Availability / Redundant aggregators
Topics to consider

20. Aggregation Patterns

21. Source Aggregation Patterns
w/o source aggregation with source aggregation

22. Aggregation Patterns
Without Source Aggregation
Pros
●
Simple Configuration
Cons
●
Fixed Aggregator endpoint address
●
Many network connections
●
High load on Aggregator

23. Aggregation Patterns
With Source Aggregation
Pros
●
Less connections
●
Lower load in aggregator
●
Less config in Containers
Cons
●
Need more resources (1 aggregate container
per host.

24. Destination
Aggregation Patterns
w/o destination aggregation with destination aggregation

25. Aggregation Patterns
Without Destination Aggregation
Pros
●
Less Nodes
●
Simpler configuration
Cons
●
Storage side changes affects collector side
●
Worse performance: many small write requests
on storage

26. Aggregation Patterns
With Destination Aggregation
Pros
●
Collector side configuration is
free from storage side changes.
●
Better performance with fine
tune on destination side
aggregator.
Cons
●
More Nodes.
●
More complex configuration.

28. Open Source Data/Log Collector
●
High Performance
●
Built-in Reliability
●
Structured Logs
●
Pluggable Architecture
●
More than 300 plugins! (input/filtering/output)

29. Architecture / Workflow

30. Full Collector/Aggregator for Containers
●
Docker Interoperability
Native Docker logging driver to use Fluentd
●
Kubernetes
Fluentd as main aggregator (notes)
●
OpenShift
Fluentd as main aggregator

31. Docker use case

32. Fluentd
Docker use case

33. Fluentd
Kubernetes use case

34. Fluentd in the Real World

35. Microsoft Use Case

37. We collect
1.6M events per second !

38. It’s a proud member of:

39. Fluentd + CNCF ?
Logging

40. Fluentd + CNCF ?
Application: Work in Process

42. Fluent Bit
Lightweight log aggregator
●
Written in C
●
High Performance
●
Pluggable Architecture
●
Built-in CPU / Memory metrics / Network TLS support
●
Event-Driven
●
Fluentd Compatible

44. Architecture

45. Library Mode

46. Library Mode

47. Library Mode

48. Library Mode

49. Library Mode

50. Library Mode

51. Library Mode

52. Library Mode

53. Library Mode

54. Library Mode

55. Library Mode

56. Messages Forwarding
Performance
500K messages per second! (1 CPU core)

57. Fluent Bit
Roadmap
●
Built-in HTTP Monitoring
●
Lua scripting support

58. Thank You!
Eduardo Silva / @edsiper
eduardo@treasure-data.com
http://fluentd.org http://fluentbit.io