Logging for Production Systems in The Container Era
â˘
2 gefällt mirâ˘1,436 views
Talk at Docker Meetup. Video: https://www.youtube.com/watch?v=4kVeHcq0nnY
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (20)
Ăhnlich wie Logging for Production Systems in The Container Era
Ăhnlich wie Logging for Production Systems in The Container Era (20)
Mehr von Sadayuki Furuhashi
Mehr von Sadayuki Furuhashi (14)
KĂźrzlich hochgeladen
KĂźrzlich hochgeladen (20)
Logging for Production Systems in The Container Era
- 1. Logging for Production Systems in The Container Era Sadayuki Furuhashi⨠Founder & Software Architect DOCKER MOUNTAIN VIEW
- 2. A little about me⌠Sadayuki Furuhashi github: @frsyuki A founder of Treasure Data, Inc. located in Silicon Valley. Fluentd - UniďŹd log collection infrastracture Embulk - Plugin-based ETL tool OSS projects I founded: An open-source hacker.
- 3. It's like JSON. but fast and small. A little about meâŚ
- 4. The Container Era Server Era Container Era Service Architecture Monolithic Microservices System Image Mutable Immutable Managed By Ops Team DevOps Team Local Data Persistent Ephemeral Log Collection syslogd / rsync ? Metrics Collection Nagios / Zabbix ?
- 5. Server Era Container Era Service Architecture Monolithic Microservices System Image Mutable Immutable Managed By Ops Team DevOps Team Local Data Persistent Ephemeral Log Collection syslogd / rsync ? Metrics Collection Nagios / Zabbix ? The Container Era How should log & metrics collection be done in The Container Era?
- 6. Problems
- 7. The traditional logrotate + rsync on containers Log Server Application Container A File FileFile Hard to analyze!! Complex text parsers Application Container C File FileFile Application Container B File FileFile High latency!! Must wait for a day Ephemeral!! Could be lost at any time
- 8. Server 1 Container A Application Container B Application Server 2 Container C Application Container D Application Kafka elasticsearch HDFS Container Container Container Container Small & many containers make storages overloaded Too many connections from micro containers!
- 9. Server 1 Container A Application Container B Application Server 2 Container C Application Container D Application Kafka elasticsearch HDFS Container Container Container Container System images are immutable Too many connections from micro containers! Embedding destination IPs⨠in ALL Docker images⨠makes management hard
- 10. Combination explosion with microservices⨠requires too many scripts for data integration LOG script to parse data cron job for loading ďŹltering script syslog script Tweet- fetching script aggregation script aggregation script script to parse data rsync server
- 11. A solution: centralized log collection service LOG Log Service
- 12. The centralized log collection service LOG
- 13. The centralized log collection service LOG We Released!⨠(Apache License)
- 14. Whatâs Fluentd? Simple core⨠+ Variety of plugins BuďŹering, HA (failover), Secondary output, etc. Like syslogd AN EXTENSIBLE & RELIABLE DATA COLLECTION TOOL
- 15. How to collect logs from⨠Docker containers
- 16. Text logging with --log-driver=ďŹuentd Server Container App FluentdSTDOUT / STDERR docker run --log-driver=fluentd ⨠--log-opt fluentd-address=localhost:24224 { âcontainer_idâ: âad6d5d32576aâ, âcontainer_nameâ: âmyappâ, âsourceâ: stdout }
- 17. Metrics collection with ďŹuent-logger Server Container App Fluentd from fluent import sender from fluent import event sender.setup('app.events', host='localhost') event.Event('purchase', { 'user_id': 21, 'item_id': 321, 'value': '1' }) tag = app.events.purchase { âuser_idâ: 21, âitem_idâ: 321 âvalueâ: 1, } fluent-logger library
- 18. Logging methods for each purpose ⢠Collecting log messages > --log-driver=ďŹuentd ⢠Application metrics > ďŹuent-logger ⢠Access logs, logs from middleware > Shared data volume ⢠System metrics (CPU usage, Disk capacity, etc.) > Fluentdâs input plugins⨠(Fluentd pulls those data periodically)
- 20. Server 1 Container A Application Container B Application Server 2 Container C Application Container D Application Kafka elasticsearch HDFS Container Container Container Container Primitive deployment⌠Too many connections from many containers! Embedding destination IPs⨠in ALL Docker images⨠makes management hard
- 21. Server 1 Container A Application Container B Application Fluentd Server 2 Container C Application Container D Application Fluentd Kafka elasticsearch HDFS Container Container Container Container destination is always localhost from appâs point of view Source aggregation decouples conďŹg from apps
- 22. Server 1 Container A Application Container B Application Fluentd Server 2 Container C Application Container D Application Fluentd active / standby / load balancing Destination aggregation makes storages scalable for high trafďŹc Aggregation server(s)
- 23. Aggregation servers ⢠Logging directly from microservices makes log storages overloaded. > Too many RX connections > Too frequent import API calls ⢠Aggregation servers make the logging infrastracture more reliable and scalable. > Connection aggregation > BuďŹering for less frequent import API calls > Data persistency during downtime > Automatic retry at recovery from downtime
- 25. Internal Architecture (simpliďŹed) Plugin Input Filter Buffer Output Plugin Plugin Plugin 2012-02-04 01:33:51 myapp.buylog{ âuserâ:âmeâ, âpathâ: â/buyItemâ, âpriceâ: 150, ârefererâ: â/landingâ } Time Tag Record
- 26. Architecture: Input Plugins HTTP+JSON (in_http) File tail (in_tail) Syslog (in_syslog) ⌠Receive logs Or pull logs from data sources In non-blocking manner Plugin Input
- 27. Filter Architecture: Filter Plugins Transform logs Filter out unnecessary logs Enrich logs Plugin Encrypt personal data Convert IP to countries Parse User-Agent âŚ
- 28. Buffer Architecture: Buffer Plugins Plugin Improve performance Provide reliability Provide thread-safety Memory (buf_memory) File (buf_ďŹle)
- 29. Architecture: Output Plugins Output Write or send event logs Plugin File (out_ďŹle) Amazon S3 (out_s3) MongoDB (out_mongo) âŚ
- 30. Buffer Architecture: Buffer Plugins Chunk Plugin Improve performance Provide reliability Provide thread-safety Input Output Chunk Chunk
- 31. Retry Error Retry Batch Stream Error Retry Retry Divide & Conquer for retry
- 32. Divide & Conquer for recovery Buffer (on-disk or in-memory) Error Overloaded!! recovery recovery + ďŹow control queued chunks
- 34. Streaming from Apache/Nginx to Elasticsearch in_tail /var/log/access.log /var/log/ďŹuentd/buffer but_ďŹle
- 35. Error Handling and Recovery in_tail /var/log/access.log /var/log/ďŹuentd/buffer but_ďŹle Buffering for any outputs Retrying automatically With exponential wait and persistence on a disk and secondary output
- 36. Tailing & parsing ďŹles Supported built-in formats: Read a log ďŹle Custom regexp Custom parser in Ruby ⢠apache ⢠apache_error ⢠apache2 ⢠nginx ⢠json ⢠csv ⢠tsv ⢠ltsv ⢠syslog ⢠multiline ⢠none pos ďŹleevents.log ? (your app)
- 37. Out to Multiple Locations Routing based on tags Copy to multiple storages buffer access.log in_tail
- 38. Example conďŹguration for real time batch combo
- 39. Data partitioning by time on HDFS / S3 access.log buffer Custom ďŹle formatter Slice ďŹles based on time 2016-01-01/01/access.log.gz 2016-01-01/02/access.log.gz 2016-01-01/03/access.log.gz ⌠in_tail
- 40. 3rd party input plugins dstat df AMQL munin jvmwatcher SQL
- 41. 3rd party output plugins AMQL Graphite
- 42. Real World Use Cases
- 43. Microsoft Operations Management Suite uses Fluentd: "The core of the agent uses an existing open source data aggregator called Fluentd. Fluentd has hundreds of existing plugins, which will make it really easy for you to add new data sources." Syslog Linux Computer Operating System Apache MySQL Containers omsconďŹg (DSC) PS DSC Providers OMI Server (CIM Server) omsagent Firewall/proxy OMSService Upload Data (HTTPS) Pull configuration (HTTPS)
- 44. Atlassian "At Atlassian, we've been impressed by Fluentd and have chosen to use it in Atlassian Cloud's logging and analytics pipeline." Kinesis Elasticsearch cluster Ingestion service
- 45. Amazon web services The architecture of Fluentd (Sponsored by Treasure Data) is very similar to Apache Flume or Facebookâs Scribe. Fluentd is easier to install and maintain and has better documentation and support than Flume and Scribe. Types of DataStoreCollect Transactional ⢠Database reads & write (OLTP) ⢠Cache Search ⢠Logs ⢠Streams File ⢠Log ďŹles (/val/log) ⢠Log collectors & frameworks Stream ⢠Log records ⢠Sensors & IoT data Web Apps IoTApplicationsLogging Mobile Apps Database Search File Storage Stream Storage
- 46. Thank you!