Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Docker and Fluentd

10.463 Aufrufe

Veröffentlicht am

At Docker Meetup Tokyo #5

Veröffentlicht in: Technologie
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier

Docker and Fluentd

  1. 1. and Docker meetup Tokyo #5
 Aug 19, 2015
  2. 2. Who are you? • Masahiro Nakagawa • github: @repeatedly • Treasure Data Inc. • Fluentd / td-agent developer • https://jobs.lever.co/treasure-data • I love OSS :) • D Language, MessagePack, The organizer of several meetups, etc…
  3. 3. Logging for containers • Storage: • should be outside of containers / hosts • Transferring: • should be over network • Aggregation: • should be done per container / per service
  4. 4. Real world…
  5. 5. Container-based world needs unified logging layer
  6. 6. Structured logging Reliable forwarding Pluggable architecturehttp://fluentd.org/
  7. 7. What’s Fluentd? • Data collector for unified logging layer • Streaming data transfer based on JSON • Written in Ruby • Gem based various plugins • http://www.fluentd.org/plugins • Working on lots of productions • http://www.fluentd.org/testimonials
  8. 8. Before…
  9. 9. After!
  10. 10. M x N → M + N Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Databases buffering / processing /
  11. 11. Core Plugins • Divide & Conquer • Buffering & Retrying • Error handling • Message routing • Parallelism • Read data • Parse data • Filter data • Format data • Write data
  12. 12. Architecture EngineInput Filter Output Buffer > grep > record_transfomer > … > Forward > File tail > ... > Forward > File > ... Output > File > Memory not pluggable FormatterParser
  13. 13. Aggregation patterns 1) 1-level aggregation 2) 2-level aggregation Pros: • simple configuration Cons: • fixed aggr, address • many connections • high load for aggr Pros: • less connections • lower load for aggr Cons: • more containers
  14. 14. Aggregation patterns 1) 1-level aggregation 2) 2-level aggregation Apps sends logs
 over fluent-logger or … There are some patterns
 - via udp / tcp
 - tail file
 - etc…
  15. 15. 2-level aggr w/ Fluentd 1. Transfer logs over network 2. Container logs & tail generated log files
  16. 16. 1) Network transferring • Apps sends logs: • using fluent-logger-* • over inter-container network • Pros: • no performance penalty • Cons: • docker logs is out of scope • a bit complex for logging over TCP app
  17. 17. 2) Tail container logs • Apps write logs to STDOUT • docker writes it to log file • fluentd container reads its file • Pros: • simple conf for apps & docker • logs include container logs • Cons: • in_tail performance penalty &
 a bit complex configuration log file app http://www.fluentd.org/guides/recipes/docker-logging
  18. 18. Seems good? But we now have smarter way!
  19. 19. Logging drivers • New from docker v1.6 • We can get docker logs directly via drivers • “fluentd" driver is coming from v1.8 • enabled by “—log-driver=fluentd” • contributed by @tagomoris
 https://github.com/docker/docker/pull/12876 • https://docs.docker.com/reference/logging/fluentd/
  20. 20. 3) Logging driver • Apps write logs to STDOUT • Docker sends its logs to
 fluentd directly • Pros: • simple conf for apps & docker • Logs includes container logs • Cons: • Do you see any concerns? over TCP app
  21. 21. Demo?
  22. 22. Fluentd docker image • Official image by fluent organization • Main maintainer is @tagomoris • https://registry.hub.docker.com/u/fluent/fluentd/ • Or build your container! • https://github.com/fluent/fluentd-docker-image
  23. 23. With Kubernetes • Kubernetes provides fluentd + Elasticsearch + Kibana image officially • https://hub.docker.com/r/kubernetes/fluentd- elasticsearch/ • http://www.dasblinkenlichten.com/logging-in- kubernetes-with-fluentd-and-elasticsearch/
  24. 24. Cloud service for entire data pipeline!

×