Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

Cloud Native Logging / Fluentd Summit Tokyo

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Wird geladen in …3
×

Hier ansehen

1 von 63 Anzeige
Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Ähnlich wie Cloud Native Logging / Fluentd Summit Tokyo (20)

Anzeige

Aktuellste (20)

Cloud Native Logging / Fluentd Summit Tokyo

  1. 1. Cloud Native Logging Fluentd Summit, June 01th, 2017 - Japan Eduardo Silva @edsiper eduardo@treasure-data.com
  2. 2. “Logging... Why?”
  3. 3. Logging: Why ? Analyze Applications Behavior
  4. 4. Logging: Why ? Analyze Applications Behavior
  5. 5. Logging: Why ?
  6. 6. Logging: Why ?
  7. 7. Logging Pipeline
  8. 8. Logging Pipeline
  9. 9. Logging Pipeline
  10. 10. It’s a mess
  11. 11. “Logging... Why?” (Micro Services)
  12. 12. OK, Where’s My Log in Container Era?
  13. 13. Logging has Never Been Easier
  14. 14. About Fluentd ● Invented by ● Now hosted at
  15. 15. Logging Pipeline
  16. 16. How Fluentd Simplifies Logging Pipeline
  17. 17. How Fluentd Simplifies Unified Logging Layer
  18. 18. About Fluentd ● More than 600 plugins available ● Pluggable Architecture ● Built-in Reliability ● Full integration with Docker and Kubernetes ● Written in Ruby + C
  19. 19. Fluentd Plugin Ecosystem
  20. 20. Fluentd Modular Architecture
  21. 21. Fluentd Structured Logging
  22. 22. Fluentd at Scale
  23. 23. Kubernetes & Fluentd
  24. 24. Kubernetes 101
  25. 25. Kubernetes 101
  26. 26. Kubernetes 101
  27. 27. Kubernetes 101
  28. 28. Kubernetes 101
  29. 29. Kubernetes 101
  30. 30. Fluentd Deployment: DaemonSet
  31. 31. Kubernetes: Fluentd Log Collection
  32. 32. Fluentd: Enrich Logs with Kubernetes Metadata
  33. 33. Storing the Logs
  34. 34. Example: Logs from the Container
  35. 35. Logs Stored by Fluentd
  36. 36. Fluentd: Error Handling
  37. 37. Fluentd: Retrying
  38. 38. Fluentd: Kubernetes DaemonSet Pre-built Yaml file https://github.com/fluent/fluentd-kubernetes-daemonset/
  39. 39. Extending the Fluentd Ecosystem
  40. 40. Fluentd Modes ● Log Forwarder ● Log Aggregator
  41. 41. Log Aggregator = (Forwarder + Buffering Capabilities)
  42. 42. Edge Nodes / Forward to Aggregators App Node 1 DB App DB
  43. 43. Edge Nodes & Costs
  44. 44. Edge Nodes & Costs ● Fluentd requires ~40MB as minimum ● Deploying a few hundred could be expensive ● Can we make Forward cheaper ?
  45. 45. Forwarder & Aggregator Log Forwarder Log Aggregator
  46. 46. About Fluent Bit ● Written in C ● Pluggable Architecture ● Built-in Reliability ● Event Driven - Async I/O
  47. 47. Why Fluent Bit as a Forwarder ● Features ○ Input, Filter and Output Plugins ○ Built-in parsing support ○ Minimum memory required 450KB
  48. 48. Edge Nodes / Forward to Aggregators
  49. 49. Cheap Forwarding
  50. 50. Cloud Native Features ● Backpressure Handling ● Security ● Kubernetes Metadata Filter ● Timestamps w/Nanoseconds (upcoming v0.12)
  51. 51. Cloud Native Features ● Backpressure
  52. 52. Cloud Native Features ● Backpressure
  53. 53. Cloud Native Features ● Backpressure
  54. 54. Cloud Native Features ● Backpressure
  55. 55. Cloud Native Features ● Backpressure Solved
  56. 56. Security ● Native integration for TLS/SSL ■ Optional for plugins with network I/O ■ Certificates support
  57. 57. Features, plugins and more ● Plugins (~30) ○ Input ■ cpu, disk, forward, mem, syslog, tcp, kmsg... ○ Filter ■ Grep, Kubernetes & Record Modifier ○ Output ■ Elasticsearch, HTTP, InfluxDB, Forward, Treasure Data
  58. 58. Fluent Bit, what else ? Networking and Co-routines Easier implementation of output plugins that interact with networking operations like socket(), connect(), read(), write(), etc. Fluent Bit provides non-blocking networking API that uses the event-loop with co-routines to implement: ● Network I/O ● TLS/SSL usage ● HTTP Client
  59. 59. Let’s “try” a demo ● Minikube! Disclaimer: I had the great idea to upgrade minutes ago!!!! :O
  60. 60. Fluent Bit Kubernetes DaemonSet Github Repository ● https://github.com/fluent/fluent-bit-kubernetes-daemonset Docker Hub Image ● fluent/fluent-bit-kubernetes-daemonset
  61. 61. Fluent Bit Project information ● Web site fluentbit.io ● Github fluent/fluent-bit Contact / Community ● Slack http://slack.fluentd.org

×