Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Centralized + Unified Logging

Vday 2014 , Fluentd Introduction

  • Loggen Sie sich ein, um Kommentare anzuzeigen.

Centralized + Unified Logging

  1. 1. Centralized + Unified Logging Gabor Kozma / gabo@ustream.tv / @kozmag82
  2. 2. Everybody wants to write logs! ✓ Application Logs (frontend / backend) ➢ php, java, ruby, python, bash ✓ Access Logs ➢ apache, nginx, tomcat, jetty ✓ System Logs ➢ syslog, hardware error log ✓ Database Logs ➢ history, transaction
  3. 3. Centralized Logging
  4. 4. Central Logging Architecture ✓ Collection ➢ file, syslog, database ✓ Transport ➢ chukwa, heka, syslog, logstash, flume, fluentd, kafka, nsq, nxlog, other custom solution. Typical: syslog-ng, rsyslog ✓ Storage / Store ➢ Amazon S3, Glacier, NAS ...
  5. 5. Central Logging Architecture ✓ Analysis (You need a way to analyze them!) ➢ Apache Hadoop + HDFS + Map-Reduce jobs ■ Hive, Pig, HBase, Impala.... ➢ Elasticsearch + Graylog2 / Kibana ➢ MongoDB + Map-Reduce/Aggregation Framework ➢ Graphite, Statsd + Dashboards ✓ Alerting (Errors almost always indicate a problem!) ➢ Airbreak/Errbit, Sentry, Honeybadger, Nagios, Zabbix, Open/PagerDuty
  6. 6. Unified Logging Layer
  7. 7. Unified Logging Layer ✓ Ubiquity ➢ Various format problem ➢ Various source and destination ➢ You must be optimize most of use case! ✓ Rigidity vs. Flexibility ➢ Apache Thrift , Apache Avro, Protocol Buffer , JSON / BSON, MessagePack
  8. 8. Unified Logging Layer ✓ Reliability and Scalability ➢ Scalable ➢ Support retryable data transfer ➢ Sync / Async data transfer ➢ Push / Pull base system ✓ Extensibility ➢ Support new input / output ■ You don’t have to modify anything else.
  9. 9. Fluentd - Pluggable architecture ✓ Input, Output, Buffer, Parser, Formatter 300+ plugins
  10. 10. Fluentd - Minimum res. require ✓ Combination of C language and Ruby ✓ 1 node ✓ 30/40 Mbyte RAM ✓ 1 CPU core 13.000 event / sec
  11. 11. Fluentd - Built-in Reliability ✓ Buffer ➢ file or memory ✓ Retrying ✓ Error handling ➢ transaction, failover, secondary node support (heartbeat)
  12. 12. Fluentd - Event structure (log) ✓ Time ➢ Second unit ➢ From data source or adding parsed time ✓ Tag ➢ for message routing ✓ Record ➢ JSON format ■ MessagePack internally :) ■ none structured
  13. 13. Fluentd - Useful plugins ✓ Output ➢ stdout, file, forest, graphite, mongo, mysql, elasticsearch, splunk, null, s3, geoip, webhdfs ✓ Input ➢ syslog, tail, http, udp, tcp, scribe ✓ Buffer ➢ memory, file ✓ Formatter and/or Parser ➢ lstv, json, multiline
  14. 14. Examples
  15. 15. Fluentd - Examples <source> type tail format /^(?<host>[^ ]*):(?<port>[^ ]*) (?<ip>[^ ]*) (?<user>[^ ]*) (?<remotelog>[^ ]*) [(?<time>[^]]*)] "(?<method>S+)(?: +(?<path>[^ ]*) +S*)?" (?<code>[^ ]*) (?<size>[^ ]*) (?: "(?<referer>[^"]*)" ""(?<agent>[^"]*)"")?(?: "(?<referer>[^"]*)" "(?<agent> [^"]*)")?$/ path /var/log/apache2/other_vhosts_access.log.* pos_file /var/log/fluent/apache2.other_vhosts_access.log.pos time_format %d/%b/%Y:%H:%M:%S %z tag apache2.access.raw read_from_head true </source>
  16. 16. Fluentd - Examples <match apache2.*.raw> type record_reformer enable_ruby false renew_record false remove_keys remotelog tag ${tag_prefix[-2]}.reformed <record> hostname ${hostname} </record> </match> <match apache2.*.reformed> type geoip geoip_lookup_key ip geoip_database /usr/share/GeoIP/GeoIPCity.dat <record> geo_city ${city['ip']} ... geo_region ${region['ip']} </record> add_tag_suffix .geoip flush_interval 5s </match>
  17. 17. Fluentd - Examples <match apache2.access.reformed.geoip> type forward flush_interval 5s buffer_type file buffer_queue_limit 512 buffer_chunk_limit 100M buffer_path /opt/fluent/buffer/apache2/ <server> name hostname host xxx.xxx.xxx.xxx weight 10 </server> ... <server> name hostname host xxx.xxx.xxx.xxx standby </server> <secondary> type file path /var/log/fluent/forward-failed/ apache2/ </secondary> </match>
  18. 18. Fluentd - Examples <match apache2.access.**> type copy <store> type file path /opt/fluent/apache2/access time_format %Y%m%dT%H%M%S%z flush_interval 60s append true compress gzip utc num_threads 4 ... ... </store> <store> type datacounter ... </store> <store> type graphite ... </store> </match>
  19. 19. Fluentd - Testimonials
  20. 20. Questions? http://www.fluentd.org http://docs.fluentd.org/ http://fluentular.herokuapp.com/ https://rubygems.org/search?query=fluent-plugin-http:// msgpack.org/

×