1.
Log Management with ELK
ELASTICSEARCH, LOGSTASH, KIBANA FOR CENTRALIZED LOGS

2.
Purpose
Centralized Log Management
◦ Collect, Parse and Filter using Logstash
◦ Store, Index and Search using Elasticsearch
◦ Visualize using Kibana
Full open source stack
◦ Use for free
◦ Support plan from Elasticsearch company

3.
Elasticsearch
Real-time search engine
◦ Based on Apache Solr/Lucene
◦ Pure Java
◦ Document database
◦ Advanced text indexing
◦ Fuzzy search
◦ Replication/Sharding for true scalability

4.
Logstash
JRuby Based log processor
Pluggable event pipeline
◦ Input plugins
◦ Filter plugins
◦ Codec plugins
◦ Output plugins
DevOps Comunity
◦ Mix of developers, operations and system administrators

5.
Kibana
Browser based dashboard for ElasticSearch
Visualization of query results
◦ Time Charts
◦ Filter any field
◦ Compare subsets

6.
Logstash pipeline
Define input, filters and outputs
Simple configuration file
Ruby syntax

7.
Logstash not just for logs
Interpretes different log formats
◦ Syslog messages
◦ Log4j with full details
◦ Apache log files
Other event types too
◦ Ganglia server monitoring events
◦ SNMP events
◦ Windows EventLog
Pre-proces before sending
◦ lumberjack

8.
Logstash Inputs
Rsyslog via TCP/UDP
Log4j appender
JMX Listener
Logstash-forwarder
File tails
SNMP
Ganglia
…

9.
Logstash Filters

10.
Grok Filter
Readable regex
Predefined patterns for common log data
Extract to properties
◦ Indexed properties

11.
Metrics Filter
Aggregate metrics
◦ Event rate using sliding windows
◦ 1 min
◦ 5 min
◦ 15 min
◦ Min/max/stddev/percentiles

12.
Logstash Outputs

13.
Statsd output
Node.js based
◦ Counters
◦ Timers
◦ Graphite frontend

14.
Alert outputs
Send alerts
◦ Email
◦ Pagerduty
◦ XMPP/Jabber
◦ Hipchat
◦ Nagios
Use treshold from metrics filter

15.
ElasticSearch output
Auto-creates new index per day
Index all recognized fields
Full text index, customizable indexer, mapper

16.
Kibana dashboard

17.
Scalability
Easy deployment using chef/puppet/docker