The ELK Stack - Get to Know Logs

2. 2 CONFIDENTIAL Agenda • Introduction. What is ELK, and why do we need it? • The ELK stack Logstash ElasticSearch Kibana •Architecture •Demo

3. 3 CONFIDENTIAL Can you check the errors from yesterday between 9:09 and 9:27

4. 4 CONFIDENTIAL So what’s a log

5. 5 CONFIDENTIAL Log = timestamp + data

6. 6 CONFIDENTIAL Lifecycle of a log Delete Real Real

7. 7 CONFIDENTIAL Default problem • Multiple log time formats •Apr 28 20:21:59 •[27/Apr/2015:07:05:28 +0000] •071012 09:27:32 •Mon, 27-Apr-15 06:27:02 UTC •2015-04-28 20:07:51 +0000 • Starts not with timestamp or without timestamp •Error messages with really unhelpful info •No rotation •No scaling Tools? • grep • awk / sed / cut •less / tail •vi / vim •regular expression •...

8. 8 CONFIDENTIAL Logging Solutions Solutions Collections Transport Parsing Storage Analysis Alerting Visualizer Commercial Logstash Logstash shipper or logstash-forwarder (Lumberjack) RPM installation Logstash shipper or logstash- forwarder (Lumberjack, encrypted transport is the default) RPM installation Output plugins Central server-master with a hot- standby in case of failure Codecs plugins Grok debugger ElasticSearch, MongoDB, AWS S3 and much more Kibana, graylog2 Riemann Kibana, graylog2 NO fluentd Input plugins Install from source or via gem Output plugins Load-balance between multiple hosts or have a master with a hot- standby in case of failure Plugins Doesn’t provide any storage tier itself but allows you to easily configure where your logs should be collected Kibana, graylog2 Riemann Kibana, graylog2 NO splunk Splunk Universal Forwarder RPM installation Splunk Universal Forwarder SSL security Splunk Splunk Splunk Splunk Splunk YES Graylog2 Graylog2 Logstash Graylog2 Logstash Graylog2 ElasticSearch Kibana, graylog2 graylog2 graylog2 NO loggly loggly loggly loggly Hosted loggly loggly loggly YES

9. 9 CONFIDENTIAL What is ELK, and why do we need it? • ELK is a stack of programs that help dealing with logs. • Includes: – Aggregation of logs – Search capabilities – Aggregation of statistics – Visualizations

10. 10 CONFIDENTIAL Logstash Unstructured Filters Outputs Documents

11. 11 CONFIDENTIAL Inputs ➔ Logs: ● Lumberjack - resilient, compressed, secure (logstash-forwarder) ● Remote syslog ● Files ➔ Devices: ● Event log, Collectd ● Netflow, WMI ➔ Event Queue: ● Redis, RabbitMQ ● Kafka, ZeroMQ ➔ Streaming APIs: ● Twitter ➔ Email (IMAP) ➔ Amazon S3, ganglia, sqs, varnishlog, etc ➔ .... http://logstash.net/docs/1.4.2/ - Full list

12. 12 CONFIDENTIAL Filters➔ grep ➔ date ➔ json ➔ grok ➔ .... http://logstash.net/docs/1.4.2/ - Full list Why Do I like Logstash? It uses Grok filter for parsing standard and non standard logs: Log Line: 27/10/14 07:39:28 [localhost-startStop-1] [] INFO com.vidmind.config.LoggingPropertyPlaceholderConfigurer - streams.limit.general = 0 Pattern: %{DATESTAMP} %{SYSLOG5424SD} ?? %{WORD:ErrorLevel} %{JAVACLASS}

13. 13 CONFIDENTIAL Outputs ➔ Storage: ● ElasticSearch ● MongoDB ● S3 ● Graphite ● File ● ... ➔ Notification: ● Zabbix ● Nagios ● Riemann ● PagerDuty ● Email ➔ Event Queue: ● Redis, RabbitMQ ● Kafka, ZeroMQ ● tcp/udp ➔ SaaS: ● AWS CloudWatch ● Hipchat ● Jira ➔ .... http://logstash.net/docs/1.4.2/ - Full list

14. 14 CONFIDENTIAL Logstash - Forwarder (Shipper) Configuration file{ "network": { "servers": [[logstash_indexers]] "timeout": 15, "ssl ca": "logstash-forwarder.crt" }, "files": [ { "paths": [ "/usr/share/tomcat7/logs/*.json.log" ], "fields": { "type": "tomcat", "server_name": "[[logstash_hostname]]", "system": "[[system]]", "server_type" : "[[server_type]]" } }, { "paths": [ "/usr/share/tomcat7/logs/*.activities.log" ], "fields": { "type": "activities", "server_name": "[[logstash_hostname]]", "system": "[[system]]", "server_type" : "[[server_type]]" } } ] }

15. 15 CONFIDENTIAL Logstash-Indexer Configuration file input { lumberjack { codec => json{} port => 5000 type => "logs" ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" } } output { elasticsearch { host => "127.0.0.1" protocol => "http" cluster => "[[elasticsearch_cluster_name]]" manage_template => false index => "logstash-%{system}-%{type}-%{+YYYY.MM.dd}" } }

16. 16 CONFIDENTIAL ElasticSearch Configuration file (yaml-based configuration) cluster.name: [[elasticsearch_cluster_name]] node.name: "[[node_name]]" node.master: false / true node.data: false / true index.number_of_replicas: 1 #Security discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: [[elasticsearch_servers]] action.disable_close_all_indices: true action.disable_delete_all_indices: true action.disable_shutdown: true script.disable_dynamic: true

17. 17 CONFIDENTIAL The ELK stack General Architecture Logstash ElasticSearch Kibana

18. 18 CONFIDENTIAL Kibana ElasticSearch Logstash-Forwarder Logstash-Indexer Logstash-Indexer Logstash-Indexer The ELK stack Our Scaled Architecture ElasticSearch ElasticSearch Kibana Kibana

The ELK Stack - Get to Know Logs

GlobalLogic Ukraine

The ELK Stack - Get to Know Logs