This presentation deals with logging in the course of mobile development, namely describing the open source logging environment built with ELK stack (ElasticSearch, Logstash and Kibana). Presentation by Igor Rudyk (Software Engineer, GlobalLogic, Lviv), delivered at Mobile TechTalk Lviv on April 28, 2015. More details - http://globallogic.com.ua/mobile-techtalk-lviv-2015-report

1. ©2015 GlobalLogic Inc. CONFIDENTIAL
The ELK stack - get to know logs
Igor Rudyk
DevOps / System Integrator

2. 2 CONFIDENTIAL
Agenda
• Introduction. What is ELK, and why do we need it?
• The ELK stack
Logstash
ElasticSearch
Kibana
•Architecture
•Demo

3. 3 CONFIDENTIAL
Can you check the errors from yesterday between 9:09 and 9:27

4. 4 CONFIDENTIAL
So what’s a log

5. 5 CONFIDENTIAL
Log = timestamp + data

6. 6 CONFIDENTIAL
Lifecycle of a log
Delete
Real Real

7. 7 CONFIDENTIAL
Default problem
• Multiple log time formats
•Apr 28 20:21:59
•[27/Apr/2015:07:05:28 +0000]
•071012 09:27:32
•Mon, 27-Apr-15 06:27:02 UTC
•2015-04-28 20:07:51 +0000
• Starts not with timestamp or without timestamp
•Error messages with really unhelpful info
•No rotation
•No scaling
Tools?
• grep
• awk / sed / cut
•less / tail
•vi / vim
•regular expression
•...

8. 8 CONFIDENTIAL
Logging Solutions
Solutions Collections Transport Parsing Storage Analysis Alerting Visualizer Commercial
Logstash Logstash shipper or
logstash-forwarder
(Lumberjack)
RPM installation
Logstash shipper or logstash-
forwarder (Lumberjack, encrypted
transport is the default)
RPM installation
Output plugins
Central server-master with a hot-
standby in case of failure
Codecs plugins
Grok debugger
ElasticSearch,
MongoDB,
AWS S3
and much more
Kibana,
graylog2
Riemann Kibana,
graylog2
NO
fluentd Input plugins
Install from source
or via gem
Output plugins
Load-balance between multiple
hosts or have a master with a hot-
standby in case of failure
Plugins Doesn’t provide any
storage tier itself but
allows you to easily
configure where your logs
should be collected
Kibana,
graylog2
Riemann Kibana,
graylog2
NO
splunk Splunk Universal
Forwarder
RPM installation
Splunk Universal Forwarder
SSL security
Splunk Splunk Splunk Splunk Splunk YES
Graylog2 Graylog2
Logstash
Graylog2
Logstash
Graylog2 ElasticSearch Kibana,
graylog2
graylog2 graylog2 NO
loggly loggly loggly loggly Hosted loggly loggly loggly YES

9. 9 CONFIDENTIAL
What is ELK, and why do we need it?
• ELK is a stack of programs that help dealing with logs.
• Includes:
– Aggregation of logs
– Search capabilities
– Aggregation of statistics
– Visualizations

10. 10 CONFIDENTIAL
Logstash
Unstructured
Filters Outputs
Documents

11. 11 CONFIDENTIAL
Inputs
➔ Logs:
● Lumberjack - resilient, compressed, secure (logstash-forwarder)
● Remote syslog
● Files
➔ Devices:
● Event log, Collectd
● Netflow, WMI
➔ Event Queue:
● Redis, RabbitMQ
● Kafka, ZeroMQ
➔ Streaming APIs:
● Twitter
➔ Email (IMAP)
➔ Amazon S3, ganglia, sqs, varnishlog, etc
➔ ....
http://logstash.net/docs/1.4.2/ - Full list

12. 12 CONFIDENTIAL
Filters➔ grep
➔ date
➔ json
➔ grok
➔ ....
http://logstash.net/docs/1.4.2/ - Full list
Why Do I like Logstash?
It uses Grok filter for parsing standard and non standard logs:
Log Line:
27/10/14 07:39:28 [localhost-startStop-1] [] INFO
com.vidmind.config.LoggingPropertyPlaceholderConfigurer -
streams.limit.general = 0
Pattern:
%{DATESTAMP} %{SYSLOG5424SD} ?? %{WORD:ErrorLevel} %{JAVACLASS}

13. 13 CONFIDENTIAL
Outputs
➔ Storage:
● ElasticSearch
● MongoDB
● S3
● Graphite
● File
● ...
➔ Notification:
● Zabbix
● Nagios
● Riemann
● PagerDuty
● Email
➔ Event Queue:
● Redis, RabbitMQ
● Kafka, ZeroMQ
● tcp/udp
➔ SaaS:
● AWS CloudWatch
● Hipchat
● Jira
➔ ....
http://logstash.net/docs/1.4.2/ - Full list

14. 14 CONFIDENTIAL
Logstash - Forwarder (Shipper)
Configuration file{
"network": {
"servers": [[logstash_indexers]]
"timeout": 15,
"ssl ca": "logstash-forwarder.crt"
},
"files": [
{
"paths": [
"/usr/share/tomcat7/logs/*.json.log"
],
"fields": { "type": "tomcat", "server_name": "[[logstash_hostname]]", "system": "[[system]]", "server_type" : "[[server_type]]" }
},
{
"paths": [
"/usr/share/tomcat7/logs/*.activities.log"
],
"fields": { "type": "activities", "server_name": "[[logstash_hostname]]", "system": "[[system]]", "server_type" : "[[server_type]]"
}
}
]
}

15. 15 CONFIDENTIAL
Logstash-Indexer
Configuration file
input {
lumberjack {
codec => json{}
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
output {
elasticsearch {
host => "127.0.0.1"
protocol => "http"
cluster => "[[elasticsearch_cluster_name]]"
manage_template => false
index => "logstash-%{system}-%{type}-%{+YYYY.MM.dd}"
}
}

16. 16 CONFIDENTIAL
ElasticSearch
Configuration file (yaml-based configuration)
cluster.name: [[elasticsearch_cluster_name]]
node.name: "[[node_name]]"
node.master: false / true
node.data: false / true
index.number_of_replicas: 1
#Security
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: [[elasticsearch_servers]]
action.disable_close_all_indices: true
action.disable_delete_all_indices: true
action.disable_shutdown: true
script.disable_dynamic: true

17. 17 CONFIDENTIAL
The ELK stack
General Architecture
Logstash ElasticSearch Kibana

18. 18 CONFIDENTIAL
Kibana
ElasticSearch
Logstash-Forwarder
Logstash-Indexer
Logstash-Indexer
Logstash-Indexer
The ELK stack
Our Scaled Architecture
ElasticSearch
ElasticSearch
Kibana
Kibana

19. ©2015 GlobalLogic Inc. CONFIDENTIAL
DEMO