1.
ELK,
a real case study
Alessandro Mazzoli
Sysadmin@Ideato
am@ideato.it
Paolo Tonin
Sysadmin@Ideato
pt@ideato.it

2.
What is ELK stack?
Elasticsearch
Logstash
Kibana
http://en.wikipedia.org/wiki/Elk

3.
Logging problems
(especially if you are a small company)

4.
Centralize, index, archive

5.
Law constraints

6.
Many log structures

7.
Log As a Service?!?!
Pretty expensive

8.
Log analysis is hard for
non technical people

9.
cat access.log | cut -d' ' -f1 | sort | uniq

10.
People don’t pay attention
to boring things.
- Brain Rules by John Medina

11.
Major logging systems
Pro

12.
Why ELK, a case study
Web Tier
Apache
Nginx
Proxy
MySQL DB
Proxy FE Database BE
NFS shared filesystem

13.
We need to scale Web Tier

14.
Why ELK, a case study
Nginx proxy LB
Apache web1 MySQL DB
Memcached,
Logstash
Web TierProxy FE Database BE
Apache web2
NFS shared filesystem

15.
Nginx proxy LB
Apache web1 MySQL DB
Memcached,
Logstash
Web TierProxy FE Database BE
Apache web2
NFS shared filesystem
Why ELK, a case study
local log
local log

16.
ELK architecture
Server 1
Log File
Logstash Shipper
Redis
Application
Logstash
Indexer
Elasticsearch

17.
Scale out any
components!

18.
Redis
Logstash
Indexer
Redis
Redis
Redis
ElasticsearchElasticsearch
Server 2
Logstash Shipper
Server 1
Logstash Shipper

19.
Logstash configuration
input {
!
}
filter {
!
}
output {
!
}
Where log come from?

20.
Logstash configuration
input {
!
}
filter {
!
}
output {
!
}
How we threat them ?
Where log come from?

21.
Logstash configuration
input {
!
}
filter {
!
}
output {
!
}
Where will be stored?
How we threat them ?
Where log come from?

22.
Example configuration
input {
file {
path => "/var/log/messages"
type => "syslog"
}
file {
path => "/var/log/apache/access.log"
type => “apache-access"
}
}

23.
Example configuration
filter {
if [type] =~ "access" {
mutate {
replace => { "type" => “apache-access” }
}
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}
date {
match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] }
}
}

24.
Example configuration
output {
elasticsearch {
host => localhost
}
}

25.
“How can I collect and
consult my application
Magento log?”

26.
Custom filter

27.
Grok
It’s is currently the best way
in logstash to parse crappy
unstructured log data into
something structured and
queryable
- http://logstash.net/docs/1.4.2/filters/grok

28.
https://github.com/ideatosrl/logstash-magento-filter
input {
file {
path => “/var/www/magento/var/log/*.log"
type => "magento"
codec => plain {
charset => "ISO-8859-1"
}
}
}
Logstash Shipper

29.
https://github.com/ideatosrl/logstash-magento-filter
filter {
if [type] == "magento" {
grok {
match => { "message" => "%
{TIMESTAMP_ISO8601:timestamp} %
{DATA:syslog_program}"}
add_field => [ "received_at", "%
{@timestamp}" ]
}
}
}
Logstash Shipper

30.
https://github.com/ideatosrl/logstash-magento-filter
output {
redis { host => “REDIS_IP"
data_type => "list"
key => "logstash"
}
}
Logstash Shipper

31.
https://github.com/ideatosrl/logstash-magento-filter
input {
redis {
host => "REDIS_IP" type => "redis-input"
data_type => “list" key => "logstash"
}
}
output {
elasticsearch {
host => “ES_PUBLIC_IP” protocol => "http"
manage_template => false
index => "logstash-%{+YYYY.MM.dd}"
}
}
Logstash Indexer

32.
Before…
2015-03-30T15:25:34.867Z SoapFault
exception: [soap:Client] Server was
unable to read request. ---> There is
an error in XML document (2, 439). --->
Input string was not in a correct
format. in /var/www/magento/vendor/
connect20/MailUp/app/code/local/MailUp/
MailUpSync/Model/Observer.php:158

33.
{
"_index": "logstash-2015.03.30",
"_type": "magento",
"_id": "AUxrSbc5UWx9I25Cgios",
"_score": null,
"_source": {
"message": "SoapFault exception: [soap:Client] Server
was unable to read request. ---> There is an error in XML
document (2, 435). ---> Input string was not in a correct
format. in /var/www/magento/vendor/connect20/MailUp/app/
code/local/MailUp/MailUpSync/Model/Observer.php:158",
"@version": "1",
"@timestamp": "2015-03-30T15:25:34.867Z",
"type": "magento",
"host": "web2",
"path": "/var/www/magento/var/log/exception.log",
},
}

34.
And display it!

35.
“Should I use ELK?”

36.
PRO
• Easy to install
!
• Opensource
!
• Not only web logs!
!
• Many output filters; S3, Google
Big Query, MongoDb etc…

37.
ConclusionsCONS
• Many moving parts, each part has
their problems and issues
!
• Quickly evolution, prepare
yourself to upgrade frequently
!
• Out-of-the-box configurations
are not suitable for medium/large
deployment

38.
Links
http://www.ideato.it/technical-articles/integrazione-logstash-
magento
https://github.com/ideatosrl/logstash-magento-filter
https://www.youtube.com/watch?v=RuUFnog29M4
http://logstash.net
http://elastic.co

39.
Questions?!?