Log files. Every server's got 'em, but how many of us make good use of them? In this meetup we are going to look at a set of open source tools that fit together to form the ELK stack. The ELK stack gives us a relatively easy way to centralize logs from many different sources, makes them searchable and lets us visualize the data. This helps us find new ways to make sense of log data and gain valuable insights into what our servers do when we are not watching. Join Peter Meth of Delvia as he walks us through the ELK Stack and shows us how it can become the first step in moving towards a DevOps conscious development shop.

1. GTA PHP
Dec 22, 2015

2. Peter Meth
• Organizer of:
‣ TrueNorth PHP
‣ DevOpsDays
‣ GTA PHP Meetup
‣ DevOps Toronto Meetup
• Find me
‣ in Acton, Ontario
‣ on Twitter @devopsmeth
‣ on Meetup
‣ on LinkedIn
‣ email pmeth@delvia.com

3. the DevOps Company

4. What is DevOps?

5. DevOps CALMS
• CULTURE
• AUTOMATION
• LEAN
• MEASUREMENT
• SHARING

6. DevOps CALMS
• CULTURE
• AUTOMATION
• LEAN
• MEASUREMENT
• SHARING

7. Elk (modified) by Alexander Mostov http://bit.ly/1Twjcjl License: CC BY-NC-SA 4.0

8. Elk (modified) by Alexander Mostov http://bit.ly/1Twjcjl License: CC BY-NC-SA 4.0

9. Logging in Finnish Lapland by Greanpeace Finland http://bit.ly/1Twjcjl License: CC BY 2.0
A bit about Logs

10. 195.154.188.29 - - [02/Dec/2015:16:23:38 -0500] "POST
/ab.asp HTTP/1.1" 404 1231 "http://example.com/11m.php"
"Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)"
157.55.39.99 - - [02/Dec/2015:16:35:04 -0500] "GET
/robots.txt HTTP/1.1" 200 2118 "-" "Mozilla/5.0
(compatible; bingbot/2.0;
+http://www.bing.com/bingbot.htm)”
[Wed Dec 02 16:52:08 2015] [error] [client
195.154.188.224] File does not exist: /var/www/ab.asp,
referer: http://example.com/11m.php
[Wed Dec 02 16:52:08 2015] [error] [client
195.154.188.224] File does not exist:
/var/www/robots.txt
Apache Access Logs
Apache Error Logs

11. MySQL Logs

12. Dec 2 17:09:01 ip-123-45-6-78 CRON[11785]:
pam_unix(cron:session): session opened for user root by (uid=0)
Dec 2 17:09:06 ip-123-45-6-78 CRON[11785]:
pam_unix(cron:session): session closed for user root
Dec 2 17:10:01 ip-123-45-6-78 CRON[11822]:
pam_unix(cron:session): session opened for user root by (uid=0)
Dec 2 17:10:08 ip-123-45-6-78 CRON[11822]:
pam_unix(cron:session): session closed for user root
Ubuntu Auth Logs

13. Application Logs
• There really are no standards
• Luckily there is usually some sort of timestamp
???? ????? ?? ?????
???? ????? ?? ?????

14. Enter ELK Stack
• Easy to install a simple stack
‣ Less than 1 hour using Digital Ocean blog post
• Made up of 3 (really 4) separate components
‣ Elasticsearch
‣ Logstash
‣ Kibana
‣ Log ship agent (rsyslog, logstash forwarder, filebeat)

15. Logger’s Playday Parade by John Lloyd https://flic.kr/p/34cyYD License: CC BY 2.0
All your logs are belong to us.
Logstash

16. Logstash
• input → filter → output workflow
• move logs from any server using many different
methods
• normalize timestamps
• powerful syntax for transformations, additions, tags
• outputs directly into Elastic Search

17. Defining Inputs
input {
lumberjack {
port => 5043
type => "logs"
ssl_certificate => “/path/file.crt”
ssl_key => "/path/file.key"
}
}
/etc/logstash/conf.d/01-lumberjack-input.conf
input {
file {
path => [“/path/to.log"]
type => "apache-access"
}
}

18. Defining Filters
filter {
if [type] == "apache-access" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
}
/etc/logstash/conf.d/10-apache-access.conf

19. Defining Outputs
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}
/etc/logstash/conf.d/30-output.conf

20. logstash-forwarder
"network": {
"servers": [ "logstash_ip:5043" ],
"ssl ca": “/path/file.crt",
"timeout": 15
},
"files": [
{
"paths": [ “/var/log/file.log” ],
"fields": { "type": "some-type" }
}
]
/etc/logstash-forwarder.conf
Note: the logstash-forwarder project has been replaced by filebeat

21. Elastic Search
bands3 by DaveBleasdale https://flic.kr/p/Bn2yw License: CC BY 2.0

22. Elastic Search
• makes log files easily searchable
• blazing fast
• scalable
• maybe you’re already using it for Dev
• lots of power, but you don’t need to tweak much for
basic ELK stack

23. G-BNIVE panel by Andy / Andrew Fogg https://flic.kr/p/5RSJp License: CC BY 2.0
Kibana

24. Kibana
• Easily search for keywords in logfiles
• Visualize your data events over time
• Drill-down into individual events
• Create dashboards
• Share dashboards

26. Live Demo
(aka Crash n Burn Time)

27. Benefits / Uses
• ELK components can scale independently
• Visualize traffic patterns & user profiles
• Root-cause analysis
• Graph any arbitrary data you want to output from your
applications
• Store log files in a central location & get them off the
individual servers
• Bring non-technical people into the conversation

28. Further Resources
• Elastic website
https://www.elastic.co/webinars/introduction-elk-stack
• Digital Ocean tutorials (search for ELK)
https://www.digitalocean.com/community/tutorials
• Digital Ocean one-click-apps
https://www.digitalocean.com/features/one-click-apps/elk/
• The Delvia team is here to help
pmeth@delvia.com

29. Questions
(hopefully I have answers)

30. Contact me
Peter Meth
pmeth@delvia.com
devopsmeth
416-677-6384
www.delvia.com