Logstash Presentation for Brisbane DevOps Meetup on 29th May 2013.

1. BRISBANE DEVOPS MEETUP
!!! LOGSTASH !!!
30TH MAY 2013
Paul Czarkowski / paul@paulcz.net /
@pczarkowski

2. WHO AM I ?
MORE OPS THAN DEV
Started out as an SA at several ISPs
Moved into Games ... IT Manager @
Pandemic (Brisbane)
Moved to BioWare to launch Star Wars:
The Old Republic
Transferred to EA, Manage Cloud Team for
Infrastruture Operations
Through M&As have 9 years tenure at EA.

3. WHO AM I ?
STUFF I LIKE TO DO ...
Automation ( and I don't just mean Puppet
and/or Chef )
Agile ( Scrum / Kanban )
Cloud ( OpenStack, Amazon )
Monitoring / Logging
Solve problems
Cook ( I would be a chef if they didn't work
so hard for such little pay)

4. WHAT IS A LOG?
A log is a human readable, machine parsable
representation of an event.
LOG = TIMESTAMP + DATA
Jan 19 13:01:13 paulcz­laptop anacron[7712]:
Normal exit (0 jobs run)
120607 14:07:00 InnoDB: Starting an apply
batch of log records to the database...
[1225306053] SERVICE ALERT:
FTPSERVER;FTP
SERVICE;OK;SOFT;2;FTP OK ­ 0.029
second response time on port 21 [220
ProFTPD 1.3.1 Server ready.]
[Sat Jan 19 01:04:25 2013] [error] [client
78.30.200.81] File does not exist:
/opt/www/vhosts/crappywebsite/html/robots.txt

5. THERE'S AN ISO FOR THAT!
ISO 8601

6. A LOG IS HUMAN READABLE...
“A human readable, machine parsable
representation of an event.”
208.115.111.74 - - [13/Jan/2013:04:28:55 -0500] "GET /robots.txt H
TTP/1.1"
301 303 "-" "Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@g
mail.com)"

8. ...BUT LOGS ARE NOT !
But they're machine parsable right?

9. LOGS ARE MACHINE PARSEABLE
208.115.111.74 - - [13/Jan/2013:04:28:55 -0500] "GET /robots.txt H
TTP/1.1"
301 303 "-" "Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@g
mail.com)"

10. LOGS ARE MACHINE PARSEABLE
Actual Regex to parse apache logs.

11. LOGS ARE MACHINE PARSEABLE
Actual Regex to parse apache logs.

12. LOGS ARE MACHINE PARSEABLE
Users will now call PERL Ninja to solve
every problem they have ­ Hero Syndrome.
Does it work for every possible log line ?
Who's going to maintain that shit ?
Is it even useful without being surrounded
by [bad] sysadmin scripts ?

13. SO WE AGREE ... THIS IS BAD.
208.115.111.74 - - [13/Jan/2013:04:28:55 -0500] "GET /robots.txt H
TTP/1.1"
301 303 "-" "Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@g
mail.com)"

14. LOGSTASH TURNS THIS
INTO THAT
208.115.111.74 - - [13/Jan/2013:04:28:55 -0500] "GET /robots.txt H
TTP/1.1"
301 303 "-" "Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@g
mail.com)"
{
"client address": "208.115.111.74",
"user": null,
"timestamp": "2013-01-13T04:28:55-0500",
"verb": "GET",
"path": "/robots.txt",
"query": null,
"http version": 1.1,
"response code": 301,
"bytes": 303,
"referrer": null
"user agent": "Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@g
mail.com)"
}

15. LOGSTASH PLUGINS...
cat chain.plugins | grep together | sed
's/like/unix/' > pipeline

16. LOGSTASH PLUGINS...
~ 25+ INPUT PLUGINS
file
tcp,udp
amqp, zeromq, redis, sqs
twitter, irc
lumberjack
...

17. LOGSTASH PLUGINS...
~ 20+ FILTER PLUGINS
date
grok
geoip
multiline
metrics
split
...

18. LOGSTASH PLUGINS...
~ 40+ OUTPUT PLUGINS
Elasticsearch
StatsD, Graphite, Nagios, PagerDuty
amqp, zeromq, redis, sqs
boundary, cloudwatch, zabbix
lumberjack
...

19. TWO VERY IMPORTANT FILTERS
Let's talk briefly about two filters that are
very important to making our logs useful

20. FILTER - DATE
takes a timestamp and makes it ISO 8601
Compliant
Turns this:
Into this:
13/Jan/2013:04:28:55 -0500
2013-01-13T04:28:55-0500

21. FILTER - GROK
Grok parses arbitrary text and structures it.
Makes complex regex patterns simple.
USERNAME [a-zA-Z0-9_-]+
USER %{USERNAME}
INT (?:[+-]?(?:[0-9]+))
MONTH b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|A
pr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(
?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)
?)b
DAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu
(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)
COMBINEDAPACHELOG %{IPORHOST:clientip} %{USER:iden
t} %{USER:auth}
[%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOT
SPACE:request}
(?: HTTP/%{NUMBER:httpversion})?|-)" %{NUMBER:re
sponse}
(?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}

22. Remember our apache log from earlier?

23. Define Inputs and Filters.
input {
file {
type => "apache"
path => ["/var/log/httpd/httpd.log"]
}
}
filter {
grok {
type => "apache"
pattern => "%{COMBINEDAPACHELOG}"
}
date {
type => "apache"
}
geoip {
type => "apache"
}
}

24. Define some outputs.
output {
statsd {
type => "apache"
increment => "apache.response.%{response}"
# Count one hit every event by response
}
elasticsearch {
type => "apache"
}
}

25. INSTANT GRAPHIFICATION !

27. KIBANA

28. KIBANA

29. ANALYZE TWITTER STREAMS
Marketing
Customer feedback
good for load testing ­ 'bieber'

31. LOGSTASH - TWITTER INPUT
input {
twitter {
type => "twitter"
keywords => ["bieber","beiber"]
user => "username"
password => "*******"
}
}
output {
elasticsearch {
type => "twitter"
}
}

32. 4% Bieber Fans can't spell his name.
10% Tweets from Blackberry ( 50ish
business execs? )
~ 200 Bieber tweets per minute.

35. ALREADY HAVE CENTRAL
RSYSLOG/SYSLOGNG SERVER?
input {
file {
type => "syslog"
path => ["/data/rsyslog/**/*.log"]
}
}
filter {
### a bunch of groks, a date, and other filters
}
output {
type => "elasticsearch"
}

36. ACT AS A CENTRAL SYSLOG SERVER
GOOD FOR APPLIANCES / SWITCHES
input {
tcp {
type => "syslog"
port => "514"
}
udp {
type => "syslog"
port => "514"
}
}
filter {
### a bunch of groks, a date, and other filter
s
}
output {
type => "elasticsearch"
}

37. IN CLOUD? DON'T OWN
NETWORK?
Use an encrypted Transport
Logstash Agent
Logstash Indexer
input { file { ... } }
output {
lumberjack {
hosts => ["logstash-indexer1", "logs
tash-indexer2"]
ssl_certificate => "/etc/ssl/logstash.crt"
}
}
input {
lumberjack {
ssl_certificate => "/etc/ssl/logstash.crt"
ssl_key => "/etc/ssl/logstash.key"
}
}
output { elasticsearch {} }

38. SYSTEM METRICS ?
input {
exec {
type => "system-loadavg"
command => "cat /proc/loadavg | awk '{print $1
,$2,$3}'"
interval => 30
}
}
filter {
grok {
type => "system-loadavg"
pattern => "%{NUMBER:load_avg_1m} %{NUMBER:loa
d_avg_5m}
%{NUMBER:load_avg_15m}"
named_captures_only => true
}
}
output {
graphite {
host => "10.10.10.10"
port => 2003
type => "system-loadavg"
metrics => [ "hosts.%{@source_host}.load_avg.1
m", "%{load_avg_1m}",
"hosts.%{@source_host}.load_avg.5m", "%{load_avg_5
m}",
"hosts.%{@source_host}.load_avg.15m", "%{load_avg_
15m}" ]
}
}

39. UNIQUE PROBLEM TO SOLVE ?
write a logstash module!
Input ­ Snmptrap
Filter ­ Translate
Can do powerful things with [ boilerplate + ] a
few lines of ruby

40. SCALING LOGSTASH
USE QUEUES ( RABBITMQ, REDIS) TO HELP
SCALE HORIZONTALLY.
Local log dir on clients = cheap queue

41. PERFORMANCE... EVENTS/SEC
HP BL460, 48Gb, 24 Cores.
Several Filters ­ Groks, Mutates, etc
jump in performance from setting 8 filter
workers (­w 8)

42. MISC CONSIDERATIONS
file limits ( lol 1024 )
Java ( use JRE7 for Performance )
Limit Memory Usage
Elasticsearch ­ ~50% RAM for Java
Heap
Redis / Rabbit ­ 1Gb
Logstash ­ 256Mb ­> 512Mb for Java
Heap
Be safe ­ avoid deadly OOM killer.
Kibana / Elasticsearch Security ­ Behind
Reverse Proxy!

43. ELASTICSEARCH
Primary Storage/Search engine for logstash
Clusterable/Scalable Search Engine
RESTful API
Lucene Syntax
Very large topic on its own...

44. FURTHER READING
http://www.logstash.net/
http://www.logstashbook.com/ [James
Turnbull]
https://github.com/paulczar/vagrant­logstash
http://jujucharms.com/charms/precise/logstash­
indexer
Logstash Puppet (github/electrical)
Logstash Chef (github/lusis)

45. OTHER TOOLS YOU SHOULD ALL BE USING...
Vagrant
Chef / Puppet ( obviously! )
FPM
Omnibus
LXC containers for lightweight VMs
OpenStack ( run a cloud locally for dev )

46. QUESTIONS?
Paul Czarkowski / paul@paulcz.net /
@pczarkowski