Don’t turn your logs into cuneiform

1. Don’t turn your logs into
cuneiform
Andrei Rebrov

3. Cuneiform

4. Logs

5. What’s the
difference?

6. Let’s dig into
• Too much logs
• Too much information inside
• They are distributed across several
machines
• We are not supposed to read blogs

7. Time for tools

8. • Open source
• Collects and parses
Logstash
http://logstash.net

9. Key feature
Too many sources?
•syslog
•nginx access log
•application logs
•database logs
How about their format?

11. З parts of Logstash
•Inputs
•Filters
•Outputs

12. http://logstash.net/docs/1.1.12/
Inputs
• file
• eventlog
• ganglia
• heroku
• syslog
• tcp

13. Filters
http://logstash.net/docs/1.1.12/
• anonymize
• date
• mutate
• grok
• grep

14. http://logstash.net/docs/1.1.12/
Outputs
• file
• graphite
• http
• irc
• email
• zabbix

15. input { stdin { type => "stdin-type"} }
output { stdout { debug => true debug_format =>
"json"} }
java -jar logstash-1.1.9-monolithic.jar agent -f
logstash-simple.conf
Easy to adopt

16. Example

17. input { stdin { type => "stdin-type"} }
filter { grok { type => "stdin-type" pattern =>
"Hello %{DATA:message}!" } }
output { stdout { debug => true debug_format =>
"json"} }
java -jar logstash-1.1.9-monolithic.jar agent -f
logstash-simple.conf
Time to parse

18. Example

19. input { stdin { type => "stdin-type" } }
output {
stdout { debug => true debug_format => "json" }
elasticsearch { embedded => true }
}
java -jar logstash-1.1.9-monolithic.jar agent -f
logstash-search.conf
How to output

20. Example

21. Where to store
ElasticSearch
http://www.elasticsearch.org

22. What is Elasticsearch?
Distibuted RESTful search server

«Real-time» search

RESTful API

Fulltext search

YAML/JSON configuration

23. Beautiful UI
http://kibana.org

24. User-friendly UI

25. Test Node
Test Node
Logstash ElasticSearch
Kibana
???
How to compose them

26. We need shippers!

27. Logstash shippers
• beaver - python, multiple outputs
• woodchuck - ruby, multiple outputs
• awesant - perl, multiple outputs supported
• lumberjack - C, encrypted+compressed
transport
• syslog-shipper - ruby, syslog tcp
• remote_syslog - ruby, syslog tcp/tls
• Message::Passing - perl, multiple inputs and
outputs
• nxlog - C, multi platform including windows,

28. Build time

29. Lumberjack installation
apt-get install rubygems
gem install fpm
export PATH=$PATH:/var/lib/gems/1.8/bin
git clone https://github.com/jordansissel/
lumberjack.git
cd lumberjack
make
make deb
dpkg -i lumberjack_0.0.8_amd64.deb

30. Logstash installation
mkdir /opt/logstash
wget
https://logstash.objects.dreamhost.com/
release/logstash-1.1.9-monolithic.jar -O /opt/
logstash/logstash.jar

31. Elasticsearch installation
wget http://download.elasticsearch.org/
elasticsearch/elasticsearch/ elasticsearch-
0.20.2.tar.gz
tar -zxf elasticsearch-0.20.2.tar.gz

32. Kibana installation
git clone --branch=kibana-ruby
https://github.com/ rashidkpc/Kibana.git
/opt/kibana
apt-get install rubygems libcurl4-openssl-
dev
export PATH=$PATH:/var/lib/gems/1.8/bin
cd /opt/kibana
bundle install

33. Lumberjack startup
/opt/lumberjack/bin/lumberjack --
host your.logstash.host --port port-for-
these-logs --ssl-ca-path /etc/ssl/
logstash.pub
Для генерации ключей на logstash
сервере:
openssl req -x509 -newkey rsa:2048 -keyout
/etc/ ssl/logstash.key -out
/etc/ssl/logstash.pub - nodes -days 365

34. Configuring Logstash #1
input {
lumberjack {
type => "apache-access"
port => 3338
ssl_certificate => "/etc/ssl/logstash.pub"
ssl_key => "/etc/ssl/logstash.key"
}
}

35. Configuring Logstash #2
filter {
date {
type => "apache-access"
timestamp => "dd/MMM/yyyy:HH:mm:ss Z"
}
}

36. Configuring Logstash #3
output {
elasticsearch {
embedded => false
cluster => logs
host => "172.28.2.2"
index => "apache-%{+YYYY.MM}"
type => "apache-access"
}
}

37. Logstash startup
/usr/bin/java -jar /opt/logstash/logstash.jar
agent -f <path-to-your.conf> -l <path-to-
where-you-want-the.log>

38. Configuring Elasticsearch
cluster.name: logs
index.number_of_replicas: 0
path.data: /elasticsearch/data
path.work: /elasticsearch/work
path.logs: /elasticsearch/logs
bootstrap.mlockall: true
discovery.zen.ping.multicast.enabled: false

39. Elasticsearch startup
./bin/elasticsearch –f
or as a daemon
./bin/elasticsearch

40. Kibana startup
ruby kibana.rb

41. Profit!

42. What to read

http://www.logstashbook.com/code/

https://github.com/logstash/logstash/blob/v1.1.12/patter
ns/grok-patterns

http://grokdebug.herokuapp.com/

http://www.infoq.com/articles/review-the-logstash-booл

http://www.elasticsearch.org/tutorials/using-
elasticsearch-for-logs/

https://lucene.apache.org/core/old_versioned_docs/ver
sions/3_5_0/queryparsersyntax.html

http://www.elasticsearch.org/tutorials/elasticsearch-on-
ec2/

http://blog.lusis.org/blog/2012/01/31/load-balancing-
logstash-with-amqp/

43. Twitter
@andrebrov
E-mail
arebrov@scrumtrek.ru
Skype
rebrov.andrey
Q&A