Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Got Logs? 
ELK stories and awesome. 
@jordansissel 
#PuppetConf 2014
Disclaimer 
I apologJizUeS fTo rK aIDnDy IoNbGn.o AxiNouIMs AaTnIiOmNatSio AnRs Ein A thMisA pZrINesGentation.
Hello friends! 
I work on Logstash 
at Elasticsearch
#PuppetApproved 
APPROVED
THE KING OF 
PAIN MOUNTAIN 
Richard Pijnenburg 
! 
Very Nice Human 
! 
Puppet Specialist 
 
Twitter: @Richardp82 — Github...
Sorry about the previous slide. 
I got a little wild.
Always be testing!
puppet testing tools? 
• rspec-puppet 
• puppet-doc-lint 
• puppet-lint 
• beaker
Elasticsearch  Puppet
Story time!
Let’s talk about ELK in the Wild!
“Oops it broke”
{ }
Complex data at high volume is 
hard, but we can help.
ELK @ Bloomberg
need: 
1.5 billion events per second
need: 
logs from thousands of servers
need: 
integration with in-house tools
Rub some 
ELK on it! 
Picture: Wikipedia - Richard Lydekker - Public Domain
10+ departments using it
ELK @
50.56.197.244 - - [13/Sep/2012:02:34:37 -0400] "GET / HTTP/1.1" 200 41687 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows ...
grep 

(?<a0>(?<a1>(?<a2>b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?| 
Jul(?:y)?|Aug(?:u...
(?<a0>(?<a1>(?<a2>b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?| 
Aug(?:u...
ELK @ CERN
-Ykb2j2ojYU
“CERN - Accelerating Science 
with Puppet - Tim Bell” 
from PuppetConf 2012
thousands of events per second
“What we really liked about Kibana, that the application 
developers can create their own dashboards, and they can 
monito...
“Kibana is well done, usable by non-experts.” 
- Gergo Horanyi @ CERN
democratize your data
“Use Elasticsearch to classify and track OpenStack gate failures" 
OpenStack elastic-recheck
Online Gaming
“Feels like Logstash is being slow”
http://upload.wikimedia.org/wikipedia/commons/5/56/India_Victor_Grigas_2011-13.jpg
Yep, that’s a bug! 
http://en.wikipedia.org/wiki/Scutelleridae
This has a measured 6.3x perf 
improvement in grok filter 
performance.
Lots of success! Hurray!
Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014
Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014
Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014
Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014
Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014
Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014
Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014
Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014
Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014
Nächste SlideShare
Wird geladen in …5
×

Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014

3.712 Aufrufe

Veröffentlicht am

Got Logs? Get Answers with Elasticsearch ELK - Jordan Sissel, Elasticsearch

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014

  1. 1. Got Logs? ELK stories and awesome. @jordansissel #PuppetConf 2014
  2. 2. Disclaimer I apologJizUeS fTo rK aIDnDy IoNbGn.o AxiNouIMs AaTnIiOmNatSio AnRs Ein A thMisA pZrINesGentation.
  3. 3. Hello friends! I work on Logstash at Elasticsearch
  4. 4. #PuppetApproved APPROVED
  5. 5. THE KING OF PAIN MOUNTAIN Richard Pijnenburg ! Very Nice Human ! Puppet Specialist  Twitter: @Richardp82 — Github and IRC: electrical
  6. 6. Sorry about the previous slide. I got a little wild.
  7. 7. Always be testing!
  8. 8. puppet testing tools? • rspec-puppet • puppet-doc-lint • puppet-lint • beaker
  9. 9. Elasticsearch  Puppet
  10. 10. Story time!
  11. 11. Let’s talk about ELK in the Wild!
  12. 12. “Oops it broke”
  13. 13. { }
  14. 14. Complex data at high volume is hard, but we can help.
  15. 15. ELK @ Bloomberg
  16. 16. need: 1.5 billion events per second
  17. 17. need: logs from thousands of servers
  18. 18. need: integration with in-house tools
  19. 19. Rub some ELK on it! Picture: Wikipedia - Richard Lydekker - Public Domain
  20. 20. 10+ departments using it
  21. 21. ELK @
  22. 22. 50.56.197.244 - - [13/Sep/2012:02:34:37 -0400] "GET / HTTP/1.1" 200 41687 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent"! 89.96.171.210 - - [13/Sep/2012:02:32:49 -0400] "GET /files/logstash/logstash-1.1.0-monolithic.jar HTTP/1.1" 200 40923996 "-" "Chef Client/0.10.10 (ruby-1.9.3-p194; ohai-0.6.4; amd64-freebsd8; +http:// opscode.com)"! 37.57.128.238 - - [13/Sep/2012:02:37:24 -0400] "GET / HTTP/1.1" 200 41687 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent"! 199.21.99.109 - - [13/Sep/2012:02:38:12 -0400] "GET /blog/tags/packaging HTTP/1.1" 200 15152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"! 180.76.6.232 - - [13/Sep/2012:02:38:23 -0400] "GET /blog/tags/wrt54gl HTTP/1.1" 200 8867 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"! 217.227.233.68 - - [13/Sep/2012:02:38:25 -0400] "GET /articles/ssh-security/ HTTP/1.1" 200 16543 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0"! 217.227.233.68 - - [13/Sep/2012:02:38:26 -0400] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/articles/ssh-security/" "Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0"! 217.227.233.68 - - [13/Sep/2012:02:38:26 -0400] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/articles/ssh-security/" "Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0"! 217.227.233.68 - - [13/Sep/2012:02:38:26 -0400] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/articles/ssh-security/" "Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/ 20100101 Firefox/15.0"! 217.227.233.68 - - [13/Sep/2012:02:38:31 -0400] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/ 20100101 Firefox/15.0"! 184.73.137.50 - - [13/Sep/2012:02:38:28 -0400] "GET /files/logstash/logstash-1.1.1-monolithic.jar HTTP/1.1" 200 53813805 "-" "Chef Client/0.10.8 (ruby-1.8.7-p334; ohai-0.6.10; i686-linux; +http:// opscode.com)"! 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/ HTTP/1.1" 200 4483 "http://news.ycombinator.com/item?id=4417660" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"! 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/bootstrap/css/bootstrap-responsive.min.css HTTP/1.1" 200 7680 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"! 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/css/style.css HTTP/1.1" 200 2715 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"! 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/css/jquery.ui.datepicker.css HTTP/1.1" 200 33035 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/ 537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"! 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/excanvas.min.js HTTP/1.1" 200 19415 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"! 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/bootstrap/css/bootstrap.min.css HTTP/1.1" 200 71463 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/ 537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"! 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.history.js HTTP/1.1" 200 6466 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"! 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/css/jquery-ui-1.8.16.custom.css HTTP/1.1" 200 50829 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/ 537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"! 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.flot.min.js HTTP/1.1" 200 37554 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"! 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.flot.selection.min.js HTTP/1.1" 200 3532 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/ 537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"! 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.smartresize.js HTTP/1.1" 200 1123 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"! 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/iso8601.min.js HTTP/1.1" 200 486 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"! 24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/safebase64.js HTTP/1.1" 200 3264 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"!
  23. 23. grep 
  24. 24. (?<a0>(?<a1>(?<a2>b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?| Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)b) +(?<a3>(?:(?: 0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])) (?<a4>(?!<[0-9])(?<a5>(?:2[0123]|[01][0-9])):(?<a6> (?:[0-5][0-9]))(?::(?<a7>(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?![0-9]))) (?<a8>(?:(?<a9>b(?: [0-9A-Za-z][0-9A-Za-z-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(.?|b))|(?<a10>(?<![0-9])(?:(?: 25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]? [0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])))) (?<a11>(?<a12>(?:[w._/%-]+))(?: [(?<a13>b(?:[1-9][0-9]*)b)])?): (?<a14>(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.] (?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]| [0-1]?[0-9]{1,2}))(?![0-9])):(?<a15>(?:[+-]?(?:[0-9]+))) [(?<a16>(?<a17>(?:(?:0[1-9])|(?:[12] [0-9])|(?:3[01])|[1-9]))/(?<a18>b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?| Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)b)/(?<a19>[0-9]+): (?<a20>(?!<[0-9])(?<a21>(?:2[0123]|[01][0-9])):(?<a22>(?:[0-5][0-9]))(?::(?<a23>(?: (?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?![0-9])).(?<a24>(?:[+-]?(?:[0-9]+))))] (?<a25>S+) (?<a26> S+)/(?<a27>S+) (?<a28>(?:[+-]?(?:[0-9]+)))/(?<a29>(?:[+-]?(?:[0-9]+)))/(?<a30>(?: [+-]?(?:[0-9]+)))/(?<a31>(?:[+-]?(?:[0-9]+)))/(?<a32>S+) (?<a33>(?:[+-]?(?:[0-9]+))) (? <a34>S+) (?<a35>.*?) (?<a36>.*?) (?<a37>S+) (?<a38>(?:[+-]?(?:[0-9]+)))/(? <a39>(?:[+-]?(?:[0-9]+)))/(?<a40>(?:[+-]?(?:[0-9]+)))/(?<a41>(?:[+-]?(?:[0-9]+)))/(? <a42>S+) (?<a43>(?:[+-]?(?:[0-9]+)))/(?<a44>(?:[+-]?(?:[0-9]+))) {(?<a45>(? <a46>.*?))} {(?<a47>(?<a48>.*?))} "(?<a49>bw+b) (?<a50>(?<a51>(?:/[A-Za- z0-9$.+!*'(){},~:;=#%_-]*)+)(?:(?<a52>?[A-Za-z0-9$.+!*'(){},~#%&/=:;_-]*))?) HTTP/(?<a53> (?:(?<a54>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:.[0-9]+)?)|(?:.[0-9]+))))))")
  25. 25. (?<a0>(?<a1>(?<a2>b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?| Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)b) +(?<a3>(?:(?:0[1-9])|(?:[12] [0-9])|(?:3[01])|[1-9])) (?<a4>(?!<[0-9])(?<a5>(?:2[0123]|[01][0-9])):(?<a6>(?:[0-5][0-9]))(?::(? <a7>(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?![0-9]))) (?<a8>(?:(?<a9>b(?:[0-9A-Za-z][0-9A-Za-z-] {0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(.?|b))|(?<a10>(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9] {1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]| [0-1]?[0-9]{1,2}))(?![0-9])))) (?<a11>(?<a12>(?:[w._/%-]+))(?:[(?<a13>b(?:[1-9][0-9]*)b)])?): (? <a14>(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?: 25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])):(?<a15>(?:[+-]?(?: [0-9]+))) [(?<a16>(?<a17>(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))/(?<a18>b(?:Jan(?:uary)?| Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?| Nov(?:ember)?|Dec(?:ember)?)b)/(?<a19>[0-9]+):(?<a20>(?!<[0-9])(?<a21>(?:2[0123]|[01] [0-9])):(?<a22>(?:[0-5][0-9]))(?::(?<a23>(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?![0-9])).(?<a24>(?:[+-]? (?:[0-9]+))))] (?<a25>S+) (?<a26>S+)/(?<a27>S+) (?<a28>(?:[+-]?(?:[0-9]+)))/(?<a29> (?:[+-]?(?:[0-9]+)))/(?<a30>(?:[+-]?(?:[0-9]+)))/(?<a31>(?:[+-]?(?:[0-9]+)))/(?<a32>S+) (? <a33>(?:[+-]?(?:[0-9]+))) (?<a34>S+) (?<a35>.*?) (?<a36>.*?) (?<a37>S+) (?<a38> (?:[+-]?(?:[0-9]+)))/(?<a39>(?:[+-]?(?:[0-9]+)))/(?<a40>(?:[+-]?(?:[0-9]+)))/(?<a41>(?:[+-]?(?: [0-9]+)))/(?<a42>S+) (?<a43>(?:[+-]?(?:[0-9]+)))/(?<a44>(?:[+-]?(?:[0-9]+))) {(?<a45>(? <a46>.*?))} {(?<a47>(?<a48>.*?))} "(?<a49>bw+b) (?<a50>(?<a51>(?:/[A-Za-z0- 9$.+!*'(){},~:;=#%_-]*)+)(?:(?<a52>?[A-Za-z0-9$.+!*'(){},~#%&/=:;_-]*))?) HTTP/(?<a53>(?:(? <a54>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:.[0-9]+)?)|(?:.[0-9]+))))))") http://upload.wikimedia.org/wikipedia/commons/7/7f/Empty-frame.png
  26. 26. ELK @ CERN
  27. 27. -Ykb2j2ojYU
  28. 28. “CERN - Accelerating Science with Puppet - Tim Bell” from PuppetConf 2012
  29. 29. thousands of events per second
  30. 30. “What we really liked about Kibana, that the application developers can create their own dashboards, and they can monitor their systems on their own, without any help from some other team” - Gergo Horanyi @ CERN
  31. 31. “Kibana is well done, usable by non-experts.” - Gergo Horanyi @ CERN
  32. 32. democratize your data
  33. 33. “Use Elasticsearch to classify and track OpenStack gate failures" OpenStack elastic-recheck
  34. 34. Online Gaming
  35. 35. “Feels like Logstash is being slow”
  36. 36. http://upload.wikimedia.org/wikipedia/commons/5/56/India_Victor_Grigas_2011-13.jpg
  37. 37. Yep, that’s a bug! http://en.wikipedia.org/wiki/Scutelleridae
  38. 38. This has a measured 6.3x perf improvement in grok filter performance.
  39. 39. Lots of success! Hurray!

×