Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

Logstash family introduction

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Nächste SlideShare
Logstash
Logstash
Wird geladen in …3
×

Hier ansehen

1 von 38 Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Ähnlich wie Logstash family introduction (20)

Anzeige

Aktuellste (20)

Logstash family introduction

  1. 1. Logstash  Family  Introduc4on Owen
  2. 2. What  is  a  log •  Oxford  Dic4onary   – a  thick  piece  of  wood  that  is  cut  from  or  has  fallen   from  a  tree   – (also  logbook)  an  official  record  of  events  during  a   par4cular  period  of  4me,  especially  a  journey  on  a   ship     •  4me  +  data        
  3. 3. In  theory,  life  cycle  of  log Record Transmit Analyze Store Delete
  4. 4. In  design,  life  cycle  of  log Record Transmit Store Delete
  5. 5. In  fact,  life  cycle  of  log Record Delete
  6. 6. Problems •  Logging  to  a  database  or  filesysytem   •  Logging  has  placed  a  load  on  the  database  and   filesystem   •  Mul4ple  log    formats   •  No  easy  way  to  search  logs   •  No  easy  method  to  gather  sta4s4cs  
  7. 7. Find  the  logs  of  16  computers  6   months  ago  ?
  8. 8. Why  use  Logstash? •  A lot choices!  •  But we want a free & high-integrality & easy to use solution •  splunk (finding your faults, just like mom) •  facebookarchive/scribe (2682 ★) •  Graylog2(Server+WUI 1683 ★) •  fluentd (2038 ★) •  logstash (2689 ★)
  9. 9. logstash  and  other  things hRps://www.youtube.com/watch?v=RuUFnog29M4
  10. 10. Logstash •  Open  Source,  Apache  Licence   •  WriRen  in  JRuby,  Runs  on  JVM   •  Plugins  easily  wriRen  in  Ruby   •  Process  mul4ple  format  (  input,  output  )   •  Logstash  Family!  (  Elas4cSearch  ,  Kibana)  
  11. 11. LogStash  Family  architecture
  12. 12. Elas4cSearch •  A  response  to  the  claim  :  “Search  is  hard”   •  Powerful  indexing  &  search  tool   •  search  &  index  data  available  Rescully  as   JSON  over  HTTP  
  13. 13. Kibana
  14. 14. All-­‐in-­‐one!
  15. 15. How logstash works? •  logstash process events, not (only) loglines! •  “The logstash agent is a processing pipeline with 3 stages: – inputs -> filters -> outputs.” – separate threads •  “Inputs generate events, filters modify them, outputs ship them elsewhere.” •  -- [the life of an event in logstash] 
  16. 16. In  my  thinking,  Event  Life  Cycle Input filter     output
  17. 17. In  fact,  Event  Life  Cycle event  (Input  -­‐>  output) event   -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐   input   filter     output
  18. 18. Logstash  is  a  wooden  tube Input Input Input filter output codec filter filter output output
  19. 19. Logstash  plugins  Workflow   •  inputs   –  How  events  get  into  LogStash.   •  codecs     –   convert  an  incoming  format  into  an  internal     representa4on   •  filters   –   processing  ac4ons  on  events  :  modify  events  or  drop   events   •  outputs   –  How  output  events  from  LogStash
  20. 20. Logstash  plugins
  21. 21. What  is  an  event!? •  A  @4mestamp  (ISO  8601  4mestamp)   •  A  messsage  field  (  data  )   •  A  @version   •  host  (  the  host  of  sender)   •  type(  syslog,  irc,  etc)  
  22. 22. Exersice:  Hello  Word! java  -­‐jar  logstash-­‐1.1.12-­‐flatjar.jar  agent  -­‐f   hello.conf java  -­‐jar  logstash.jar  agent  -­‐f  hello.conf
  23. 23. Input   •  tcp   •  udp   •  unix   •  file   •  syslog   •  redis   •  logstash-­‐fowarder(former  Lumberjack)
  24. 24. Codecs •  plain   •  json   •  rubydebug   •  mul4line
  25. 25. Outputs   •  mongodb   •  elas4cSearch   •  email   •  file   •  jira  
  26. 26. Exercise:  Mul4ple  input  &  output
  27. 27. logstash-­‐forwader •  ♫  I'm  a  lumberjack  and  I'm  ok!  I  sleep  when   idle,  then  I  ship  logs  all  day!  I  parse  your  logs,  I   eat  the  JVM  agent  for  lunch!  ♫   •  WriRen  in  Go   •  lumberjack  is  reserved  for  protocol   •  Resource  Usage  Concerns   •  Need  an  SSL  CA  to  verify  the  server  
  28. 28. lumberjack •  Encryp4on  &  Authen4ca4on  (TLS)   •  Compression  (  reduce  bandwidth)   •  Sequence  &  ack  behavior  like  TCP   •  Low  latency   •  Reliable  Aplica4on-­‐Level  message  transport
  29. 29. Forwarder  Sample
  30. 30. Filters   •  date   •  grok   •  drop   •  geoIP   •  mutate     •  mul4line  
  31. 31. Exercise:  Parse  Data
  32. 32. filter  config  
  33. 33. powerful  grok •  Parse  arbitrary  text  and  structure  it.   •  The  syntax  for  a  grok  paRern  is       –  %{SYNTAX:SEMANTIC}   •  55.3.244.1        GET          /index.html          15824   –  %{IP:client}     –  %{WORD:method}     –  %{URIPATHPARAM:request}     –  %{NUMBER:bytes}   •  hRps://github.com/elas4csearch/logstash/blob/ v1.4.2/paRerns/grok-­‐paRerns  
  34. 34. grok  sample  
  35. 35. drop
  36. 36. mutate   •  Muta4ons  on   fields.     –   rename   –   remove   – replace   – join   – split   – upper   – lower
  37. 37. mul4line •  Codecs  &  filter
  38. 38. Reference •  hRps://www.digitalocean.com/community/ tutorials/how-­‐to-­‐use-­‐logstash-­‐and-­‐kibana-­‐to-­‐ centralize-­‐and-­‐visualize-­‐logs-­‐on-­‐ubuntu-­‐14-­‐04   •  hRp://www.vmdoh.com/blog/centralizing-­‐ logs-­‐lumberjack-­‐logstash-­‐and-­‐elas4csearch   •  hRp://jpmens.net/2012/08/09/i-­‐grok-­‐how-­‐to-­‐ mutate-­‐a-­‐file-­‐with-­‐logstash/   •  hRp://gleenders.blogspot.tw/2014/02/ logstash-­‐glassfish.html  

×