Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige

Hier ansehen

1 von 24 Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Anzeige

Ähnlich wie Logstash (20)

Aktuellste (20)

Anzeige

Logstash

  1. 1. Goal of Centralized log collection • Collect, parse and store log events • Make log events searchable • Analyze log events
  2. 2. Log collection strategy Indexer Transporter/Broker Agent Machine Instance UI Storage and Search engine Storage Log collector and log shipper
  3. 3. Agent • Agent works as log collector and log shipper • It tails the log on regular interval from log files • And ships the log to transporter or broker.
  4. 4. Transporter/broker • Event shipping rate at agent vary with time. • Transportor or broker makes sure that log shipping rate to indexer is same all the time.
  5. 5. Indexer • Indexer fetches logs from transporter/broker. • Indexer indexes the log events. • Indexer calls the API of “Storage and search engine” to write the data on storage.
  6. 6. Storage and search engine • Stores the indexed data at specific location in a well defined format • It provides API for storage, search and analysis of log events.
  7. 7. Tools for Log collection • Splunk • Logstash • Graylog2 • Fluentd
  8. 8. Splunk Architecture Splunk Search heads Splunk Indexer Splunk UI Forwarder Server Forwarder with load balancer Splunk Search heads Splunk Indexer Storage http://docs.splunk.com/Documentation/Splun k/latest/Deploy/Distributedoverview
  9. 9. Logstash Architecture Logstash Indexer (10.10.10.215) Redis server (10.10.10.215) Logstash (10.10.10.73) Kibana Elastic search (10.10.10.215) Storage Log collector and log shipper Logstash (10.10.10.77) http://logstash.net/docs/1.2.1/tutorials/gettin g-started-centralized
  10. 10. Log format • "[DEBUG] [2013-10-13 22:56:20,191] [http-38219-4] [annotation.DefaultAnnotationHandlerMapping] - [Mapping [/useractivity/addUserActivity] to handler 'com.firstrain.rest.controller.UserActivityController@44 8d5a91']” Loglevel Date format Java Thread Java Class msg
  11. 11. Logstash overview • Input ( 37) – Collects logs from logsource • Filter ( 39) – Applies regex to fragment the logs • Output (51) – Writes parsed logs to destination
  12. 12. Logstash shipper configuration input { file { path => " "/frlogdir/fruseractivity/cpflogs/fruseractivity.log“ type => "log4j" tags => “ua" codec => multiline { pattern => "^[" negate => true what => "previous" multiline_tag => "exception" } } } filter { if [type] == "log4j" { grok { patterns_dir => "/opt/logstash/patterns" match => ["message", "[%{FRWORDS:loglevel}] [%{FRTIMESTAMP:frtimestamp}] [%{FRWORDS:javathread}] [%{FRWORDS:javaclass}] - %{GREEDYDATA:msg}"] } date { match => ["frtimestamp", "YYYY-MM-dd HH:mm:ss,SSS"] } } output { ## stdout { codec => rubydebug } redis { host => "10.10.10.215" data_type => "list" key => "logstash" } }
  13. 13. Logstash Indexer configuration input { redis { host => "127.0.0.1" # these settings should match the output of the agent data_type => "list" key => "logstash" # We use the 'json' codec here because we expect to read # json events from redis. codec => json } } output { stdout { debug => true debug_format => "json"} elasticsearch { host => "127.0.0.1" } }
  14. 14. Example of a log event Log event :: { "@timestamp" => "2013-10-14T05:56:20.191Z", "message" => "[DEBUG] [2013-10-13 22:56:20,191] [http-38219-4] [annotation.DefaultAnnotationHandlerMapping] - [Mapping [/useractivity/addUserActivity] to handler 'com.firstrain.rest.controller.UserActivityController@448d5a91']", "@version" => "1", "type" => "log4j", "tags" => [ [0] “ua" ], "host" => "pfrontend2-rl.ca.firstrain.net", "path" => "/frlogdir/fruseractivity/cpflogs/fruseractivity.log", "loglevel" => "DEBUG", "frtimestamp" => "2013-10-13 22:56:20,191", "javathread" => "http-38219-4", "javaclass" => "annotation.DefaultAnnotationHandlerMapping", "msg" => "[Mapping [/useractivity/addUserActivity] to handler 'com.firstrain.rest.controller.UserActivityController@448d5a91']" }
  15. 15. App’s Tags • Useractivity (ua) • Usercollobaration (uc) • Chartservice (cs) • Dataprovider (dp) • Restservice (rs) • Webapp/portal (wa) • Solr (solr) • Searchemailcontentgenerator (secg) • Searchemailcontentgeneratornode2 (secgn2) • Searchemaildispatcher (sed) • Searchemailfilter (sef)
  16. 16. Component’s Tags • Hpthepoint (hpthepoint) • Hp (hp) • Pfizer (pfizer) • Sfdc (sfdc) . . . • Combined perf logs of components (combinedperf) • CpfLogs (cpf) • Corecpf (corecpf) • ISPN (ispn) • Matcher (matcher) • Access log of components (components)
  17. 17. Types • Tomcat logs (log4j) • Garbage collector (gc) • Performance logs (perf) • Mysql (mysql) • Tomcat and Http access log (accesslog)
  18. 18. Dashboard
  19. 19. Features • Centralized log management console (GUI) • Log search and analysis system • Histogram, Graphs and Charts • Log shipping for windows and Linux machine • Near Real time log processing • Offline log processing • Dashboard view • Data retention and storage policy • Addons and modules • Alerting system
  20. 20. Splunk vs Logstash • Paid vs free • Installation, Configuration and setup • Regex based field extraction during search from UI • Regex based field extraction during log shipping • Correlation graph of multiple graphs • Zoom in/ Zoom out • Alerts • Easy url sharing • Modules and add-ons support • Disk space usage
  21. 21. Demo • Find the ip when login is failed
  22. 22. Demo • Find the activity logs for ip where login is failed
  23. 23. References • http://docs.splunk.com/Documentation/Splunk • http://logstash.net/docs/1.2.1/ • http://semicomplete.com/presentations/logstash-scale11x/#/ 1
  24. 24. Questions & Answers

×