Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

Log analysis using Logstash,ElasticSearch and Kibana

Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Anzeige
Nächste SlideShare
Elastic Stack Introduction
Elastic Stack Introduction
Wird geladen in …3
×

Hier ansehen

1 von 20 Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (20)

Ähnlich wie Log analysis using Logstash,ElasticSearch and Kibana (20)

Anzeige

Weitere von Avinash Ramineni (10)

Aktuellste (20)

Anzeige

Log analysis using Logstash,ElasticSearch and Kibana

  1. 1. Log Analysis – Logstash, Elastic Search, Kibana Avinash Ramineni Shantanu Mirajkar
  2. 2. • Logging • Pains of Log Management • Introducing Logstash • Elasticsearch • Kibana • Demo • Installing Logstash, Elasticsearch Kibana • Questions Agenda
  3. 3. • Why do we need Logging ? – Troubleshoot Issues – Security • Analyze logs to detect patterns • Detect Malware Activity - Intrusion Detection, Denial of Service • Unauthorized Resource Usage – Monitoring • Monitor Resource Usage • Developers and Logging – Logging Aids in Development ? – Forget about Production !!!!! Logging
  4. 4. • “Capture-it-all” Approach • What to Log? Everything  • DevOps Movement • Logs are archived for years • Big Data • Application Usage Statistics Logging
  5. 5. • Searching the logs – Command line, cat, tail, sed, grep, awk – Regular Expressions • Multiple Servers behind the load balancer • Multi-Tier Architecture – Web Application – Service Layer – Correlation between various components in a System • Geographically distributed – Timestamps Log management
  6. 6. • Centralize all the Logs – Too much information to go through – Increasingly hard to correlate the contextual Data • Add Searching and Indexing Technology – grep – Custom logging frameworks , custom integration of logging, searching technologies • Monitor the Logs Log management
  7. 7. • Logstash to the Rescue –Integration Framework • Log Collection • Centralization • Parsing • Storage and Search Logstash
  8. 8. • JRuby – Run on Java Virtual Machine (JVM) – Simple Message Based Architecture – Single Agent that can be configured for multiple things – OPEN SOURCE • Four Components – Shipper – Broker and Indexer – Search and Storage – Web Interface Logstash
  9. 9. Architecture Image courtesy of Logstashbook
  10. 10. Architecture - Broker • Acts as Temp Buffer between Logstash Agents and the Central server – Enhance Performance by providing caching buffer for log events – Adds Resiliency • Incase the Indexing fails, the events are held in a queue instead of getting lost • AMQP,0MQ, Redis
  11. 11. • Indexing and Searching Tool – Built on Lucene • Search and Index data available Restfully as JSON over HTTP • Comes bundled with Logstash – embedded • Text indexing Search Engine – Searches on the Index rather than on the content • Creates Indexes of the incoming content – Uses Apache Lucene to create Indexes • ElasticSearch can have a schema – Fields on which Indexes are created ElasticSearch
  12. 12. • Indexes are stored in Lucene Instances called “Shards” • ElasticSearch can have multiple nodes • Two Types of Shards – Primary – Replica • Replicas of Primary Shards – Protect the data – Make Searches Faster ElasticSearch
  13. 13. • Wouldn’t it be good to have a webpage to do search on ElasticSearch instead of searching it through a Service • Kibana provides a Simple but Powerful web Interface – Customizable Dashboards – Search the log events • Support Lucene Query Syntax – Creation of tables, graphs and sophisticated visualizations Kibana
  14. 14. Kibana
  15. 15. Kibana
  16. 16. Demo
  17. 17. • Send Alerts – Emails – Instant Messaging – Other Monitoring System • Collect and Deliver Metrics to metric engine Alerts / Monitoring Support
  18. 18. • Small VMs with limited memory • Outsourced managed servers • Java not installed • Alternatives – Syslog • Rsyslog • Syslogd • Syslog-NG – Logstash Forwarder (Lumber Jack) Shipping Logs with Logstash Agent
  19. 19. • Scale each component as needed • Can be built into using chef and puppet scripts Scaling / Deployment
  20. 20. Industry ExperienceQuestions ? avinash@clairvoyantsoft.com Twitter:@avinashramineni shantanu@clairvoyantsoft.com

Hinweis der Redaktion

  • DevOps -- the kind of guys who have both a developer and an operator hat making sure that custom developed applications are running smoothly

×