Our applications are continuously sending log messages about their state. We usually have many different solutions/applications, which means, that we have many different logs. And that is never good. In short, your data is mutating, evolving and growing, but traditional analyses are no long in step with these changes.
Business forte is all about sourcing, sculpting and analyzing this variety of data. The talk will focus on technology that makes this possible.
URL: https://www.youtube.com/watch?v=Se2rVAg4EVs
2. Agenda
1 Problem in validating logs
2 How Logstash can help
3 ELK Stack (Elastic Search, Logstash, Kibana)
4 Some hands on
5 How we used ELK stack in our automation framework
6 World beyond
3. Problem in validating logs
Follow active log files.
Logs keep growing and are rotated.
Collating multiline logs in single event is difficult task.
We have different kinds of applications and hence different
kinds of logs. And that have different formats.
10. How Logstash can help
LogStash is a data pipeline that helps you process logs from
a variety of systems.
Logstash allows you to parse data and converge on a
common format.
Logstash provides a fast and convenient way to custom logic
for parsing these logs
Support for multiple plugins
11. LogStash
Input Section Filter Section Output Section
• File
• Stdin
• Syslog
• SNMP Traps
• TCP/UDP
• and many more
• Grok
• Mutate
• Geoip
• Drop
• and many more
• Elastic Search
• File
• Email
• and many more
13. Logstash-forwarder
A tool to collect logs locally
for processing elsewhere
Secure, low latency, low
resource usage, and
reliable.
Another option: Log-courier
Logstash-forwarder
Logstash
14. ELK Stack
Elasticsearch, Logstash and Kibana
End-to-end stack that delivers actionable insights in real time
from almost any type of structured and unstructured data
source
I. Logstash is used for cooking data
II. Elastic Search is used for storing this cooked data
III. Kibana gives shape to your data
Each one is packed and fully self contained in a jar and easy
to use
24. World Beyond
Analytics - count things and summarize your data.
Crawling and Document Processing
1. For crawling, people are using both Scrapy and Nutch together
with Elasticsearch.
Variety of companies are using ELK stack to pump their
search infrastructure.
1. Wikimedia
2. Empowers GitHub's 4 million members through providing
search across GitHub's 8 million+ code repositories.