The document discusses log analysis using open source scalable systems. It outlines challenges of analyzing logs from distributed systems which are generated in different formats by various components. It proposes using logstash to collect logs, Redis for queuing, Elasticsearch for storage and searching, Kibana for dashboards and visualization, and other tools like Beaver and Lumberjack. The architecture involves collecting, processing, and analyzing logs in a centralized manner for easier access and monitoring of distributed systems.

1. Log Analysis using Open Source
Scalable Systems
Gurvinder Singh
Verktøysamling Oct'2013

2. Motivation
●
Distributed Systems
●
In the moment of heat
●
Centeralized interface to logs
●
Easier access
http://www.themeparkreview.com/tatsumediaday/tatsumediaday57.jpg
10/2/13 SLIDE 2

3. Challenges
●
Almost every component generates logs
●
Different Formats and logging methods
●
Different requirements for processing
●
Dashboards
●
Alerts
http://img72.imageshack.us/img72/3885/nephew2logs.jpg
10/2/13 SLIDE 3

4. Components
●
Logstash
●
Redis
●
Elasticsearch
●
Kibana
●
Beaver
●
lumberjack
●
.....
10/2/13 SLIDE 4

5. Architecture

6. Architecture

7. Architecture

8. Architecture

9. Architecture

10. Demo

11. Possible Input/Processing/Output Options
10/2/13
SLIDE
11

12. Thanks:-)
Questions?
10/2/13
SLIDE
12