Introduciton of Logstash, Elasticsearch and Kibana.

1. Logstash & Elasticsearch
Tatooine

2. Agenda
• Introduction
• What is logstash
• logstash in action
• What is Elasticsearch
• Elasticsearch in action

3. Introduction
• Issue : find out TPS based on ELB ’s log files
• What is the goals
• make life is easier
• No repeat to write program again and agains
• Make log event searchable
• Able to analyse log event

5. • Free & open source
• Event processing log file
• Support multiple sources and destinations
• logstash can manipulate data
• pipeline = input + (filter) + output
• codec is change data representation
• Flexible configuration
• Not limit only process log event

6. • Middle guy that sit between sources and destinations that
• manage event and logs
• collect data
• parse data
• enrich data
• store data (search & visualise)

7. logstash = inputs + filters + outputs

8. Elastic search
• NoSQL: Document Oriented
- Insert, delete, update, retrieve, analytic and search
• Built on top of apache lucene
- lucene is most popular java based full text search index implement
• Distributed text search engine
- Inverted Index
- Cluster

9. Ancient Search Engine

10. Why
Free & open source
Easy to scale (distributed)
Everything is one JSON call (Restful API)
Unleash power of Lucene under the hood
•
•
•
•
• Excellent query DSL
• Support for advance search features
(full text search)
Document oriented
Schema free
•
•
• Active community

11. What does it added to lucene?
• RESTFUL Service
- JSON API over HTTP
• High Available & Performance
- node form cluster
- distributed data using shard
- replicas request load, fault tolerance
• Long terms persistency
- write through persistent storage system

12. Document Oriented
name address hired_date department
Ball Paris 22/06/2015 Business
JenJa Tokyo 18/01/2016 Accounting
Kook London 1/04/2017 Marketing
{
…..
“hit”: [
{
“_index”: “general”
“_type”: “employee”,
“name”: “Ball”,
“address”: “Paris”,
“hired_date”: “22/06/2015”,
“department”: “business”
},
{
“_index”: “general”
“_type”: “employee”,
“name”: “JenJa”,
“address”: “Tokyo”,
“hired_date”: “18/01/2016”,
“department”: “Accounting”
},
…..
]
}
Table: employee
Database: general

13. Elastic search
Elasticsearch Relational MySQL
Index Database
Type Table
Document Row
Field Column

14. Who use elasticsearch ?

15. Conclusion
• Logstash used to load,parse and structured data
to elasticsearch
• Elasticsearch used to find number of TPS for each
API

16. Q & A
Thanks you