SlideShare ist ein Scribd-Unternehmen logo
1 von 16
Downloaden Sie, um offline zu lesen
Introducing (B)ELK stackIntroducing (B)ELK stack
BBeatseats
EElasticSearchlasticSearch
LLogStashogStash
KKibanaibana
Bart Van Bos - 11/07/2016
(B)ELK – General Terminology(B)ELK – General Terminology
● Beats - ElasticSearch – LogStash – Kibana
(B)ELK – Functional Flow(B)ELK – Functional Flow
● Back pressure – buffer points (Kafka) !!!
(B)ELK – Architecture(B)ELK – Architecture
● ELK Architecture @ LinkedIn
– Ref: http://www.slideshare.net/TinLe1/elk-atlinked-in
Step 1 – BeatsStep 1 – Beats
● Beats are lightweight shippers for (log) data
● Packetbeats for analysing complex distributed
applications and troubleshooting
● Topbeats for shipping resource utilization
metrics
● Filebeats for shipping log files
● Community beats
– httpbeat, pingbeat, apachebeat, dockerbeat,
nginxbeat, uwsgibeat, phpfpmbeat
Step 1 – Beats – PacketbeatsStep 1 – Beats – Packetbeats
● Packetbeat use cases (example demo here)
– REST API monitoring: response times, HTTP error
codes, …
– DB monitoring: 10 slowest SQL queries
● Protocol support: DNS, HTTP, MySQL, PgSQL,
MongoDB, Memcache, Redis, Thrift-RPC
Step 1 – Beats – PacketbeatsStep 1 – Beats – Packetbeats
● Packetbeat caveat – performance impact
● Traffic capturing options
– pcap / af_packet / pf_ring: use af_packet on AWS!
– memory mapped sniffing
– 200k packets per second before dropping packets
Step 1 – Beats – TopbeatsStep 1 – Beats – Topbeats
● Topbeat use cases
– System wide stats: hooked onto the Linux top
command for system load, used/idle times,
free/used memory
– Per process stats: Process name, PID, CPU time,
memory size
– File system stats: Device name, mount point,
available disk space, used disk space
Step 1 – Beats – FilebeatsStep 1 – Beats – Filebeats
● Filebeat components
Step 1 – Beats – FilebeatsStep 1 – Beats – Filebeats
● Filebeat properties
● Send at least once by confirmation
● Handles log rotation
● Last reading state in case you restart your
system of LogStash is not reachable => upon
revive it will send all missing logs
● By default send new log lines every 10
seconds
Step 2 – LogStash – IntroductionStep 2 – LogStash – Introduction
● LogStash functional flow
– Inputs: beats, syslog, stdin, S3, Redis, Kafka, ...
– Filters: using GROK (regex templating)
– Outputs: ElasticSearch, eMail, exec, Redis, Kafka,
Zabbix, ...
Step 2 – LogStash – TipsStep 2 – LogStash – Tips
● LogStash Tips
– Check predefined GROK patterns (don’t re-invent
the wheel)
● http://grokconstructor.appspot.com/groklib/grok-patterns
– Use online tool to test your GROK filters!
● http://grokconstructor.appspot.com/do/match
– Don’t forget the Kibana re-indexing feature before
making new visualizations!
● https://rafaelmt.net/en/2015/09/01/kibana-tutorial/#refresh-
fields
– Keep logstash configuration files (c)lean
Step 2 – LogStash – ConfigurationStep 2 – LogStash – Configuration
● LogStash: configuration example
Step 3 – ElasticSearchStep 3 – ElasticSearch
● ElasticSearch
– Distributed, open source search and analytics engine
– Uses JSON Documents, is schema-less and RESTful
– Based on Lucene (Java): reverse indexing
– Performance profile:
● Slow in write (re-indexing)
● Fast in read => analysis
Step 4 – KibanaStep 4 – Kibana
● Kibana
– Open source data visualization platform
– Interact with your data through powerful graphics
– Ongoing battle against Apache Solr
● Kibana dashboards per client => a 4x win
– DevOps (ssh/grep/alerting)
– Developers (performance analysis, API optimization)
– PM (pro-active vs. fire extinguishing)
– Customers => new revenue streams!
● Technical SEO
● Business Intelligence
DEMO TIMEDEMO TIME
Bart Van Bos - 11/07/2016

Weitere ähnliche Inhalte

Was ist angesagt?

Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and Logstash
Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and LogstashKeeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and Logstash
Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and LogstashAmazon Web Services
 
ELK Elasticsearch Logstash and Kibana Stack for Log Management
ELK Elasticsearch Logstash and Kibana Stack for Log ManagementELK Elasticsearch Logstash and Kibana Stack for Log Management
ELK Elasticsearch Logstash and Kibana Stack for Log ManagementEl Mahdi Benzekri
 
Centralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stackCentralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stackRich Lee
 
Log analysis with the elk stack
Log analysis with the elk stackLog analysis with the elk stack
Log analysis with the elk stackVikrant Chauhan
 
ELK in Security Analytics
ELK in Security Analytics ELK in Security Analytics
ELK in Security Analytics nullowaspmumbai
 
Centralized Logging System Using ELK Stack
Centralized Logging System Using ELK StackCentralized Logging System Using ELK Stack
Centralized Logging System Using ELK StackRohit Sharma
 
Monitoring using Prometheus and Grafana
Monitoring using Prometheus and GrafanaMonitoring using Prometheus and Grafana
Monitoring using Prometheus and GrafanaArvind Kumar G.S
 
Log management with ELK
Log management with ELKLog management with ELK
Log management with ELKGeert Pante
 
Introduction to Kibana
Introduction to KibanaIntroduction to Kibana
Introduction to KibanaVineet .
 
Centralised logging with ELK stack
Centralised logging with ELK stackCentralised logging with ELK stack
Centralised logging with ELK stackSimon Hanmer
 
ELK stack introduction
ELK stack introduction ELK stack introduction
ELK stack introduction abenyeung1
 
Logging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & KibanaLogging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & KibanaAmazee Labs
 
Monitoring With Prometheus
Monitoring With PrometheusMonitoring With Prometheus
Monitoring With PrometheusKnoldus Inc.
 
ELK, a real case study
ELK,  a real case studyELK,  a real case study
ELK, a real case studyPaolo Tonin
 

Was ist angesagt? (20)

Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and Logstash
Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and LogstashKeeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and Logstash
Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and Logstash
 
ELK Elasticsearch Logstash and Kibana Stack for Log Management
ELK Elasticsearch Logstash and Kibana Stack for Log ManagementELK Elasticsearch Logstash and Kibana Stack for Log Management
ELK Elasticsearch Logstash and Kibana Stack for Log Management
 
ELK Stack
ELK StackELK Stack
ELK Stack
 
Log analysis with elastic stack
Log analysis with elastic stackLog analysis with elastic stack
Log analysis with elastic stack
 
Centralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stackCentralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stack
 
Log analysis with the elk stack
Log analysis with the elk stackLog analysis with the elk stack
Log analysis with the elk stack
 
ELK in Security Analytics
ELK in Security Analytics ELK in Security Analytics
ELK in Security Analytics
 
Elk - An introduction
Elk - An introductionElk - An introduction
Elk - An introduction
 
Introduction to ELK
Introduction to ELKIntroduction to ELK
Introduction to ELK
 
Centralized Logging System Using ELK Stack
Centralized Logging System Using ELK StackCentralized Logging System Using ELK Stack
Centralized Logging System Using ELK Stack
 
Monitoring using Prometheus and Grafana
Monitoring using Prometheus and GrafanaMonitoring using Prometheus and Grafana
Monitoring using Prometheus and Grafana
 
Log management with ELK
Log management with ELKLog management with ELK
Log management with ELK
 
Introduction to Kibana
Introduction to KibanaIntroduction to Kibana
Introduction to Kibana
 
Centralised logging with ELK stack
Centralised logging with ELK stackCentralised logging with ELK stack
Centralised logging with ELK stack
 
ELK stack introduction
ELK stack introduction ELK stack introduction
ELK stack introduction
 
Logging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & KibanaLogging with Elasticsearch, Logstash & Kibana
Logging with Elasticsearch, Logstash & Kibana
 
Elk stack
Elk stackElk stack
Elk stack
 
Monitoring With Prometheus
Monitoring With PrometheusMonitoring With Prometheus
Monitoring With Prometheus
 
ELK, a real case study
ELK,  a real case studyELK,  a real case study
ELK, a real case study
 
ELK introduction
ELK introductionELK introduction
ELK introduction
 

Andere mochten auch

Experiences in ELK with D3.js for Large Log Analysis and Visualization
Experiences in ELK with D3.js  for Large Log Analysis  and VisualizationExperiences in ELK with D3.js  for Large Log Analysis  and Visualization
Experiences in ELK with D3.js for Large Log Analysis and VisualizationSurasak Sanguanpong
 
Interactive learning analytics dashboards with ELK (Elasticsearch Logstash Ki...
Interactive learning analytics dashboards with ELK (Elasticsearch Logstash Ki...Interactive learning analytics dashboards with ELK (Elasticsearch Logstash Ki...
Interactive learning analytics dashboards with ELK (Elasticsearch Logstash Ki...Andrii Vozniuk
 
Dreaming of IoCs Adding Time Context to Threat Intelligence
Dreaming of IoCs Adding Time Context to Threat IntelligenceDreaming of IoCs Adding Time Context to Threat Intelligence
Dreaming of IoCs Adding Time Context to Threat IntelligencePriyanka Aash
 
A sample data visualisation web application
A sample data visualisation web applicationA sample data visualisation web application
A sample data visualisation web applicationsandugandhi
 
From Config Management Sucks to #cfgmgmtlove
From Config Management Sucks to #cfgmgmtlove From Config Management Sucks to #cfgmgmtlove
From Config Management Sucks to #cfgmgmtlove Kris Buytaert
 
Building Product from ground up using Open Source Technologies
Building Product from ground up using Open Source TechnologiesBuilding Product from ground up using Open Source Technologies
Building Product from ground up using Open Source TechnologiesAmit Goel
 
Data science team, a practice to setup
Data science team, a practice to setupData science team, a practice to setup
Data science team, a practice to setupOmid Mogharian
 
Send that (damn) elevator down !
Send that (damn) elevator down !Send that (damn) elevator down !
Send that (damn) elevator down !Ekta Grover
 
ELK: Moose-ively scaling your log system
ELK: Moose-ively scaling your log systemELK: Moose-ively scaling your log system
ELK: Moose-ively scaling your log systemAvleen Vig
 
Real-time data analysis using ELK
Real-time data analysis using ELKReal-time data analysis using ELK
Real-time data analysis using ELKJettro Coenradie
 
Open Source Logging and Monitoring Tools
Open Source Logging and Monitoring ToolsOpen Source Logging and Monitoring Tools
Open Source Logging and Monitoring ToolsPhase2
 
Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)
Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)
Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)Cohesive Networks
 
Monitoring with ElasticSearch
Monitoring with ElasticSearch Monitoring with ElasticSearch
Monitoring with ElasticSearch Kris Buytaert
 
Elastic Stackにハマった話
Elastic Stackにハマった話Elastic Stackにハマった話
Elastic Stackにハマった話Kazuhiro Kosaka
 
Monitoring using Open source technologies
Monitoring using Open source technologiesMonitoring using Open source technologies
Monitoring using Open source technologiesUTKARSH BHATNAGAR
 
The Rise of Real Time
The Rise of Real TimeThe Rise of Real Time
The Rise of Real Timeconfluent
 
Keystone - Leverage Big Data 2016
Keystone - Leverage Big Data 2016Keystone - Leverage Big Data 2016
Keystone - Leverage Big Data 2016Peter Bakas
 
How Did BuzzFeed Harvest One Million Email Subscribers?
How Did BuzzFeed Harvest One Million Email Subscribers?How Did BuzzFeed Harvest One Million Email Subscribers?
How Did BuzzFeed Harvest One Million Email Subscribers?Wildcard Digital Inc
 

Andere mochten auch (20)

Experiences in ELK with D3.js for Large Log Analysis and Visualization
Experiences in ELK with D3.js  for Large Log Analysis  and VisualizationExperiences in ELK with D3.js  for Large Log Analysis  and Visualization
Experiences in ELK with D3.js for Large Log Analysis and Visualization
 
Interactive learning analytics dashboards with ELK (Elasticsearch Logstash Ki...
Interactive learning analytics dashboards with ELK (Elasticsearch Logstash Ki...Interactive learning analytics dashboards with ELK (Elasticsearch Logstash Ki...
Interactive learning analytics dashboards with ELK (Elasticsearch Logstash Ki...
 
Dreaming of IoCs Adding Time Context to Threat Intelligence
Dreaming of IoCs Adding Time Context to Threat IntelligenceDreaming of IoCs Adding Time Context to Threat Intelligence
Dreaming of IoCs Adding Time Context to Threat Intelligence
 
A sample data visualisation web application
A sample data visualisation web applicationA sample data visualisation web application
A sample data visualisation web application
 
Rootconf
RootconfRootconf
Rootconf
 
From Config Management Sucks to #cfgmgmtlove
From Config Management Sucks to #cfgmgmtlove From Config Management Sucks to #cfgmgmtlove
From Config Management Sucks to #cfgmgmtlove
 
Mesoscon 2015
Mesoscon 2015Mesoscon 2015
Mesoscon 2015
 
Building Product from ground up using Open Source Technologies
Building Product from ground up using Open Source TechnologiesBuilding Product from ground up using Open Source Technologies
Building Product from ground up using Open Source Technologies
 
Data science team, a practice to setup
Data science team, a practice to setupData science team, a practice to setup
Data science team, a practice to setup
 
Send that (damn) elevator down !
Send that (damn) elevator down !Send that (damn) elevator down !
Send that (damn) elevator down !
 
ELK: Moose-ively scaling your log system
ELK: Moose-ively scaling your log systemELK: Moose-ively scaling your log system
ELK: Moose-ively scaling your log system
 
Real-time data analysis using ELK
Real-time data analysis using ELKReal-time data analysis using ELK
Real-time data analysis using ELK
 
Open Source Logging and Monitoring Tools
Open Source Logging and Monitoring ToolsOpen Source Logging and Monitoring Tools
Open Source Logging and Monitoring Tools
 
Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)
Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)
Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)
 
Monitoring with ElasticSearch
Monitoring with ElasticSearch Monitoring with ElasticSearch
Monitoring with ElasticSearch
 
Elastic Stackにハマった話
Elastic Stackにハマった話Elastic Stackにハマった話
Elastic Stackにハマった話
 
Monitoring using Open source technologies
Monitoring using Open source technologiesMonitoring using Open source technologies
Monitoring using Open source technologies
 
The Rise of Real Time
The Rise of Real TimeThe Rise of Real Time
The Rise of Real Time
 
Keystone - Leverage Big Data 2016
Keystone - Leverage Big Data 2016Keystone - Leverage Big Data 2016
Keystone - Leverage Big Data 2016
 
How Did BuzzFeed Harvest One Million Email Subscribers?
How Did BuzzFeed Harvest One Million Email Subscribers?How Did BuzzFeed Harvest One Million Email Subscribers?
How Did BuzzFeed Harvest One Million Email Subscribers?
 

Ähnlich wie Introducing ELK

Logs aggregation and analysis
Logs aggregation and analysisLogs aggregation and analysis
Logs aggregation and analysisDivante
 
Scaling an ELK stack at bol.com
Scaling an ELK stack at bol.comScaling an ELK stack at bol.com
Scaling an ELK stack at bol.comRenzo Tomà
 
Configuring Syslog by Octavio
Configuring Syslog by OctavioConfiguring Syslog by Octavio
Configuring Syslog by OctavioRowell Dionicio
 
Logging in The World of DevOps
Logging in The World of DevOps Logging in The World of DevOps
Logging in The World of DevOps DevOps Indonesia
 
Elk for applications on k8s
Elk for applications on k8sElk for applications on k8s
Elk for applications on k8sChe-Chia Chang
 
Paper_Scalable database logging for multicores
Paper_Scalable database logging for multicoresPaper_Scalable database logging for multicores
Paper_Scalable database logging for multicoresHyo jeong Lee
 
Functioning incessantly of Data Science Platform with Kubeflow - Albert Lewan...
Functioning incessantly of Data Science Platform with Kubeflow - Albert Lewan...Functioning incessantly of Data Science Platform with Kubeflow - Albert Lewan...
Functioning incessantly of Data Science Platform with Kubeflow - Albert Lewan...GetInData
 
Efficient kernel backporting
Efficient kernel backportingEfficient kernel backporting
Efficient kernel backportingLF Events
 
Introduction to apache kafka
Introduction to apache kafkaIntroduction to apache kafka
Introduction to apache kafkaSamuel Kerrien
 
Road to sbt 1.0 paved with server
Road to sbt 1.0   paved with serverRoad to sbt 1.0   paved with server
Road to sbt 1.0 paved with serverEugene Yokota
 
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly SolarWinds Loggly
 
Technology behind-real-time-log-analytics
Technology behind-real-time-log-analytics Technology behind-real-time-log-analytics
Technology behind-real-time-log-analytics Data Science Thailand
 
Apache Camel K - Fredericia
Apache Camel K - FredericiaApache Camel K - Fredericia
Apache Camel K - FredericiaClaus Ibsen
 
OSDC 2016 - Ingesting Logs with Style by Pere Urbon-Bayes
OSDC 2016 - Ingesting Logs with Style by Pere Urbon-BayesOSDC 2016 - Ingesting Logs with Style by Pere Urbon-Bayes
OSDC 2016 - Ingesting Logs with Style by Pere Urbon-BayesNETWAYS
 
Sonar qube to impove code quality
Sonar qube   to impove code qualitySonar qube   to impove code quality
Sonar qube to impove code qualityMani Sarkar
 
How YugaByte DB Implements Distributed PostgreSQL
How YugaByte DB Implements Distributed PostgreSQLHow YugaByte DB Implements Distributed PostgreSQL
How YugaByte DB Implements Distributed PostgreSQLYugabyte
 
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...Chester Chen
 
Circonus: Design failures - A Case Study
Circonus: Design failures - A Case StudyCirconus: Design failures - A Case Study
Circonus: Design failures - A Case StudyHeinrich Hartmann
 
Type safe, versioned, and rewindable stream processing with Apache {Avro, K...
Type safe, versioned, and rewindable stream processing  with  Apache {Avro, K...Type safe, versioned, and rewindable stream processing  with  Apache {Avro, K...
Type safe, versioned, and rewindable stream processing with Apache {Avro, K...Hisham Mardam-Bey
 
Hadoop con2016 - Implement Real-time Centralized logging System by Elastic Stack
Hadoop con2016 - Implement Real-time Centralized logging System by Elastic StackHadoop con2016 - Implement Real-time Centralized logging System by Elastic Stack
Hadoop con2016 - Implement Real-time Centralized logging System by Elastic StackLen Chang
 

Ähnlich wie Introducing ELK (20)

Logs aggregation and analysis
Logs aggregation and analysisLogs aggregation and analysis
Logs aggregation and analysis
 
Scaling an ELK stack at bol.com
Scaling an ELK stack at bol.comScaling an ELK stack at bol.com
Scaling an ELK stack at bol.com
 
Configuring Syslog by Octavio
Configuring Syslog by OctavioConfiguring Syslog by Octavio
Configuring Syslog by Octavio
 
Logging in The World of DevOps
Logging in The World of DevOps Logging in The World of DevOps
Logging in The World of DevOps
 
Elk for applications on k8s
Elk for applications on k8sElk for applications on k8s
Elk for applications on k8s
 
Paper_Scalable database logging for multicores
Paper_Scalable database logging for multicoresPaper_Scalable database logging for multicores
Paper_Scalable database logging for multicores
 
Functioning incessantly of Data Science Platform with Kubeflow - Albert Lewan...
Functioning incessantly of Data Science Platform with Kubeflow - Albert Lewan...Functioning incessantly of Data Science Platform with Kubeflow - Albert Lewan...
Functioning incessantly of Data Science Platform with Kubeflow - Albert Lewan...
 
Efficient kernel backporting
Efficient kernel backportingEfficient kernel backporting
Efficient kernel backporting
 
Introduction to apache kafka
Introduction to apache kafkaIntroduction to apache kafka
Introduction to apache kafka
 
Road to sbt 1.0 paved with server
Road to sbt 1.0   paved with serverRoad to sbt 1.0   paved with server
Road to sbt 1.0 paved with server
 
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
AWS re:Invent presentation: Unmeltable Infrastructure at Scale by Loggly
 
Technology behind-real-time-log-analytics
Technology behind-real-time-log-analytics Technology behind-real-time-log-analytics
Technology behind-real-time-log-analytics
 
Apache Camel K - Fredericia
Apache Camel K - FredericiaApache Camel K - Fredericia
Apache Camel K - Fredericia
 
OSDC 2016 - Ingesting Logs with Style by Pere Urbon-Bayes
OSDC 2016 - Ingesting Logs with Style by Pere Urbon-BayesOSDC 2016 - Ingesting Logs with Style by Pere Urbon-Bayes
OSDC 2016 - Ingesting Logs with Style by Pere Urbon-Bayes
 
Sonar qube to impove code quality
Sonar qube   to impove code qualitySonar qube   to impove code quality
Sonar qube to impove code quality
 
How YugaByte DB Implements Distributed PostgreSQL
How YugaByte DB Implements Distributed PostgreSQLHow YugaByte DB Implements Distributed PostgreSQL
How YugaByte DB Implements Distributed PostgreSQL
 
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...
 
Circonus: Design failures - A Case Study
Circonus: Design failures - A Case StudyCirconus: Design failures - A Case Study
Circonus: Design failures - A Case Study
 
Type safe, versioned, and rewindable stream processing with Apache {Avro, K...
Type safe, versioned, and rewindable stream processing  with  Apache {Avro, K...Type safe, versioned, and rewindable stream processing  with  Apache {Avro, K...
Type safe, versioned, and rewindable stream processing with Apache {Avro, K...
 
Hadoop con2016 - Implement Real-time Centralized logging System by Elastic Stack
Hadoop con2016 - Implement Real-time Centralized logging System by Elastic StackHadoop con2016 - Implement Real-time Centralized logging System by Elastic Stack
Hadoop con2016 - Implement Real-time Centralized logging System by Elastic Stack
 

Introducing ELK

  • 1. Introducing (B)ELK stackIntroducing (B)ELK stack BBeatseats EElasticSearchlasticSearch LLogStashogStash KKibanaibana Bart Van Bos - 11/07/2016
  • 2. (B)ELK – General Terminology(B)ELK – General Terminology ● Beats - ElasticSearch – LogStash – Kibana
  • 3. (B)ELK – Functional Flow(B)ELK – Functional Flow ● Back pressure – buffer points (Kafka) !!!
  • 4. (B)ELK – Architecture(B)ELK – Architecture ● ELK Architecture @ LinkedIn – Ref: http://www.slideshare.net/TinLe1/elk-atlinked-in
  • 5. Step 1 – BeatsStep 1 – Beats ● Beats are lightweight shippers for (log) data ● Packetbeats for analysing complex distributed applications and troubleshooting ● Topbeats for shipping resource utilization metrics ● Filebeats for shipping log files ● Community beats – httpbeat, pingbeat, apachebeat, dockerbeat, nginxbeat, uwsgibeat, phpfpmbeat
  • 6. Step 1 – Beats – PacketbeatsStep 1 – Beats – Packetbeats ● Packetbeat use cases (example demo here) – REST API monitoring: response times, HTTP error codes, … – DB monitoring: 10 slowest SQL queries ● Protocol support: DNS, HTTP, MySQL, PgSQL, MongoDB, Memcache, Redis, Thrift-RPC
  • 7. Step 1 – Beats – PacketbeatsStep 1 – Beats – Packetbeats ● Packetbeat caveat – performance impact ● Traffic capturing options – pcap / af_packet / pf_ring: use af_packet on AWS! – memory mapped sniffing – 200k packets per second before dropping packets
  • 8. Step 1 – Beats – TopbeatsStep 1 – Beats – Topbeats ● Topbeat use cases – System wide stats: hooked onto the Linux top command for system load, used/idle times, free/used memory – Per process stats: Process name, PID, CPU time, memory size – File system stats: Device name, mount point, available disk space, used disk space
  • 9. Step 1 – Beats – FilebeatsStep 1 – Beats – Filebeats ● Filebeat components
  • 10. Step 1 – Beats – FilebeatsStep 1 – Beats – Filebeats ● Filebeat properties ● Send at least once by confirmation ● Handles log rotation ● Last reading state in case you restart your system of LogStash is not reachable => upon revive it will send all missing logs ● By default send new log lines every 10 seconds
  • 11. Step 2 – LogStash – IntroductionStep 2 – LogStash – Introduction ● LogStash functional flow – Inputs: beats, syslog, stdin, S3, Redis, Kafka, ... – Filters: using GROK (regex templating) – Outputs: ElasticSearch, eMail, exec, Redis, Kafka, Zabbix, ...
  • 12. Step 2 – LogStash – TipsStep 2 – LogStash – Tips ● LogStash Tips – Check predefined GROK patterns (don’t re-invent the wheel) ● http://grokconstructor.appspot.com/groklib/grok-patterns – Use online tool to test your GROK filters! ● http://grokconstructor.appspot.com/do/match – Don’t forget the Kibana re-indexing feature before making new visualizations! ● https://rafaelmt.net/en/2015/09/01/kibana-tutorial/#refresh- fields – Keep logstash configuration files (c)lean
  • 13. Step 2 – LogStash – ConfigurationStep 2 – LogStash – Configuration ● LogStash: configuration example
  • 14. Step 3 – ElasticSearchStep 3 – ElasticSearch ● ElasticSearch – Distributed, open source search and analytics engine – Uses JSON Documents, is schema-less and RESTful – Based on Lucene (Java): reverse indexing – Performance profile: ● Slow in write (re-indexing) ● Fast in read => analysis
  • 15. Step 4 – KibanaStep 4 – Kibana ● Kibana – Open source data visualization platform – Interact with your data through powerful graphics – Ongoing battle against Apache Solr ● Kibana dashboards per client => a 4x win – DevOps (ssh/grep/alerting) – Developers (performance analysis, API optimization) – PM (pro-active vs. fire extinguishing) – Customers => new revenue streams! ● Technical SEO ● Business Intelligence
  • 16. DEMO TIMEDEMO TIME Bart Van Bos - 11/07/2016