SlideShare ist ein Scribd-Unternehmen logo
1 von 22
Downloaden Sie, um offline zu lesen
Open Source Monitoring Conference 2017
Log monitoring with
Logstash and Icinga
An introduction to log monitoring
Open Source Monitoring Conference 2017
Who are we
• Walter Heck, Founder & CTO of
OlinData (http://www.olindata.com)
• Oliver Lowe, DevOps Consultant of
OlinData (http://www.olindata.com)
Open Source Monitoring Conference 2017
Overview
• What is Elastic/Logstash/Filebeat and why should you
care?
• Logstash outputs
• Outputting to Icinga, why?
• An example scenario: backups
• Alternatives
• Demo!
Open Source Monitoring Conference 2017
What is Elastic and why should you care?
• Open Source ‘fancy’ fulltext search engine
• Among other things used for centralised logging
• Just as monitoring should be a first class citizen in a
modern infrastructure, so should logging
Open Source Monitoring Conference 2017
• Essentially tails a given
set of files and sends new
events to defined outputs
• Output can be straight
to Elastic, but
commonly first to
Logstash
• Specify the type of
document or input to
allow for further
processing at a later stage
What is Filebeat and why should you care?
filebeat.prospectors:
- input_type: log
paths:
- /var/log/backup.log
output.logstash:
hosts: ["localhost:5044"]
Open Source Monitoring Conference 2017
• Takes events from 1+
inputs
• Filters events to mold
the data
• Sends processed events
on to 1+ outputs
• Mutate the inputs into
our desired outputs
What is Logstash and why should you care?
# /etc/logstash/conf.d/logstash.conf
input {
<..some inputs here (see
https://www.elastic.co/guide/en/logstash/current/input-p
lugins.html) ..>
}
filter {
<..some filters here (see
https://www.elastic.co/guide/en/logstash/current/filter-
plugins.html)..>
}
output {
<..some outputs here (see
https://www.elastic.co/guide/en/logstash/current/output-
plugins.html)..>
}
Open Source Monitoring Conference 2017
Logstash plugins
• Allows fully custom inputs, filters and outputs
• Input plugins define where data is coming from
• eg. filebeat, AWS Cloudwatch, syslog, tcp, udp
• Filter plugins define how to manipulate data
• eg. json, geoip, dns, grok, json, ruby(!)
• Output plugins define where to send data
• eg. cloudwatch, datadog, elasticsearch, statsd
Open Source Monitoring Conference 2017
The Logstash Icinga Output plugin
• Allows taking action in Icinga based on events
encountered in logstash
• Uses Icinga API to accomplish this
https://github.com/Icinga/logstash-output-icinga/
Open Source Monitoring Conference 2017
Open Source Monitoring Conference 2017
• Monitoring of backup
• Logs are indexed in Elasticsearch
• Logstash to filter logs for keywords for specific actions:
• Assign maintenance window to icinga node
• Alert for start of backup
• Alert for successful or failed backup
• Remove maintenance window from icinga node
DEMO: using official icinga2x-elastic vagrant box:
https://github.com/Icinga/icinga-vagrant
Possible Scenarios
Open Source Monitoring Conference 2017
process-check-result
● Allows for manipulating
the state of a check
○ Set a dummy
`catchall` service to
critical if we see the
word “error” in a
logstash log
○ Set a dummy
`backup` service to
ok after finishing a
backup
Setting Input type Req?
exit_status number, For services: 0=OK,
1=WARNING, 2=CRITICAL,
3=UNKNOWN, for hosts: 0=OK,
1=CRITICAL.
Yes
plugin_output string, The plugins main output.
Does not contain the performance
data.
Yes
performance_data array, The performance data. No
check_command array, The first entry should be the
check commands path, then one
entry for each command line
option followed by an entry for
each of its argument.
No
check_source string, Usually the name of the
command_endpoint
No
Open Source Monitoring Conference 2017
send-custom-notification
• Sends a custom
notification
• Eg. a custom host
notification
announcing a global
maintenance to host
owners
Setting Input type Req?
author string, Name of the author. Yes
comment string, Comment text. Yes
force boolean, Default: false. If true, the
notification is sent regardless of
downtimes or whether notifications
are enabled or not.
No
Open Source Monitoring Conference 2017
add / remove-comment
• Allows adding or
removing a comment
from a service or a host
• Removing requires
passing the author
of the comment
(removes all
comments by that
author :( )
Setting Input type Req?
author string, Name of the author. Yes
comment string, Comment text. Yes
Open Source Monitoring Conference 2017
schedule / remove-downtime
• Allows adding or
removing downtime
Setting Input type Req
?
author string, Name of the author. Yes
comment string, Comment text. Yes
start_time timestamp (epoc), Timestamp marking
the beginning of the downtime.
Yes
end_time timestamp (epoc), Timestamp marking
the end of the downtime.
Yes
fixed boolean, Defaults to true. If true, the
downtime is fixed otherwise flexible.
No
duration number, Duration of the downtime in
seconds if fixed is set to false.
Yes/N
o
trigger_name string, Sets the trigger for a triggered
downtime.
No
child_options number, Schedule child downtimes. 0
does not do anything, 1 schedules child
downtimes triggered by this downtime, 2
schedules non-triggered downtimes.
Defaults to 0.
No
Open Source Monitoring Conference 2017
Installing the logstash output
# download the gem
wget
https://github.com/Icinga/logstash-output-icinga/releases/download/v1.1.0/logstas
h-output-icinga-1.1.0.gem
# install the plugin
/usr/share/logstash/bin/logstash-plugin install --local
./logstash-output-icinga-1.1.0.gem
Validating ./logstash-output-icinga-1.1.0.gem
Installing logstash-output-icinga
Installation successful
Soon to be replaced by a published gem (thanks jordansissel!)
Open Source Monitoring Conference 2017
Configure Icinga
object ApiUser "icinga" {
password = "icinga"
client_cn = NodeName
permissions = [ "*" ]
}
apply Service "backup" {
import "generic-service"
check_command = "dummy"
assign where host.address
}
Open Source Monitoring Conference 2017
input {
beats {
port => 5044
type => "logs"
}
}
filter {
if [syslog_severity] == "error" {
mutate {
replace => { "exit_status" => "2" }
}
}
if [source] == "/var/log/mypreciouslog.json" {
json {
source => "message"
}
}
}
Configure logstash
output {
icinga {
host => 'localhost'
user => 'icinga'
password => 'icinga'
ssl_verify => false
action => 'process-check-result'
action_config => {
exit_status => "%{exit_status}"
plugin_output => "%{message}"
}
icinga_host => "%{hostname}"
icinga_service => "backup"
}
elasticsearch {
hosts => [ "localhost:9200" ]
}
stdout {
}
}
Open Source Monitoring Conference 2017
Demo time!
(brace yourselves..)
Open Source Monitoring Conference 2017
Up for improvement
● Multiple icinga outputs needed for each action
○ Action_config fields should be dynamic
● Repeated configuration
Open Source Monitoring Conference 2017
• Elasticsearch Watcher (operates on elasticsearch level!)
• Provided via X-Pack (commercial)
• Provides alerting on cluster and index events
• Allows notification via Slack, email or supported
mechanisms
• Splunk?
Alternative(s)
Open Source Monitoring Conference 2017
We’re hiring!
NL based consultants
jobs@olindata.com
Open Source Monitoring Conference 2017
Questions?
@walterheck / @olindata
http://www.olindata.com
walterheck@olindata.com
oliver@olindata.com
http://github.com/olindata

Weitere ähnliche Inhalte

Was ist angesagt?

Getting started with salt stack
Getting started with salt stackGetting started with salt stack
Getting started with salt stack
Suresh Paulraj
 
How we found a firewall vendor bug using Teleport as a bastion jump host
How we found a firewall vendor bug using Teleport as a bastion jump hostHow we found a firewall vendor bug using Teleport as a bastion jump host
How we found a firewall vendor bug using Teleport as a bastion jump host
Faelix Ltd
 

Was ist angesagt? (20)

Nagios Conference 2014 - Janice Singh - Real World Uses for Nagios APIs
Nagios Conference 2014 - Janice Singh - Real World Uses for Nagios APIsNagios Conference 2014 - Janice Singh - Real World Uses for Nagios APIs
Nagios Conference 2014 - Janice Singh - Real World Uses for Nagios APIs
 
Vagrant in 15 minutes
Vagrant in 15 minutesVagrant in 15 minutes
Vagrant in 15 minutes
 
2020 ADDO Spring Break OWASP ZAP Automation
2020 ADDO Spring Break OWASP ZAP Automation2020 ADDO Spring Break OWASP ZAP Automation
2020 ADDO Spring Break OWASP ZAP Automation
 
OSMC 2018 | Current State of Icinga by Bernd Erk
OSMC 2018 | Current State of Icinga by Bernd ErkOSMC 2018 | Current State of Icinga by Bernd Erk
OSMC 2018 | Current State of Icinga by Bernd Erk
 
sensu
sensusensu
sensu
 
OSMC 2018 | SLA Monitoring mit Icinga & Prometheus by Moritz Tanzer
OSMC 2018 | SLA Monitoring mit Icinga & Prometheus by Moritz TanzerOSMC 2018 | SLA Monitoring mit Icinga & Prometheus by Moritz Tanzer
OSMC 2018 | SLA Monitoring mit Icinga & Prometheus by Moritz Tanzer
 
How Yelp Uses Sensu to Monitor Services in a SOA World
How Yelp Uses Sensu to Monitor Services in a SOA WorldHow Yelp Uses Sensu to Monitor Services in a SOA World
How Yelp Uses Sensu to Monitor Services in a SOA World
 
Getting started with salt stack
Getting started with salt stackGetting started with salt stack
Getting started with salt stack
 
Maximizing PHP Performance with NGINX
Maximizing PHP Performance with NGINXMaximizing PHP Performance with NGINX
Maximizing PHP Performance with NGINX
 
How we found a firewall vendor bug using Teleport as a bastion jump host
How we found a firewall vendor bug using Teleport as a bastion jump hostHow we found a firewall vendor bug using Teleport as a bastion jump host
How we found a firewall vendor bug using Teleport as a bastion jump host
 
Nagios Conference 2013 - Eric Stanley and Andy Brist - API and Nagios
Nagios Conference 2013 - Eric Stanley and Andy Brist - API and NagiosNagios Conference 2013 - Eric Stanley and Andy Brist - API and Nagios
Nagios Conference 2013 - Eric Stanley and Andy Brist - API and Nagios
 
Code lifecycle on the Acquia Cloud Platform
Code lifecycle on the Acquia Cloud PlatformCode lifecycle on the Acquia Cloud Platform
Code lifecycle on the Acquia Cloud Platform
 
Prometheus + Grafana = Awesome Monitoring
Prometheus + Grafana = Awesome MonitoringPrometheus + Grafana = Awesome Monitoring
Prometheus + Grafana = Awesome Monitoring
 
Rundeck Open Source Workflow Automation
Rundeck Open Source Workflow AutomationRundeck Open Source Workflow Automation
Rundeck Open Source Workflow Automation
 
Salt conf15 presentation-william-cannon
Salt conf15 presentation-william-cannonSalt conf15 presentation-william-cannon
Salt conf15 presentation-william-cannon
 
Monitoring Cloud Native Applications with Prometheus
Monitoring Cloud Native Applications with PrometheusMonitoring Cloud Native Applications with Prometheus
Monitoring Cloud Native Applications with Prometheus
 
Nagios Conference 2014 - Jack Chu - How to Think With Nagios to Solve Monitor...
Nagios Conference 2014 - Jack Chu - How to Think With Nagios to Solve Monitor...Nagios Conference 2014 - Jack Chu - How to Think With Nagios to Solve Monitor...
Nagios Conference 2014 - Jack Chu - How to Think With Nagios to Solve Monitor...
 
Dimitri Bellini and Pietro Antonacci - Manage Zabbix Proxies in Remote Networ...
Dimitri Bellini and Pietro Antonacci - Manage Zabbix Proxies in Remote Networ...Dimitri Bellini and Pietro Antonacci - Manage Zabbix Proxies in Remote Networ...
Dimitri Bellini and Pietro Antonacci - Manage Zabbix Proxies in Remote Networ...
 
OSMC 2018 | Integrating Check_MK agent into Thruk – Windows monitoring made e...
OSMC 2018 | Integrating Check_MK agent into Thruk – Windows monitoring made e...OSMC 2018 | Integrating Check_MK agent into Thruk – Windows monitoring made e...
OSMC 2018 | Integrating Check_MK agent into Thruk – Windows monitoring made e...
 
Thomas Schmainda - Tracking Boeing Satellites With Nagios - Nagios World Conf...
Thomas Schmainda - Tracking Boeing Satellites With Nagios - Nagios World Conf...Thomas Schmainda - Tracking Boeing Satellites With Nagios - Nagios World Conf...
Thomas Schmainda - Tracking Boeing Satellites With Nagios - Nagios World Conf...
 

Ähnlich wie OSMC 2017 | Log Monitoring with Logstash and Icinga by Walter Heck

Ähnlich wie OSMC 2017 | Log Monitoring with Logstash and Icinga by Walter Heck (20)

Fuzzing - Part 2
Fuzzing - Part 2Fuzzing - Part 2
Fuzzing - Part 2
 
How to Use OWASP Security Logging
How to Use OWASP Security LoggingHow to Use OWASP Security Logging
How to Use OWASP Security Logging
 
.NET @ apache.org
 .NET @ apache.org .NET @ apache.org
.NET @ apache.org
 
Docker Logging and analysing with Elastic Stack - Jakub Hajek
Docker Logging and analysing with Elastic Stack - Jakub Hajek Docker Logging and analysing with Elastic Stack - Jakub Hajek
Docker Logging and analysing with Elastic Stack - Jakub Hajek
 
Docker Logging and analysing with Elastic Stack
Docker Logging and analysing with Elastic StackDocker Logging and analysing with Elastic Stack
Docker Logging and analysing with Elastic Stack
 
Action! Development and Operations for Sticker Shop
Action! Development and  Operations for Sticker ShopAction! Development and  Operations for Sticker Shop
Action! Development and Operations for Sticker Shop
 
JS Fest 2018. Никита Галкин. Микросервисная архитектура с переиспользуемыми к...
JS Fest 2018. Никита Галкин. Микросервисная архитектура с переиспользуемыми к...JS Fest 2018. Никита Галкин. Микросервисная архитектура с переиспользуемыми к...
JS Fest 2018. Никита Галкин. Микросервисная архитектура с переиспользуемыми к...
 
DevOps - A Purpose for an Institution.pdf
DevOps - A Purpose for an Institution.pdfDevOps - A Purpose for an Institution.pdf
DevOps - A Purpose for an Institution.pdf
 
Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...
Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...
Eko10 - Security Monitoring for Big Infrastructures without a Million Dollar ...
 
Eko10 workshop - OPEN SOURCE DATABASE MONITORING
Eko10 workshop - OPEN SOURCE DATABASE MONITORINGEko10 workshop - OPEN SOURCE DATABASE MONITORING
Eko10 workshop - OPEN SOURCE DATABASE MONITORING
 
Cross-platform logging and analytics
Cross-platform logging and analyticsCross-platform logging and analytics
Cross-platform logging and analytics
 
EuroPython 2013 - Python3 TurboGears Training
EuroPython 2013 - Python3 TurboGears TrainingEuroPython 2013 - Python3 TurboGears Training
EuroPython 2013 - Python3 TurboGears Training
 
Thinking DevOps in the era of the Cloud - Demi Ben-Ari
Thinking DevOps in the era of the Cloud - Demi Ben-AriThinking DevOps in the era of the Cloud - Demi Ben-Ari
Thinking DevOps in the era of the Cloud - Demi Ben-Ari
 
Monitoring in Big Data Platform - Albert Lewandowski, GetInData
Monitoring in Big Data Platform - Albert Lewandowski, GetInDataMonitoring in Big Data Platform - Albert Lewandowski, GetInData
Monitoring in Big Data Platform - Albert Lewandowski, GetInData
 
Eko10 Workshop Opensource Database Auditing
Eko10  Workshop Opensource Database AuditingEko10  Workshop Opensource Database Auditing
Eko10 Workshop Opensource Database Auditing
 
Serverless for High Performance Computing
Serverless for High Performance ComputingServerless for High Performance Computing
Serverless for High Performance Computing
 
PyCon AU 2012 - Debugging Live Python Web Applications
PyCon AU 2012 - Debugging Live Python Web ApplicationsPyCon AU 2012 - Debugging Live Python Web Applications
PyCon AU 2012 - Debugging Live Python Web Applications
 
Cashing in on logging and exception data
Cashing in on logging and exception dataCashing in on logging and exception data
Cashing in on logging and exception data
 
What you wanted to know about MySQL, but could not find using inernal instrum...
What you wanted to know about MySQL, but could not find using inernal instrum...What you wanted to know about MySQL, but could not find using inernal instrum...
What you wanted to know about MySQL, but could not find using inernal instrum...
 
Taking AppSec to 11 - BSides Austin 2016
Taking AppSec to 11 - BSides Austin 2016Taking AppSec to 11 - BSides Austin 2016
Taking AppSec to 11 - BSides Austin 2016
 

Kürzlich hochgeladen

Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Lisi Hocke
 

Kürzlich hochgeladen (20)

WSO2CON 2024 - How to Run a Security Program
WSO2CON 2024 - How to Run a Security ProgramWSO2CON 2024 - How to Run a Security Program
WSO2CON 2024 - How to Run a Security Program
 
The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)
 
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open SourceWSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
 
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
 
WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...
WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...
WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...
 
WSO2CON 2024 - Not Just Microservices: Rightsize Your Services!
WSO2CON 2024 - Not Just Microservices: Rightsize Your Services!WSO2CON 2024 - Not Just Microservices: Rightsize Your Services!
WSO2CON 2024 - Not Just Microservices: Rightsize Your Services!
 
WSO2CON 2024 - How CSI Piemonte Is Apifying the Public Administration
WSO2CON 2024 - How CSI Piemonte Is Apifying the Public AdministrationWSO2CON 2024 - How CSI Piemonte Is Apifying the Public Administration
WSO2CON 2024 - How CSI Piemonte Is Apifying the Public Administration
 
WSO2CON 2024 - How CSI Piemonte Is Apifying the Public Administration
WSO2CON 2024 - How CSI Piemonte Is Apifying the Public AdministrationWSO2CON 2024 - How CSI Piemonte Is Apifying the Public Administration
WSO2CON 2024 - How CSI Piemonte Is Apifying the Public Administration
 
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
 
WSO2CON 2024 - IoT Needs CIAM: The Importance of Centralized IAM in a Growing...
WSO2CON 2024 - IoT Needs CIAM: The Importance of Centralized IAM in a Growing...WSO2CON 2024 - IoT Needs CIAM: The Importance of Centralized IAM in a Growing...
WSO2CON 2024 - IoT Needs CIAM: The Importance of Centralized IAM in a Growing...
 
WSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go PlatformlessWSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go Platformless
 
From Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST APIFrom Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST API
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
 
WSO2Con204 - Hard Rock Presentation - Keynote
WSO2Con204 - Hard Rock Presentation - KeynoteWSO2Con204 - Hard Rock Presentation - Keynote
WSO2Con204 - Hard Rock Presentation - Keynote
 
What Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the SituationWhat Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the Situation
 
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
 
WSO2Con2024 - Unleashing the Financial Potential of 13 Million People
WSO2Con2024 - Unleashing the Financial Potential of 13 Million PeopleWSO2Con2024 - Unleashing the Financial Potential of 13 Million People
WSO2Con2024 - Unleashing the Financial Potential of 13 Million People
 
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
 
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdfAzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
 
WSO2CON2024 - Why Should You Consider Ballerina for Your Next Integration
WSO2CON2024 - Why Should You Consider Ballerina for Your Next IntegrationWSO2CON2024 - Why Should You Consider Ballerina for Your Next Integration
WSO2CON2024 - Why Should You Consider Ballerina for Your Next Integration
 

OSMC 2017 | Log Monitoring with Logstash and Icinga by Walter Heck

  • 1. Open Source Monitoring Conference 2017 Log monitoring with Logstash and Icinga An introduction to log monitoring
  • 2. Open Source Monitoring Conference 2017 Who are we • Walter Heck, Founder & CTO of OlinData (http://www.olindata.com) • Oliver Lowe, DevOps Consultant of OlinData (http://www.olindata.com)
  • 3. Open Source Monitoring Conference 2017 Overview • What is Elastic/Logstash/Filebeat and why should you care? • Logstash outputs • Outputting to Icinga, why? • An example scenario: backups • Alternatives • Demo!
  • 4. Open Source Monitoring Conference 2017 What is Elastic and why should you care? • Open Source ‘fancy’ fulltext search engine • Among other things used for centralised logging • Just as monitoring should be a first class citizen in a modern infrastructure, so should logging
  • 5. Open Source Monitoring Conference 2017 • Essentially tails a given set of files and sends new events to defined outputs • Output can be straight to Elastic, but commonly first to Logstash • Specify the type of document or input to allow for further processing at a later stage What is Filebeat and why should you care? filebeat.prospectors: - input_type: log paths: - /var/log/backup.log output.logstash: hosts: ["localhost:5044"]
  • 6. Open Source Monitoring Conference 2017 • Takes events from 1+ inputs • Filters events to mold the data • Sends processed events on to 1+ outputs • Mutate the inputs into our desired outputs What is Logstash and why should you care? # /etc/logstash/conf.d/logstash.conf input { <..some inputs here (see https://www.elastic.co/guide/en/logstash/current/input-p lugins.html) ..> } filter { <..some filters here (see https://www.elastic.co/guide/en/logstash/current/filter- plugins.html)..> } output { <..some outputs here (see https://www.elastic.co/guide/en/logstash/current/output- plugins.html)..> }
  • 7. Open Source Monitoring Conference 2017 Logstash plugins • Allows fully custom inputs, filters and outputs • Input plugins define where data is coming from • eg. filebeat, AWS Cloudwatch, syslog, tcp, udp • Filter plugins define how to manipulate data • eg. json, geoip, dns, grok, json, ruby(!) • Output plugins define where to send data • eg. cloudwatch, datadog, elasticsearch, statsd
  • 8. Open Source Monitoring Conference 2017 The Logstash Icinga Output plugin • Allows taking action in Icinga based on events encountered in logstash • Uses Icinga API to accomplish this https://github.com/Icinga/logstash-output-icinga/
  • 9. Open Source Monitoring Conference 2017
  • 10. Open Source Monitoring Conference 2017 • Monitoring of backup • Logs are indexed in Elasticsearch • Logstash to filter logs for keywords for specific actions: • Assign maintenance window to icinga node • Alert for start of backup • Alert for successful or failed backup • Remove maintenance window from icinga node DEMO: using official icinga2x-elastic vagrant box: https://github.com/Icinga/icinga-vagrant Possible Scenarios
  • 11. Open Source Monitoring Conference 2017 process-check-result ● Allows for manipulating the state of a check ○ Set a dummy `catchall` service to critical if we see the word “error” in a logstash log ○ Set a dummy `backup` service to ok after finishing a backup Setting Input type Req? exit_status number, For services: 0=OK, 1=WARNING, 2=CRITICAL, 3=UNKNOWN, for hosts: 0=OK, 1=CRITICAL. Yes plugin_output string, The plugins main output. Does not contain the performance data. Yes performance_data array, The performance data. No check_command array, The first entry should be the check commands path, then one entry for each command line option followed by an entry for each of its argument. No check_source string, Usually the name of the command_endpoint No
  • 12. Open Source Monitoring Conference 2017 send-custom-notification • Sends a custom notification • Eg. a custom host notification announcing a global maintenance to host owners Setting Input type Req? author string, Name of the author. Yes comment string, Comment text. Yes force boolean, Default: false. If true, the notification is sent regardless of downtimes or whether notifications are enabled or not. No
  • 13. Open Source Monitoring Conference 2017 add / remove-comment • Allows adding or removing a comment from a service or a host • Removing requires passing the author of the comment (removes all comments by that author :( ) Setting Input type Req? author string, Name of the author. Yes comment string, Comment text. Yes
  • 14. Open Source Monitoring Conference 2017 schedule / remove-downtime • Allows adding or removing downtime Setting Input type Req ? author string, Name of the author. Yes comment string, Comment text. Yes start_time timestamp (epoc), Timestamp marking the beginning of the downtime. Yes end_time timestamp (epoc), Timestamp marking the end of the downtime. Yes fixed boolean, Defaults to true. If true, the downtime is fixed otherwise flexible. No duration number, Duration of the downtime in seconds if fixed is set to false. Yes/N o trigger_name string, Sets the trigger for a triggered downtime. No child_options number, Schedule child downtimes. 0 does not do anything, 1 schedules child downtimes triggered by this downtime, 2 schedules non-triggered downtimes. Defaults to 0. No
  • 15. Open Source Monitoring Conference 2017 Installing the logstash output # download the gem wget https://github.com/Icinga/logstash-output-icinga/releases/download/v1.1.0/logstas h-output-icinga-1.1.0.gem # install the plugin /usr/share/logstash/bin/logstash-plugin install --local ./logstash-output-icinga-1.1.0.gem Validating ./logstash-output-icinga-1.1.0.gem Installing logstash-output-icinga Installation successful Soon to be replaced by a published gem (thanks jordansissel!)
  • 16. Open Source Monitoring Conference 2017 Configure Icinga object ApiUser "icinga" { password = "icinga" client_cn = NodeName permissions = [ "*" ] } apply Service "backup" { import "generic-service" check_command = "dummy" assign where host.address }
  • 17. Open Source Monitoring Conference 2017 input { beats { port => 5044 type => "logs" } } filter { if [syslog_severity] == "error" { mutate { replace => { "exit_status" => "2" } } } if [source] == "/var/log/mypreciouslog.json" { json { source => "message" } } } Configure logstash output { icinga { host => 'localhost' user => 'icinga' password => 'icinga' ssl_verify => false action => 'process-check-result' action_config => { exit_status => "%{exit_status}" plugin_output => "%{message}" } icinga_host => "%{hostname}" icinga_service => "backup" } elasticsearch { hosts => [ "localhost:9200" ] } stdout { } }
  • 18. Open Source Monitoring Conference 2017 Demo time! (brace yourselves..)
  • 19. Open Source Monitoring Conference 2017 Up for improvement ● Multiple icinga outputs needed for each action ○ Action_config fields should be dynamic ● Repeated configuration
  • 20. Open Source Monitoring Conference 2017 • Elasticsearch Watcher (operates on elasticsearch level!) • Provided via X-Pack (commercial) • Provides alerting on cluster and index events • Allows notification via Slack, email or supported mechanisms • Splunk? Alternative(s)
  • 21. Open Source Monitoring Conference 2017 We’re hiring! NL based consultants jobs@olindata.com
  • 22. Open Source Monitoring Conference 2017 Questions? @walterheck / @olindata http://www.olindata.com walterheck@olindata.com oliver@olindata.com http://github.com/olindata