SlideShare ist ein Scribd-Unternehmen logo
1 von 68
Downloaden Sie, um offline zu lesen
@
1© Würth Phoenix … more than software
event processing with tornado
Francesco Cina’ - Patrick Zambelli
Würth Phoenix GmbH
@
2
Patrick Zambelli
… more than software© Würth Phoenix
 Monitoring projects consultant in Würth Phoenix
 A decade of passion for contributing to the monitoring world
 Customer installations of 10.000 Hosts and a couple more
services
 Variegated list of active and passive service checks
 A long to-do list of other devices, services and unforeseen
events to keep under control
@
 IT and Consulting Company of the Würth-Group
 Headquarter in Italy, European-wide presence, more than 160 highly skilled employees
 International experience in Business Software and IT Management
 Core competencies in trading processes, wholesale distribution and logistics
 Microsoft Gold Certified Partner, ITIL certified, OTRS Preferred Partner
 Icinga partner and provider of IT system monitoring platform NetEye
3
ABOUT WÜRTH PHOENIX
Facts & figures
 More than 1.200 customers
worldwide
 Over 70 successfully
implemented ERP and CRM
projects
 400 NetEye customers
 HQ in Italy
We improve business productivity by
delivering world class software
solutions and a team of highly
motivated and skilled IT experts
© Würth Phoenix … more than software
@
 Monitoring Challenges
 Poll vs. Event
 Why a new event processor ?
 Use case of email processing
4
agenda
… more than software© Würth Phoenix
@
 Polling to collect monitoring data
5
Monitoring approach challenge
… more than software© Würth Phoenix
How is your
disk usage
OK: usage of 37%
on c:
@
Polling
 Schedule a check on static defined time intervals to get a state
 Well defined results, graphs, alerts
 Centralized configuration and control
 Examples:
 Agents i.e. NSClient++
 SSH
 SNMP
 WMI
Historical, this was the way to go
6
Monitoring approach challenge
… more than software© Würth Phoenix
How is your disk
usage
OK: usage of
37% on c:
@
Event
 Accept metrics at any time
 Interpretation on event collection
 Examples
 SNMP Traps
 Email
 Syslog
 Telemetry,
stream data from remote points to monitoring systems
 Netflow
 WebHook
7
Monitoring approach challenge
… more than software© Würth Phoenix
That’s bad
news!
Hey, I’ve got a
broken disk
Act exactly when the event happens
@
 Monitoring via Polling or Event processing ?
8
Poll vs. event: Pros and Cons
… more than software© Würth Phoenix
How is your
disk usage
Hey, I’ve got a
broken disk
@
Polling Pros
 Control when a check should be executed
 Get only the data which I’m interested in
 Knowing the context I should get
(context = host, service, performance data)
Polling Cons
 Static configuration for monitored architecture
 Continuous cost of resource usage
 Not all data is retrievable via polling
9
Poll vs. event
… more than software© Würth Phoenix
Event Pros
 Real-time react when event happens
 No need to know how to fetch data
Support the channel (i.e. syslog, email, trap,)
 Dynamic on fast changing architectures:
no action for new deployed hosts, devices,
applications
Event Cons
 Need to face large amounts of data (peaks)
 Lack for filtering at source
 Guaranty for receiving event ( i.e.SNMP Trap)
 Not the right approach for host alive or
service availability check (exceptions exists)
@
10
Poll vs. event: Need we both ?
… more than software© Würth Phoenix
How is your
disk usage
Hey, I’ve got a
broken disk
@
 Polling can be configured fast within an IT infrastructure
 Standard checks for availability and health monitoring
 Templates for reusable monitoring packages
 Monitoring Automation with Icinga2 provides dynamics to adapt to changing architectures
 Event based monitoring as complement to polling
 Experience in event based monitoring since 2013: the “Event Handler”
 Rule based (Regex)
 Support for multiple channels: Email, SNMP-Trap, Syslog, SMS
 Associate action to matched event
11
Poll vs. event: Need we both ?
… more than software© Würth Phoenix
The combination of both worlds makes a winning team !
@
 Good experience to study the concept of a daemon, able to run rules against incoming events
 New channels we want to evaluate
 NetFlow
 Telemetry
 DNS
 Webhooks
 Not scaling to address present and especially the
future amount of events
 Volume
 Velocity
 Variety: Architecture limiting further extension
12
Event Processing experience
… more than software© Würth Phoenix
BUT
@
 Short history of system complexity and monitoring
14
Let’s focus on event based monitoring
… more than software© Würth Phoenix
@
 Vertical Scaling -> More CPU, RAM
15
199x – SYSTEMs COMPLEXITY AND PERFORMANCE
… more than software© Würth Phoenix
@
 Vertical Scaling -> More CPU, RAM
16
199x – SYSTEMs COMPLEXITY AND PERFORMANCE
… more than software© Würth Phoenix
It’s SLOW! You should buy
new hardware!
@
 Vertical Scaling -> No monitoring
17
199x – Systems MONITORING
… more than software© Würth Phoenix
@
 Vertical Scaling -> No monitoring
18
199x – Systems MONITORING
… more than software© Würth Phoenix
Is it working? Well… the system is
up…
@
 Horizontal Scaling -> More Threads
19
200x – SYSTEMs COMPLEXITY AND PERFORMANCE
… more than software© Würth Phoenix
@
 Horizontal Scaling -> More Threads
20
200x – SYSTEMs COMPLEXITY AND PERFORMANCE
… more than software© Würth Phoenix
It’s SLOW! I should parallelize
the load!
@
 Horizontal Scaling -> Simple monitoring systems / scripts
21
200x – Systems MONITORING
… more than software© Würth Phoenix
@
 Horizontal Scaling -> Simple monitoring systems / scripts
22
200x – Systems MONITORING
… more than software© Würth Phoenix
Is it working? Well… I can ping
it…
@
 Distributed Systems -> More Machines
23
201x – SYSTEMs COMPLEXITY AND PERFORMANCE
… more than software© Würth Phoenix
@
 Distributed Systems -> More Machines
24
201x – SYSTEMs COMPLEXITY AND PERFORMANCE
… more than software© Würth Phoenix
It’s SLOW! I should distribute
the load!
@
 Distributed Systems -> Advanced monolithic monitoring systems / dashboards
25
201x – Systems MONITORING
… more than software© Würth Phoenix
Event
Event
Event
Event
Event
Event
@
 Distributed Systems -> Advanced monolithic monitoring systems / dashboards
26
201x – Systems MONITORING
… more than software© Würth Phoenix
Is it working? Well… the board is
green…
@
 Distributed “Distributed Systems” -> More Distributed System
27
202x – SYSTEMs COMPLEXITY AND PERFORMANCE
… more than software© Würth Phoenix
@
 Distributed “Distributed Systems” -> More Distributed System
28
202x – SYSTEMs COMPLEXITY AND PERFORMANCE
… more than software© Würth Phoenix
It’s SLOW! I should find a new
job!
@
 Distributed “Distributed System” -> Distributed monitoring systems
29
202x – Systems MONITORING
… more than software© Würth Phoenix
Event
Event
Event
Event
Event
Event
@
 Distributed “Distributed System” -> Distributed monitoring systems
30
202x – Systems MONITORING
… more than software© Würth Phoenix
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
Event
@
 Distributed “Distributed System” -> Distributed monitoring systems
31
202x – Systems MONITORING
… more than software© Würth Phoenix
@
 Distributed “Distributed System” -> Distributed monitoring systems
32
202x – Systems MONITORING
… more than software© Würth Phoenix
@
 Distributed “Distributed System” -> Distributed monitoring systems
33
202x – Systems MONITORING
… more than software© Würth Phoenix
@
 Distributed “Distributed System” -> Distributed monitoring systems
34
202x – Systems MONITORING
… more than software© Würth Phoenix
Is it working? Well… nobody is
complaining…
@
 How to handle this huge load of not homogenous events?
35
new challenge
… more than software© Würth Phoenix
@
 Scale horizontally your monitoring software
36
handling the increased load – Solution 1: Scale
… more than software© Würth Phoenix
xK
events
@
 Scale horizontally your monitoring software
37
handling the increased load – Solution 1: Scale
… more than software© Würth Phoenix
3xK
events
@
 Scale horizontally your monitoring software
 Pro:
 Could be cheap
 Could work out of the box
38
handling the increased load – Solution 1: Scale
… more than software© Würth Phoenix
@
 Scale horizontally your monitoring software
 Pro:
 Could be cheap
 Could work out of the box
 Cons
 It is not cheap
39
handling the increased load – Solution 1: Scale
… more than software© Würth Phoenix
@
 Scale horizontally your monitoring software
 Pro:
 Could be cheap
 Could work out of the box
 Cons
 It is not cheap
 It does not work out of the box
40
handling the increased load – Solution 1: Scale
… more than software© Würth Phoenix
@
 Scale horizontally your monitoring software
 Pro:
 Could be cheap
 Could work out of the box
 Cons
 It is not cheap
 It does not work out of the box
 Throughput does not grow linearly
41
handling the increased load – Solution 1: Scale
… more than software© Würth Phoenix
@
 Solution 2: Use a big data system
42
handling the increased load – Solution 2: Big Data System
… more than software© Würth Phoenix
xK
events
@
 Solution 2: Use a big data system
43
handling the increased load – Solution 2: Big Data System
… more than software© Würth Phoenix
xM
events
@
 Solution 2: Use a big data system
44
handling the increased load – Solution 2: Big Data System
… more than software© Würth Phoenix
xM events
xK
events
@
“Lots of people struggle with the complexities of getting big data systems up
and running, when they possibly shouldn’t be using the systems in the first
place.”
http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html
Processing a 128 billion edge graph
Spark cluster of 128 node => 1784 seconds
Single threaded local process => 15 seconds
45
handling the increased load – Solution 2: Big Data System
… more than software© Würth Phoenix
@
 Solution 2: Use a big data system
 Pro:
 It is a real and mature solution
 Cons:
 It adds tons of complexity
 High resources needed
 You probably don’t need it
46
handling the increased load – Solution 2: Big Data System
… more than software© Würth Phoenix
@
 We don’t want this one
47
…
… more than software© Würth Phoenix
3xK
events
@
 We don’t want this one
48
…
… more than software© Würth Phoenix
xM events xK
events
@
 What we want
49
…
… more than software© Würth Phoenix
xM events
xK
events
Something simple,
lightweight,
cheap…
@
 Let me introduce you… TORNADO!
50
TORNADO
… more than software© Würth Phoenix
xM events
xK
events
A simple “Complex Event Processing” engine
@
 The solution we desire should:
 Handle millions of events
 Scale linearly
 Multiple event formats and sources
 Take decisions based on the event
content
 Be simple
 Be easy
 Be cheap
 Robust
51
TORNADo
… more than software© Würth Phoenix
Tornado:
 Can handle millions of events per
second per CPU
@
 The solution we desire should:
 Handle millions of events
 Scale linearly
 Multiple event formats and sources
 Take decisions based on the event
content
 Be simple
 Be easy
 Be cheap
 Robust
52
TORNADo
… more than software© Würth Phoenix
Tornado:
 Stateless
 Cloud ready
@
 The solution we desire should:
 Handle millions of events
 Scale linearly
 Multiple event formats and sources
 Take decisions based on the event
content
 Be simple
 Be easy
 Be cheap
 Robust
53
TORNADo
… more than software© Würth Phoenix
Tornado:
 Handles a single event format
 Has collectors
 Small components that translate
events from a format X to the
Tornado Event format
@
 The solution we desire should:
 Handle millions of events
 Scale linearly
 Multiple event formats and sources
 Take decisions based on the event
content
 Be simple
 Be easy
 Be cheap
 Robust
54
TORNADo
… more than software© Würth Phoenix
Tornado:
 Has Pipelines, Filters and Rules
@
 The solution we desire should:
 Handle millions of events
 Scale linearly
 Multiple event formats and sources
 Take decisions based on the event
content
 Be simple
 Be easy
 Be cheap
 Robust
55
TORNADo
… more than software© Würth Phoenix
Tornado:
 Single executable
 Single configuration file
 Accepts a single event format
{
“type”: "your_event_type",
“created_ms”: 1554130814854,
“payload”:{
“key1”: "value1",
“key2”: true,
“key3”: ["something", else]
}
}
@
 The solution we desire should:
 Handle millions of events
 Scale linearly
 Multiple event formats and sources
 Take decisions based on the event
content
 Be simple
 Be easy
 Be cheap
 Robust
56
TORNADo
… more than software© Würth Phoenix
Tornado:
 Millions events per second on
commodity hardware
 No need for new hardware
 Lightweight, run with a couples of
RAM MB
 Highly optimized, Multithreaded,
Non-blocking IO
 Intensively tested
 Modular source code in Rust
@
57
TORNADO ARCHITECTURE
… more than software© Würth Phoenix
DatasourceS Tornado Collectors
Pipelines ExecutorsIcinga2 API Icinga2 API collector
SNMP Traps
Email - procmail
rsyslog
WebHooks
…
snmptrapd collector
Email collector
subscribe
stream
call
rsyslog collector
stream
WebHooks collector
API call
… collector
Matching
Extracting
Dispatching
Icinga
Icinga2 API Call
Archive
Save to file system
Script
Execute a script
Logger
Log to file system
…
@
58
TORNADO ARCHITECTURE
… more than software© Würth Phoenix
DatasourceS Tornado Collectors
Pipelines ExecutorsIcinga2 API Icinga2 API collector
SNMP Traps
Email - procmail
rsyslog
WebHooks
…
snmptrapd collector
Email collector
subscribe
stream
call
rsyslog collector
stream
WebHooks collector
API call
… collector
Matching
Extracting
Dispatching
Icinga
Icinga2 API Call
Archive
Save to file system
Script
Execute a script
Logger
Log to file system
…
@
59
TORNADO ARCHITECTURE
… more than software© Würth Phoenix
DatasourceS Tornado Collectors
Pipelines ExecutorsIcinga2 API Icinga2 API collector
SNMP Traps
Email - procmail
rsyslog
WebHooks
…
snmptrapd collector
Email collector
subscribe
stream
call
rsyslog collector
stream
WebHooks collector
API call
… collector
Matching
Extracting
Dispatching
Icinga
Icinga2 API Call
Archive
Save to file system
Script
Execute a script
Logger
Log to file system
…
@
 Tornado Collector Example: SNMPtrapd collector
60
TORNADO ARCHITECTURE - Collectors
… more than software© Würth Phoenix
PDU INFO:
version 1
errorstatus 0
community public
receivedfrom UDP: [127.0.1.1]:41543->[127.0.2.2]:162
transactionid 1
errorindex 0
messageid 0
requestid 414568963
notificationtype TRAP
VARBINDS:
iso.3.6.1.2.1.1.3.0 type=67 value=Timeticks: (1166403) 3:14:24.03
iso.3.6.1.6.3.1.1.4.1.0 type=6 value=OID: iso.3.6.1.4.1.8072.2.3.0.1
iso.3.6.1.4.1.8072.2.3.2.1 type=2 value=INTEGER: 123456
snmptrapd collector
{
"type":"snmptrapd",
"created_ms":"1553765890000",
"payload":{
"protocol":"UDP",
"src_ip":"127.0.1.1",
"src_port":"41543",
"dest_ip":"127.0.2.2",
"PDUInfo":{
"version":"1",
"errorstatus":"0",
"community":"public",
"transactionid":"1",
"errorindex":"0",
"messageid":"0",
"requestid":"414568963",
"notificationtype":"TRAP"
},
"oids":{
"iso.3.6.1.2.1.1.3.0":"67",
"iso.3.6.1.6.3.1.1.4.1.0":"6",
"iso.3.6.1.4.1.8072.2.3.2.1":"2"
}
}
}
From SNMPtrapd
To Tornado
@
61
TORNADO ARCHITECTURE
… more than software© Würth Phoenix
DatasourceS Tornado Collectors
Pipelines ExecutorsIcinga2 API Icinga2 API collector
SNMP Traps
Email - procmail
rsyslog
WebHooks
…
snmptrapd collector
Email collector
subscribe
stream
call
rsyslog collector
stream
WebHooks collector
API call
… collector
Matching
Extracting
Dispatching
Icinga
Icinga2 API Call
Archive
Save to file system
Script
Execute a script
Logger
Log to file system
…
@
62
TORNADO ARCHITECTURE
… more than software© Würth Phoenix
DatasourceS Tornado Collectors
Pipelines ExecutorsIcinga2 API Icinga2 API collector
SNMP Traps
Email - procmail
rsyslog
WebHooks
…
snmptrapd collector
Email collector
subscribe
stream
call
rsyslog collector
stream
WebHooks collector
API call
… collector
Matching
Extracting
Dispatching
Icinga
Icinga2 API Call
Archive
Save to file system
Script
Execute a script
Logger
Log to file system
…
@
 Pipelines, Filters, Rules
63
TORNADO architecture
… more than software© Würth Phoenix
Email Filter
Snmptrapd
filter
Rule 1
Rule 2
Rule n
Host X Host Y
Rule 1
Rule 2
Rule 1
Rule 2
Rsyslog
filter
Rule 1
Rule 2
Rule n
Hostgroup
Linux1 filter
Rule 1
Rule 2
Rule n
Pass all
Filter
Event
“type”: “email”
Filter Matched
Filter Matched
@
 Pipelines, Filters, Rules
64
TORNADO architecture
… more than software© Würth Phoenix
Email Filter
Rule 1
Rule 2
Rule n
Pass all
Filter
Event
“type”: “email”
Filter Matched
Filter Matched
{
"description": "This filter allows every event",
"active": true
}
{
"description": "This filter allows events of type 'email'",
"active": true,
"filter": {
"type": "equal",
"first": "${event.type}",
"second": "email"
}
}
@
 Pipelines, Filters, Rules
65
TORNADO architecture
… more than software© Würth Phoenix
Email Filter
Rule 1
Rule 2
Rule n
Pass all
Filter
Event
“type”: “email”
Action: Icinga Service CRITICAL
Action: Execute Script
Action: Icinga Service OK
Action …
Filter Matched
Filter Matched
Rule Matched Action Dispatched
@
 Pipelines, Filters, Rules
66
TORNADO architecture
… more than software© Würth Phoenix
Email Filter
Rule 2
Rule Matched
{
"description": "Normal temperature of a server.",
"constraint": {
"WHERE": {
"type": "AND",
"operators": [
{
"type": "equal",
"first": "${event.type}",
"second": “email"
},
{
"type": "lt",
"first": "${event.payload.temperature}",
"second": "50"
}
]
}
},
"actions": [
{
"id": "icinga2",
"payload": {
"icinga2_action_name": "process-check-result",
"icinga2_action_payload": {
"exit_status": "0",
"plugin_output": "OK - The temperature is ${event.payload.temperature}",
"filter": "host.name=="${event.body.server_name}"",
"type": “Host"
}
}
}
]
}
@
67
Usecase: email event collection
… more than software© Würth Phoenix
Demo time
@
68
Usecase: email event collection
… more than software© Würth Phoenix
DatasourceS Tornado Collectors
Pipelines ExecutorsIcinga2 API Icinga2 API collector
SNMP Traps
Email - procmail
rsyslog
WebHooks
snmptrapd collector
Email collector
subscribe
stream
call
rsyslog collector
stream
WebHooks collector
API call
Matching
Extracting
Dispatching
Icinga
Icinga2 API Call
Archive
Save to file system
Script
Execute a script
Logger
Log to file system
…
@
71
JOIN US
… more than software© Würth Phoenix
The SIMPLE Complex Event Processing Engine
https://github.com/WuerthPhoenix/tornado
mr.francesco.cina@gmail.com
https://github.com/ufoscout
patrick.zambelli@wuerth-phoenix.com
https://github.com/zampat

Weitere ähnliche Inhalte

Was ist angesagt?

Tulinx introduction 20130622 detailed
Tulinx introduction 20130622   detailedTulinx introduction 20130622   detailed
Tulinx introduction 20130622 detailedarjen1970
 
Managing a Widely Distributed Network
Managing a Widely Distributed NetworkManaging a Widely Distributed Network
Managing a Widely Distributed Network Savvius, Inc
 
OpManager training - Device discovery and classification.
OpManager training - Device discovery and classification.OpManager training - Device discovery and classification.
OpManager training - Device discovery and classification.ManageEngine, Zoho Corporation
 
IMCSummit 2015 - Day 1 IT Business Track - Designing a Big Data Analytics Pla...
IMCSummit 2015 - Day 1 IT Business Track - Designing a Big Data Analytics Pla...IMCSummit 2015 - Day 1 IT Business Track - Designing a Big Data Analytics Pla...
IMCSummit 2015 - Day 1 IT Business Track - Designing a Big Data Analytics Pla...In-Memory Computing Summit
 
Openstack Summit Vancouver 2015 - Maintaining and Operating Swift at Public C...
Openstack Summit Vancouver 2015 - Maintaining and Operating Swift at Public C...Openstack Summit Vancouver 2015 - Maintaining and Operating Swift at Public C...
Openstack Summit Vancouver 2015 - Maintaining and Operating Swift at Public C...donaghmccabe
 
UniVerse11.2 Audit Logging
UniVerse11.2 Audit LoggingUniVerse11.2 Audit Logging
UniVerse11.2 Audit LoggingRocket Software
 
Monitoring federation open stack infrastructure
Monitoring federation open stack infrastructureMonitoring federation open stack infrastructure
Monitoring federation open stack infrastructureFernando Lopez Aguilar
 
OSMC 2015: The Assimilation Project by Alan Robertson
OSMC 2015: The Assimilation Project by Alan RobertsonOSMC 2015: The Assimilation Project by Alan Robertson
OSMC 2015: The Assimilation Project by Alan RobertsonNETWAYS
 

Was ist angesagt? (9)

Hp open view
Hp open viewHp open view
Hp open view
 
Tulinx introduction 20130622 detailed
Tulinx introduction 20130622   detailedTulinx introduction 20130622   detailed
Tulinx introduction 20130622 detailed
 
Managing a Widely Distributed Network
Managing a Widely Distributed NetworkManaging a Widely Distributed Network
Managing a Widely Distributed Network
 
OpManager training - Device discovery and classification.
OpManager training - Device discovery and classification.OpManager training - Device discovery and classification.
OpManager training - Device discovery and classification.
 
IMCSummit 2015 - Day 1 IT Business Track - Designing a Big Data Analytics Pla...
IMCSummit 2015 - Day 1 IT Business Track - Designing a Big Data Analytics Pla...IMCSummit 2015 - Day 1 IT Business Track - Designing a Big Data Analytics Pla...
IMCSummit 2015 - Day 1 IT Business Track - Designing a Big Data Analytics Pla...
 
Openstack Summit Vancouver 2015 - Maintaining and Operating Swift at Public C...
Openstack Summit Vancouver 2015 - Maintaining and Operating Swift at Public C...Openstack Summit Vancouver 2015 - Maintaining and Operating Swift at Public C...
Openstack Summit Vancouver 2015 - Maintaining and Operating Swift at Public C...
 
UniVerse11.2 Audit Logging
UniVerse11.2 Audit LoggingUniVerse11.2 Audit Logging
UniVerse11.2 Audit Logging
 
Monitoring federation open stack infrastructure
Monitoring federation open stack infrastructureMonitoring federation open stack infrastructure
Monitoring federation open stack infrastructure
 
OSMC 2015: The Assimilation Project by Alan Robertson
OSMC 2015: The Assimilation Project by Alan RobertsonOSMC 2015: The Assimilation Project by Alan Robertson
OSMC 2015: The Assimilation Project by Alan Robertson
 

Ähnlich wie OSMC 2019 | Tornado – Extend Icinga2 for Active and passive Monitoring of complex heterogeneous IT Environments by Francesco Cina and Patrick Zambelli

SciPy Stack vs. InfluxDB and Grafana
SciPy Stack vs. InfluxDB and GrafanaSciPy Stack vs. InfluxDB and Grafana
SciPy Stack vs. InfluxDB and GrafanaInfluxData
 
NetEye 4 based on Icinga 2 - Icinga Camp Milan 2019
NetEye 4 based on Icinga 2 - Icinga Camp Milan 2019NetEye 4 based on Icinga 2 - Icinga Camp Milan 2019
NetEye 4 based on Icinga 2 - Icinga Camp Milan 2019Icinga
 
Icinga Camp, Berlin 2019
Icinga Camp, Berlin 2019Icinga Camp, Berlin 2019
Icinga Camp, Berlin 2019Francesca Papa
 
SFScon19 - Michele Santuari - Tornado A new Complex Event Processing Engine d...
SFScon19 - Michele Santuari - Tornado A new Complex Event Processing Engine d...SFScon19 - Michele Santuari - Tornado A new Complex Event Processing Engine d...
SFScon19 - Michele Santuari - Tornado A new Complex Event Processing Engine d...South Tyrol Free Software Conference
 
SplunkLive! Splunk App for VMware
SplunkLive! Splunk App for VMwareSplunkLive! Splunk App for VMware
SplunkLive! Splunk App for VMwareSplunk
 
David Henthorn [Rose-Hulman Institute of Technology] | Illuminating the Dark ...
David Henthorn [Rose-Hulman Institute of Technology] | Illuminating the Dark ...David Henthorn [Rose-Hulman Institute of Technology] | Illuminating the Dark ...
David Henthorn [Rose-Hulman Institute of Technology] | Illuminating the Dark ...InfluxData
 
SFScon15 - Jürgen Vigna: " Application Performance Monitoring auf Open Source...
SFScon15 - Jürgen Vigna: " Application Performance Monitoring auf Open Source...SFScon15 - Jürgen Vigna: " Application Performance Monitoring auf Open Source...
SFScon15 - Jürgen Vigna: " Application Performance Monitoring auf Open Source...South Tyrol Free Software Conference
 
2009-05-05 A Customer's Perspective on Making Enterprise Linux Deployable, Sc...
2009-05-05 A Customer's Perspective on Making Enterprise Linux Deployable, Sc...2009-05-05 A Customer's Perspective on Making Enterprise Linux Deployable, Sc...
2009-05-05 A Customer's Perspective on Making Enterprise Linux Deployable, Sc...Shawn Wells
 
Splunk for vmware virtualization customer presentation
Splunk for vmware virtualization customer presentationSplunk for vmware virtualization customer presentation
Splunk for vmware virtualization customer presentationGreg Hanchin
 
Radiology Partner invitation - Join us in offering powerful solutions to the ...
Radiology Partner invitation - Join us in offering powerful solutions to the ...Radiology Partner invitation - Join us in offering powerful solutions to the ...
Radiology Partner invitation - Join us in offering powerful solutions to the ...Joachim Surich
 
SplunkLive! Minneapolis April 2013 - Digital River
SplunkLive! Minneapolis April 2013 - Digital RiverSplunkLive! Minneapolis April 2013 - Digital River
SplunkLive! Minneapolis April 2013 - Digital RiverSplunk
 
OSMC 2015 | End-to-End Monitoring mit Alyvix by Jürgen Vigna
OSMC 2015 | End-to-End Monitoring mit Alyvix by Jürgen VignaOSMC 2015 | End-to-End Monitoring mit Alyvix by Jürgen Vigna
OSMC 2015 | End-to-End Monitoring mit Alyvix by Jürgen VignaNETWAYS
 
OSMC 2015: End to End Monitoring mit Alyvix-Jürgen Vigna
OSMC 2015: End to End Monitoring mit Alyvix-Jürgen VignaOSMC 2015: End to End Monitoring mit Alyvix-Jürgen Vigna
OSMC 2015: End to End Monitoring mit Alyvix-Jürgen VignaNETWAYS
 
Case study: How Cozy Cloud monitors every layer of its activity using OVH Met...
Case study: How Cozy Cloud monitors every layer of its activity using OVH Met...Case study: How Cozy Cloud monitors every layer of its activity using OVH Met...
Case study: How Cozy Cloud monitors every layer of its activity using OVH Met...OVHcloud
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsSafe Software
 
Smau Milano 2016 - Retelit
Smau Milano 2016 - RetelitSmau Milano 2016 - Retelit
Smau Milano 2016 - RetelitSMAU
 
Acceleration_and_Security_draft_v2
Acceleration_and_Security_draft_v2Acceleration_and_Security_draft_v2
Acceleration_and_Security_draft_v2Srinivasa Addepalli
 

Ähnlich wie OSMC 2019 | Tornado – Extend Icinga2 for Active and passive Monitoring of complex heterogeneous IT Environments by Francesco Cina and Patrick Zambelli (20)

SciPy Stack vs. InfluxDB and Grafana
SciPy Stack vs. InfluxDB and GrafanaSciPy Stack vs. InfluxDB and Grafana
SciPy Stack vs. InfluxDB and Grafana
 
Influxdays
InfluxdaysInfluxdays
Influxdays
 
GrafanaCon EU 2018
GrafanaCon EU 2018GrafanaCon EU 2018
GrafanaCon EU 2018
 
NetEye 4 based on Icinga 2 - Icinga Camp Milan 2019
NetEye 4 based on Icinga 2 - Icinga Camp Milan 2019NetEye 4 based on Icinga 2 - Icinga Camp Milan 2019
NetEye 4 based on Icinga 2 - Icinga Camp Milan 2019
 
Icinga Camp, Berlin 2019
Icinga Camp, Berlin 2019Icinga Camp, Berlin 2019
Icinga Camp, Berlin 2019
 
SFScon19 - Michele Santuari - Tornado A new Complex Event Processing Engine d...
SFScon19 - Michele Santuari - Tornado A new Complex Event Processing Engine d...SFScon19 - Michele Santuari - Tornado A new Complex Event Processing Engine d...
SFScon19 - Michele Santuari - Tornado A new Complex Event Processing Engine d...
 
SplunkLive! Splunk App for VMware
SplunkLive! Splunk App for VMwareSplunkLive! Splunk App for VMware
SplunkLive! Splunk App for VMware
 
David Henthorn [Rose-Hulman Institute of Technology] | Illuminating the Dark ...
David Henthorn [Rose-Hulman Institute of Technology] | Illuminating the Dark ...David Henthorn [Rose-Hulman Institute of Technology] | Illuminating the Dark ...
David Henthorn [Rose-Hulman Institute of Technology] | Illuminating the Dark ...
 
SFScon15 - Jürgen Vigna: " Application Performance Monitoring auf Open Source...
SFScon15 - Jürgen Vigna: " Application Performance Monitoring auf Open Source...SFScon15 - Jürgen Vigna: " Application Performance Monitoring auf Open Source...
SFScon15 - Jürgen Vigna: " Application Performance Monitoring auf Open Source...
 
2009-05-05 A Customer's Perspective on Making Enterprise Linux Deployable, Sc...
2009-05-05 A Customer's Perspective on Making Enterprise Linux Deployable, Sc...2009-05-05 A Customer's Perspective on Making Enterprise Linux Deployable, Sc...
2009-05-05 A Customer's Perspective on Making Enterprise Linux Deployable, Sc...
 
Splunk for vmware virtualization customer presentation
Splunk for vmware virtualization customer presentationSplunk for vmware virtualization customer presentation
Splunk for vmware virtualization customer presentation
 
Radiology Partner invitation - Join us in offering powerful solutions to the ...
Radiology Partner invitation - Join us in offering powerful solutions to the ...Radiology Partner invitation - Join us in offering powerful solutions to the ...
Radiology Partner invitation - Join us in offering powerful solutions to the ...
 
SplunkLive! Minneapolis April 2013 - Digital River
SplunkLive! Minneapolis April 2013 - Digital RiverSplunkLive! Minneapolis April 2013 - Digital River
SplunkLive! Minneapolis April 2013 - Digital River
 
OSMC 2015 | End-to-End Monitoring mit Alyvix by Jürgen Vigna
OSMC 2015 | End-to-End Monitoring mit Alyvix by Jürgen VignaOSMC 2015 | End-to-End Monitoring mit Alyvix by Jürgen Vigna
OSMC 2015 | End-to-End Monitoring mit Alyvix by Jürgen Vigna
 
OSMC 2015: End to End Monitoring mit Alyvix-Jürgen Vigna
OSMC 2015: End to End Monitoring mit Alyvix-Jürgen VignaOSMC 2015: End to End Monitoring mit Alyvix-Jürgen Vigna
OSMC 2015: End to End Monitoring mit Alyvix-Jürgen Vigna
 
Case study: How Cozy Cloud monitors every layer of its activity using OVH Met...
Case study: How Cozy Cloud monitors every layer of its activity using OVH Met...Case study: How Cozy Cloud monitors every layer of its activity using OVH Met...
Case study: How Cozy Cloud monitors every layer of its activity using OVH Met...
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data Streams
 
Smau Milano 2016 - Retelit
Smau Milano 2016 - RetelitSmau Milano 2016 - Retelit
Smau Milano 2016 - Retelit
 
Acceleration_and_Security_draft_v2
Acceleration_and_Security_draft_v2Acceleration_and_Security_draft_v2
Acceleration_and_Security_draft_v2
 
Rotronic RMS Catalog
Rotronic RMS CatalogRotronic RMS Catalog
Rotronic RMS Catalog
 

Kürzlich hochgeladen

Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024Andreas Granig
 
OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024Shane Coughlan
 
Malaysia E-Invoice digital signature docpptx
Malaysia E-Invoice digital signature docpptxMalaysia E-Invoice digital signature docpptx
Malaysia E-Invoice digital signature docpptxMok TH
 
What need to be mastered as AI-Powered Java Developers
What need to be mastered as AI-Powered Java DevelopersWhat need to be mastered as AI-Powered Java Developers
What need to be mastered as AI-Powered Java DevelopersEmilyJiang23
 
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1KnowledgeSeed
 
A Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data MigrationA Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data MigrationHelp Desk Migration
 
Lessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdfLessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdfSrushith Repakula
 
How to install and activate eGrabber JobGrabber
How to install and activate eGrabber JobGrabberHow to install and activate eGrabber JobGrabber
How to install and activate eGrabber JobGrabbereGrabber
 
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAGAI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAGAlluxio, Inc.
 
JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)Max Lee
 
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Gáspár Nagy
 
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product UpdatesGraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product UpdatesNeo4j
 
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...OnePlan Solutions
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfDeskTrack
 
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdfStrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdfsteffenkarlsson2
 
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfImplementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfVictor Lopez
 
Naer Toolbar Redesign - Usability Research Synthesis
Naer Toolbar Redesign - Usability Research SynthesisNaer Toolbar Redesign - Usability Research Synthesis
Naer Toolbar Redesign - Usability Research Synthesisparimabajra
 
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...naitiksharma1124
 

Kürzlich hochgeladen (20)

Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024
 
OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024
 
Malaysia E-Invoice digital signature docpptx
Malaysia E-Invoice digital signature docpptxMalaysia E-Invoice digital signature docpptx
Malaysia E-Invoice digital signature docpptx
 
What need to be mastered as AI-Powered Java Developers
What need to be mastered as AI-Powered Java DevelopersWhat need to be mastered as AI-Powered Java Developers
What need to be mastered as AI-Powered Java Developers
 
Top Mobile App Development Companies 2024
Top Mobile App Development Companies 2024Top Mobile App Development Companies 2024
Top Mobile App Development Companies 2024
 
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
 
A Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data MigrationA Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data Migration
 
Lessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdfLessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdf
 
How to install and activate eGrabber JobGrabber
How to install and activate eGrabber JobGrabberHow to install and activate eGrabber JobGrabber
How to install and activate eGrabber JobGrabber
 
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAGAI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
 
JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)
 
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
 
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product UpdatesGraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
 
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdf
 
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdfStrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf
 
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfImplementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
 
Naer Toolbar Redesign - Usability Research Synthesis
Naer Toolbar Redesign - Usability Research SynthesisNaer Toolbar Redesign - Usability Research Synthesis
Naer Toolbar Redesign - Usability Research Synthesis
 
AI Hackathon.pptx
AI                        Hackathon.pptxAI                        Hackathon.pptx
AI Hackathon.pptx
 
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
 

OSMC 2019 | Tornado – Extend Icinga2 for Active and passive Monitoring of complex heterogeneous IT Environments by Francesco Cina and Patrick Zambelli

  • 1. @ 1© Würth Phoenix … more than software event processing with tornado Francesco Cina’ - Patrick Zambelli Würth Phoenix GmbH
  • 2. @ 2 Patrick Zambelli … more than software© Würth Phoenix  Monitoring projects consultant in Würth Phoenix  A decade of passion for contributing to the monitoring world  Customer installations of 10.000 Hosts and a couple more services  Variegated list of active and passive service checks  A long to-do list of other devices, services and unforeseen events to keep under control
  • 3. @  IT and Consulting Company of the Würth-Group  Headquarter in Italy, European-wide presence, more than 160 highly skilled employees  International experience in Business Software and IT Management  Core competencies in trading processes, wholesale distribution and logistics  Microsoft Gold Certified Partner, ITIL certified, OTRS Preferred Partner  Icinga partner and provider of IT system monitoring platform NetEye 3 ABOUT WÜRTH PHOENIX Facts & figures  More than 1.200 customers worldwide  Over 70 successfully implemented ERP and CRM projects  400 NetEye customers  HQ in Italy We improve business productivity by delivering world class software solutions and a team of highly motivated and skilled IT experts © Würth Phoenix … more than software
  • 4. @  Monitoring Challenges  Poll vs. Event  Why a new event processor ?  Use case of email processing 4 agenda … more than software© Würth Phoenix
  • 5. @  Polling to collect monitoring data 5 Monitoring approach challenge … more than software© Würth Phoenix How is your disk usage OK: usage of 37% on c:
  • 6. @ Polling  Schedule a check on static defined time intervals to get a state  Well defined results, graphs, alerts  Centralized configuration and control  Examples:  Agents i.e. NSClient++  SSH  SNMP  WMI Historical, this was the way to go 6 Monitoring approach challenge … more than software© Würth Phoenix How is your disk usage OK: usage of 37% on c:
  • 7. @ Event  Accept metrics at any time  Interpretation on event collection  Examples  SNMP Traps  Email  Syslog  Telemetry, stream data from remote points to monitoring systems  Netflow  WebHook 7 Monitoring approach challenge … more than software© Würth Phoenix That’s bad news! Hey, I’ve got a broken disk Act exactly when the event happens
  • 8. @  Monitoring via Polling or Event processing ? 8 Poll vs. event: Pros and Cons … more than software© Würth Phoenix How is your disk usage Hey, I’ve got a broken disk
  • 9. @ Polling Pros  Control when a check should be executed  Get only the data which I’m interested in  Knowing the context I should get (context = host, service, performance data) Polling Cons  Static configuration for monitored architecture  Continuous cost of resource usage  Not all data is retrievable via polling 9 Poll vs. event … more than software© Würth Phoenix Event Pros  Real-time react when event happens  No need to know how to fetch data Support the channel (i.e. syslog, email, trap,)  Dynamic on fast changing architectures: no action for new deployed hosts, devices, applications Event Cons  Need to face large amounts of data (peaks)  Lack for filtering at source  Guaranty for receiving event ( i.e.SNMP Trap)  Not the right approach for host alive or service availability check (exceptions exists)
  • 10. @ 10 Poll vs. event: Need we both ? … more than software© Würth Phoenix How is your disk usage Hey, I’ve got a broken disk
  • 11. @  Polling can be configured fast within an IT infrastructure  Standard checks for availability and health monitoring  Templates for reusable monitoring packages  Monitoring Automation with Icinga2 provides dynamics to adapt to changing architectures  Event based monitoring as complement to polling  Experience in event based monitoring since 2013: the “Event Handler”  Rule based (Regex)  Support for multiple channels: Email, SNMP-Trap, Syslog, SMS  Associate action to matched event 11 Poll vs. event: Need we both ? … more than software© Würth Phoenix The combination of both worlds makes a winning team !
  • 12. @  Good experience to study the concept of a daemon, able to run rules against incoming events  New channels we want to evaluate  NetFlow  Telemetry  DNS  Webhooks  Not scaling to address present and especially the future amount of events  Volume  Velocity  Variety: Architecture limiting further extension 12 Event Processing experience … more than software© Würth Phoenix BUT
  • 13. @  Short history of system complexity and monitoring 14 Let’s focus on event based monitoring … more than software© Würth Phoenix
  • 14. @  Vertical Scaling -> More CPU, RAM 15 199x – SYSTEMs COMPLEXITY AND PERFORMANCE … more than software© Würth Phoenix
  • 15. @  Vertical Scaling -> More CPU, RAM 16 199x – SYSTEMs COMPLEXITY AND PERFORMANCE … more than software© Würth Phoenix It’s SLOW! You should buy new hardware!
  • 16. @  Vertical Scaling -> No monitoring 17 199x – Systems MONITORING … more than software© Würth Phoenix
  • 17. @  Vertical Scaling -> No monitoring 18 199x – Systems MONITORING … more than software© Würth Phoenix Is it working? Well… the system is up…
  • 18. @  Horizontal Scaling -> More Threads 19 200x – SYSTEMs COMPLEXITY AND PERFORMANCE … more than software© Würth Phoenix
  • 19. @  Horizontal Scaling -> More Threads 20 200x – SYSTEMs COMPLEXITY AND PERFORMANCE … more than software© Würth Phoenix It’s SLOW! I should parallelize the load!
  • 20. @  Horizontal Scaling -> Simple monitoring systems / scripts 21 200x – Systems MONITORING … more than software© Würth Phoenix
  • 21. @  Horizontal Scaling -> Simple monitoring systems / scripts 22 200x – Systems MONITORING … more than software© Würth Phoenix Is it working? Well… I can ping it…
  • 22. @  Distributed Systems -> More Machines 23 201x – SYSTEMs COMPLEXITY AND PERFORMANCE … more than software© Würth Phoenix
  • 23. @  Distributed Systems -> More Machines 24 201x – SYSTEMs COMPLEXITY AND PERFORMANCE … more than software© Würth Phoenix It’s SLOW! I should distribute the load!
  • 24. @  Distributed Systems -> Advanced monolithic monitoring systems / dashboards 25 201x – Systems MONITORING … more than software© Würth Phoenix Event Event Event Event Event Event
  • 25. @  Distributed Systems -> Advanced monolithic monitoring systems / dashboards 26 201x – Systems MONITORING … more than software© Würth Phoenix Is it working? Well… the board is green…
  • 26. @  Distributed “Distributed Systems” -> More Distributed System 27 202x – SYSTEMs COMPLEXITY AND PERFORMANCE … more than software© Würth Phoenix
  • 27. @  Distributed “Distributed Systems” -> More Distributed System 28 202x – SYSTEMs COMPLEXITY AND PERFORMANCE … more than software© Würth Phoenix It’s SLOW! I should find a new job!
  • 28. @  Distributed “Distributed System” -> Distributed monitoring systems 29 202x – Systems MONITORING … more than software© Würth Phoenix Event Event Event Event Event Event
  • 29. @  Distributed “Distributed System” -> Distributed monitoring systems 30 202x – Systems MONITORING … more than software© Würth Phoenix Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event Event
  • 30. @  Distributed “Distributed System” -> Distributed monitoring systems 31 202x – Systems MONITORING … more than software© Würth Phoenix
  • 31. @  Distributed “Distributed System” -> Distributed monitoring systems 32 202x – Systems MONITORING … more than software© Würth Phoenix
  • 32. @  Distributed “Distributed System” -> Distributed monitoring systems 33 202x – Systems MONITORING … more than software© Würth Phoenix
  • 33. @  Distributed “Distributed System” -> Distributed monitoring systems 34 202x – Systems MONITORING … more than software© Würth Phoenix Is it working? Well… nobody is complaining…
  • 34. @  How to handle this huge load of not homogenous events? 35 new challenge … more than software© Würth Phoenix
  • 35. @  Scale horizontally your monitoring software 36 handling the increased load – Solution 1: Scale … more than software© Würth Phoenix xK events
  • 36. @  Scale horizontally your monitoring software 37 handling the increased load – Solution 1: Scale … more than software© Würth Phoenix 3xK events
  • 37. @  Scale horizontally your monitoring software  Pro:  Could be cheap  Could work out of the box 38 handling the increased load – Solution 1: Scale … more than software© Würth Phoenix
  • 38. @  Scale horizontally your monitoring software  Pro:  Could be cheap  Could work out of the box  Cons  It is not cheap 39 handling the increased load – Solution 1: Scale … more than software© Würth Phoenix
  • 39. @  Scale horizontally your monitoring software  Pro:  Could be cheap  Could work out of the box  Cons  It is not cheap  It does not work out of the box 40 handling the increased load – Solution 1: Scale … more than software© Würth Phoenix
  • 40. @  Scale horizontally your monitoring software  Pro:  Could be cheap  Could work out of the box  Cons  It is not cheap  It does not work out of the box  Throughput does not grow linearly 41 handling the increased load – Solution 1: Scale … more than software© Würth Phoenix
  • 41. @  Solution 2: Use a big data system 42 handling the increased load – Solution 2: Big Data System … more than software© Würth Phoenix xK events
  • 42. @  Solution 2: Use a big data system 43 handling the increased load – Solution 2: Big Data System … more than software© Würth Phoenix xM events
  • 43. @  Solution 2: Use a big data system 44 handling the increased load – Solution 2: Big Data System … more than software© Würth Phoenix xM events xK events
  • 44. @ “Lots of people struggle with the complexities of getting big data systems up and running, when they possibly shouldn’t be using the systems in the first place.” http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html Processing a 128 billion edge graph Spark cluster of 128 node => 1784 seconds Single threaded local process => 15 seconds 45 handling the increased load – Solution 2: Big Data System … more than software© Würth Phoenix
  • 45. @  Solution 2: Use a big data system  Pro:  It is a real and mature solution  Cons:  It adds tons of complexity  High resources needed  You probably don’t need it 46 handling the increased load – Solution 2: Big Data System … more than software© Würth Phoenix
  • 46. @  We don’t want this one 47 … … more than software© Würth Phoenix 3xK events
  • 47. @  We don’t want this one 48 … … more than software© Würth Phoenix xM events xK events
  • 48. @  What we want 49 … … more than software© Würth Phoenix xM events xK events Something simple, lightweight, cheap…
  • 49. @  Let me introduce you… TORNADO! 50 TORNADO … more than software© Würth Phoenix xM events xK events A simple “Complex Event Processing” engine
  • 50. @  The solution we desire should:  Handle millions of events  Scale linearly  Multiple event formats and sources  Take decisions based on the event content  Be simple  Be easy  Be cheap  Robust 51 TORNADo … more than software© Würth Phoenix Tornado:  Can handle millions of events per second per CPU
  • 51. @  The solution we desire should:  Handle millions of events  Scale linearly  Multiple event formats and sources  Take decisions based on the event content  Be simple  Be easy  Be cheap  Robust 52 TORNADo … more than software© Würth Phoenix Tornado:  Stateless  Cloud ready
  • 52. @  The solution we desire should:  Handle millions of events  Scale linearly  Multiple event formats and sources  Take decisions based on the event content  Be simple  Be easy  Be cheap  Robust 53 TORNADo … more than software© Würth Phoenix Tornado:  Handles a single event format  Has collectors  Small components that translate events from a format X to the Tornado Event format
  • 53. @  The solution we desire should:  Handle millions of events  Scale linearly  Multiple event formats and sources  Take decisions based on the event content  Be simple  Be easy  Be cheap  Robust 54 TORNADo … more than software© Würth Phoenix Tornado:  Has Pipelines, Filters and Rules
  • 54. @  The solution we desire should:  Handle millions of events  Scale linearly  Multiple event formats and sources  Take decisions based on the event content  Be simple  Be easy  Be cheap  Robust 55 TORNADo … more than software© Würth Phoenix Tornado:  Single executable  Single configuration file  Accepts a single event format { “type”: "your_event_type", “created_ms”: 1554130814854, “payload”:{ “key1”: "value1", “key2”: true, “key3”: ["something", else] } }
  • 55. @  The solution we desire should:  Handle millions of events  Scale linearly  Multiple event formats and sources  Take decisions based on the event content  Be simple  Be easy  Be cheap  Robust 56 TORNADo … more than software© Würth Phoenix Tornado:  Millions events per second on commodity hardware  No need for new hardware  Lightweight, run with a couples of RAM MB  Highly optimized, Multithreaded, Non-blocking IO  Intensively tested  Modular source code in Rust
  • 56. @ 57 TORNADO ARCHITECTURE … more than software© Würth Phoenix DatasourceS Tornado Collectors Pipelines ExecutorsIcinga2 API Icinga2 API collector SNMP Traps Email - procmail rsyslog WebHooks … snmptrapd collector Email collector subscribe stream call rsyslog collector stream WebHooks collector API call … collector Matching Extracting Dispatching Icinga Icinga2 API Call Archive Save to file system Script Execute a script Logger Log to file system …
  • 57. @ 58 TORNADO ARCHITECTURE … more than software© Würth Phoenix DatasourceS Tornado Collectors Pipelines ExecutorsIcinga2 API Icinga2 API collector SNMP Traps Email - procmail rsyslog WebHooks … snmptrapd collector Email collector subscribe stream call rsyslog collector stream WebHooks collector API call … collector Matching Extracting Dispatching Icinga Icinga2 API Call Archive Save to file system Script Execute a script Logger Log to file system …
  • 58. @ 59 TORNADO ARCHITECTURE … more than software© Würth Phoenix DatasourceS Tornado Collectors Pipelines ExecutorsIcinga2 API Icinga2 API collector SNMP Traps Email - procmail rsyslog WebHooks … snmptrapd collector Email collector subscribe stream call rsyslog collector stream WebHooks collector API call … collector Matching Extracting Dispatching Icinga Icinga2 API Call Archive Save to file system Script Execute a script Logger Log to file system …
  • 59. @  Tornado Collector Example: SNMPtrapd collector 60 TORNADO ARCHITECTURE - Collectors … more than software© Würth Phoenix PDU INFO: version 1 errorstatus 0 community public receivedfrom UDP: [127.0.1.1]:41543->[127.0.2.2]:162 transactionid 1 errorindex 0 messageid 0 requestid 414568963 notificationtype TRAP VARBINDS: iso.3.6.1.2.1.1.3.0 type=67 value=Timeticks: (1166403) 3:14:24.03 iso.3.6.1.6.3.1.1.4.1.0 type=6 value=OID: iso.3.6.1.4.1.8072.2.3.0.1 iso.3.6.1.4.1.8072.2.3.2.1 type=2 value=INTEGER: 123456 snmptrapd collector { "type":"snmptrapd", "created_ms":"1553765890000", "payload":{ "protocol":"UDP", "src_ip":"127.0.1.1", "src_port":"41543", "dest_ip":"127.0.2.2", "PDUInfo":{ "version":"1", "errorstatus":"0", "community":"public", "transactionid":"1", "errorindex":"0", "messageid":"0", "requestid":"414568963", "notificationtype":"TRAP" }, "oids":{ "iso.3.6.1.2.1.1.3.0":"67", "iso.3.6.1.6.3.1.1.4.1.0":"6", "iso.3.6.1.4.1.8072.2.3.2.1":"2" } } } From SNMPtrapd To Tornado
  • 60. @ 61 TORNADO ARCHITECTURE … more than software© Würth Phoenix DatasourceS Tornado Collectors Pipelines ExecutorsIcinga2 API Icinga2 API collector SNMP Traps Email - procmail rsyslog WebHooks … snmptrapd collector Email collector subscribe stream call rsyslog collector stream WebHooks collector API call … collector Matching Extracting Dispatching Icinga Icinga2 API Call Archive Save to file system Script Execute a script Logger Log to file system …
  • 61. @ 62 TORNADO ARCHITECTURE … more than software© Würth Phoenix DatasourceS Tornado Collectors Pipelines ExecutorsIcinga2 API Icinga2 API collector SNMP Traps Email - procmail rsyslog WebHooks … snmptrapd collector Email collector subscribe stream call rsyslog collector stream WebHooks collector API call … collector Matching Extracting Dispatching Icinga Icinga2 API Call Archive Save to file system Script Execute a script Logger Log to file system …
  • 62. @  Pipelines, Filters, Rules 63 TORNADO architecture … more than software© Würth Phoenix Email Filter Snmptrapd filter Rule 1 Rule 2 Rule n Host X Host Y Rule 1 Rule 2 Rule 1 Rule 2 Rsyslog filter Rule 1 Rule 2 Rule n Hostgroup Linux1 filter Rule 1 Rule 2 Rule n Pass all Filter Event “type”: “email” Filter Matched Filter Matched
  • 63. @  Pipelines, Filters, Rules 64 TORNADO architecture … more than software© Würth Phoenix Email Filter Rule 1 Rule 2 Rule n Pass all Filter Event “type”: “email” Filter Matched Filter Matched { "description": "This filter allows every event", "active": true } { "description": "This filter allows events of type 'email'", "active": true, "filter": { "type": "equal", "first": "${event.type}", "second": "email" } }
  • 64. @  Pipelines, Filters, Rules 65 TORNADO architecture … more than software© Würth Phoenix Email Filter Rule 1 Rule 2 Rule n Pass all Filter Event “type”: “email” Action: Icinga Service CRITICAL Action: Execute Script Action: Icinga Service OK Action … Filter Matched Filter Matched Rule Matched Action Dispatched
  • 65. @  Pipelines, Filters, Rules 66 TORNADO architecture … more than software© Würth Phoenix Email Filter Rule 2 Rule Matched { "description": "Normal temperature of a server.", "constraint": { "WHERE": { "type": "AND", "operators": [ { "type": "equal", "first": "${event.type}", "second": “email" }, { "type": "lt", "first": "${event.payload.temperature}", "second": "50" } ] } }, "actions": [ { "id": "icinga2", "payload": { "icinga2_action_name": "process-check-result", "icinga2_action_payload": { "exit_status": "0", "plugin_output": "OK - The temperature is ${event.payload.temperature}", "filter": "host.name=="${event.body.server_name}"", "type": “Host" } } } ] }
  • 66. @ 67 Usecase: email event collection … more than software© Würth Phoenix Demo time
  • 67. @ 68 Usecase: email event collection … more than software© Würth Phoenix DatasourceS Tornado Collectors Pipelines ExecutorsIcinga2 API Icinga2 API collector SNMP Traps Email - procmail rsyslog WebHooks snmptrapd collector Email collector subscribe stream call rsyslog collector stream WebHooks collector API call Matching Extracting Dispatching Icinga Icinga2 API Call Archive Save to file system Script Execute a script Logger Log to file system …
  • 68. @ 71 JOIN US … more than software© Würth Phoenix The SIMPLE Complex Event Processing Engine https://github.com/WuerthPhoenix/tornado mr.francesco.cina@gmail.com https://github.com/ufoscout patrick.zambelli@wuerth-phoenix.com https://github.com/zampat