This document describes Siddhi, an open source complex event processing (CEP) engine. It discusses the objectives of developing Siddhi as a high performing CEP engine. It outlines Siddhi's architecture including its input/output adapters, compiler, Siddhi manager, and processors. It provides details on Siddhi's implementation and capabilities such as filtering, pattern matching, joins, windows, and aggregators. It also evaluates Siddhi's performance compared to other CEP engines and discusses its deployments, website, documentation, and future plans.
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
High Performance Complex Event Processing Engine
1. A HIGH PERFORMANCE COMPLEX EVENT
PROCESSING ENGINE
http://siddhi.sourceforge.net/
2. Siddhi and NBQSA
• Bronze award under Tertiary Category at the
National Best Quality Software Awards 2011
• Held on 7th October 2011 in Sri Lanka
3. INTRODUCTION
Project Group Members
• Suhothayan S. (Suho)
• Loku Narangoda I.U. (Isuru)
• Chaturanga H.W.G.S (Subash)
• Gajasinghe K.C.B. (Kasun)
Project Supervisors
• Dr. Srinath Perera
• Ms.Vishaka Nanayakkara
4. WHAT IS CEP ?
Complex Event Processing (CEP) is identifying meaningful
patterns, relationships and data abstractions among unrelated
events and fire an immediate response.
Database Applications Event-driven Applications
Query
Ad-hoc queries or requests Continuous standing queries
Paradigm
Latency Seconds, hours, days Milliseconds or less
Tens of thousands of
Data Rate Hundreds of events/sec
events/sec or more
Event
request output
input stream
response stream
5. SCENARIOS OF EVENT PROCESSING
Latency
Months
Days CEP Target Scenarios
Relational Database
hours Operational Analytics
Applications
Applications, e.g.,
Minutes Data Warehousing Logistics, etc.
Applications
Seconds
Web Analytics Applications
100 ms Monitoring Manufacturing Financial
Applications Applications Trading
< 1ms Applications
0 10 100 1000 10000 100000 ~1million
Aggregate Data Rate (Events/sec.)
11. PROJECT OBJECTIVES
• Carry out research to come up with effective algorithms
and approaches to implement A High Performing CEP
engine.
• Come up with an effective architecture for a CEP Engine
which can detect events in a computationally
efficient way.
12. FROM RESEARCH
For State Machine
SASE: High-Performance Complex Event Processing
over Streams
For Pipeline Architecture
Aurora: a new model and architecture for data stream
management
For Query Plan Management
Query Processing, Resource Management,
and Approximation in a Data Stream
Management
30. Our PROGRESS
• Initial research
• System Design
• 1st iteration
• Web site
• 2nd iteration
• All major functionalities are implemented
• Profiling and Performance Testing
• Improved Siddhi API
• Siddhi Benchmark
• 3rd iteration
• Adding new features for user requests
• Profiling and Performance Testing
38. ADDITIONAL FEATURES
• Improvements in the Selection Operator
- using MVEL
• Support conditions on the aggregation functions
-having, group-by
• Event quantifications in sequential patterns
-kleene closure (*)
• Unique function
40. PERFORMANCE TESTING
Details of the Machine
Hardware : Intel- Core 2 Duo 2.10 GHz
Memory 1.9GB
OS : Ubuntu
Release 9.10
Kernel Linux 2.6.31-14-generic
GNOME - 2.28.1
41. PERFORMANCE COMPARISON WITH ESPER
Performance Comparison for a simple filter without a
window
Events Siddhi Esper
(ms) (ms)
10 4 5
100 5.6 9.8
1000 34.5 55.2
10,000 252.6 491
100,000 961.4 1669.2
1000,000 3730.6 5546.8
42. PERFORMANCE COMPARISON WITH ESPER
Performance Comparison for a timed window query for
average calculation for a given symbol
Events Siddhi Esper
(ms) (ms)
10 5 9
100 10.6 16.2
1000 52 109.2
10,000 419.2 1159.2
100,000 1302.8 2724
1000,000 7883.2 11589.8
43. PERFORMANCE COMPARISON WITH ESPER
Performance Comparison for a State machine query
Events Siddhi Esper
(ms) (ms)
10 5.8 30
100 11 58.4
1000 98.6 333
10,000 550.4 2285.6
100,000 1606 9663.2
1000,000 12563 77641
53. LITERATURE REVIEW
• T. J. Owens, "Survey of event processing. Technical report, Air Force Research
Laboratory, Information Directorate ," 2007.
• G. Tóth, R. Rácz, J. Pánczél, T. Gergely, A. Beszédes, L. Farkas, L.J. Fülöp, "Survey
on Complex Event Processing and Predictive Analytics," , 2010.
• Alex Cheung, Guoli Li, Balasubramaneyam Maniymaran, Vinod Muthusamy, Reza
Sherafat Kazemzadeh Hans-Arno Jacobsen, "System, The PADRES
Publish/Subscribe,".
• Air Force Research Laboratory, "Survey of Event Processing ," , 2007.
• J. Nagy, "User-Centric Personalized Extensibility for Data-Driven Web Applications ,
IF/AFOSR Minigrant Proposal," , 2007.
• J. Gehrke, B. Panda, M. Riedewald, V. Sharma, and W. White A. Demers, "Cayuga: A
General Purpose Event Monitoring System," , Asolimar, California, January 2007.
• J. Gehrke, M. Hong, M. Riedewald, and W. White A. Demers, "A General Algebra
and Implementation for Monitoring Event Streams," 2005.
• K. Vikram, "FingerLakes: A Distributed Event Stream Monitoring System,".
• D Abadi, D Carney, U. Cetintemel, and M. et al. Cherniack, "Aurora: a data stream
management system," , San Diego, California, 2003.
54. LITERATURE REVIEW
• H. Balakrishnan, M. Balazinska, D. Carney, and U. et al. Cetintemel, "Retrospective on Aurora," vol. 13, no. 4, 2004.
• D. Abadi, Y. Ahmad, and M. Balazinska: U. Cetintemel et al. al., "The Design of the Borealis Stream Processing Engine,"
, 2005.
• S. Zdonik, M. Stonebraker, M. Cherniack, and U. Centintemel et al., "The Aurora and Medusa Projects," , 2003.
• S. Chandrasekaran, O. Cooper, A. Deshpande, and M. Franklin et al., "TelegraphCQ: Continuous Dataflow Processing for
an Uncertain World," , 2003.
• S. Chandrasekaran, O. Cooper, A. Deshpande, M. Franklin, J. Hellerstein, W. Hong, S. Madden, F. Reiss, M. Shah, S.
Krishnamurthy, "TelegraphCQ: An Architectural Status Report," IEEE Data Engineering Bulletin, vol. 26, no. 1, 2003.
• T. Sellis, "Multiple Query Optimization," , 1988.
• N., Sanghai, S., Roy, P., Sudarshan, S. Dalvi, "Pipelining in Multi-Query Optimization," , 2001.
• A., Sudarshan, S., Viswanathan, S. Gupta, "Query Scheduling in Multi Query Optimization," , 2001.
• P., Seshadri, A., Sudarshan, A., Bhobhe, S. Roy, "Efficient and Extensible Algorithms For Multi Query Optimization," ,
2000.
• S. Zdonik. (2006, March) Stream Processing Overview [presentation] Workshop on Event Processing, Hawthorne, New
York.
• M. Liu, L. Ding, E.A. Rundensteiner, and M. Mani M. Li, "Event Stream Processing with Out-of-Order Data Arrival," ,
2007.
• Y. Diao, D. Gyllstrom, and N. Immerman J. Agrawal, "Efficient pattern matching over event streams," , 2008.
• T. Shafeek, "Aurora: A New Model And Architecture For Data Stream Management, B.Tech Seminar Report,
Government Engineering College, Thrissur," 2010.