1. Complex Event Processing
Across Web Applications
Mike Clymer - Founder, CoBaLU
Glue Conference 2009
http://www.clynetic.com

2. Agenda
Problem Space and Solutions
CEP and Web Application Use Cases
Complex Event Processing (CEP)
Overview
Approaching CEP Design for the Web
Final Thoughts
Q&A

3. Problems
Web as a platform is growing radically
Consumer Web is increasingly about
real-time information sharing
Information overload for the consumer
Too complex - data loses meaning
Efficiency vs effectiveness in data
consumption
Your time is valuable

4. Current Solutions
Feed subscription ser vices / tools
Too many data streams, lose focus
Constant monitoring
Aggregators
Narrows the focus some
Still a fire-hose of data
Too noisy

5. Current Solutions
Grouping, tagging, and filtering (for
previous solutions)
Still requires too much management and setup
Semantic applications *
Are becoming more available
Relatively new to the consumer conscience
Top-Down approach has similar goals to CEP
* http://www.readwriteweb.com/archives/10_semantic_apps_to_watch.php

6. Goals for Improvement
Require the user to monitor less data
Guide the user with intelligently filtered
data for context and relevance
Discovery and analysis
Create situational knowledge that is
actionable from many data streams
The Implicit Web
Paying attention to what is important - context

7. Agenda
Problem Space and Solutions
CEP and Web Application Use Cases
Complex Event Processing (CEP)
Overview
Approaching CEP Design for the Web
Final Thoughts
Q&A

8. CEP and Web Apps
The various tools, ser vices, and content
consumed create a seemingly unrelated
net work of data nodes
Interested in computing events related
to “real life”, or web usage
Could build event models from data
related to the social graph

9. Web App CEP Use Cases
Yelp recommendations for vendors and
services based on FB status, Amazon
purchases, and Google calendar
Media getting news leads faster
(Twitter) *
Tracking global health trends
http://www.google.org/flutrends/
Use location to warn of epidemic hotspots
* http://www.readwriteweb.com/archives/can_media_take_tips_from_t witter_techmemes_experie.php

10. Web App CEP Use Cases
Professional net work user lists
employer as one recently acquired
Notifies user to update contacts / endorsements
Grabs relevant job listings from additional
sources
Customer Development application
tracking message boards, blogs and
Twitter about customer problems, needs,
wants, likes, complaints, reviews

11. Web App CEP Use Cases
Conference management app
Connects attendees, speakers, and domain
experts based on event calendars, session
attendance, Twitter trends, and user profiles
Uses location to increase net working
opportunities from previous correlation
Helps conference organizers manage conference
hotspots based on location of attendees, Twitter
trends, service usage

12. Agenda
Problem Space and Solutions
CEP and Web Application Use Cases
Complex Event Processing (CEP)
Overview
Approaching CEP Design for the Web
Final Thoughts
Q&A

13. CEP / EP Visually
Detected &
Real-Time Situation Predicted
Events Refinement
Situations
Patterns from
Historical Data
* Adapted by Mike Clymer from (Tim Bass):
http://www.slideshare.net/TimBassCEP/getting-started-in-cep-how-to-build-an-event-processing-application-presentation-717795

14. CEP / EP Concepts
The main focus of CEP is to build a
net work technology that creates
knowledge that is both actionable and
situational
CEP is made for processing large
amounts of real-time data with
dynamic analysis and low-latency
Power comes from processing real-time
and historical data in an event cloud

15. CEP / EP Concepts
CEP uses raw events as input and
processes the event data to detect
actionable situations. CEP then triggers
the right action, at the right time
Typical CEP progression
event >> correlation >> assessment >> decision >>
action
* Adapted by Mike Clymer from (Tim Bass):
http://www.thecepblog.com/2007 /05/14/what-is-complex-event-processing-part-1/

16. CEP / EP Concepts
Many different definitions for CEP
CEP: Architecture vs Algorithm
Architectures for the event transport
Can be built as EDA, ESP
Needed to supply CEP with the input events
Algorithms for event processing and pattern
detection (situation recognition)
Define an Event Processing Language (patterns)

17. CEP / EP Concepts
Event is the atomic element for analysis
Event is some activity that happened
Events are context dependent
Event patterns related to timing and
causal relationships bet ween events

18. CEP / EP Concepts
Events are represented by abstractions
Raw data should be modeled as events
Events are organized into hierarchies
representing abstractions of real-life
hierarchies
Event hierarchies provide the ability to
apply computation and analysis to
related sets of events

19. CEP / EP Concepts
Goal of CEP w.r.t. ESP:
Identify the meaningful events within those
event streams
Employ techniques such as:
Detection of complex patterns of many events
Event correlation and abstraction
Building event hierarchies, and relationships
bet ween events such as causality,
membership, and timing
* Adapted by Mike Clymer from (Wikipedia):
http://en.wikipedia.org/wiki/Event_Stream_Processing

20. Common CEP / EP Uses
Algorithmic trading - Finance, Energy
Fraud detection - Web Commerce, Credit
Track and trace - Patients, Packages
Sensor net works - RFID, GPS
MMOGs - Player monitoring
Net work security - Intrusion detection
* Adapted by Mike Clymer from (Tim Bass):
http://www.slideshare.net/TimBassCEP/getting-started-in-cep-how-to-build-an-event-processing-application-presentation-717795

21. Agenda
Problem Space and Solutions
Complex Event Processing (CEP)
Overview
CEP and Web Application Use Cases
Approaching CEP Design for the Web
Final Thoughts
Q&A

22. Architectural Approach
Normalize with Build and Track Analyze Events Compare, Correlate,
Common Event Data Event State Context / Scenario Analyze Scenarios
Event Sources CEP Consumers
User Interface
Web
Data
Web
Level One Level Two Level Three
Collection
API
Event
Agent
Pre-Process Event Situation Predictive
Tracking Detection Analysis
Data
Collection Messaging Backbone GUI
Agent
CEP Engine
DB Management
Level Four
Data Profiles
Process Historical
Collection and Pub / Sub
Refinement Data
Agent Patterns
Take Action
from Detected
Situations
* Adapted by Mike Clymer from (Tim Bass):
http://www.slideshare.net/TimBassCEP/getting-started-in-cep-how-to-build-an-event-processing-application-presentation-717795

23. CEP Design Thoughts
Build a messaging backbone to direct
communication bet ween collecting
events, processing events, and
analyzing situations (ner vous system)
Message queues
Jabber like application ser vers for pub / sub
Define sets of event patterns, or a
pattern language (EPLs)
Possibly use DSL - Spec for specific problem

24. CEP Design Thoughts
Normalize incoming data streams
Define and adhere to standard formats
Identify common event data and event pattern
sets
Try to leverage data standards
Improves data transport, eases event
comparison
Utilize data snapshots for historical
and pattern analysis

25. CEP Design Thoughts
Make use of domain specific agentry
Agents to collect from heterogeneous data
sources (sensors)
Agents to handle basic event processing and
normalization (pre-processing)
Agents to apply rules and perform analysis
(Bayesian nets, neural nets, semantic rules, etc)
Possibly use agents to drive actions from
detected situations

26. CEP Design Thoughts
Design for low-latency, high throughput
Situation and context detection are often
associated with a time element
Data collection agentry will often produce high-
volumes of data based on the diversity of data
nodes
Messaging backbone should be scalable for
concurrency

27. Agenda
Problem Space and Common Solutions
Complex Event Processing (CEP)
Overview
Applying CEP to the Consumer Web
Approaching CEP Design for the Web
Final Thoughts
Q&A

28. Final Thoughts
Design CEP applications with passive
user experience in mind
Value is derived from helping user figure out what
is important
UX is applicable to thirdparty ser vice consumers
Standards, protocols, and reference
architectures can be “good enough”
Definitely needed for common info exchange
Practical application > Optimal theory

29. Final Thoughts
Design with goal adding true data
utility
Data pass-through is not enough
Context is the key to data enrichment
Tim O'Reilly Quote: quot;Create more value
than you capture.quot;
Different context, but relevant to our discussion

30. Agenda
Problem Space and Common Solutions
Complex Event Processing (CEP)
Overview
Applying CEP to the Consumer Web
Approaching CEP Design for the Web
Final Thoughts
Q&A

31. Further Exploration
http://complexevents.com/
http://www.thecepblog.com/2007 /05/14/what-
is-complex-event-processing-part-1/
http://www.slideshare.net/TimBassCEP/slideshows
http://en.wikipedia.org/wiki/
Complex_event_processing
http://apama.typepad.com/my_weblog/
http://epthinking.blogspot.com/

32. Further Exploration
http://www.amazon.com/Distributed-Event-
Based-Systems-Gero-M%C3%BChl/dp/
3540326510/ref=sr_1_3?
ie=UTF8&s=books&qid=1241925043&sr=8-3
http://www.amazon.com/XMPP-Definitive-Real-
Time-Applications-Technologies/dp/059652126X/
ref=sr_1_1?
ie=UTF8&s=books&qid=1241973202&sr=8-1
http://www.amazon.com/Power-Events-
Introduction-Processing-Distributed/dp/
0201727897 /ref=sr_1_1?
ie=UTF8&s=books&qid=1241973250&sr=8-1