Understand what Complex Event Processing (CEP) is Where does CEP fit in SOA/EDA/BAM/BI/RDBMS Learn about CEP use cases Discover CEP adoption paths

1. Complex Event Processing
Quoi, Pourquoi, Comment?
Fabien Coppens
Alexandre Vasseur
©OCTO Technology – Université d’été du Système d’information

2. Objectives
• Understand what Complex Event Processing (CEP) is
• Where does CEP fit in SOA/EDA/BAM/BI/RDBMS
• Learn about CEP use cases
• Discover CEP adoption paths

3. Speaker Qualifications
• Fabien Coppens
• Senior Architect
• CEP user
• Alexandre Vasseur
• Middleware technology evangelist
• Esper co-lead & early CEP fan

4. The « Right-Time » Enterprise
Situation
• Business is data-driven
– More data, more sources Events
– More complex situational scenarios
• Right-time is Time
– A business imperative
– A technology challenge Push
• Strong coupling! Loosely coupled
• Request/Response!
Time &
• Data stored and later analyzed
Event-Driven
• Moving requirements
Action/Decision

5. SOA + EDA + XTP + Edge + BAM/BI
Convergence: Event Processing
« By 2011, a new generation of application platforms stemming from
the convergence of extreme transaction processing (XTP)
technologies will supersede Java EE and .NET as the platform of
choice for large-scale, business-critical operational applications.“
Gartner, July 2007
Large scale SOAs
RFID BAM & monitoring
Financial Services Sensors
Fraud detection Location/presence based services
Security
Right-Time + Events

6. Event Processing
Simple, Streaming, Complex / Business EP
• Simple • Consumer/Producer, JMS, ESBs, EIP
– Route, Filter, Enrich – Stateless & Timeless
• Streaming • How?
– min,max,avg – « Event Stream Processing »
– Sliding data windows – Statefu-less / Timefu-less
• Complex / Business • How??
– Many streams – Continuous joins and indexing
– Causality – Time as first class citizen
A followed by B • Temporal requests
within t minutes
– Statefull / Timefull
– Different rates
– Historical data and real time data
When [CEP] then […]

7. CEP’s Friends
• Edge
– Produce zillions of events
• SOA
– Interconnect, mediate, leverage and reuse
• XTP
– Scale throughput and data volumes as required
– Right-time down to sub-millisecond where needed
• Rules engine
When [CEP] then […] If … then […]
CEP
Situational & Actionable Awareness
• BAM & BI (or back to SOA)
– Operational applications

8. CEP ~ Database 2.0 ?
• RDBMS • CEP
– Store data (a lot) – Store rules
– Handle queries – Handle data
– SQL (ie abstracted) – EPL: Event Processing Language
• Request/Response • Subscribe/Notify
• Concept of Time • Time & causality
• Right time • « Continuous query »
CEP Server
Database
EPL Queries
SQL Queries
CEP

9. Event Processing Language & CEP Server
• EPL: For right-time events, what SQL is to historical data
– Best practices, rapid prototyping, tooling, less custom code
• CEP server: Middleware to deal with recurrent complexity
– indexing, state management, scalability, management, connectivity etc
Transports Is EPL expressive enough
Dashboards (build vs buy)
In & Out
Is EPL a standard
CEP Application
(vs SQL)
Source + CEP as EPL + Sink Is it a specialized RDBMS or a
true engine? Does it require a
Runtime Management
Runtime Management
Record / Replay / Test
Record / Replay / Test
Time & RDBMS?
Filter, Aggregate
Causality
Manageable?
Sliding windows Scalable
Continuous execution
Interoperate?
State management, threading
Interrop. High Availability

10. #1: Algorithmic trading
• Volume Weighted Average
select ticker, avg(price), count(*), sum(price)
from MarketData(ticker=‘GOOG').win:length(1000)
Simple Event Processing
Stream Processing (Timefull)
• Example CEP server performance figures
– 1 CEP server, Java based, TCP in/out
– 1000 CEP statements
– 500 000 evt/s [100 Mbit/s]
– Latency: avg 2.5µs, 99% < 10µs
– 4 core 2GHz, 65% load avg

11. #1: Build vs Buy?
Avg price / last 2 ticks, per ticker
MarketData(GOOG, 50) list = new MyList(GOOG,2)
/*2 = size*/
// on event
MarketData(MSFT, 25)
list.add(tick)
// some tick discarded
// as list size was
MarketData(GOOG, 30) // harcoded with size 2
//TODO: do not hardcode
GOOG:40
// recompute avg
MarketData(GOOG, 40) //TODO: optimize this
send computeAvg(list)
GOOG:35
Receive market data
• Functional specification?
• Abstraction for user defined rules? Output / trigger actions
• Performance & monitoring etc?

12. #2: Fraud and compliance, risk management
Context
– FX trading
– Event { Trade direction, amount, user id, account, currencies traded }
– 200 000 evt/day, 50 rules, ~5% of triggering
Two (or more) buy trades followed by ( ) a sale within the next hour
between 2 parties
select e1, e2, final from pattern [
every e1=FXTrade(userId in (100,101))
(e2=FXTrade(userId in (100,101), dir = e1.dir)
where timer:within(8 hours)
) final=FXTrade(userId in (100,101), dir != e1.dir)
where timer:within(1 hour) ]

13. #3: Location based services
Context:
• Mobile asset management (Command & Control systems)
• Tracking delay between real time position and planned position
– 20% time left to reach destination, 10% delta from planned position
• Continuous join with reference (passive) data
select id, lat, lon, time-ref.time,
Utils.distance(lat,lon,ref.lat,ref.lon)
as distance
from GPSPosition,
sql:RefDB [
select lat, lon, time from Plan
where id={id} and stage=‘JFK’
] as ref
where time<0.8*ref.time and
Utils.distance(lat,lon,ref.lat,ref.lon)>1.1*SPEED*(0.8*ref.time-time)

14. #4: Trading as SaaS : Strateer
• Algo trading as SaaS
• CEP is a core enabler
• Users can configure their own trading strategies
• ~20 CEP rules templates provided
– Each configurable by end users
• Power users can create their own rules
• Esper CEP + JMS + MarketData connectors + RIA

15. #4: Strateer use case
1.
Drag & Drop 2.
your own
automated Test it the way
strategy fund managers
3.
do
5.
Then run it 4. Share and
on Strateer: compare
Get trading
1000s of data results with
alerts to you
sources, real- community
anytime,
time engine.
anywhere.
Strateer is algorithmic trading for individual
investors

16. #4: Strateer architecture

17. Solutions (excerpt)
Single use case platform (trading, Event Processing = Middleware
compliance, BI ..)
xxx Event Processing x = Simple / Stream / Complex / Rules
?
Mandates a DB? In-memory vs hidden DB vs DB
Latency / Throughput 1 to 1000 if not more range
Of the shelf? Interoperate? Open platform?
Proven / Innovating …
Maturity …
Cost (Buy + Learn + Operate) …

18. Adoption: Risks & Benefits
“CEP is mature? CEP is really not ESP? CEP is
really event-driven SOA? CEP is really real-time
BI? CEP is really low latency, high throughput,
white-box COTs algo trading? CEP is really not a
type of BPM? CEP is not really for detecting
complex events? Complex does not really mean
complex?” [as seen on blogs]
Use case driven
?
Build vs Buy
SOA/BI/BAM related
Orthogonal processing
(discover & monitor)
vs
path critical processing
(act upon)

19. Links
• http://esper.codehaus.org
• http://www.espertech.com
• http://complexevents.com
• And many other googling around or asking your favorite
vendors (!)