Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Complex Event Processing: What?, Why?, How?

5.596 Aufrufe

Veröffentlicht am

July 2008, Introduction session to CEP from various angles, and short case studies around Esper

Veröffentlicht in: Technologie, Wirtschaft & Finanzen
  • Als Erste(r) kommentieren

Complex Event Processing: What?, Why?, How?

  1. 1. Complex Event Processing Quoi, Pourquoi, Comment? Fabien Coppens Alexandre Vasseur ©OCTO Technology – Université d’été du Système d’information
  2. 2. Objectives <ul><li>Understand what Complex Event Processing (CEP) is </li></ul><ul><li>Where does CEP fit in SOA/EDA/BAM/BI/RDBMS </li></ul><ul><li>Learn about CEP use cases </li></ul><ul><li>Discover CEP adoption paths </li></ul>
  3. 3. Speaker Qualifications <ul><li>Fabien Coppens </li></ul><ul><li>Senior Architect </li></ul><ul><li>CEP user </li></ul><ul><li>Alexandre Vasseur </li></ul><ul><li>Middleware technology evangelist </li></ul><ul><li>Esper co-lead & early CEP fan </li></ul>
  4. 4. The « Right-Time » Enterprise <ul><li>Business is data-driven </li></ul><ul><ul><li>More data, more sources </li></ul></ul><ul><ul><li>More complex situational scenarios </li></ul></ul><ul><li>Right-time is </li></ul><ul><ul><li>A business imperative </li></ul></ul><ul><ul><li>A technology challenge </li></ul></ul><ul><ul><ul><li>Strong coupling! </li></ul></ul></ul><ul><ul><ul><li>Request/Response! </li></ul></ul></ul><ul><ul><ul><li>Data stored and later analyzed </li></ul></ul></ul><ul><ul><ul><li>Moving requirements </li></ul></ul></ul>Situation Action/Decision Events Time Push Loosely coupled Time & Event-Driven
  5. 5. <ul><li>« By 2011, a new generation of application platforms stemming from the convergence of extreme transaction processing (XTP) technologies will supersede Java EE and .NET as the platform of choice for large-scale, business-critical operational applications .“ Gartner, July 2007 </li></ul>SOA + EDA + XTP + Edge + BAM/BI Convergence: Event Processing Financial Services Fraud detection RFID BAM & monitoring Sensors Location/presence based services Security Large scale SOAs Right-Time + Events
  6. 6. Event Processing Simple, Streaming, Complex / Business EP <ul><li>Simple </li></ul><ul><ul><li>Route, Filter, Enrich </li></ul></ul><ul><li>Streaming </li></ul><ul><ul><li>min,max,avg </li></ul></ul><ul><ul><li>Sliding data windows </li></ul></ul><ul><li>Complex / Business </li></ul><ul><ul><li>Many streams </li></ul></ul><ul><ul><li>Causality A followed by B within t minutes </li></ul></ul><ul><ul><li>Different rates </li></ul></ul><ul><ul><li>Historical data and real time data </li></ul></ul><ul><li>Consumer/Producer, JMS, ESBs, EIP </li></ul><ul><ul><li>Stateless & Timeless </li></ul></ul><ul><li>How? </li></ul><ul><ul><li>« Event Stream Processing » </li></ul></ul><ul><ul><li>Statefu-less / Timefu-less </li></ul></ul><ul><li>How?? </li></ul><ul><ul><li>Continuous joins and indexing </li></ul></ul><ul><ul><li>Time as first class citizen </li></ul></ul><ul><ul><ul><li>Temporal requests </li></ul></ul></ul><ul><ul><li>Statefull / Timefull </li></ul></ul>When [CEP] then […]
  7. 7. CEP’s Friends <ul><li>Edge </li></ul><ul><ul><li>Produce zillions of events </li></ul></ul><ul><li>SOA </li></ul><ul><ul><li>Interconnect, mediate, leverage and reuse </li></ul></ul><ul><li>XTP </li></ul><ul><ul><li>Scale throughput and data volumes as required </li></ul></ul><ul><ul><li>Right-time down to sub-millisecond where needed </li></ul></ul><ul><li>BAM & BI (or back to SOA) </li></ul><ul><ul><li>Operational applications </li></ul></ul>CEP Situational & Actionable Awareness <ul><li>Rules engine </li></ul>When [CEP] then […] If … then […]
  8. 8. CEP ~ Database 2.0 ? <ul><li>RDBMS </li></ul><ul><ul><li>Store data (a lot) </li></ul></ul><ul><ul><li>Handle queries </li></ul></ul><ul><ul><li>SQL (ie abstracted) </li></ul></ul><ul><li>Request/Response </li></ul><ul><li>Concept of Time </li></ul><ul><li>Right time </li></ul><ul><li>CEP </li></ul><ul><ul><li>Store rules </li></ul></ul><ul><ul><li>Handle data </li></ul></ul><ul><ul><li>EPL: Event Processing Language </li></ul></ul><ul><li>Subscribe/Notify </li></ul><ul><li>Time & causality </li></ul><ul><li>« Continuous query » </li></ul>SQL Queries CEP Server EPL Queries CEP Database
  9. 9. Event Processing Language & CEP Server <ul><li>EPL: For right-time events, what SQL is to historical data </li></ul><ul><ul><li>Best practices, rapid prototyping, tooling, less custom code </li></ul></ul><ul><li>CEP server: Middleware to deal with recurrent complexity </li></ul><ul><ul><li>indexing, state management, scalability, management, connectivity etc </li></ul></ul>? CEP Application Source + CEP as EPL + Sink Is EPL expressive enough (build vs buy) Is EPL a standard (vs SQL) Is it a specialized RDBMS or a true engine? Does it require a RDBMS? Manageable? Scalable Interoperate? Time & Causality Filter, Aggregate Sliding windows Continuous execution State management, threading Transports In & Out Interrop. Dashboards High Availability Runtime Management Record / Replay / Test
  10. 10. #1: Algorithmic trading <ul><li>Volume Weighted Average </li></ul><ul><li>select ticker, avg (price), count (*), sum (price) </li></ul><ul><li>from MarketData(ticker=‘GOOG'). win:length (1000) </li></ul><ul><li>Example CEP server performance figures </li></ul><ul><ul><li>1 CEP server, Java based, TCP in/out </li></ul></ul><ul><ul><li>1000 CEP statements </li></ul></ul><ul><ul><li>500 000 evt/s [100 Mbit/s] </li></ul></ul><ul><ul><li>Latency: avg 2.5 µ s, 99% < 10 µ s </li></ul></ul><ul><ul><li>4 core 2GHz, 65% load avg </li></ul></ul>Simple Event Processing Stream Processing (Timefull)
  11. 11. #1: Build vs Buy? MarketData(GOOG, 50) MarketData(MSFT, 25) MarketData(GOOG, 30) MarketData(GOOG, 40) Avg price / last 2 ticks, per ticker GOOG:40 GOOG:35 Receive market data Output / trigger actions <ul><li>Functional specification? </li></ul><ul><li>Abstraction for user defined rules? </li></ul><ul><li>Performance & monitoring etc? </li></ul>list = new MyList(GOOG,2) /*2 = size*/ // on event list.add(tick) // some tick discarded // as list size was // harcoded with size 2 //TODO: do not hardcode // recompute avg //TODO: optimize this send computeAvg(list)
  12. 12. #2: Fraud and compliance, risk management <ul><li>Context </li></ul><ul><ul><li>FX trading </li></ul></ul><ul><ul><li>Event { Trade direction, amount, user id, account, currencies traded } </li></ul></ul><ul><ul><li>200 000 evt/day, 50 rules, ~5% of triggering </li></ul></ul><ul><li>Two (or more) buy trades followed by (  ) a sale within the next hour between 2 parties </li></ul><ul><li>select e1, e2, final from pattern [ </li></ul><ul><li>every e1=FXTrade(userId in (100,101)) </li></ul><ul><li> (e2=FXTrade(userId in (100,101), dir = e1.dir) </li></ul><ul><li>where timer:within (8 hours ) </li></ul><ul><li>)  final=FXTrade(userId in (100,101), dir != e1.dir) </li></ul><ul><li>where timer:within (1 hour ) ] </li></ul>
  13. 13. #3: Location based services <ul><li>Context: </li></ul><ul><li>Mobile asset management (Command & Control systems) </li></ul><ul><li>Tracking delay between real time position and planned position </li></ul><ul><ul><li>20% time left to reach destination, 10% delta from planned position </li></ul></ul><ul><li>Continuous join with reference (passive) data </li></ul><ul><li>select id, lat, lon, time-ref.time, </li></ul><ul><li>Utils.distance (lat,lon,ref.lat,ref.lon) </li></ul><ul><li>as distance </li></ul><ul><li>from GPSPosition, </li></ul><ul><li>sql:RefDB [ </li></ul><ul><li>select lat, lon, time from Plan </li></ul><ul><li>where id={id} and stage=‘JFK’ </li></ul><ul><li>] as ref </li></ul><ul><li>where time<0.8*ref.time and </li></ul><ul><li>Utils.distance(lat,lon,ref.lat,ref.lon)>1.1*SPEED*(0.8*ref.time-time) </li></ul>
  14. 14. #4: Trading as SaaS : Strateer <ul><li>Algo trading as SaaS </li></ul><ul><li>CEP is a core enabler </li></ul><ul><li>Users can configure their own trading strategies </li></ul><ul><li>~20 CEP rules templates provided </li></ul><ul><ul><li>Each configurable by end users </li></ul></ul><ul><li>Power users can create their own rules </li></ul><ul><li>Esper CEP + JMS + MarketData connectors + RIA </li></ul>
  15. 15. #4: Strateer use case Strateer is algorithmic trading for individual investors 1. Drag & Drop your own automated strategy 2. Test it the way fund managers do 3. Then run it on Strateer: 1000s of data sources, real-time engine. 4. Get trading alerts to you anytime, anywhere. 5. Share and compare results with community
  16. 16. #4: Strateer architecture
  17. 17. Solutions (excerpt) Single use case platform (trading, compliance, BI ..) xxx Event Processing Mandates a DB? Latency / Throughput Of the shelf? Proven / Innovating Maturity Cost (Buy + Learn + Operate) Event Processing = Middleware x = Simple / Stream / Complex / Rules In-memory vs hidden DB vs DB 1 to 1000 if not more range Interoperate? Open platform? … … … ?
  18. 18. Adoption: Risks & Benefits ? “ CEP is mature? CEP is really not ESP? CEP is really event-driven SOA? CEP is really real-time BI? CEP is really low latency, high throughput, white-box COTs algo trading? CEP is really not a type of BPM? CEP is not really for detecting complex events? Complex does not really mean complex?” [as seen on blogs] Use case driven Build vs Buy SOA/BI/BAM related Orthogonal processing (discover & monitor) vs path critical processing (act upon)
  19. 19. Links <ul><li>http://esper.codehaus.org </li></ul><ul><li>http://www.espertech.com </li></ul><ul><li>http://complexevents.com </li></ul><ul><li>And many other googling around or asking your favorite vendors (!) </li></ul>