Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
2013 © Trivadis
BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENN...
2013 © Trivadis
Guido Schmutz
•  Working for Trivadis for more than 17 years
•  Oracle ACE Director for Fusion Middleware ...
2013 © Trivadis
AGENDA
1.  Introduction
2.  Twitter Use Case
3.  Processing with Oracle Event Processing (CEP)
4.  Visuali...
2013 © Trivadis
Big Data Definition (4 Vs)
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
4
+ Tim...
2013 © Trivadis
The world is changing …
The model of Generating/Consuming Data has changed ….
Old Model: few companies are...
2013 © Trivadis
Who is generating Big Data?
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
6
The ...
2013 © Trivadis
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
7
2013 © Trivadis
Internet Of Things – Sensors
are/will be everywhere
There are more devices tapping into the
internet than ...
2013 © Trivadis
Data as an Asset - Store Anything?
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)...
2013 © Trivadis
AGENDA
1.  Introduction
2.  Twitter Use Case
3.  Processing with Oracle Event Processing (CEP)
4.  Visuali...
2013 © Trivadis
Retrieve Tweets and Visualize
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
11
2013 © Trivadis
Access to Tweets
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
12
Quelle
Source ...
2013 © Trivadis
How to design a stream (event) processing system?
19.02.2014
Processing Twitter Stream with Oracle Event P...
2013 © Trivadis
AGENDA
1.  Introduction
2.  Twitter Use Case
3.  Processing with Oracle Event Processing (CEP)
4.  Visuali...
2013 © Trivadis
Oracle Event Processing (OEP) - Engine
Lightweight Java Application Server
•  Full environment for running...
2013 © Trivadis
Oracle Event Processing – Event Processing Network
Concept
19.02.2014
Processing Twitter Stream with Oracl...
2013 © Trivadis
Oracle Event Processing – In Memory, Continuous
Queries
Event Processing Output
§  Filtering
-  New strea...
2013 © Trivadis
Oracle Event Processing - CQL
Initiative for a complete “continuous” query language
Start with SQL ’99 plu...
2013 © Trivadis
Oracle Event Processing – Visual Development Tools
19.02.2014
Processing Twitter Stream with Oracle Event ...
2013 © Trivadis
Oracle Event Processing – Operation & Management
19.02.2014
Processing Twitter Stream with Oracle Event Pr...
2013 © Trivadis
Implementation – complete picture
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
...
2013 © Trivadis
1) Creating a Twitter Adapter
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
22
T...
2013 © Trivadis
2) Send Tweets to BAM
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
23
Twitter
A...
2013 © Trivadis
3) Extract interesting information from Tweet
19.02.2014
Processing Twitter Stream with Oracle Event Proce...
2013 © Trivadis
4) Count occurrences within period
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)...
2013 © Trivadis
5) Adding Cassandra NoSQL for storing results
19.02.2014
Processing Twitter Stream with Oracle Event Proce...
2013 © Trivadis
Implementing in Oracle Event Processing
19.02.2014
Processing Twitter Stream with Oracle Event Processing ...
2013 © Trivadis
1) Creating Twitter Adapter –
Connecting to Twitter Stream
19.02.2014
Processing Twitter Stream with Oracl...
2013 © Trivadis
1) Creating Twitter Adapter –
Tweet Event
19.02.2014
Processing Twitter Stream with Oracle Event Processin...
2013 © Trivadis
1) Creating Twitter Adapter –
Adapter Factory
19.02.2014
Processing Twitter Stream with Oracle Event Proce...
2013 © Trivadis
1) Creating Twitter Adapter –
Assembly
19.02.2014
Processing Twitter Stream with Oracle Event Processing (...
2013 © Trivadis
1) Creating Twitter Adapter –
Export Adapter to server
19.02.2014
Processing Twitter Stream with Oracle Ev...
2013 © Trivadis
1) Creating Twitter Adapter –
Using Twitter Adapter
19.02.2014
Processing Twitter Stream with Oracle Event...
2013 © Trivadis
2) Sending Tweets to BAM
Using Oracle BAM Enterprise Message Sources (JMS) interface
19.02.2014
Processing...
2013 © Trivadis
2) Sending Tweets to BAM –
Convert event to JMS MapMessage
19.02.2014
Processing Twitter Stream with Oracl...
2013 © Trivadis
3) Extract information from Tweet –
Extract Hashtags from TweetEvent
19.02.2014
Processing Twitter Stream ...
2013 © Trivadis
3) Extract information from Tweet –
Extract Hashtags from TweetEvent
19.02.2014
Processing Twitter Stream ...
2013 © Trivadis
4) Count occurrences within
period - Using CQL
19.02.2014
Processing Twitter Stream with Oracle Event Proc...
2013 © Trivadis
Implementation – Complete Picture
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
...
2013 © Trivadis
AGENDA
1.  Introduction
2.  Twitter Use Case
3.  Processing with Oracle Event Processing (CEP)
4.  Visuali...
2013 © Trivadis
Oracle BAM: Architected for Integration and
Visualization
Processing Twitter Stream with Oracle Event Proc...
2013 © Trivadis
Oracle BAM – Create a
Data Object
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
...
2013 © Trivadis
Oracle BAM Enterprise Message
Source Configuration
19.02.2014
Processing Twitter Stream with Oracle Event ...
2013 © Trivadis
AGENDA
1.  Introduction
2.  Twitter Use Case
3.  Processing with Oracle Event Processing (CEP)
4.  Visuali...
2013 © Trivadis
Implementation – Storing information in NoSQL
database
19.02.2014
Processing Twitter Stream with Oracle Ev...
2013 © Trivadis
Event Processing Network in OEP
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
46
2013 © Trivadis
The world is changing …
new data stores
Problem of traditional (R)DBMS approach:
§  Complex object graph
...
2013 © Trivadis
Apache Cassandra – NoSQL database
•  Developed at Facebook
•  Open source distributed database management ...
2013 © Trivadis
Apache Cassandra - NoSQL Database
• Don’t think of relational table => more of a sorted map
• Know your ap...
2013 © Trivadis
Apache Cassandra – NoSQL database
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
...
2013 © Trivadis
Apache Cassandra – NoSQL database
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
...
2013 © Trivadis
AGENDA
1.  Introduction
2.  Twitter Use Case
3.  Processing with Oracle Event Processing (CEP)
4.  Visuali...
2013 © Trivadis
Big Data Reference Architecture – Combine Streaming
and Batch
19.02.2014
Processing Twitter Stream with Or...
2013 © Trivadis
Summary
Cassandra
•  No single point of failure
•  Forget your data modeling skills
•  Model around the qu...
2013 © Trivadis
Questions and answers ...
2013 © Trivadis
BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FRE...
Nächste SlideShare
Wird geladen in …5
×

Processing Twitter Stream with Oracle Event Processing (OEP)

967 Aufrufe

Veröffentlicht am

Processing Twitter Stream with Oracle Event Processing (OEP)
Präsentation von Guido Schmutz, Trivadis AG

Veröffentlicht in: Technologie
  • My struggles with my dissertation were long gone since the day I contacted Emily for my dissertation help. Great assistance by guys from ⇒⇒⇒WRITE-MY-PAPER.net ⇐⇐⇐
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier

Processing Twitter Stream with Oracle Event Processing (OEP)

  1. 1. 2013 © Trivadis BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA
 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) Guido Schmutz OFM Partner Forum Malta 19.2.2014 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 1
  2. 2. 2013 © Trivadis Guido Schmutz •  Working for Trivadis for more than 17 years •  Oracle ACE Director for Fusion Middleware and SOA •  Co-Author of different books •  Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data •  Member of Trivadis Architecture Board •  Technology Manager @ Trivadis •  More than 25 years of software development experience •  Contact: guido.schmutz@trivadis.com •  Blog: http://guidoschmutz.wordpress.com •  Twitter: gschmutz 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 2
  3. 3. 2013 © Trivadis AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (CEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 3
  4. 4. 2013 © Trivadis Big Data Definition (4 Vs) 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 4 + Time to action ? – Big Data + Event Processing = Fast Data Characteristics of Big Data: Its Volume, Velocity and Variety in combination
  5. 5. 2013 © Trivadis The world is changing … The model of Generating/Consuming Data has changed …. Old Model: few companies are generating data, all others are consuming data New Model: all of use are generating data, and all of us are consuming data 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 5
  6. 6. 2013 © Trivadis Who is generating Big Data? 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 6 The progress and innovation is no longer hindered by the ability to collect data But by the ability to manage, analyze, summarize, visualize and discover knowledge from the collected data in a timely manner and in a scalable fashion Social media and networks (all of us are generating data) Scientific instruments (collecting all sorts of data) Mobile devices (tracking all objects all the time) Sensor technology and networks (measuring all kinds of data)
  7. 7. 2013 © Trivadis 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 7
  8. 8. 2013 © Trivadis Internet Of Things – Sensors are/will be everywhere There are more devices tapping into the internet than people on earth How do we prepare our systems/architecture for the future? 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 8 Source: CiscoSource: The Economist
  9. 9. 2013 © Trivadis Data as an Asset - Store Anything? 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 9 But then data is
 just too valuable
 to delete!
 We must 
 store anything! Nonsense! Just 
 store the data 
 you know 
 you need today! It depends … but Big Data technologies allow to store the raw information from both new data sources as well as existing ones so that you can later use it to create new data-driven products, you would not have thought about today!
  10. 10. 2013 © Trivadis AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (CEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 10
  11. 11. 2013 © Trivadis Retrieve Tweets and Visualize 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 11
  12. 12. 2013 © Trivadis Access to Tweets 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 12 Quelle Source Limitations Cost Twitter’s Search API 3200 / user 5000 / keyword 180 requests / 15 minutes free Twitter’s Streaming API 1%-40% of total volume free DataSift none 0.15 -0.20$ / unit Gnip none On request
  13. 13. 2013 © Trivadis How to design a stream (event) processing system? 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 13 Twitter Stream tweet Sensor tweet Persist (Queue) Twitter Stream tweet Sensor tweet Processing tweet Processing result result Twitter Stream tweet Receiving/ Processing result
  14. 14. 2013 © Trivadis AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (CEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 14
  15. 15. 2013 © Trivadis Oracle Event Processing (OEP) - Engine Lightweight Java Application Server •  Full environment for running Java applications •  Module Framework - OSGi High Throughput •  Hundreds of thousands of events/second Event Processing Infrastructure Easy-to-use development environment •  Service Framework – Spring DM, POJO Enterprise Web 2.0 & Eclipse-based tooling Multiple-choice VM •  JRockit or WebLogic RealTime 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 15
  16. 16. 2013 © Trivadis Oracle Event Processing – Event Processing Network Concept 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 16
  17. 17. 2013 © Trivadis Oracle Event Processing – In Memory, Continuous Queries Event Processing Output §  Filtering -  New stream filtered for specific criteria, e.g. stock price > $22 §  Correlation & Aggregation -  Scrolling, time-based window metrics, e.g. average # of stock trades in the last hour §  Pattern Matching -  Notification of detected event patterns, e.g. price changes A, B and C occurred within 15 minute window 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 17
  18. 18. 2013 © Trivadis Oracle Event Processing - CQL Initiative for a complete “continuous” query language Start with SQL ’99 plus “continuous” query extensions §  Based on Stanford University research Industry standards discussions §  Event Processing Technical Society (EPTS) §  ANSI SQL §  OMG Adoption Today §  ANSI SQL Standards Proposal for CQL Pattern Matching -  Oracle, IBM, Stanford University §  OpenSource Adoption of CQL §  Oracle Complex Event Processor (CEP) Releaseà Available in 11g 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 18
  19. 19. 2013 © Trivadis Oracle Event Processing – Visual Development Tools 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 19
  20. 20. 2013 © Trivadis Oracle Event Processing – Operation & Management 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 20
  21. 21. 2013 © Trivadis Implementation – complete picture 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 21 Mention Extractor Twitter Adapter Counter
 Processor Hashtag
 Extractor Author Extractor Cassandra Counter BAM Tweet Cassandra Tweet BAM Counter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity robertvanmolken #ofmforum #malta #ofmforum,5 #malta,2 Robertvanmolken,1 @SOASimone,1 @SOACommunity,5 JMS JMS Twitter range 30 seconds
 slide 30 seconds
  22. 22. 2013 © Trivadis 1) Creating a Twitter Adapter 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 22 Twitter Adapter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta Twitter
  23. 23. 2013 © Trivadis 2) Send Tweets to BAM 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 23 Twitter Adapter BAM Tweet @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta JMS Twitter
  24. 24. 2013 © Trivadis 3) Extract interesting information from Tweet 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 24 Mention Extractor Twitter Adapter Hashtag
 Extractor Author Extractor BAM Tweet @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity robertvanmolken #ofmforum #malta JMS Twitter
  25. 25. 2013 © Trivadis 4) Count occurrences within period 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 25 Mention Extractor Twitter Adapter Counter
 Processor Hashtag
 Extractor Author Extractor BAM Tweet BAM Counter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta #ofmforum,5 #malta,2 Robertvanmolken,1 @SOASimone,1 @SOACommunity,5 JMS JMS Twitter range 30 seconds
 slide 30 seconds @SOASimone @SOACommunity robertvanmolken #ofmforum #malta
  26. 26. 2013 © Trivadis 5) Adding Cassandra NoSQL for storing results 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 26 Mention Extractor Twitter Adapter Counter
 Processor Hashtag
 Extractor Author Extractor Cassandra Counter BAM Tweet Cassandra Tweet BAM Counter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta #ofmforum,5 #malta,2 Robertvanmolken,1 @SOASimone,1 @SOACommunity,5 JMS JMS Twitter range 30 seconds
 slide 30 seconds @SOASimone @SOACommunity robertvanmolken #ofmforum #malta
  27. 27. 2013 © Trivadis Implementing in Oracle Event Processing 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 27 Mention Extractor Twitter Adapter Counter
 Processor Hashtag
 Extractor Author Extractor BAM Tweet BAM Counter JMS JMS Twitter range 30 seconds
 slide 30 seconds
  28. 28. 2013 © Trivadis 1) Creating Twitter Adapter – Connecting to Twitter Stream 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 28
  29. 29. 2013 © Trivadis 1) Creating Twitter Adapter – Tweet Event 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 29
  30. 30. 2013 © Trivadis 1) Creating Twitter Adapter – Adapter Factory 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 30
  31. 31. 2013 © Trivadis 1) Creating Twitter Adapter – Assembly 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 31
  32. 32. 2013 © Trivadis 1) Creating Twitter Adapter – Export Adapter to server 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 32
  33. 33. 2013 © Trivadis 1) Creating Twitter Adapter – Using Twitter Adapter 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 33
  34. 34. 2013 © Trivadis 2) Sending Tweets to BAM Using Oracle BAM Enterprise Message Sources (JMS) interface 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 34
  35. 35. 2013 © Trivadis 2) Sending Tweets to BAM – Convert event to JMS MapMessage 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 35
  36. 36. 2013 © Trivadis 3) Extract information from Tweet – Extract Hashtags from TweetEvent 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 36
  37. 37. 2013 © Trivadis 3) Extract information from Tweet – Extract Hashtags from TweetEvent 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 37
  38. 38. 2013 © Trivadis 4) Count occurrences within period - Using CQL 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 38
  39. 39. 2013 © Trivadis Implementation – Complete Picture 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 39
  40. 40. 2013 © Trivadis AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (CEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 40
  41. 41. 2013 © Trivadis Oracle BAM: Architected for Integration and Visualization Processing Twitter Stream with Oracle Event Processing (OEP) Internet BAM Dashboards WebApplications StartPage ActiveViewer ActiveStudio Architect Administrator ReportServer iCommand Oracle Database (Grid) BAM Data & Metadata External Data Objects WebServices Internet Enterprise Integration Framework Application Server BI Web Services JMS Connector BAM Adapter ADF BAM DataControl ADF Pages with DVT BAM ServerEventEngine Actions & Escalations Notification Services ReportCache Snapshots & Change Lists Memory / Disk ActiveDataCache ViewSets API Kernel DataSets DataStorageEngine ODI Databases OLTP & Data Warehouses Mobile Devices Data & Metadata Import & Export BPEL BPM Message Queues CEP OESB 19.02.2014 41
  42. 42. 2013 © Trivadis Oracle BAM – Create a Data Object 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 42
  43. 43. 2013 © Trivadis Oracle BAM Enterprise Message Source Configuration 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 43
  44. 44. 2013 © Trivadis AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (CEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 44
  45. 45. 2013 © Trivadis Implementation – Storing information in NoSQL database 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 45 Mention Extractor Twitter Adapter Counter
 Processor Hashtag
 Extractor Author Extractor Cassandra Counter BAM Tweet Cassandra Tweet BAM Counter JMS JMS Twitter range 30 seconds
 slide 30 seconds
  46. 46. 2013 © Trivadis Event Processing Network in OEP 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 46
  47. 47. 2013 © Trivadis The world is changing … new data stores Problem of traditional (R)DBMS approach: §  Complex object graph §  Schema evolution §  Semi-structured data §  Scaling Polyglot persistence §  Using multiple data storage technologies (RDMBS + NoSQL + NewSQL + In- Memory) §  Selected based on the way data is being used by individual applications •  Why using an RDBMS if there are better storage alternatives? •  Key/Value, Column Family, Document, Graph-oriented, Relational, … §  Can occur both over the enterprise as well as within a single application 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 47 ORDER ADDRESS CUSTOMER ORDER_LINES Order ID: 1001 Order Date: 15.9.2012 Line Items Customer First Name: Peter Last Name: Sample Billing Address Street: Somestreet 10 City: Somewhere Postal Code: 55901 Name Ipod Touch Monster Beat Apple Mouse Quantity 1 2 1 Price 220.95 190.00 69.90
  48. 48. 2013 © Trivadis Apache Cassandra – NoSQL database •  Developed at Facebook •  Open source distributed database management system •  Professional grade support from company called DataStax •  Main Features §  Real-Time §  Highly Distributed §  Support for Multiple Data Center §  Highly Scalable §  No Single Point of Failure §  Fault Tolerant §  Tunable Consistency §  Cassandra Query Language (CQL) 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 48
  49. 49. 2013 © Trivadis Apache Cassandra - NoSQL Database • Don’t think of relational table => more of a sorted map • Know your application => model around the queries • De-normalize and duplicate for read performance • Index is not an afterthought, anymore=> index upfront • Think of physical storage structure 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 49 2013-08 Day-1, keyword-1=>100 Day-2, keyword-1=>150 Day-3, keyword-1=>170 …. Day-31, keyword-1 =>170 2013-08-31 Hour-1, keyword-1 =>10 Hour-2, keyword-1 =>15 Hour-3, keyword-1 =>17 …. Hour-24, keyword-1 =>17 2013-08-31-10 Minute-1, keyword-1=>2 Minute-2, keyword-1=>3 Minute-3, keyword-1 =>5 …. Minute-60, keyword-1=>2 Row-key Columns à
  50. 50. 2013 © Trivadis Apache Cassandra – NoSQL database 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 50
  51. 51. 2013 © Trivadis Apache Cassandra – NoSQL database 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 51
  52. 52. 2013 © Trivadis AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (CEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 52
  53. 53. 2013 © Trivadis Big Data Reference Architecture – Combine Streaming and Batch 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 53
  54. 54. 2013 © Trivadis Summary Cassandra •  No single point of failure •  Forget your data modeling skills •  Model around the queries •  Query Language Oracle Event Processing •  Very light weight server •  Very easy to write adapters •  Very strong CQL language Oracle Business Activity Monitoring §  11g version a bit “old fashioned” §  Easy to integrate through JMS 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 54
  55. 55. 2013 © Trivadis Questions and answers ... 2013 © Trivadis BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA
 Guido Schmutz Technology Manager guido.schmutz@trivadis.com 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 55

×