Processing Twitter Stream with Oracle Event Processing (OEP)
•
2 gefällt mir•3,883 views
This session will present how to connect to the Twitter Streaming API and process and analyse the tweets in real-time by using Oracle Event Processing. We will see how typical analytics on a such social media data can be done directly on the event stream using Java and/or the Continuous Query Language (CQL). How do we best store the results, so that they can be made available continously to a dashboard for presentation will be shown as well. Additionally the so-called Lambda architecture will be presented, where such an event streaming solution plays a critical part, in combination with a Big Data batch processing layer.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (20)
Ähnlich wie Processing Twitter Stream with Oracle Event Processing (OEP)
Ähnlich wie Processing Twitter Stream with Oracle Event Processing (OEP) (20)
Mehr von Guido Schmutz
Mehr von Guido Schmutz (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Processing Twitter Stream with Oracle Event Processing (OEP)
- 1. Processing Twitter Stream with Oracle Event Processing (OEP) Guido Schmutz OFM Partner Forum Malta 19.2.2014 BASEL 1 BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014 FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA
- 2. Guido Schmutz • • Working for Trivadis for more than 17 years Oracle ACE Director for Fusion Middleware and SOA • • Co-Author of different books Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data • • Member of Trivadis Architecture Board Technology Manager @ Trivadis • More than 25 years of software development experience • Contact: guido.schmutz@trivadis.com • • Blog: http://guidoschmutz.wordpress.com Twitter: gschmutz 2 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 3. AGENDA 1. Introduction 2. Twitter Use Case 3. Processing with Oracle Event Processing (OEP) 4. Visualization with Oracle Business Activity Monitoring (BAM) 5. Store Information in Apache Cassandra 6. Summary 3 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 4. Big Data Definition (4 Vs) Characteristics of Big Data: Its Volume, Velocity and Variety in combination + Time to action ? – Big Data + Event Processing = Fast Data 4 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 5. The world is changing … The model of Generating/Consuming Data has changed …. Old Model: few companies are generating data, all others are consuming data New Model: all of use are generating data, and all of us are consuming data 5 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 6. Who is generating Big Data? Mobile devices (tracking all objects all the time) Social media and networks (all of us are generating data) Scientific instruments (collecting all sorts of data) Sensor technology and networks (measuring all kinds of data) The progress and innovation is no longer hindered by the ability to collect data But by the ability to manage, analyze, summarize, visualize and discover knowledge from the collected data in a timely manner and in a scalable fashion 6 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 7. 7 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 8. Internet Of Things – Sensors are/will be everywhere There are more devices tapping into the internet than people on earth How do we prepare our systems/architecture for the future? 8 2013 © Trivadis Source: The Economist Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014 Source: Cisco
- 9. Data as an Asset - Store Anything? But then data is just too valuable to delete! We must store anything! 9 It depends … but Big Data technologies allow to store the raw information from both new data sources as well as existing ones so that you can later use it to create new data-driven products, you would not have thought about today! 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014 Nonsense! Just store the data you know you need today!
- 10. AGENDA 1. Introduction 2. Twitter Use Case 3. Processing with Oracle Event Processing (OEP) 4. Visualization with Oracle Business Activity Monitoring (BAM) 5. Store Information in Apache Cassandra 6. Summary 10 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 11. Retrieve Tweets and Visualize 11 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 12. Access to Tweets Source Twitter’s Search API Twitter’s Streaming API DataSift Limitations Cost 3200 / user 5000 / keyword 180 requests / 15 minutes free 1%-40% of total volume free Quelle none Gnip 12 none 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014 0.15 -0.20$ / unit On request
- 13. How to design a stream (event) processing system? Twitter Stream Twitter Stream tweet Twitter Stream 13 tweet tweet Receiving/ Processing Sensor Sensor result tweet tweet Processing Persist (Queue) 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014 result tweet Processing result
- 14. AGENDA 1. Introduction 2. Twitter Use Case 3. Processing with Oracle Event Processing (OEP) 4. Visualization with Oracle Business Activity Monitoring (BAM) 5. Store Information in Apache Cassandra 6. Summary 14 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 15. Oracle Event Processing (OEP) - Engine Lightweight Java Application Server • Full environment for running Java applications • Module Framework - OSGi High Throughput • Hundreds of thousands of events/second Event Processing Infrastructure Easy-to-use development environment • Service Framework – Spring DM, POJO Enterprise Web 2.0 & Eclipse-based tooling Multiple-choice VM • JRockit or WebLogic RealTime 15 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 16. Oracle Event Processing – Event Processing Network Concept 16 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 17. Oracle Event Processing – In Memory, Continuous Queries Event Processing Output § Filtering - New stream filtered for specific criteria, e.g. stock price > $22 § Correlation & Aggregation - Scrolling, time-based window metrics, e.g. average # of stock trades in the last hour § Pattern Matching - 17 Notification of detected event patterns, e.g. price changes A, B and C occurred within 15 minute window 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 18. Oracle Event Processing - CQL Initiative for a complete “continuous” query language Start with SQL ’99 plus “continuous” query extensions § Based on Stanford University research Industry standards discussions § Event Processing Technical Society (EPTS) § ANSI SQL § OMG Adoption Today § ANSI SQL Standards Proposal for CQL Pattern Matching - Oracle, IBM, Stanford University § OpenSource Adoption of CQL § Oracle Complex Event Processor (CEP) Releaseà Available in 11g 18 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 19. Oracle Event Processing – Visual Development Tools 19 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 20. Oracle Event Processing – Operation & Management 20 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 21. Implementation – complete picture @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta BAM Tweet JMS Cassandra Tweet Twitter #ofmforum Hashtag #malta Extractor Twitter Adapter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta 21 Cassandra @SOACommunity,5 Counter @SOASimone Counter #ofmforum,5 Mention @SOACommunity Processor Extractor #malta,2 Robertvanmolken,1 BAM range 30 seconds Counter slide 30 seconds Author Extractor robertvanmolken 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014 @SOASimone,1 JMS
- 22. 1) Creating a Twitter Adapter Twitter Twitter Adapter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta 22 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 23. 2) Send Tweets to BAM @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta Twitter Twitter Adapter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta 23 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014 BAM Tweet JMS
- 24. 3) Extract interesting information from Tweet @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta Twitter #ofmforum Hashtag #malta Extractor @SOASimone Mention @SOACommunity Extractor Twitter Adapter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta 24 Author Extractor robertvanmolken 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014 BAM Tweet JMS
- 25. 4) Count occurrences within period @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta BAM Tweet JMS BAM Counter JMS Twitter #ofmforum Hashtag #malta Extractor @SOASimone,1 @SOACommunity,5 @SOASimone Mention #ofmforum,5 Counter @SOACommunity Processor Extractor #malta,2 Twitter Adapter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta 25 Robertvanmolken,1 range 30 seconds slide 30 seconds Author Extractor robertvanmolken 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 26. 5) Adding Cassandra NoSQL for storing results @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta BAM Tweet JMS Cassandra Tweet Twitter #ofmforum Hashtag #malta Extractor Twitter Adapter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta 26 Cassandra @SOACommunity,5 Counter @SOASimone Counter #ofmforum,5 Mention @SOACommunity Processor Extractor #malta,2 Robertvanmolken,1 BAM range 30 seconds Counter slide 30 seconds Author Extractor robertvanmolken 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014 @SOASimone,1 JMS
- 27. Implementing in Oracle Event Processing BAM Tweet JMS BAM Counter JMS Twitter Hashtag Extractor Mention Extractor Twitter Adapter Author Extractor 27 Counter Processor range 30 seconds slide 30 seconds 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 28. 1) Creating Twitter Adapter – Connecting to Twitter Stream 28 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 29. 1) Creating Twitter Adapter – Tweet Event 29 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 30. 1) Creating Twitter Adapter – Adapter Factory 30 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 31. 1) Creating Twitter Adapter – Assembly 31 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 32. 1) Creating Twitter Adapter – Export Adapter to server 32 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 33. 1) Creating Twitter Adapter – Using Twitter Adapter 33 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 34. 2) Sending Tweets to BAM Using Oracle BAM Enterprise Message Sources (JMS) interface 34 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 35. 2) Sending Tweets to BAM – Convert event to JMS MapMessage 35 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 36. 3) Extract information from Tweet – Extract Hashtags from TweetEvent 36 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 37. 3) Extract information from Tweet – Extract Hashtags from TweetEvent 37 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 38. 4) Count occurrences within period - Using CQL 38 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 39. Implementation – Complete Picture 39 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 40. AGENDA 1. Introduction 2. Twitter Use Case 3. Processing with Oracle Event Processing (OEP) 4. Visualization with Oracle Business Activity Monitoring (BAM) 5. Store Information in Apache Cassandra 6. Summary 40 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 41. Oracle BAM: Architected for Integration and Visualization Application Server Message Queues CEP WebApplications BAM Server Enterprise Integration Framework ActiveDataCache BPM ActiveViewer Actions & Escalations API Architect Administrator Kernel WebServices Web Services Internet Internet ReportCache ReportServer ViewSets ADF Pages with DVT DataSets Snapshots & Change Lists DataStorageEngine Memory / Disk ADF ODI BAM DataControl External Data Objects iCommand BI Data & Metadata Import & Export OLTP & Data Warehouses BAM Data & Metadata Databases 41 BAM Dashboards Notification Services BAM Adapter BPEL StartPage ActiveStudio JMS Connector OESB Mobile Devices EventEngine Oracle Database (Grid) 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 42. Oracle BAM – Create a Data Object 42 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 43. Oracle BAM Enterprise Message Source Configuration 43 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 44. AGENDA 1. Introduction 2. Twitter Use Case 3. Processing with Oracle Event Processing (OEP) 4. Visualization with Oracle Business Activity Monitoring (BAM) 5. Store Information in Apache Cassandra 6. Summary 44 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 45. Implementation – Storing information in NoSQL database BAM Tweet JMS Cassandra Tweet Twitter Hashtag Extractor Mention Extractor Twitter Adapter Author Extractor 45 Cassandra Counter Counter Processor range 30 seconds slide 30 seconds 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014 BAM Counter JMS
- 46. Event Processing Network in OEP 46 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 47. The world is changing … new data stores ORDER Order ID: 1001 Order Date: 15.9.2012 Problem of traditional (R)DBMS approach: § § § § Complex object graph Schema evolution Semi-structured data Scaling Customer CUSTOMER First Name: Peter Last Name: Sample Billing Address Street: Somestreet 10 City: Somewhere Postal Code: 55901 ADDRESS Line Items Quantity Price Ipod Touch 1 220.95 Monster Beat 2 190.00 Apple Mouse 1 69.90 Name ORDER_LINES Polyglot persistence § Using multiple data storage technologies (RDMBS + NoSQL + NewSQL + InMemory) § Selected based on the way data is being used by individual applications • Why using an RDBMS if there are better storage alternatives? • Key/Value, Column Family, Document, Graph-oriented, Relational, … § Can occur both over the enterprise as well as within a single application 47 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 48. Apache Cassandra – NoSQL database • Developed at Facebook • Open source distributed database management system • Professional grade support from company called DataStax • Main Features § § § § § § § § 48 Real-Time Highly Distributed Support for Multiple Data Center Highly Scalable No Single Point of Failure Fault Tolerant Tunable Consistency Cassandra Query Language (CQL) 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 49. Apache Cassandra - NoSQL Database • Don’t think of relational table => more of a sorted map • Know your application => model around the queries • De-normalize and duplicate for read performance • Index is not an afterthought, anymore=> index upfront • Think of physical storage structure Row-key Columns à 2013-08 Day-1, keyword-1=>100 Day-2, keyword-1=>150 Day-3, keyword-1=>170 …. Day-31, keyword-1 =>170 2013-08-31 Hour-1, keyword-1 =>10 Hour-2, keyword-1 =>15 Hour-3, keyword-1 =>17 …. Hour-24, keyword-1 =>17 2013-08-31-10 Minute-1, keyword-1=>2 Minute-2, keyword-1=>3 Minute-3, keyword-1 =>5 …. Minute-60, keyword-1=>2 49 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 50. Apache Cassandra – NoSQL database 50 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 51. Apache Cassandra – NoSQL database 51 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 52. AGENDA 1. Introduction 2. Twitter Use Case 3. Processing with Oracle Event Processing (OEP) 4. Visualization with Oracle Business Activity Monitoring (BAM) 5. Store Information in Apache Cassandra 6. Summary 52 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 53. Big Data Reference Architecture – Combine Streaming and Batch 53 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 54. Summary Oracle Event Processing Cassandra • Very light weight server • No single point of failure • Very easy to write adapters • Forget your data modeling skills • Very strong CQL language • Model around the queries • Query Language Oracle Business Activity Monitoring § 11g version a bit “old fashioned” § Easy to integrate through JMS 54 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
- 55. Questions and answers ... Guido Schmutz Technology Manager guido.schmutz@trivadis.com BASEL 55 BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014 FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA