Processing Twitter Stream with Oracle Event Processing (OEP) Präsentation von Guido Schmutz, Trivadis AG

1. 2013 © Trivadis
BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA
2013 © Trivadis
Processing Twitter Stream with
Oracle Event Processing (OEP)
Guido Schmutz
OFM Partner Forum Malta
19.2.2014
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
1

2. 2013 © Trivadis
Guido Schmutz
• Working for Trivadis for more than 17 years
• Oracle ACE Director for Fusion Middleware and SOA
• Co-Author of different books
• Consultant, Trainer Software Architect for Java, Oracle, SOA and
Big Data / Fast Data
• Member of Trivadis Architecture Board
• Technology Manager @ Trivadis
• More than 25 years of software development
experience
• Contact: guido.schmutz@trivadis.com
• Blog: http://guidoschmutz.wordpress.com
• Twitter: gschmutz
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
2

3. 2013 © Trivadis
AGENDA
1. Introduction
2. Twitter Use Case
3. Processing with Oracle Event Processing (CEP)
4. Visualization with Oracle Business Activity Monitoring (BAM)
5. Store Information in Apache Cassandra
6. Summary
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
3

4. 2013 © Trivadis
Big Data Definition (4 Vs)
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
4
+ Time to action ? – Big Data + Event
Processing = Fast Data
Characteristics of Big Data: Its Volume,
Velocity and Variety in combination

5. 2013 © Trivadis
The world is changing …
The model of Generating/Consuming Data has changed ….
Old Model: few companies are generating data, all others are consuming
data
New Model: all of use are generating data, and all of us are consuming
data
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
5

6. 2013 © Trivadis
Who is generating Big Data?
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
6
The progress and innovation is no longer hindered by the ability to collect data
But by the ability to manage, analyze, summarize, visualize and discover knowledge
from the collected data in a timely manner and in a scalable fashion
Social media and networks
(all of us are generating data)
Scientific instruments
(collecting all sorts of data)
Mobile devices
(tracking all objects all the time)
Sensor technology and
networks
(measuring all kinds of data)

7. 2013 © Trivadis
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
7

8. 2013 © Trivadis
Internet Of Things – Sensors
are/will be everywhere
There are more devices tapping into the
internet than people on earth
How do we prepare our
systems/architecture for the future?
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
8
Source: CiscoSource: The Economist

9. 2013 © Trivadis
Data as an Asset - Store Anything?
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
9
But then data is
just too valuable
to delete!
We must
store anything!
Nonsense! Just
store the data
you know
you need today!
It depends … but Big Data
technologies allow to store the
raw information from both new
data sources as well as existing
ones so that you can later use it to
create new data-driven products,
you would not have thought
about today!

10. 2013 © Trivadis
AGENDA
1. Introduction
2. Twitter Use Case
3. Processing with Oracle Event Processing (CEP)
4. Visualization with Oracle Business Activity Monitoring (BAM)
5. Store Information in Apache Cassandra
6. Summary
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
10

11. 2013 © Trivadis
Retrieve Tweets and Visualize
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
11

12. 2013 © Trivadis
Access to Tweets
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
12
Quelle
Source Limitations Cost
Twitter’s Search API 3200 / user
5000 / keyword
180 requests / 15 minutes
free
Twitter’s Streaming API 1%-40% of total volume free
DataSift
none
0.15 -0.20$ /
unit
Gnip none On request

13. 2013 © Trivadis
How to design a stream (event) processing system?
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
13
Twitter
Stream
tweet
Sensor
tweet
Persist
(Queue)
Twitter
Stream
tweet
Sensor
tweet
Processing
tweet
Processing
result
result
Twitter
Stream
tweet
Receiving/
Processing
result

14. 2013 © Trivadis
AGENDA
1. Introduction
2. Twitter Use Case
3. Processing with Oracle Event Processing (CEP)
4. Visualization with Oracle Business Activity Monitoring (BAM)
5. Store Information in Apache Cassandra
6. Summary
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
14

15. 2013 © Trivadis
Oracle Event Processing (OEP) - Engine
Lightweight Java Application Server
• Full environment for running Java applications
• Module Framework - OSGi
High Throughput
• Hundreds of thousands of events/second
Event Processing Infrastructure
Easy-to-use development environment
• Service Framework – Spring DM, POJO
Enterprise Web 2.0 & Eclipse-based tooling
Multiple-choice VM
• JRockit or WebLogic RealTime
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
15

16. 2013 © Trivadis
Oracle Event Processing – Event Processing Network
Concept
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
16

17. 2013 © Trivadis
Oracle Event Processing – In Memory, Continuous
Queries
Event Processing Output
§ Filtering
- New stream filtered for specific criteria, e.g. stock price > $22
§ Correlation & Aggregation
- Scrolling, time-based window metrics, e.g. average # of stock trades in the last hour
§ Pattern Matching
- Notification of detected event patterns, e.g. price changes A, B and C occurred within 15 minute window
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
17

18. 2013 © Trivadis
Oracle Event Processing - CQL
Initiative for a complete “continuous” query language
Start with SQL ’99 plus “continuous” query extensions
§ Based on Stanford University research
Industry standards discussions
§ Event Processing Technical Society (EPTS)
§ ANSI SQL
§ OMG
Adoption Today
§ ANSI SQL Standards Proposal for CQL Pattern Matching
- Oracle, IBM, Stanford University
§ OpenSource Adoption of CQL
§ Oracle Complex Event Processor (CEP) Releaseà Available in 11g
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
18

19. 2013 © Trivadis
Oracle Event Processing – Visual Development Tools
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
19

20. 2013 © Trivadis
Oracle Event Processing – Operation & Management
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
20

21. 2013 © Trivadis
Implementation – complete picture
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
21
Mention
Extractor
Twitter
Adapter
Counter
Processor
Hashtag
Extractor
Author
Extractor
Cassandra
Counter
BAM
Tweet
Cassandra
Tweet
BAM
Counter
@SOASimone @SOACommunity
heard you couldn’t make it. We
miss you! #ofmforum #malta
@SOASimone @SOACommunity
heard you couldn’t make it. We
miss you! #ofmforum #malta
@SOASimone
@SOACommunity heard you
couldn’t make it. We miss
you! #ofmforum #malta
@SOASimone
@SOACommunity
robertvanmolken
#ofmforum
#malta
#ofmforum,5
#malta,2
Robertvanmolken,1
@SOASimone,1
@SOACommunity,5
JMS
JMS
Twitter
range 30 seconds
slide 30 seconds

22. 2013 © Trivadis
1) Creating a Twitter Adapter
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
22
Twitter
Adapter
@SOASimone
@SOACommunity heard you
couldn’t make it. We miss
you! #ofmforum #malta
Twitter

23. 2013 © Trivadis
2) Send Tweets to BAM
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
23
Twitter
Adapter
BAM
Tweet
@SOASimone @SOACommunity
heard you couldn’t make it. We
miss you! #ofmforum #malta
@SOASimone
@SOACommunity heard you
couldn’t make it. We miss
you! #ofmforum #malta
JMS
Twitter

24. 2013 © Trivadis
3) Extract interesting information from Tweet
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
24
Mention
Extractor
Twitter
Adapter
Hashtag
Extractor
Author
Extractor
BAM
Tweet
@SOASimone @SOACommunity
heard you couldn’t make it. We
miss you! #ofmforum #malta
@SOASimone
@SOACommunity heard you
couldn’t make it. We miss
you! #ofmforum #malta
@SOASimone
@SOACommunity
robertvanmolken
#ofmforum
#malta
JMS
Twitter

25. 2013 © Trivadis
4) Count occurrences within period
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
25
Mention
Extractor
Twitter
Adapter
Counter
Processor
Hashtag
Extractor
Author
Extractor
BAM
Tweet
BAM
Counter
@SOASimone @SOACommunity
heard you couldn’t make it. We
miss you! #ofmforum #malta
@SOASimone
@SOACommunity heard you
couldn’t make it. We miss
you! #ofmforum #malta
#ofmforum,5
#malta,2
Robertvanmolken,1
@SOASimone,1
@SOACommunity,5
JMS
JMS
Twitter
range 30 seconds
slide 30 seconds
@SOASimone
@SOACommunity
robertvanmolken
#ofmforum
#malta

26. 2013 © Trivadis
5) Adding Cassandra NoSQL for storing results
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
26
Mention
Extractor
Twitter
Adapter
Counter
Processor
Hashtag
Extractor
Author
Extractor
Cassandra
Counter
BAM
Tweet
Cassandra
Tweet
BAM
Counter
@SOASimone @SOACommunity
heard you couldn’t make it. We
miss you! #ofmforum #malta
@SOASimone @SOACommunity
heard you couldn’t make it. We
miss you! #ofmforum #malta
@SOASimone
@SOACommunity heard you
couldn’t make it. We miss
you! #ofmforum #malta
#ofmforum,5
#malta,2
Robertvanmolken,1
@SOASimone,1
@SOACommunity,5
JMS
JMS
Twitter
range 30 seconds
slide 30 seconds
@SOASimone
@SOACommunity
robertvanmolken
#ofmforum
#malta

27. 2013 © Trivadis
Implementing in Oracle Event Processing
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
27
Mention
Extractor
Twitter
Adapter
Counter
Processor
Hashtag
Extractor
Author
Extractor
BAM
Tweet
BAM
Counter
JMS
JMS
Twitter
range 30 seconds
slide 30 seconds

28. 2013 © Trivadis
1) Creating Twitter Adapter –
Connecting to Twitter Stream
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
28

29. 2013 © Trivadis
1) Creating Twitter Adapter –
Tweet Event
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
29

30. 2013 © Trivadis
1) Creating Twitter Adapter –
Adapter Factory
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
30

31. 2013 © Trivadis
1) Creating Twitter Adapter –
Assembly
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
31

32. 2013 © Trivadis
1) Creating Twitter Adapter –
Export Adapter to server
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
32

33. 2013 © Trivadis
1) Creating Twitter Adapter –
Using Twitter Adapter
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
33

34. 2013 © Trivadis
2) Sending Tweets to BAM
Using Oracle BAM Enterprise Message Sources (JMS) interface
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
34

35. 2013 © Trivadis
2) Sending Tweets to BAM –
Convert event to JMS MapMessage
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
35

36. 2013 © Trivadis
3) Extract information from Tweet –
Extract Hashtags from TweetEvent
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
36

37. 2013 © Trivadis
3) Extract information from Tweet –
Extract Hashtags from TweetEvent
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
37

38. 2013 © Trivadis
4) Count occurrences within
period - Using CQL
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
38

39. 2013 © Trivadis
Implementation – Complete Picture
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
39

40. 2013 © Trivadis
AGENDA
1. Introduction
2. Twitter Use Case
3. Processing with Oracle Event Processing (CEP)
4. Visualization with Oracle Business Activity Monitoring (BAM)
5. Store Information in Apache Cassandra
6. Summary
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
40

41. 2013 © Trivadis
Oracle BAM: Architected for Integration and
Visualization
Processing Twitter Stream with Oracle Event Processing (OEP)
Internet
BAM Dashboards
WebApplications
StartPage
ActiveViewer
ActiveStudio
Architect
Administrator
ReportServer
iCommand
Oracle Database
(Grid)
BAM Data &
Metadata
External Data Objects
WebServices
Internet
Enterprise
Integration
Framework
Application Server
BI
Web Services
JMS Connector
BAM Adapter
ADF
BAM DataControl
ADF Pages with DVT
BAM ServerEventEngine
Actions & Escalations
Notification Services
ReportCache
Snapshots &
Change Lists
Memory / Disk
ActiveDataCache
ViewSets
API
Kernel
DataSets
DataStorageEngine
ODI
Databases
OLTP &
Data Warehouses
Mobile Devices
Data & Metadata
Import & Export
BPEL
BPM
Message
Queues
CEP
OESB
19.02.2014
41

42. 2013 © Trivadis
Oracle BAM – Create a
Data Object
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
42

43. 2013 © Trivadis
Oracle BAM Enterprise Message
Source Configuration
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
43

44. 2013 © Trivadis
AGENDA
1. Introduction
2. Twitter Use Case
3. Processing with Oracle Event Processing (CEP)
4. Visualization with Oracle Business Activity Monitoring (BAM)
5. Store Information in Apache Cassandra
6. Summary
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
44

45. 2013 © Trivadis
Implementation – Storing information in NoSQL
database
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
45
Mention
Extractor
Twitter
Adapter
Counter
Processor
Hashtag
Extractor
Author
Extractor
Cassandra
Counter
BAM
Tweet
Cassandra
Tweet
BAM
Counter
JMS
JMS
Twitter
range 30 seconds
slide 30 seconds

46. 2013 © Trivadis
Event Processing Network in OEP
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
46

47. 2013 © Trivadis
The world is changing …
new data stores
Problem of traditional (R)DBMS approach:
§ Complex object graph
§ Schema evolution
§ Semi-structured data
§ Scaling
Polyglot persistence
§ Using multiple data storage technologies (RDMBS + NoSQL + NewSQL + In-
Memory)
§ Selected based on the way data is being used by individual applications
• Why using an RDBMS if there are better storage alternatives?
• Key/Value, Column Family, Document, Graph-oriented, Relational, …
§ Can occur both over the enterprise as well as within a single application
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
47
ORDER
ADDRESS
CUSTOMER
ORDER_LINES
Order
ID: 1001
Order Date: 15.9.2012
Line Items
Customer
First Name: Peter
Last Name: Sample
Billing Address
Street: Somestreet 10
City: Somewhere
Postal Code: 55901
Name
Ipod Touch
Monster Beat
Apple Mouse
Quantity
1
2
1
Price
220.95
190.00
69.90

48. 2013 © Trivadis
Apache Cassandra – NoSQL database
• Developed at Facebook
• Open source distributed database management system
• Professional grade support from company called DataStax
• Main Features
§ Real-Time
§ Highly Distributed
§ Support for Multiple Data Center
§ Highly Scalable
§ No Single Point of Failure
§ Fault Tolerant
§ Tunable Consistency
§ Cassandra Query Language (CQL)
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
48

49. 2013 © Trivadis
Apache Cassandra - NoSQL Database
• Don’t think of relational table => more of a sorted map
• Know your application => model around the queries
• De-normalize and duplicate for read performance
• Index is not an afterthought, anymore=> index upfront
• Think of physical storage structure
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
49
2013-08
Day-1,
keyword-1=>100
Day-2,
keyword-1=>150
Day-3,
keyword-1=>170
….
Day-31,
keyword-1 =>170
2013-08-31
Hour-1,
keyword-1 =>10
Hour-2,
keyword-1 =>15
Hour-3,
keyword-1 =>17
….
Hour-24,
keyword-1 =>17
2013-08-31-10
Minute-1,
keyword-1=>2
Minute-2,
keyword-1=>3
Minute-3,
keyword-1 =>5
….
Minute-60,
keyword-1=>2
Row-key Columns à

50. 2013 © Trivadis
Apache Cassandra – NoSQL database
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
50

51. 2013 © Trivadis
Apache Cassandra – NoSQL database
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
51

52. 2013 © Trivadis
AGENDA
1. Introduction
2. Twitter Use Case
3. Processing with Oracle Event Processing (CEP)
4. Visualization with Oracle Business Activity Monitoring (BAM)
5. Store Information in Apache Cassandra
6. Summary
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
52

53. 2013 © Trivadis
Big Data Reference Architecture – Combine Streaming
and Batch
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
53

54. 2013 © Trivadis
Summary
Cassandra
• No single point of failure
• Forget your data modeling skills
• Model around the queries
• Query Language
Oracle Event Processing
• Very light weight server
• Very easy to write adapters
• Very strong CQL language
Oracle Business Activity
Monitoring
§ 11g version a bit “old fashioned”
§ Easy to integrate through JMS
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
54

55. 2013 © Trivadis
Questions and answers ...
2013 © Trivadis
BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA
Guido Schmutz
Technology Manager
guido.schmutz@trivadis.com
19.02.2014
Processing Twitter Stream with Oracle Event Processing (OEP)
55