Organizational success depends on our ability to sense the environment, grab opportunities and eliminate threats that are present in real-time. Such real-time processing is now available to all organizations (with or without a big data background) through the new WSO2 Stream Processor. This slides presents WSO2 Stream Processor’s new features and improvements and explains how they make an organization excel in the current competitive marketplace. Some key features we will consider are: * WSO2 Stream Processor’s highly productive developer environment, with graphical drag-and-drop, and the Streaming SQL query editor * The ability to process real-time queries that span from seconds to years * Its interactive visualization and dashboarding features with improved widget generation * Its ability to processing at scale via distributed deployments with full observability * Default support for HTTP analytics, distributed message trace analytics, and Twitter analytics

1. WSO2 Stream Processor:
Graphical Editor, HTTP & Message Trace Analytics and More
Sriskandarajah Suhothayan
Director, WSO2

2. WSO2 Stream Processor
An open source, cloud native analytics product optimized
to create real-time, actionable insights for agile digital
businesses.
2

3. Stream Processing
3
● Need to write code
● Complex deployment (5 - 6 nodes)
● Inability to change fast
Challenges
Source : http://cdn.business2community.com/wp-content/uploads/2012/08/Invest_Money_Photoxpress_2938045.jpg

4. WSO2 Stream Processor
4
● Need to write code
○ Streaming SQL + Graphical Editor
● Complex deployment (5 - 6 nodes)
○ 2 node minimum HA deployment (scale beyond with
Kafka)
● Inability to change fast
○ Query templates, editor
The solution

5. WSO2 Stream Processor

6. • Lightweight and lean
• Easy-to-learn streaming SQL with graphical editor
• High performance analytics with just 2 nodes (HA)
• Native support for streaming machine learning
• Long-term aggregations from seconds to years
• Highly scalable deployment with exactly-once processing
• Tools for development and monitoring
• Tools for business users to write their own rules
Overview of WSO2 Stream Processor

7. Market Recognition
● Named as a Strong Performer in The Forrester Wave™: Big Data
Streaming Analytics, Q1 2016
● Highest score possible in 'Acquisition and Pricing' criteria, and among
second highest scores in 'Ability to Execute' criteria
● The Forrester report notes:
“WSO2 is an open source middleware provider that includes a full spectrum of
architected-as-one components such as application servers, message brokers, enterprise
service bus, and many others.
Its streaming analytics solution follows the complex event processor architectural
approach, so it provides very low-latency analytics. Enterprises that already use WSO2
middleware can add CEP seamlessly. Enterprises looking for a full middleware stack that
includes streaming analytics will find a place for WSO2 on their shortlist as well.”
7

8. New Features

9. Stream Processor Studio

10. Editor,
Debugger,
Simulation,
and Testing
all-in-one
Development Studio for Siddhi Apps

11. Graphical Query Editor
Drag & drop
query builder

12. 1. Streaming data preprocessing and transformation
2. Data store integration
3. Streaming data summarization from seconds to years
4. KPI analysis and alerts
5. Event correlation and trend analysis
6. Real-time prediction and streaming machine learning
7. Service integration
8. Improved JSON processing support
Supported Streaming Analytics Patterns

13. • Source and Sinks
– HTTP, Kafka, TCP, Email, JMS, File, RabbitMQ,
MQTT, Web-Socket, Twitter
• Message Formats
– JSON, XML, Text, Binary, Key-value, CSV
• Data Stores
– RDBMS, Solr, MongoDB, HBase, Cassandra,
Elasticsearch
Supported connectors

14. Analytics Extensions
https://store.wso2.com/store/assets/analyticsextension/lis
t

15. Business Dashboard

16. • Generate dashboard and widgets
• Fine grained permissions
– Dashboard level
– Widget level
– Data level
• Localization support
• Inter-widget communication
• Shareable dashboards with widget state persistence
Dashboards

17. Dashboards

18. Widget Generation

19. Status Dashboard

20. • Understand system performance via
– Throughput
– Latency
– CPU, memory utilizations
• Monitor in various scales
– Node level
– Siddhi app level
– Siddhi query level
Status Dashboard
Monitor resource nodes and Siddhi apps

21. 21
Status Dashboard

22. Self Service & Omniscience

23. • Define and modify rules using config forms
• Generate templates with
Editor
Business Rules for Non Technical Users

24. • Build template using Graphical UI or JSON schema
Create Business Rules Templates

25. Default Analytics Support

26. HTTP Analytics
Request-Response
and error analysis
over time

27. Distributed Message Tracing
Open Tracing Support

28. Twitter Analytics
Word cloud and
sentiment
analytics

29. Distributed Deployment

30. • High performance
– Process around 100k events/sec
– While most other stream
processing systems need around
5+ nodes
• Zero downtime
• Zero event loss
• Simple deployment
with RDBMS coordination (no
ZooKeeper, Kafka, etc.)
• Multi data center support
Minimum HA with 2 Nodes
Stream Processor
Stream Processor
Event Sources
Dashboard
Notification
Invocation
Data Source
Siddhi App
Siddhi App
Siddhi App
Siddhi App
Siddhi App
Siddhi App
Event
Store

31. Distributed Deployment with Kafka
• Exactly-once processing
• Fault tolerance
• Highly scalable
• No back pressure
• Distributed development configurations via annotations
• Pluggable distribution options (YARN, K8, etc.)

32. Distributed Deployment

33. Distributed Deployment with Kafka
Data
base
Event
Source
Event
Sink
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Kafka Topic
Kafka Topic
Kafka Topic
Kafka
Topic
Kafka
Topic

34. Sample Distributed Siddhi App
@source(type = ‘kafka’, …, @map(type = ‘json’))
define stream ProductionStream (name string, amount double, factoryId int);
@dist(parallel = ‘4’, execGroup = ‘gp1’)
from ProductionStream[amount > 100]
select *
insert into HighProductionStream ;
@dist(parallel = ‘2’, execGroup = ‘gp2’)
partition with (factoryId of HighProductionStream)
begin
from HighProductionStream#window.timeBatch(1 min)
select factoryId, sum(amount) as amount
group by factoryId
insert into ProdRateStream ;
end;
Filter
Source
FilterFilterFilter
PartitionPartition

35. User Stories

36. Success Stories
37
a
TFL used WSO2 real time streaming to create next generation transport
systems.
Uber detected fraud in real time, processing over 400K events per
second
CSI uses WSO2 streaming capabilities to integrate people, systems,
and things.
State of Arizona monitors and manages their Private PaaS in real time
to improve efficiencies and trim costs.

37. Download and Tryout
38
WSO2 Stream Processor
http://wso2.com/analytics/
Documentation
https://docs.wso2.com/display/SP420

38. THANK YOU
wso2.com