1. API and Big data Solution
Patterns
Srinath Perera
Senior Software Architect

2. About
WSO2
• Providing
the
only
complete
open
source
componen:zed
cloud
pla=orm
– Dedicated
to
removing
all
the
stumbling
blocks
to
enterprise
agility
– Enabling
you
to
focus
on
business
logic
and
business
value
• Recognized
by
leading
analyst
firms
as
visionaries
and
leaders
– Gartner
cites
WSO2
as
visionaries
in
all
3
categories
of
applica:on
infrastructure
– Forrester
places
WSO2
in
top
2
for
API
Management
• Global
corpora:on
with
offices
in
USA,
UK
&
Sri
Lanka
– 200+
employees
and
growing
• Business
model
of
selling
comprehensive
support
&
maintenance
for
our
products

3. 150+
globally
posi:oned
support
customers

4. Outline
• What
is
API
Management
• What
is
Big
Data?
• API
manager
+
Bigdata
solu:on
• API
+
Big
data
PaVerns
• Conclusion
Photo
by
John
Trainoron
Flickr
hVp://www.flickr.com/photos/trainor/
2902023575/,
Licensed
under
CC

5. APIs
• An API is a business capability delivered over the
Internet to internal or external consumers
– Network accessible function
– Available using standard web protocols
– With well-defined interfaces
– Designed for access by third-parties
• A Managed API is:
– Actively advertised and subscribe-able
– Available with SLAs
– Secured, authenticated, authorized and protected
– Monitored and monetized with analytics

6. Typical Use Cases
• Expose APIs for public consumption
– Extend your business through APIs.
• Expose APIs for internal consumption
– Manage the APIs used in internal applications
– Detect Usage Patterns
– Internal Monetization
• Control Access to Cloud Services
– Manage and Secure access from internal
applications to cloud services (SalesForce, Google
Apps, etc.)

7. WSO2 API
Management
Platform
WSO2’s API Management Vision
• Create APIs
– WSO2 Application Server, Data Services Server and ESB
• Find and subscribe/buy APIs
– API Store and Governance
• Manage, secure and protect APIs
– API Management and Gateway
• Monitor and Monetize APIs
– API Monitoring and Analytics
• Develop, host and run API-based applications in a
Platform-as-a-Service
– WSO2 Stratos and Cloud Development Platform
– API Cloud

8. API Manager Components

9. API Gateway Processing Flow

10. API Manager Product and Platform

11. What is Big data?
• There is lot of data
available
– E.g. Internet of things
• We have computing
power
• We have technology
• Goal To know
– To Explain
– To predict
• Challenge is the full
lifecycle

12. Drivers of Big Data

13. Big Data Architecture

14. Collecting Data
• Data collected at sensors and sent to big data
system via events or flat files
• Event Streams: we name the events by its
content/ originator
• Get
data
through
– Point
to
Point
– Event
Bus
(WSO2
MB)
• E.g.
Data
bridge
(come
with
WSO2
CEP/
BAM)
– a
thrib
based
transport
that
do
about
400k
events/
sec

15. Storing Data
• Historically we used databases
– Scale is a challenge: replication,
sharding
• Scalable options
– NoSQL (Cassandra, Hbase) [If
data is structured]
• Column families Gaining Ground
– Distributed file systems (e.g.
HDFS) [If data is unstructured]
• New SQL
– In Memory computing, VoltDB
• Specialized data structures
– Graph Databases, Data structure
servers hVp://www.flickr.com/photos/keso/
363133967/

16. Making Sense of Data
• To know (what happened?)
– Basic analytics +
visualizations
– Interactive drill down
– E.g. MapReduce, Hive, CEP
• To explain (why)
– Data mining, classifications,
building models, clustering
(e.g. Mahout, Map Reduce)
• To forecast
– Neural networks, decision
models (e.g. Mahout)

17. WSO2 Big Data Story

18. API Gateway + Big Data

19. API + Big Data Patterns
• Big Data as API enabler
– Monitor and control: API analytics for an
organization
– Analytics: Billing and Metering
– API Store (Market Place)
• Targeted Marketing (Recommendations,
Deals, Search)
• Context sensitive prizing
– Deployment optimization
– Governance: Planning and Proactive
action
• APIs as a Big Data Enabler
– Easy way to find and get access to data
– Data Market Place
– API Façade for exposing Analytics/ Data
– Controlled delegation of data
hVp://www.flickr.com/photos/ooohoooh/
1350774047/

20. Pattern 1: Monitor and Control

21. Monitor and Control (Contd.)
• API Manager as single entry point to come in and go
out
• Capture data – client, what API, when? $$ revenue ..
• Analyze and Monitor KPIs
– CIO Dashboard
– Alerts
• React to trends faster
• Greater control
– Audit
– Subscriptions policies
– Usage tiers
• Policy enforcement
– Make sure everything is managed

22. Example: Business value driven Org
• Customer facing site is a
composite (e.g. like Gmail page)
• Each widget back by API or APIs
• Each widget added removed in
runtime
• Widget shown based on business
value
• Each API owner get resource
compared to business value
• We can use API management as
the tool to realize this

23. Pattern 2: Billing and Metering
• API store detect all access
• Contains information about
subscriptions, amount of data
transferred etc.
• We can use this to
– Meter, control and optimize
– Enforce usage tiers (e.g. Quota
via CEP)
• Generate Billing and reports
• Supported by WSO2 API
manager now

24. Pattern 3: API Store (Marketplace)

25. Pattern 4: Deployment Optimization

26. App Factory - Applications Management

27. App Factory (Contd.)
• Applications Management
– Creation
– Continuous Build / Deploy
– Promotion across environments
• Team Management
– Add developers, QA engineers and DevOps for your
application
• Bugs/Features Tracking
• Integrated with API Store
• Activity Dashboards
• Built on top of Jaggery framework
• Skinnable

28. Pattern 5:API lifecycle Governance
• Take control of your API life cycle
(develop, test, production) with
WSO2 App Factory
• You can apply analytics to your
API lifecycle
– How long does it take to develop and
API?
– How often are they updated?
– What are the dependencies of my
API?
– What are most effective versions?
– What steps takes most time?
– What APIs are actively developed?
– What one have more issues, how fast
are they get fixed?

29. Pattern 6: Find datasets
• Most big data apps need
Good data
• Hard to find good data
– No good way to find
– Hard to access
– No clear way to buy
• APIs is a great way to find
useful data
• Also to provide controlled
access
e.g. Andy Stanford-Clark used public data about
cruisers to track where availability.

30. Pattern 7: Data Marketplace
• Find, (buy), use data
• Great way to monetize the
output of big data projects
– Sell the analyzed traffic data
– Sell stock quote analytics
• API subscribe model can help
• Usage tiers can be based on
frequency or resolution of
data

31. Pattern 8: Controlled Data Delegation
• Only expose the part of your data set of a processing
without exposing credentials of the database to the
user

32. Big data in API Store

33. APIs in Big data System

34. Usecase
• Trigger an alert when new version is 10% more
slower than the older versions
– Expose the API through API store
– Configure API manager with BAM location
– BAM integration will pump stats (api, version, latency,
time)
– Write a BAM query to calculate the latency through
time and store it in a DB
– Write a CEP query to detect if new version is slower
Define eventTable APIStatHistory ..
Select from APIStat as s join APIStatHistory as h on (h.latency -
s.latency)<=h.latency*.1

35. Conclusion
• APIs and Big Data has received much
attentions
• We looked at WSO2 offerings in both
cases
– WSO2 API Manager, WSO2 App Factory
– WSO2 BAM, WSO2 CEP
• We looked at some patterns on how both
can work together
• We looked at how we can implement them

36. Engage
with
WSO2
• Helping
you
get
the
most
out
of
your
deployments
• From
project
evalua:on
and
incep:on
to
development
and
going
into
produc:on,
WSO2
is
your
partner
in
ensuring
100%
project
success

37. Questions?