2. Cardinality Overview
q Cardinality
are
a
group
of
wireless
telecommunica5ons
engineers,
architects,
data
scien5sts,
and
so8ware
developers.
q The
largest
Big
Data
Hadoop
Solu5on
deployed
in
any
telecommunica5ons
operator
in
Europe.
q Cardinality
have
extensive
radio
design,
op5misa5on
and
opera5onal
support
system
(OSS)
experience
q Leveraging
innova5ve
methodologies
and
pro-‐ac5ve
approach
3. Cardinality is….
q Innova5ve
Start-‐Up,
with
referenceable
customers
&
sound
financial
backing
q Founders
have
real
world
Big
Data
experience
with
Mobile
Network
Operators
(MNOs)
q Using
proven
open
source
architectures,
which
has
evolved
based
on
analy5cs
experience
gained
from
over
4
years
using
Hadoop
clusters
in
mul5ple
operators
q Tier
One
MNO
Big
Data
contact
won
in
summer
2015
(via
open
RFP,
against
large
/
established
“big
data”
companies)
q Ini5al
volumes:
q Big
Data
is
tainted
with
hype
–
Cardinality
iden5fy,
target,
real
business
problems
&
solve
them
ETL
Throughput
100K
events
per
sec
(per
Feed)
Big
Data
Storage
10-‐12
billion
complex
rows/day
(40
TB
uncompressed
per
day)
Cache
~200
million
complex
rows
(100GB)
5. .
Collect Data from all the network elements
MonetiseStore Visualise
Data Lake: Collect Everything
AnalyseCorrelate
6. This is a sample text.
Insert your desired
text here.
here.
This is a sample text.
Insert your desired
text here.
Data Accessibility
Data Transfer
Data Storage
Data Correlation
Either (real)time
data is not
available or there
is license cost
q too costly to transfer the
data from source to DW
q too late by the time data
reaches the DW
q Complex Correlation
needed
q Batch processing does
not complete in tine
q Not real time
q Existing DW doesn’t
have enough space to
store all data sets
q It’s too expensive to
store all the data sets in
existing DW
Common Issues
7.
Data Lake
ETL
Engine
Analysis
Engine
Data Lake
Cardinality Perception
Visualisa5on
Unconnected
&
Un-‐enriched
data
silo
approach
repeated
in
New
Data
Lakes
create
unusable
big
data
sets
Data
Science
driven
enriched
and
connected
data
sets
imported
in
Data
Lakes
using
Cardinality’s
Percep5on
Plaborm
can
be
easily
accessed
by
users
and
machines
Data Silos Vs. Enriched Data Sets
Tradi&onal Approach
8. DESIGN
Big Data
Strategy
Data Lake
Big Data
Analytics
Architecture
CREATE
BUILD
DELIVER
Cardinality
Perception
How can we help?
9. Full
stack
deployment
of
Cardinality’s
Percep5on
plaborm
including
a
Hadoop
based
data
lake.
Greenfield
Deployment
of
Cardinality’s
Percep5on
plaborm
around
the
exis5ng
Hadoop
cluster
Exis5ng
Hadoop
Cluster
Offerings
Deployment
of
Cardinality
use
cases
and
data
visualisa5on,
using
exis5ng
Hadoop
Cluster
and
ETL
engine
Use
Cases/Data
Analy5cs
Big
Data
Consul5ng
Leverage
Cardinality’s
exper5se
in
crea5ng
organisa5on
wide
strategies,
designing
big
data
architectures
and
end-‐2-‐end
data
analysis/use
cases
Works with existing hadoop cluster
and flexible options incl. full stack
10. Open Approach to Infrastructure
Cloud
(Public,
Private,
or
Hybrid)
Appliance
Model
(Pre-‐Built)
Data
Centre
✔
✔
✔
11. Components Deployed
• ETL Engine
• Data Lake: New Hadoop Cluster
• Magnitude: Cache, SQL, NoSQL
• Marvel: Data Analysis/Use
Cases
• Scene: Visualisation
Data
Collec*on
(SFTP,
TCP,
WebSocket)
Data/Event
Stream
Data
Parser,
Enrichment
&
Analysis
DATALAKE&STORAGE
Batch
Analysis
Spark
Jobs
APPLICATIONS
Enterprise
Applications
Interactive Web &
Mobile Applications
BI / Reporting, Ad
Hoc Analysis
Statistical Analysis
DATASOURCE
Probes CDRs Control
Plane
Sensor
Data
Geo-
location
Data
Network
Data
User Plane
Full Stack Deployment
12. Deployed Components
• ETL Engine
• Magnitude: Cache, SQL,
NoSQL
• Data Analysis/Use Cases
• Visualisation
Data
Collec*on
(SFTP,
TCP,
WebSocket)
Data/Event
Stream
Data
Parser,
Enrichment
&
Analysis
DATALAKE&STORAGE
Batch
Analysis
Spark
Jobs
APPLICATIONS
Enterprise
Applications
Interactive Web &
Mobile Applications
BI / Reporting, Ad
Hoc Analysis
Statistical Analysis
DATASOURCE
Probes CDRs Control
Plane
Sensor
Data
Geo-
location
Data
Network
Data
User Plane
Exis5ng
Hadoop
Cluster
Existing Hadoop Cluster
13. Components Deployed
• Magnitude: Cache, SQL,
NoSQL
• Marvel: Data Analysis/
Cardinality use cases
• Scene: Visualisation
Cardinality’s
Data
PipeLine
(ETL)
DATALAKE&STORAGE
Batch
Analysis
Spark
Jobs
APPLICATIONS
Enterprise
Applications
Interactive Web &
Mobile Applications
BI / Reporting, Ad
Hoc Analysis
Statistical Analysis
DATASOURCE
Probes CDRs Control
Plane
Sensor
Data
Geo-
location
Data
Network
Data
User Plane
Exis*ng
ETL
Exis5ng
Hadoop
Cluster
Use Case / Data Analysis
14. Cardinality Perception
MME
Mul5-‐Vendor
Probes
Data
Proxy
Firewall
Scene
Voice,
SMS,
Data
(Mediated
&
Billed
CDRs)
Marvel
(Data
Analysis)
Other
Data
Sources
Data
Pipeline
Data
Fetcher
Data
Parser
Magnitude
(Data
Storage)
HDFS
NO
SQL
SQL
Cache
Management
Server
Manage
ment
Cluster
Manager
Component
Repository
Map
Reduce
Impala
(SQL)
Spark
• Enterprise Ready (Scale, Robustness,
Secure, Performance) Platform
• Utilising Enterprise Grade Open
Source Components
• Vendor Independent, Best in Class
component selection
• Each component is wrapped for simple
upgrades and version control
• Architected using Micro Services to
keep platform AGILE
Security
API
Open
Access
(Resbul
&
SOAP)
Access
Subscriber
Driven
Coverage
Sta5s5cal
Repor5ng
Data
Visualisa5on
15. Real Time
Decision
EngineData
Feeds
File
Sources
ETLEngine
API’s
Machine
Learning
Data
Hadoop
RealTimePrivacyFilter
(MSISDN,IPaddress,UAAgent,Domainsetc.)
Streaming
Sources
Process
Feeds
API
Interface
File
Handler
Stream
Monitor
API
Monitor
Encryption
&Enrichment
Data
Link
Data
Feed
File
Drop
Administration & Monitoring
Analysis Layer
Presentation Layer
Technology Overview
17. Data Sources
This is a sample text.
Insert your desired text
here.
This is a sample text.
Insert your desired text
here.
SAMPLE TEXT
This is a sample text.
Insert your desired text
here.
This is a sample text.
Insert your desired text
here.
This is a sample text.
Insert your desired text
here.
Data Sources
User Plane Data
• DPI
• Proxy
• Probes
Control Plane Data
• MME/GGSN
• Probes
• PCRF
Other Network Data
• Subscriber Location
• Probes
• Network Performance
CDRs
• Voice
• SMS
• Data
• Roaming
Reference Data
• Tariff Data
• Cell Site Reference
• Demographics
Other Data Sources
• Marketing Promotions
• Customer Feedback
18. DATA
Network
q Radio Analytics
q Congestion Analytics
q Video Analytics
q Location Based Analysis
q Network Performance
Business & Finance
q Cell Profitability
q Tariff Profitability
q Future Network Investment
Analysis
Customer Services
q Near RT customer issues
q Churn Prediction & Prevention
q Customer Loyalty
Marketing
q Customer Segmentation
q Segment of One
q Device & Tariff Upsell
q New Tariffs
q Content Upsell
q OTT impact
360° use of Big Data
19. Network
Conges5on
Hotspot
Analysis
Heat
Maps
Web
&
Video
Analy5cs
Performance
&
Capacity
Analy5cs
Profitability
Marke5ng
User
Segmenta5on
Sales
&
Upsell
Targe5ng
Tariff
Proposi5ons
Handset
&
Consump5on
Analy5cs
OTT
Service
Package
Analysis
Social
Media
Impact
Customer
Services
Churn
&
Reten5on
Proac5ve
Subscriber
Engagement
Billshock
&
Selfcare
Event
Monitoring
(fes5vals,
spor5ng
events)
Finance
&
Fraud
Fraud
Predic5on
Customer
Profitability
Targeted
Investment
Analysis
Board
Repor5ng
KPI
Metrics
Dashboards
Compliance
Security
Monitoring
Criminal
Ac5vity
Data
Privacy
Freedom
of
Informa5on
Requests
Compliance
Reports
IT
Data
Warehouse
Bill
Verifica5on
Business
Intelligence
Example use cases
21. Example Use Case Value
• Device
Analy5cs
–
Ability
to
iden5fy
and
monitor
“rouge”
device
in
the
network,
and
inves5gate
device
upsell
opportuni5es
• Data
&
Voice
QoE
–
Ability
to
monitor
QoE
across
the
network,
iden5fica5on
of
poten5al
op5miza5ons,
and
to
standardise
published
QoE
metrics
• Subscriber
Ac5ve
States
–
Ability
to
iden5fy
the
subscriber
migra5on
and
ac5va5on
paherns
based
on
the
prepaid
datasets(top-‐up
&
ac5va5ons).
22. Use Case: Device AnalyticsDevice Analytics
Type of Devices
• Smartphones
• Feature phones]
• IoT
Manufacturer Distribution
• Samsung
• Apple
• HTC etc.
Operating System Distribution
• Android
• iOS
• Other
Rogue Device Identification
• Non Standard Devices
• Poor Voice Experience
• Poor Data Experience
• Anomaly Detection
Device Market Performance
Device Up Sell Target
23. Application QoE
• Facebook, YouTube etc.
• Feature phones]
• IoT
Data QoE
• Latency
• Speed
• Traffic Analysis
Operating System Distribution
• Android
• iOS
• Other
Rogue Device Identification
• Non Standard Devices
• Poor Voice Experience
• Poor Data Experience
• Anomaly Detection
Voice Experience
• Call Drop Analysis
• Location based Call Experience
• Rogue Device Analysis
Use Case: Data & Voice quality of
experience
24. Prepaid Patterns
• Top-up distribution
• Up-Sell opportunities
Subscriber Activation Pattern
Distribution
• Promotion to activation co-relation
• Network Activation stats
Network Experience Co-
relations
• Churn to network experience
• Promotion to NT to Churn
Prepaid Promotion
Analysis
• Competitor Promotion co-
relation
• Subscriber churn analysis
Use Case: Subscriber Active
States
25. Data Discovery Workshop
Deployment of Cardinality’s
Perception Platform in the
cloud
Transfer/Load encrypted data
in Perception
Reports and Actionable
analytics
Proof-of-Concept (Cloud)
