Data Science Powered Apps for Internet of Things

1© Copyright 2016 Pivotal. All rights reserved.
Data Science-Powered Apps for
the Internet of Things
Chris Rawles1 and Jarrod Vawdrey2
1. Sr. Data Scientist in New York, New York
2. Sr. Data Scientist in Atlanta, Georgia

Today’s talk
1. A real-time data science app
A. The app: a live demonstration
B. How can a data scientist build a data science application?
C. Revisiting the app
2. Generalizing the framework: Solving new data science
challenges
A. Internet of Things – Creating a smart app to prevent oil spill disasters
B. Financial data - How can retail banks influence their cardholders’
behavior?

Today’s talk
challenges
behavior?

App

Today’s talk
challenges
A. Internet of Things – creating a smart app
behavior?

Training
app
Model
Scoring as
a service
Model
Training as
a service
Sensor
app
Scoring
app
Dashboard
app
Data science workflow: Movement classification
1. Sensor + Dashboard
2. Redis
3. Training app
4. Scoring app

here is my source code
run it on the cloud for me
- Onsi Fakhouri
@onsijoe
i do not care how

cf push
 CF determines app type (Java, Python, Ruby, …)
 Installs necessary environment
 Provisions and binds data services
 Creates domain, routing, and load balancing
 Continual app health checks and restarts

Data ingestion: Accelerometric data
 Accelerometric data streamed from
mobile phone at 15 Hz (15x / second)
 Other sensor data: gyroscopic data,
magnetometer data, lon/lat, etc.
Accelerometer axes

 For real-time applications, low-latency data ingestion into
the data store is essential
 WebSocket protocol - socket.io
– Mobile phone  Webserver
– Webserver  Dashboard
 socket.io  redis
Data ingestion
Training
app
Sensor
app

Data storage
 We are using a redis store for:
– Storing training data
– Model persistence
– Storing a micro-batch of scoring data
 Other storage systems include GemFire, HAWQ/Hadoop,
Greenplum Database, PostgreSQL, …

Modeling
Scalable machine learning applications in Pivotal
Cloud Foundry
1. Training app
2. Scoring app

Modeling – Training app
 Goal: build a data-driven model that learns accelerometric
motions associated with each activity
Feature Engineering
• Time-domain
transformations
• Fast Fourier Transform
analysis
Machine Learning
Classification Model
• Random Forest Model
using 2 second time
windows (30 samples)
…
Training data
Trained
model

Model building
 20 seconds per
training activity
 Two second moving
window on training
data
 Features: time-
domain summary
statistics and Fourier
transform coefficients

Model training approaches
1. Near-real-time model training
– Use small batches to train model
2. Real-time model training
– Online machine learning algorithm : continually update model
using each new data point
3. Offline model training
– Build a model offline using batches
– Useful for models requiring finer model tuning and calibration

Feature Engineering
• Time-domain
transformations
• Fast Fourier Transform
analysis
Machine Learning
Classification Model
• Random Forest Model
using 2 second time
windows (30 samples)
Trained model
Streaming input window
Model
Prediction
API Call
Model
prediction
PCF App:
Scoring app
• Real-time model scoring
• The dashboard initiates a request via
an API call and receives a model
prediction
{ "channel": "1234",
"label": ”walking",
"label_value": 0.746 }

1. Application auto-scaling
– As the data grows, the model scales
2. Application autonomy
– The model application is independent of other applications = faster
development iterations
– Faster development = rapid feedback loop
3. Multiple applications can access model scoring app
Operationalizing scalable data science applications
Model scoring as a service
Why?

Today’s talk
challenges
A. Internet of Things – creating a smart app
behavior?

App

Today’s talk
challenges
behavior?

Gene Sequencing
Smart Grids
COST TO SEQUENCE
ONE GENOME
HAS FALLEN FROM
$100M IN
2001
TO $10K IN 2011
TO $1K IN 2014
READING SMART METERS
EVERY 15 MINUTES IS
3000X MORE
DATA INTENSIVE
Stock Market
Social Media
FACEBOOK UPLOADS
250 MILLION
PHOTOS EACH DAY
In all industries billions of data points represent
opportunities for the Internet of Things
Oil Exploration
Video Surveillance
OIL RIGS GENERATE
25000
DATA POINTS
PER SECOND
Medical Imaging
Mobile Sensors

How can we use data
to help prevent
accidents like the Macondo
Disaster ?

Training
app
Model
Scoring as
a service
Model
Training as
a service
Sensor
app
Scoring
app
Dashboard
app
Data science workflow: Movement classification

Training
app
Model
Scoring as
a service
Model
Training as
a service
Sensor
app
Scoring
app
Dashboard
app
Data science workflow: Creating a smart app to
prevent oil spill disasters • Alert operator
• Send signal to control system
to change operating
parameters
• Replace old machinery
• Shut down plant

Training
app
Model
Scoring as
a service
Model
Training as
a service
Sensor
app
Scoring
app
Dashboard
app
Data science workflow: How can retail banks influence their
cardholders’ behavior? • Provide customized services
and promotions
• Next best offer
• Characterize and improve
customer satisfaction

Thank you
Questions and comments
crawles@pivotal.io

Data Science Powered Apps for Internet of Things

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (13)

Ähnlich wie Data Science Powered Apps for Internet of Things

Ähnlich wie Data Science Powered Apps for Internet of Things (20)

Mehr von VMware Tanzu

Mehr von VMware Tanzu (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Data Science Powered Apps for Internet of Things