Presented at #H2OWorld 2017 in Mountain View, CA. Learn more about H2O.ai: https://www.h2o.ai/. Follow @h2oai: https://twitter.com/h2oai. - - - Effective volume anomaly detection presents unique challenges when monitoring customer transaction volumes across thousands of platforms and systems. We overcome this by using H2O, building on open source tools, and delivering machine learning anomaly detection for enterprise scale. Hear how we model, visualize then automatically alert on anomalous Mobile app volumes in real-time. Donald Gennetten has over 15 years experience supporting digital channels in the Financial Services industry. In his current role as a Data Engineer for Capital One’s Monitoring Intelligence team, he leads a cross-functional group of Data, Business, and Engineering subject matter experts to deliver Advanced Analytics solutions for real-time customer transaction monitoring and issue detection. Rahul Gupta is a Data Engineer in Capital One's Center for Machine Learning, focusing heavily on back-end development and model creation. His primary efforts include building an Algorithmic IT Operations (AIOps) platform that utilizes a combination of batch and streaming data with Machine Learning capabilities to improve the stability of Capital One services and overall customer experience.

2. Donald Gennetten
Rahul Gupta
Data Engineer
Data Engineer

3. Using H2O for Mobile Transaction Forecasting &
Anomaly Detection

4. Problems are usually identifiable through
elevated failures or volume anomalies
Easy to detect, measure, and alert Hard to detect, measure, and alert
Elevated Failure Rate Low Volume Anomaly
No
AlertAlert

5. Why not set volume alerts?
Unlike failure alerts, volume-based thresholds
vary by event type, hour, minute, day of week,
week of the year, holiday, and much more.
100+ customer event types
x
24 hours/day
x
7 days/week
x
52 weeks/year
Over 873k distinct thresholds to calculate, set
and maintain.
Machine Learning should be used
when:
• You cannot effectively code the solution
• You cannot scale

6. Solving the problem required going
beyond modeling
Visualize/Alert Pilot
Develop
Platform
ModelingDefine DataIdentify Business Case
Our goal was to deliver Machine Learning for Production Monitoring that:
• Followed Governance Requirements
• Used Available Data Science and Machine Learning Resources
• Leveraged Platform Engineering and Open Source Technology
• Ensured Usability and Scalability

7. Sparkling Water allowed us to rapidly test
and deploy machine learning
• Sparkling Water combines the fast, scalable ML algorithms of H2O, the H2O Flow UI, Scala, and
Python with the capabilities of Apache Spark
• In-memory processing supports big data environment needs
• Spark + Python + Scala enables a unified coding pipeline
• Grid search options allow for greater efficiency
• Test models
• Optimize hyperparameters
• H2O Flow facilitates ad-hoc experimentation
• REST API is easily integrated into production software

8. GBM provided greater flexibility and
benefits over traditional methods
• Traditional time series techniques assume stationary data (no trends/seasonality), constant variance
over time
• Univariate time series consists of single, sequential observations over equal time increments
• GBM model accepts external explanatory variables
• # accounts having payment due
• Incidents
• Change orders
• Payment due dates
• GBM also enables data filtering/exclusion (e.g., incident data for training set)

9. We developed an open source, cloud-
based platform for rapid delivery
1 Retrieve volumes for training
2 Provide holiday and other static
data
3 Store forecast with
actual volumes
4 Detect & flag anomalies
Amazon S3 Sparkling Water InfluxDB Amazon EC2 Grafana
5 Display volumes, forecast & anomalies

10. What does it look like?
Monitoring teams are easily able to visually inspect forecasted and actual volumes in real-time
Forecasts are
available for future
dates to aid in capacity
planning
Now

11. What does anomalous volume look like?
Small changes in expected volume are easy to detect, measure, and alert
~12% of expected
events were missing
after a planned change
to the streaming data
platform
Alerts triggered due to lower than expected volume; Root cause analysis determined a platform
release was casing dropped data and a code roll back was required to resolve the issue

12. Does it improve incident detection times?
Anomaly detection alerts are sent ahead of escalation and detection times, including when other
alarms aren't triggered
Anomaly detected at
11:15 p.m. when Login
volumes spiked ~20k
higher than expected
Incident response teams were alerted at 11:17 p.m., more than 4 minutes before other incident
alarms

13. Solar events as a predictor?
Variation from predicted login volume was easily quantified during the August 21st solar eclipse;
Interest appears to have been lost within 15 minutes of totality
A
A. 12:06 p.m. EDT
(9:06 a.m. PDT)
the solar eclipse
starts in Salem,
Oregon
B. 2:41 p.m. EDT
(11:41 a.m. PDT)
totality begins in
Columbia, South
Carolina
C. 4:06 p.m. EDT
(1:06 p.m. PDT)
eclipse ends
B C
Variation from forecast