Development and test organizations face major challenges when building robust automated tests around their mobile applications. With limited testing resources and increasingly more complex projects, stakeholders worry about the risk and quality of mobile products. So how do you plan a mobile test automation project to prioritize testing resources and efforts? Tarun Bhatia used big data analytics to understand where customers spend most of their time on their apps out in the wild. See how you can analyze massive amounts of mobile usage data to create an operational model of carriers, devices, networks, countries, and OS versions. Based on real-user data, they developed automation strategies to create better tests and focus on the right priorities. Learn how you can use big data analytics to apply mobile automation in areas of continuous integration, performance, benchmarking, compatibility, stress, and performance testing.

1.
W8
Session
4/15/2015
1:00
PM
"Mobile Test Automation with
Big Data Analytics"
Presented by:
Tarun Bhatia
Microsoft
Brought
to
you
by:
340
Corporate
Way,
Suite
300,
Orange
Park,
FL
32073
888-­‐268-­‐8770
·∙
904-­‐278-­‐0524
·∙
sqeinfo@sqe.com
·∙
www.sqe.com

2. Tarun Bhatia
Microsoft
Tarun Bhatia is a technical program manager in charge of driving the best breed
of performance measurements and analysis for Microsoft Online Office Division.
Tarun leads innovative strategies—analytics, performance, benchmarking, and
compatibility—and guides the team to create an effective, reliable, and robust
monitoring architecture. With more than seven years of software development
experience in quality and service assurance, Tarun shows that taking initiative
and thinking outside the box can deliver big results—both personally and for the
company.

3. 4/8/15
1
Tarun Bhatia
Mobile Test Automation Using Big Data Analytics
Introduction
quality
as·∙sur·∙ance:
A
program
for
the
systema<c
monitoring
and
evalua<on
of
the
various
aspects
of
a
project,
service,
or
facility
to
ensure
that
standards
of
quality
are
being
met
Source:
hCp://www.merriam-­‐webster.com/
dic<onary/quality%20assurance

4. 4/8/15
2
Staged Rollout with Active Monitoring
• Crash
Reports
• User
Reviews
20%
User
Base
• Crash
Reports
• User
Reviews
50%
User
Base
• Crash
Reports
• User
Reviews
100%
User
Base
Manage
Analyze
ExtractValue
Value
What is Big Data ?
MB, GB,TB, PB
Records
Transactions
Tables, Files
Volume
Batch
Near-time
Real-time
Streams
Velocity
Structured
Unstructured
Semi-Structured
All theAbove
Variety
Source: Celent
The 4Vs
of Big
Data

5. 4/8/15
3
Data Everywhere
Trends in Tech Salary Reaffirm
Source: http://marketing.dice.com/pdf/Dice_TechSalarySurvey_2015.pdf

6. 4/8/15
4
“
”
If you think you can,
or if you think you can’t,
you are correct. – Henry Ford
Question
Your confidence level in current mobile automation
architecture?
Cost of Finding Bugs
0
20
40
60
80
100
120
140
160
Req Design Code UnitTesting Integration
Testing
System
Testing
Test Prod
Cost

7. 4/8/15
5
How it Starts!
Stage 1
• Company needs mobile presence
• They hire Mobile Devs andTesters (usually manual)
Stage 2
• App becomes too complex to cover all the permutations via
manual testing
• Company hiresAutomation Engineers (SDET) and are told to
“automate everything”!
Stage 3
• Full-on effort to catch-up and automate all features
• SDET burnout!
Creating a Plan
Successful
Automation
Plan
Device Lab
Automation
Framework
Prioritize
FeatureTest
Cases
Stress/
Performance/
Other Additional
Testing

8. 4/8/15
6
Creating a Device Lab
Creating a Device Lab (Using Big Data)
Total # of
Devices
Devices with
most # of
reported bugs
Your most
Popular
Devices
Time box and
add bug to
your backlog
Buy/Loan/
Rent device
and bring it in-
house

9. 4/8/15
7
Creating a Device Lab
30%
17%
13%
5%
4%
4%
3%
3%
3%
3%
2%
2% 2%
2% 2%
5%
Apple
LG MS770
Samsung Galaxy SIII
Microsoft
Coolpad Quatro 4G
ZTE N9210
Samsung GalaxyAdmire 4G
Droid RAZR
Samsung Galaxy Note II
LG Esteem
LG MS870
SamsungAdmire
Samsung Epic 4G
Samsung Galaxy SII
Samsung Omnia II
Other
Total # of Devices > 1850!!
Pick an Automation Test Framework

10. 4/8/15
8
Prioritize
KPI
Customer
Usage Data
Finance
(Revenue
Stream)
Data
Marketing/
Social Data
User Usage Pattern
Home Screen,
40%
1 Detail Screen,
20%
2 Detail Screen,
15%
3 Detail Screen,
10%
All OtherValues,
15%

11. 4/8/15
9
Tests
Real User, Marketing and Finance
Data
Stress
ServerVs. UI Data
New Features
Performance
System Under
Test
Production data
Test Results
RunTests
Quality
Assessment
Stress Testing
— Find
Resource
Leaks
— Find
App’s
Capacity
and
Capabili:es
— Find
Memory
and
Ba>ery
Consump:on
Trends

12. 4/8/15
10
Server Vs. UI Testing
Server
Client Test Framework
• Verify
data
is
in-­‐sync
during
tes:ng
• Ensure
no
data
loss
during
test
progress
• Detect
UI
TTL
(Time
to
Load)
on
devices
under
various
condi:ons
Performance Testing (Analyze and Record KPIs)

13. 4/8/15
11
Effective Testing
Write
Once,Test
Anywhere
Active
Monitoring
Test Re-Use
Performance
Availability
Conclusion