Almost every DevOps team struggles with which platforms and devices to test. With a constant flow of new devices, operating systems, and desktop browsers, plus ever-changing user conditions, knowing exactly what to test can be a challenge for web and mobile app teams. With constant change, how can you plan your tests strategically? Stop the test-coverage guesswork and begin a testing strategy based off the latest data. In this webinar, you’ll learn how to develop a test coverage strategy using industry data and device statistics from around the world. Attend this 45-minute webinar to learn: • Key market insights for 2019, including 5G, iOS 13, Android Q, and foldable phones. • How to develop a test coverage plan that reflects your actual users. • How to size a lab and calculate the ROI of test coverage. • The key mobile and operating system permutations to test against. • How to stay ahead of the market by testing on new devices, operating systems, and browsers.

1. 1 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
How To Develop a Cross-Platform Test Coverage Strategy

2. 2 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
2© 2018, Perfecto Mobile Ltd. All Rights Reserved.
Housekeeping
1 2
Ask us questions in
the Q&A panel!
Join our live poll
question.
3
We will send you the
recording and slides.

3. 3 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
Today’s Speaker:
Eran Kinsbruner
• Chief Evangelist and author at Perfecto
• Blogger and speaker
• 19+ years in development & testing
• Author of
“The Digital Quality Handbook”
and “Continuous Testing for
DevOps Professionals”
• Twitter: @ek121268

4. 4 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
Today’s Agenda
1
2
3
Key market insights for 2019.
How to develop a test coverage plan that reflects your actual users.
How to size a lab and calculate the ROI of test coverage.
4 Q&A

5. 5 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
Poll Question
The number of tests and the time it
takes for each one.
The time-windows it takes to complete
the test cycle.
The right number of unique platforms.
All of the above.
What is the most important factor in sizing a digital lab?

6. 6 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io

7. 7 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
Key Trends in Digital for 2019

8. 8 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
Growing Trends in Mobile

9. 9 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
Current Mobile Market Stats

10. 10 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
Introduction to Android Q (10)
Gives users more
control over
location.
Foldable device support
includes changes to
onResume and onPause to
support multi-resume and
notify your app when it has
focus.
New settings panel API
allows apps to show settings
to users in the context of
their app.
More privacy protections
and more control over
apps and access to shared
files prevents apps from
launching an activity while
in the background.

11. 11 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
Introduction to iOS 13
Dark Mode & UI Changes
New UI settings, “undo”
gesture, split view, app bar,
removal of volume changing
popup, and more.
User Productivity Changes
Better Siri support in driving
scenarios, iOS to remember
Wi-Fi connections based on
locations, merging MacOS
and iOS apps, iPad
enhancements, and more.
Bug fixes and
performance
Improvements
As in every iOS
release.
Removed Support
for Older Devices
iPhone 5S, SE, 6, 6S,
6+, iPad Mini 2, and
iPad Mini 3.

12. 12 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
The First Wave of Foldables Is Here
Z T E A X O N M H U A W E I M AT E X
L G V 5 0 T H I N Q ( + A C C E S S O R Y ) S A M S U N G G A L A X Y F O L D *

13. 13 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
The Evolution of Software Testing Toward AI
Source: Hackernoon
Manual Testing
1980 – 1990
Waterfall Methodology
Bulky Automation Tools
More Robust Automation Tools
+ Open Source Frameworks More About Scale
Autonomous Testing,
Machine Learning, and AI
1990 – 2000
Experimentation with
different development
approaches
2000 – 2010
Agile approaches,
faster release cycles
2010 – 2018
DevOps, continuous
testing, CI/CD
The Future
Collaborative,
smart testing

14. 14 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
How to Develop a Test Coverage Plan That Reflects Your Actual Users

15. 15 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
Test Coverage Fundamentals
Web Traffic AnalyticsMost Used Mobile Platforms
Selecting the Right Test Cases

16. 16 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
Test Coverage Fundamentals Continued
Source: Stat Counter

17. 17 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
Environment Conditions

18. 18 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
Recent Mobile
Coverage
Recommendations
– U.S, Q2 2019

19. 19 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
How to Size a Lab and Calculate
the ROI of Test Coverage.

20. 20 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
Step 1: Acknowledge Your Pipeline Testing Requirements
P L AT F O R M C O V E R A G E T H R O U G H O U T T H E D E V O P S P I P E L I N E

21. 21 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
Step 2: Gather Testing Productivity Metrics
• Measurable Metrics:
• Test suite size — Number of unique test cases (Unit, Regression, Non-Functional, etc.).
• Average time per test — Time in minutes (2-3 minutes is a best practice).
• Test execution — Number of hours a cycle should run (Nightly, per build).
• Soft Metrics:
• Platforms specific — Defects history, unique features support, etc.
• Analytics data
• Tests specific — Test flakiness and inconsistency.

22. 22 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
Step 3: Size Your Digital Lab
Coverage
Bucket
Number of
Unique Tests
(Regression
Suite)
Average Time
Per Test
Execution
Window
(Hours)
Test Execution
Time
(Serial)
Parallel Test
Execution
Requirement
Cost Avoidance
(Business Tester
Annual Salary
Input)
Essential
(Top 10)
150 3 Minutes 8 Hours 4500 Minutes
(75 Hours)
9 67 Hours Saved
($3,500 per
cycle)
Enhanced
(Top 25)
150 3 Minutes 8 Hours 11,250 Minutes
(187.5 Hours)
23 180 Hours Saved
($8640 per cycle)
Extended
(Top 32)
150 3 Minutes 8 Hours 14,400 Minutes
(240 Hours)
30 232 Hours Saved
($11,136 per
cycle)

23. 23 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
Generic Testing ROI and Measurements
• Test authoring (time to create test,
ease of use, script reusability).
• Test execution abilities (parallel testing,
connectivity, and integration
to standard tools and frameworks).
• Test results analysis time.
• Test maintenance simplicity.
• Software license cost per user/floating.
• Number of resources required for the above.
T R A C K T H E F O L L O W I N G : C R E AT I O N  E X E C U T I O N  L A B  A N A LY S I S

24. 24 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
Continuous Testing Cost Measurements — Example (DIY Web Testing)

25. 25 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
• Test coverage is more than just platforms.
• Web & mobile require constant assessment (market dynamics).
• Size the lab based on the right platforms and required
capacity/time constrains.
• In order to measure ROI of testing, focus on four pillars:
Summary

26. 26 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
Perfecto’s Smart Automation for Continuous Testing
SMART Test Execution
Fast and parallel test execution with
multi-team orchestration abilities and
management.
SMART Test Reporting
"Single pane of glass" provides
visibility and scales to support
millions of test results.
SMART Test Creation
Automated creation that matches your
team’s skillset (Appium, Espresso,
XCUITest, Quantum BDD).
SMART Cloud Lab
Always on and stable. Always up
to date. Supports all mobile OS
and platforms.
The Perfecto
human factor
increases your
chance to succeed.
Black Belt
Testing Experts
Training
Dedicated
Success Manager
24/7 VIP
Support

27. 27 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
About Perfecto

28. 28 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
Perfecto Is a Perforce Company
A G I L E
P L A N N I N G
C O D E M G M T. &
C O L L A B O R AT I O N
E M B E D D E D
C O M P O N E N T S
O P E R AT I O N S
T O O L S
A U T O M AT E D
T E S T I N G

29. 29 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io

30. 30 | How to Develop a Cross-Platform Testing Coverage Strategy perfecto.io
Thank You
Visit perfecto.io or follow us @perfectomobile