The mobile app landscape is incredibly fragmented across iOS and Android markets. And all OS/device combinations need to be tested thoroughly for the best end user experiences. So, where do you start? Simulators and emulators are fast, but there’s a tradeoff — they can’t catch as many bugs as real devices. Organizations need a risk-based approach to enjoy the power of both platforms. Using both real and virtual devices allows you to achieve the scale of testing and app quality you seek. Learn how to leverage the strengths of each and consider the level of risk you are willing to take with escaped defects. Join this webinar, led by Perfecto’s Chief Evangelist Eran Kinsbruner and VP of Product, Tzvika Shahaf, to understand how to build a strong mobile testing strategy that mitigates risk of escaped defects by using the correct mix of real and virtual devices. Join this webinar to: - Understand the differences between testing on simulators, emulators, and real devices. - Learn platform considerations unique to testing on Android and iOS devices. - Get recommendations on what platforms to test on through each stage of the SDLC. Learn more at https://www.perfecto.io.

1. Perfecto by Perforce © 2020 Perforce Software, Inc.
How to Create a Risk-Based Testing Strategy
With Simulators, Emulators, & Real Devices

2. 2 | Advanced Codeless Testing for Web Apps
Today’s Speakers:
Eran Kinsbruner
• Chief Evangelist and author at Perfecto
• Blogger and speaker
• 19+ years in development & testing
• Author of “The Digital Quality Handbook”
and “Continuous Testing for DevOps
Professionals”
• @ek121268
• https://www.linkedin.com/in/eran-kinsbruner-4b47a81/
Tzvika Shahaf • VP Product Management at Perfecto
• Software professional with over 12 years in product
management, development, and testing
• Expert in implementing and advising on test strategies for
Fortune 500 enterprises and leading tech companies
• http://linkedin.com/in/tzvikashahaf/
• @Tzvika_Shahaf

3. perfecto.io3 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Today’s Agenda
1
2
Test automation coverage considerations
Definitions of emulators and simulators
3 Differences between iOS/Android virtual platforms
6 Q&A
5 Live demo: testing in parallel and at scale on virtual platforms
https://ter.li/y7jfmv
4 Strategy for testing on both real and virtual platforms

4. perfecto.io4 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Balancing Test Automation Coverage
TEST SCENARIO ONE
Test Case – 1 Test Case – 2 Test Case – 3 Test Case – N
2 sides of the same coin

5. perfecto.io5 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Digital Platform Coverage Strategy
Android OS Market Share
Android 6.x (Marshmallow) Android 7.x (Nougat) Android 8.x (Oreo)
Android 9 (Pie) Android 10 (Q)

6. perfecto.io6 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Real Devices Run Different Custom OS Versions
Device Model Custom OS Implementation Versions Base Stock OS
Samsung Galaxy S20 Ultra One UI 2 Android 10
Samsung Galaxy S7 TouchWiz UI Android 8.0
Samsung S8 One UI Android 9.0
Samsung Note 9 One UI 2 Android 10.0
Google Pixel 4 XL STOCK Android 10
Huawei Mate 20 Pro EMUI 9.1 Android 9.0
LG G8 ThinQ LG UX 9.0 Android 10.0
Oppo Reno 5G Color OS 6 Android 10
Huawei Y9 Prime Magic UI 2.1 Android 10
Huawei Enjoy 20 Pro EMUI 10.1 Android 10

7. perfecto.io7 | Perfecto by Perforce © 2020 Perforce Software, Inc.
The Agile Testing Manifesto
T H E D E V O P S M A N I F E S T O
How to determine test value?
 Continuous testing over testing at the end.
 Embracing all testing activities over only automated functional testing.
 Testing what gives value over testing everything.
 Testing across the team over testing in siloed testing departments.
 Product coverage over code coverage.

8. perfecto.io8 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Virtual Platform Definitions
• Simulators
• iOS (Apple)
• A simulator is software that imitates the workings of
software on your system that was made for mobile.
• Mobile simulators can be used in app testing and cross-
browser or responsive testing to simulate a mobile device
and running your app or website in it.
• Emulators
• Android (Google)
• An emulator is software that imitates the working of
software on your system that was made for mobile.
• Emulators also take into consideration the hardware part
of the mobile device (memory, processor, battery
performance, etc.).
Real Device Behavior

9. perfecto.io9 | Perfecto by Perforce © 2019 Perforce Software, Inc.
Differences Between Android & iOS Virtual Platforms

10. perfecto.io10 | Perfecto by Perforce © 2019 Perforce Software, Inc.
Real Device Issues — Logs, Analysis, and Reporting Can Help
It is not working on my Samsung s10
BAD UPDATE !!! Now the APP does not open on my Xiaomi Mi A1
Source: App Annie

11. perfecto.io11 | Perfecto by Perforce © 2019 Perforce Software, Inc.
Real Device Issues: COVID-19 PathCheck App

12. perfecto.io12 | Perfecto by Perforce © 2019 Perforce Software, Inc.
Working Together: Test Data is Generated by Multiple Personas
D E V Q A / S D E Ts N F T/ O t h e r

13. perfecto.io13 | Perfecto by Perforce © 2020 Perforce Software, Inc.
US Mobile Carrier Performance, Q1 2020 Carriers Vary
— Test Against Edge Performances
Source: OpenSignal

14. perfecto.io14 | Perfecto by Perforce © 2020 Perforce Software, Inc.
How to Keep Your Test Coverage In Sync With Your End Users

15. perfecto.io15 | Perfecto by Perforce © 2019 Perforce Software, Inc.
Perfecto’s Mobile Testing Pyramid Combo
Testing Pyramid Within Your DevOps Pipeline Requirements
U N I T T E S T S
M A N UA L
T E S T S
Testing isolated units of code.
I N T EG R AT I O N T E S T S
Testing across integrations of more than one system.
U I T E S T S
Testing through a user interface.
Per code commit/ Post commit
Build Acceptance/Regression Testing
E2E Testing
Exploratory/Manual Testing
Dev
Triggered
Mix Real Devices & Emu./Simu.
CI
Triggered+
Real Devices Only
CI
Triggered
Real Devices OnlyProd. Monitoring
Non Functional Testing:
Performance (UX)
Testing/Security
Real Devices Only

16. perfecto.io16 | Perfecto by Perforce © 2019 Perforce Software, Inc.
Leverage Tagging for Test Impact Analysis (TIA) and Fast Debugging
Tags are key enablers for slicing
and dicing test data, reducing
MTTR (Mean Time To Resolution),
and making data-driven decisions.

17. perfecto.io17 | Perfecto by Perforce © 2019 Perforce Software, Inc.
Considerations
for Using iOS and
Android Virtual
Platforms vs. Real
Devices
Stage and activity in
the software
development cycle
Dev
Test
Debug
Design
Cost avoidance
Coverage
Platforms
Real user
conditions
Test types
Functional/Regression
/Integration Testing
Unit
Non Functional
Production Monitoring

18. perfecto.io18 | Perfecto by Perforce © 2019 Perforce Software, Inc.
Requirements for Perfecto’s Mobile App Testing
Pyramid Combo Within Your DevOps Pipeline
Early SDLC stages fit the use of
virtual platforms
Easy setup
Fast feedback
Cost effective
Unit testing types
Build acceptance and
integration testing should
involve a mix
Advanced scenarios across iOS/Android
are limited on virtual platforms (no app
store, diff HW)
Real user experience required to guide
next software development phase
Such phase requires a combination of
environments (real & virtual)
UI and UAT phases fit the use
of only real devices
Real UX feedback
Greater mix of platform coverage against
official OS versions (Index)
Risk-free release gateway decision
Please Refer to Perfecto’s Test Coverage Index

19. perfecto.io19 | Perfecto by Perforce © 2019 Perforce Software, Inc.
Coverage Index Across Geographies

20. perfecto.io20 | Perfecto by Perforce © 2019 Perforce Software, Inc.
Pipeline Testing Coverage Example
Virtual Platforms
Mix of
Virtual Platforms & Real Devices
Only Real Devices

21. LIVE DEMO – Virtual Platform Testing With Perfecto

22. Thank You!
Visit perfecto.io or follow us @perfectomobile