Teams generate huge amounts of test reporting data daily. Unfortunately, most open source frameworks do not come with reporting, making the analysis mostly manual and tedious. That’s where a test reporting platform can help you. To find bugs faster, you need something that can filter out noise and false negatives. That way, you can focus on the real issues for a quick MTTR. An efficient test reporting platform helps you find and fix bugs quickly and release higher-quality apps faster. Join this webinar, led by Perfecto Chief Evangelist Eran Kinsbruner and VP of Product Management Tzvika Shahaf, to: - Understand how efficient analysis helps you test faster. - Learn what to look for in an effective reporting platform. - See a live demo of test failure analysis and debugging.

1. Perfecto by Perforce © 2020 Perforce Software, Inc.
Fast Data, Fast Delivery
How Smart Analysis Accelerates App Testing

2. Today’s Speakers:
Eran Kinsbruner
Tzvika Shahaf
• Chief Evangelist and author at Perfecto
• Blogger and speaker
• 19+ years in development & testing
• Author of “The Digital Quality Handbook”
and “Continuous Testing for DevOps
Professionals”
• @ek121268
• https://www.linkedin.com/in/eran-kinsbruner-4b47a81/
• VP Product Management at Perfecto
• Software professional with over 12 years in product
management, development, and testing
• Expert in implementing and advising on test strategies for
Fortune 500 enterprises and leading tech companies
• http://linkedin.com/in/tzvikashahaf/
• @Tzvika_Shahaf

3. perfecto.io3 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Today’s Agenda
1
2
The need for smarter reporting and analysis
How efficient analysis can help organizations test faster
https://www.perfecto.io/resources/state-test-automation
3 Pillars of an effective test reporting platform for mobile and web app testing
5 Q&A
4 Live demo: Failure analysis and noise reduction in large test data

4. perfecto.io4 | Perfecto by Perforce © 2020 Perforce Software, Inc.
T H E D E V O P S M A N I F E S T O
The Agile Testing Manifesto
How do you determine test value?
 Continuous testing over testing at the end.
 Embracing all testing activities over only automated functional testing.
 Testing what gives value over testing everything.
 Testing across the team over testing in siloed testing departments.
 Product coverage over code coverage.

5. perfecto.io5 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Working Together: Test Data Is Generated by Multiple Personas
D E V Q A / S D E Ts N F T/ O t h e r

6. perfecto.io6 | Perfecto by Perforce © 2020 Perforce Software, Inc.
OPTIMIZED
DELIVERY PIPELINE Innovation
Throughput
Quality of
Output
Time Cost
Today — DevOps Process Probably Looks Like This
Process
Impact
Organizational
Impact
2-3 WEEKS 1-3 WEEKS
❌ Unstable
❌ Labor-intensive
❌ Cluttered
❌ Slow
• Manual testing.
• Unreliable and flaky
executions.
• Long time to analyze results
and fix issues.
• Slows time to release.
• Increases risk and reduces
flexibility during the cycle.
• Reduces innovation time
versus bug fixes time.
• QA and security often not
part of the daily cycle.
• Testing holds back
innovation.
”End of
Cycle”
Testing

7. perfecto.io7 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Contributing Factors: Common Challenges in Test Automation
Automation Skillset
Merging Tests Into Pipeline
Escaped Defects Due to Noise
Advanced Automation Scenarios
Challenge to Set Up Test Environment Designed for
Testability
Time Spent to Analyze Reports
Testing Is Done Separately
Test Maintenance & Digital Platforms Coverage Lack of Time to Automate
It all boils down to people, processes, and/or technology.

8. perfecto.io8 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Minimum capabilities needed:
”Single pane of glass" for visibility and fast triaging
Scripts & FW Issues (40%)
Backend Issues (30%)
Lab Issues (15%)
Orchestration Issues (15%)
OBJECTS (40%) | CODING (20%) | TIME (20%) | OTHER (20%)
AVAILABILITY (40%) | NETWORK (20%) | DATA (30%) |
OTHER (10%)
NETWORK (30%) | STABILITY (30%) | CONFIGURATION (20%) | LOCK (10%) |
OTHER (10%)
IN-USE (40%) | DISCONNECTED (30%) | LICENSES (20%) |
OTHER (10%)
40%30%
15%15%
What’s wrong with
my scripts?
What’s wrong with
my backend?
What’s wrong with
my lab?
What’s wrong with
my executions?
Test Reporting & Analysis — Common Test Automation Pitfalls

9. Real Life Test Automation Failures & Recommendations

10. perfecto.io10 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Create Your Own Failure Reasons

11. perfecto.io11 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Capability Not Supported
Ensure you properly set WebDriver
desired capabilities and that such a
configuration is supported.

12. perfecto.io12 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Element Not Found
Debug your scripts across multiple platforms,
use strong object locators — exclude test upon
recurring failures for stability.

13. perfecto.io13 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Device Not Found/Device In Use
Ensure using dynamic device searches vs. hard-
coded devices to minimize ”race condition” for
platforms.

14. perfecto.io14 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Leverage Tagging for Test Impact Analysis (TIA) and Fast Debugging
Tags are key enablers for slicing and dicing test data,
reducing MTTR (Mean Time To Resolution), and
making data-driven decisions.

15. perfecto.io15 | Perfecto by Perforce © 2020 Perforce Software, Inc.
U N I T T E S T S
M A N UA L
T E S T S
Testing isolated units of code.
I N T E G R AT I O N T E S T S
Testing across integrations of more than one system
U I T E S T S
Testing through a user interface.
Per code commit/Post commit
Build Acceptance/Regression Testing
E2E Testing
Exploratory/Manual Testing
Simulator/Emulator
Dev
Triggered
Mix Real Devices & Emu./Simu.
CI
Triggered+
Real Devices Only
CI
Triggered
Real Devices OnlyProd. Monitoring
Non Functional Testing:
Performance (UX)
Testing/Security
Perfecto’s Mobile Testing Pyramid ComboTesting Pyramid Within DevOps Pipeline Requirements
Real Devices Only

16. perfecto.io16 | Perfecto by Perforce © 2020 Perforce Software, Inc.
It is not working on my Samsung s10
BAD UPDATE !!! Now the APP does not open on my Xiaomi Mi A1
Real Device Issues — Logs, Analysis, and Reporting Can Help
Source: App Annie

17. Pillars for Effective Test Reporting and Analysis

18. perfecto.io18 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Pipeline Example: What Good Looks Like
Efficient Reporting Allows You to Meet Testing Time Windows
Relevant Unit Testing
High-Value Smoke Testing:
Functional, API, Integration, Component, and Security Testing
Regression on Real Devices:
Mixed Functional, Non Functional, Performance, and Security Testing

19. perfecto.io19 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Perfecto’s Reporting and Analysis — Key Features

20. perfecto.io20 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Live Execution
Dashboard Allows
Realtime Monitoring
of Test Activities

21. perfecto.io21 | Perfecto by Perforce © 2020 Perforce Software, Inc.
Rich and Actionable Test Reporting Artifacts and Evidence
Device Logs, Accessibility Reports, Screenshots,
Videos, Assertions Reports, Network HAR Files
Can Expedite Defects Resolution

22. perfecto.io22 | Perfecto by Perforce © 2020 Perforce Software, Inc.
v
K E Y C O M P O N E N T S
Continuous Testing Visibility — Fast & Smart Feedback
Cross-platform visual
validation dashboard
allows quick understanding
of functional/UI defects.
Executive overview of
testing throughout the
Continuous Integration
pipeline in real-time and
with trends.
Heatmap dashboard for
risk/focus area mapping
that provides executive
notion of emerging issues
in testing.
ANALYSIS

23. perfecto.io23 | Perfecto by Perforce © 2020 Perforce Software, Inc.
v
K E Y C O M P O N E N T S
Continuous Testing Visibility — Fast & Smart Feedback
Noise reduction through
automation detection and
classification (AI-based) of
errors/failure.
Report Library list for
effective triaging (slicing
and dicing of big data).
Single test report
for detailed root-
cause-
analysis with a rich
set of artifacts.
ANALYSIS

24. perfecto.io24 | Perfecto by Perforce © 2020 Perforce Software, Inc.
v
K E Y C O M P O N E N T S
Create Your Own Failure Reasons
System-supplied
failure reasons.
Overview of how
to add a “custom”
failure reason.
Tooltip shows ID
of non-system
failure reasons.
ANALYSIS

25. Live Reporting Platform Demo

26. perfecto.io26 | Perfecto by Perforce © 2020 Perforce Software, Inc.
The 4 Key Pillars of Continuous Testing in DevOps
Automation for effective use of Time, Tools, & Resources
This is where value is being
realized, quality is improved
Unified functionality + security approach serves DevOps with greater effectiveness.
Automated test analysis
• Fast feedback
• Root cause analysis
• Risk based coverage
Test creation and
maintenance
• Authoring tool
• Validations
• Accuracy
• Coverage

27. Thank You!
Visit perfecto.io or follow us @perfectomobile