Analytics-driven testing uses data on user behavior and software usage to prioritize testing efforts on the most critical areas. This allows testing teams to significantly reduce regression testing time by focusing on the most popular features and environments rather than exhaustively testing all possible combinations. Analytics can provide insights into actual usage patterns to determine the optimal risk-based testing strategy and achieve high test coverage within constrained timelines.
3. 7y in HPE Software
in various managerial positions
Started in testing
Today:
Inbound Product
Manager
StormRunner Functional
ABOUT ME
4. AGENDA • Applications Development Overview
• Today’s Testing Challenges
• What are the risks?
• Real World Examples
• How can you empower your testing by
using Analytics
5. The World of Software is changing
Mega-trends create demands on modern applications
MODERN APPLICATION DEVELOPMENT
Reduce costs
Increase customer attraction/retention
Increase the value of your brand
Get to market faster
300,000
tweets
200 million+
emails sent
220,000
new photos posted
50,000 apps
downloaded
$80,000
in online sales
72 hours
of new video content uploaded
2.5 million
pieces of content shared
Agile
Every minute*…
* Source: “The Data Explosion in 2014 Minute-by-Minute”by ACI InformationGroup.
http://aci.info/2014/07/12/the-data-explosion-in-2014-minute-by-minute-infographic/
Mobile
Cloud
Digital
DevOps
13. Example • My product: 30 features
• Supported Matrix: 10 environments
(OS/Browser)
• My testing team: 10 testers
• Testing a feature on an environment = 1 day
• Target regression cycle: <3 days!
14. 30 features X 10 combinations = 300 days
300 days / 10 testers = 30 days
15. Google Analytics • A freemium web analytics service offered by Google
that tracks and reports website traffic.
• Google launched the service in November 2005
after acquiring Urchin.
• Google Analytics is now the most widely used web
analytics service on the Internet.
• Note: I’m not affiliated with Google Analytics in
anyway
* Source: https://w3techs.com/technologies/overview/traffic_analysis/all
16. Environment
Optimization • Focus on the majority of combinations (OS x
Browser)
• Reduce your regression risk to a minimum
• Wait for customer feedback on the other areas
20. User Behavior • Focus on the majority of the functionality
• Target the most active areas
• Wait for customer feedback on other areas (% of escaped
defects)
• Calculated risk(!)
24. Building a risk
calculator • Test everything (100%) = 300 days
• All functionality, top combinations = 60 days
• Main features, top combinations = 24 days
• Add additional levels based on your needs and
time
25. Building a risk
calculator
Top flows
1 OS +
Browser
Main
Functionalitie
s
top
combinations
All
Functionalitie
s
top
combinations
All
Functionalitie
s
Additional
combinations
Test
Everything
50%
75%
90%
100%
95%
28. Don’t have
analytics? • Use market analysis and statistics
• https://netmarketshare.com/
• https://www.w3schools.com/browsers/
• Many more!
• Conduct annual user survey
• CustomersValidations – before & after
30. Re-use calculator for
Automation
Top flows
1 OS +
Browser
Main
Functionalitie
s
top
combinations
All
Functionalitie
s
top
combinations
All
Functionalitie
s
Additional
combinations
Test
Everything
50%
75%
90%
100%
95%
CI
Nightly
Full
regression
32. Summary • Don’t test everything!
• Get to know your users
• Calculate your risk
• Measure your un-tested areas
• % of Escaped defects
• Customer Satisfaction