By Carson Underwood at SmartBear Connect.

1. Maximize Test Automation Initiatives
With a Risk Based Approach
Carson Underwood

2. Introduction
● 12 years in software development
● Started in game development
● Have been doing test automation for four years
● I automate my house (I might have a problem)
● Long time musician

3. Agenda
● What is Risk Based development and testing
● Benefits
● Types of risks
● Reducing risk with automation
● Quality Gates / Quality Debt
● Feedback / Metrics / Continuous Evaluation

4. What Is Risk Based Testing?
● Risk based testing is an approach that reduces
product & project risks
● Identifies the risks of a product or project
● Uses risk levels to guide the testing process
● Streamlines the test planning process
● Allows for better planning, reporting and high
quality software

5. Benefits Of Risk Based Testing
● Improved productivity
● Improved quality
● Clear metrics for test coverage
● Reduced redundancy
● Faster release cycles

6. Types Of Risks
Project Risks
● Organizational
○ Skill training, resources
○ Communication
○ Collaboration / Department Silos
● Technical
○ Business Requirements
○ Environments
○ Low quality code, data or tests
● Third Party
○ Third party services, contracts etc…
● Integration / Interoperability
Product Risks
● Failure-prone builds
● Potential to cause harm to the
company
○ Legal, loss of money, downtime
● Poor software characteristics
○ Functionality, reliability, usability,
performance
● Poor data integrity and quality
● Software that does not meet
requirements

7. Risk Ranking Levels
● Level 1: Spelling, minor UI issues, etc...
● Level 2: Inconsistent behavior, minor performance
issue
● Level 3: Negative testing, app misbehaves with
unexpected input
● Level 4: App does perform critical functions correctly
● Level 5: App crashes, complete loss of usability

8. Example Approach

9. Reducing Risk With Test Automation
● Create “Quality Gates” in CI / CD environments
● Reduce time to feedback
● Automate appropriate tests
● Increase coverage
● Staggered test types
● Automate along side development

10. Example Project
● Sign In for our POS Changed drastically
● Determined tests that were no longer valid
○ Updated the high risk tests first
● Automated new tests as development work finished
● Branched automation work with development
● Skipped automating low risk tests for the first
release

11. Quality Gates

12. Quality Gates: Code Quality
● Continuous monitoring of overal code quality
● Use tools to monitor commits & provide code
reviews
● Create unit tests
● Provide development, testing, and production
environments

13. Quality Gates: Integration Tests
● Verify expected services are running
● Verify application is responsive
● Perform data driven functional API testing
● Ensure needed eternal services are available

14. Quality Gates: UI & Security /
Performance
● Run high level functional UI tests as smoke tests
● Ensure these pass
● Run a suite of full UI regression tests based on risk
● Run security scans for each build
● Run light load tests to ensure performance has not
decreased

15. Reduce Time To Feedback
● Create high level automated smoke tests
● Include only high risk areas of the application
● Run with each build / deploy
● Provides immediate feedback to development /
stakeholders
● Automate in dev branch for new features / changes

16. What To (and not to) Automate
● Automate high risk tests - Level 4 & 5 Risks
○ Data integrity
○ Potential to lose customers / cost company money
● Security tests
○ Security tests at multiple levels
○ Also High Impact to business / customers
● Do not automate subjective tests - Level 1 risks
○ Appearance
○ Frequently changing tests that cause false positives
○ Don’t increase the risk of Automated tests breaking for low ROI

17. Reduce Automation Risk
● Automate appropriately to reduce risk of false positives
● Create robust frameworks
○ Environment and data agnostic
○ Reusable test assets
○ Report good metrics
● Reduce redundancy to reduce project time risk
● Ensure tests can be cherry picked at run time
● Happy path tests vs edge case tests

18. Metrics
● Use test case management and issue tracking tools
○ Number of and lists of tests ran
○ Number of failed tests
○ Reasons for failures
○ Failure history
● Opened issues, rejected issues, issues found post release
● Link issues to tests, and components of the application
● Differentiate between environment, data, application, test case...

19. Continuous Evaluation
● Monitor and report metrics
● Build test plans based on commonly failed scenarios
● Exclude Low risk / stable tests if there is no code change
● Increase coverage but not time
● Constantly re-evaluate risk ratings based on failures / passes

20. Conclusion
● Risk Based testing is about minimizing risk, not eliminating
● Use risk rankings to plan development and testing
● Quality Gates and Metrics are a must to reduce Quality Debt
● Robust and reliable automation is a must
● Questions?