The challenge for every product is to ship bug-free code as often as possible. Whether you are an early stage startup with a pilot application or a large corporation with myriad services, you’re dealing with this problem every day. We usually end up with either too little or too much testing and it’s hard to find the sweet spot. Too little testing and you have bugs and application instability, leading to time spent fixing bugs and manually regression testing your apps. You’re asking yourself, “isn’t there an easier way to do this?” Too much testing and you have slow release times and high automation maintenance costs. In this scenario, you’re asking yourself, “are the bugs I’m catching worth the time I’m spending maintaining this code?” In this webinar, software engineer Kate Green will go over a framework for evaluating your testing situation in order to find your organization’s sweet spot. Key Takeaways - Understanding where you are today - Identifying weak, brittle, or buggy parts of your application - Figuring out where to test first, and with what types of tests - How to pare down an excessively large automation suite Measuring test effectiveness

1. Just Enough
(Automated) Testing
Kate Green
@4kategreen
kategreen.codes

2. What Will I Cover?
All testing budgets are smaller than we want. Let’s use those resources wisely.
We will talk about:
Methods of analyzing your system
Prioritizing testing of different issues
Applying the correct test type for the problem
Novel testing approaches
@4kategreen
kategreen.codes
•
•
•
•

3. Company Risk Tolerance Profile
Higher Tolerance Lower Tolerance
Industry
Stage
Goals can give insights into particular problems
1. Increase quality to retain customers
2.
3.
4.
This gives you an idea of what
to expect from your organization,
it may not reflect reality!
@4kategreen
kategreen.codes

4. Brittle Code
Code that Changes Often Has Privacy or Security Concerns
High Traffic Areas
Where to Focus Your Efforts
Places with lots of bugs
Code with lots of dependencies
Code with hard-coded values
Places your team experiments
Formulae that change regularly
Login
Sign up
Password Changes
Search
Sorting and filtering
Checkout
API endpoints that handle user data
Forms or other user input fields
Permissions and role-based access control
@4kategreen
kategreen.codes

5. Ways to Analyze Your System
Bugs
Code
Traffic
Customer Feedback
Tools
Find Code Smells
Dependency Vulnerabilities
Current Test Focii and Coverage
Static Code Analysis
Unit Test Coverage Reports
Find Themes
Understand Current Fest Focii (manual and automated)
Business Analytics Software
Monitoring
Heavily-Trafficked Parts of the Application
Understand User Workflows
Find Usability Issues
User Research
Business Analytics Software
@4kategreen
kategreen.codes
1

6. Find the Risky Parts
Brittle Code Code that Changes Often
Has Privacy or Security Concerns High Traffic Areas
Identify parts of your system that fall into each category
@4kategreen
kategreen.codes

7. Prioritize Your Risks
The next step is to prioritize the risks you identified on the last page. There are two ways to evaluate the
risk. First, evaluate the severity. Then, evaluate simplicity of fixing the issue. Evaluate on a scale of 1-10.
Risk SimplicitySeverity
1.
2.
3.
4.
Average
@4kategreen
kategreen.codes

8. Match Your Testing to Your Problems
End-to-End Testing
API Testing
Integration Testing
Unit Testing
Test Speed Maintenance & BrittlenessUse Cases
verify the app as a
whole works correctly
slow
fast
fast
fast
high
low
medium:
coupling to the system is
a risk
low to medium:
mocking and fixtures
increase brittleness and
maintenance
test related sections of
the system
demonstrate correctness
of functions, methods,
business logic
verify data layer requests
and responses
@4kategreen
kategreen.codes

9. Creative Test Approaches
To minimize dependencies on fixtures, try Property-Based Testing. Property-based testing applies many
random inputs based on expected types.
To get a better look at the effectiveness of unit tests, look at Mutation Testing. It applies source code
variants to your unit tests and documents which cases were missed.
To make API Testing more comprehensive, try Consumer-Driven Contract Testing. Testing with contracts
enables to test both the request and response parts of an API call.
If your testing requires creativity, then you need to evaluate which parts of your system should be
automated and which should be manual.
@4kategreen
If your needs are around observability or tracking performance, you should look at Monitoring.
kategreen.codes

10. Apply Test Types to Your Problem Areas
Problem Test Type Why?
1.
2.
3.
4.
@4kategreen
kategreen.codes