The data tsunami is coming—or maybe it’s already here. Data science, big data, and machine learning are the buzzwords of the day. Data is changing our products and the way we build them, so we should also change the way we verify our products. In a world of increasing connectivity and accelerated deadlines, data can provide an edge. But what role should data play in assessing the quality of software? Where does it make sense to use data, and where is it inappropriate? Steve Rowe covers the history of how data fits into testing, explains why data is an important tool to have in your quality toolkit, and presents strategies for adding data to your testing plans and using it more effectively in your testing.

1. T10
The Criticality of Test Data
10/19/2017 1:30:00 PM
Amp Up Your Testing by Harnessing
Test Data
Presented by:
Steve Rowe
Microsoft
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888-­‐268-­‐8770 ·∙ 904-­‐278-­‐0524 - info@techwell.com - https://www.techwell.com/

2. Steve Rowe
Microsoft
Steve Rowe is a graduate of the University of Illinois and a twenty-year veteran at
Microsoft. He spent most of that time in the testing and quality organizations
around Windows and worked on multimedia, helping bring DVD playback to the
PC, introducing Media Player, redesigning Windows audio, and working on
hardware accelerated video playback. After that he helped create Windows
Runtime, the application and API model used by Universal Windows Applications.
In the past several years Steve has focused on using data science to understand
product quality. He was heavily involved in the shift within the Windows team
from automated testing to assessing quality through telemetry. Steve's blog can
be found at http://steverowe.net.

3. Amp Up Your Testing
With Data
Steve Rowe
Data Science Manager @ Microsoft

4. Goals of this talk
• Explain why data driven quality is an important tool in your belt
• Provide strategies to start using data in your quality assessment
• This talk will not teach you to be a data scientist

5. Why Data Driven
Quality?

6. What is
Quality?
The goal of testing is to assess the quality of the object under
test.
So what is Quality?
“The form is the part of the world over which we have
control…. The context is that part of the world which puts
demands on this form…. Fitness is the relation of mutual
acceptability between these two.”
- Christopher Alexander, Notes on the Synthesis of Form
“The test of the machine is the satisfaction it gives you. There
isn't any other test. If the machine produces tranquility it's
right. If it disturbs you it's wrong until either the machine or
your mind is changed.”
- Robert Pirsig, Zen and the Art of Motorcycle Maintenance

7. Defining Quality
• Quality is a match of fitness to function
• There is a user
• Who has a task
• A piece of software is high quality if it accomplishes its intended task well
• There is no assessment of quality without the context of a user
• Quality can include:
• Simplicity
• Functionality
• Time to market
• Standards compliance
• …
• To test software we must determine the characteristics of the intended task and then assess whether they
are achieved.

8. A Brief History of Testing
Manual
Testing
Test
Automation
Test Tooling
Data Driven
Quality

9. Wave 1: Manual Testing
• Exploratory Testing or Proscribed Testing
• Carried out by individuals
• Benefits:
• Highly connected to user.
• Low ramp-up cost.
• Detractors:
• Doesn’t scale. Especially for software with long shelf life.
• Often repetitive.

10. Wave 2: Test Automation
• Write software to test the product
• Benefits:
• Can scale very well.
• Detractors:
• Hard to keep up with triage of results.
• Expensive to write. Requires developers.
• Tends to focus on spec, not customers.

11. Wave 3: Test Tooling
• Software that writes tests
• Example: Model Based Testing
• Benefits:
• Very thorough
• Detractors:
• Difficult to determine failures (test oracles).
• Hard to understand the impact of failures.
• Long lead time.
• Totally disconnected from user perception of quality.

12. Wave 4: Data Driven Quality
• Utilize customer data to understand what customers are experiencing
• Benefits:
• Directly connected to customer experience
• Gains benefit from scale
• Detractors:
• Requires data infrastructure
• Requires a large enough user base

13. Why Data Driven
Quality?

14. Data Empowers Us to Delight Our Customers
• Do you understand how your features are being used by customers?
• How they are *actually* used not how you intend them to be used?
• What would it change if you did?
• Could you plan better?
• Could you fix the right bugs?
• Customers tell us in their actions what they want.
• Are we listening?

15. Data Allows Us to Ship Faster
• There is a risk/speed tradeoff in software
• If we take more risk, we can ship faster
• If we can lower risk, we can ship faster
• Risk is a factor of likelihood and cost
• Cost is duration x impact
• Data Driven Quality helps lower risk
• Earlier detection (and fixing) lowers duration
Duration
Impact

16. Implementing Data
Driven Quality
How do we get there?

17. The Data Journey
Data
Oblivious
• Decisions made
by intuition.
Data
Affirmation
• Decisions made by
intuition. Data
only used to
reinforce decisions.
Data Driven
• Decisions are
validated by data.
Data Centric
• Decisions cannot
be made without
data. Products
may incorporate
data.
Intuition is Source of Truth
Data is Source of Truth

18. Two Paths
• Someone in
management says,
“We are going to be
data driven”
Top
Down
Edict
• Teams build data
excellence locally
• Other teams start
following
Bottom
Up
Organic

19. Start with the Right Questions
• What do you want to know?
• What action will you take when you know the answer?
• Begin with your business questions
• Let the questions inform the necessary data and type of analysis
• Do not start with data and ask what questions you might answer

20. The Siren Song of Data
“Far better an approximate answer to
the right question, which is often vague,
than the exact answer to the wrong
question, which can always be made
precise.” – John Tukey

21. Use the Data You Already Have
Analyze test results
• Test Results
• Logs
01
Add
instrumentation
• Measure how the code
behaves under test
load
02
Add customer data
• Use the same data
points
03

22. Create
Metrics
Does the product work as you expect?
• Functionality
• Performance
• Reliability
• …
Health
Metrics
Does the product have the effect on customers you
wanted?
• Are they using it more?
• Are they spending more money?
• Are they accomplishing their task faster?
Success
Metrics

23. Good Metrics
• Always know what failing looks like before you start.
Set Targets ahead of time
• Not Number of Successes, but % of Attempts that Fail
• Aggregate by users or machines not instances
Metrics should be ratios

24. Don’t Get Too Fancy
Start with counting (in ratio form)
Move to Summary Statistics
(mean, median, P90)
Use statistical methods to
evaluate accuracy
Use predictive models like linear
regression or classification trees

25. Data Driven Quality Is Not Magic
• It is a tool that accomplishes a particular job
• If your circumstances are different, it may not work
• Don’t replace everything, especially all at once.
• Mix it in with your tests. Cross-fade over time.
• What is your risk profile?
• If you work on airplanes or medical equipment, failure costs are very high
• Moving to data driven quality is a long process
• Don’t turn back half way

26. Data
Oblivious
• Decisions made
by intuition.
Data
Affirmation
• Decisions made by
intuition. Data
only used to
reinforce decisions.
Data Driven
• Decisions are
validated by data.
Data Centric
• Decisions cannot
be made without
data. Products
may incorporate
data.
This is just the beginning
Intuition is Source of Truth
Data is Source of Truth