Has your organization ever considered replacing a tester that did not write, for example, 15 test cases per day? Is the testing team blamed if defect leakage is greater than 5% into production? What drives decisions like these? The common thread in these examples is “Test Metrics” Test Metrics... Everyone has an opinion about them. Some believe they are the most valuable way to communicate the results of testing. Some think that they are useless, misleading, and damaging to the communication of test results. Some believe that without measurement you are not managing the effort. And some believe that bad metrics are worse than no metrics at all. Where does your organization fit in the metrics and measurement debates? Is your team aligned? Do you agree with the team? Do you use a reporting process for test results? Are you forced to report on metrics you don't believe are valuable? Do you have dozens of metrics that you are reporting periodically that no one looks at, and when they do look at them, there is room for misinterpretation? In this session, Mike Lyles and Jay Philips will challenge the audience to discuss the topic of metrics and measurement, review multiple viewpoints on the topic, and address many of the questions that organizations have today around metrics and measurement. Takeaways: - Top metrics that are misused or misunderstood in most every organization. - Metrics that you should you get rid of ASAP! - Best and Worst metrics - based on opinions of the speakers & audience. - Metrics that everyone should use – and how they compare to your organization’s metrics. - Tools and processes that can help your organization better measure your testing. ** Presentation given at STPCon Spring 2014

1. Session 205
The Good, The Bad,
and The Metrics
Jay Philips & Mike Lyles
1

2. About Us
Jay Philips - CEO & President of
Project Realms, Inc. an IT consulting
firm focusing on software quality. Jay
is also the CEO & President of
TeamQualityPro, which is a real-time
integrated dashboard platform used to
evaluate the entire ecosystem of
application projects and resources.
TeamQualityPro is code agnostic so all
organizations can implement a real-
time executive dashboard system.
Mike Lyles - Sr. QA Manager with
20+ years of IT experience. He has
held multiple roles in QA, from
functional testing to test
environments and data management.
He is currently leading Performance
testing, Test Automation, and Service
Virtualization for all business
communities within his
organization.
2
2

3. What is
your definition
of metrics?
3
3

4. What Is Your Opinion?
• Some believe they are the most
valuable way to communicate the
results of testing
• Some think that they are useless,
misleading, and damaging to the
communication of test results
• Some believe that without
measurement you are not
managing the effort.
• Some believe that bad metrics
are worse than no metrics at all.
4
4

5. 5
5

6. How is Your Organization
Using Metrics Today?
• Is your team aligned? Do you
agree?
• Do you use a reporting process
for test results?
• Are you forced to report on
metrics you don't believe are
valuable?
• Do you have dozens of metrics
that you are reporting
periodically that no one reads?
6
6

7. 7
7

8. Do you
understand
your
metrics?
8
8

9. 9
9

10. Discussion Topic #1
The Good,
The
Misunderstood,
and
The Bad Metrics…
10
10

11. Comparing the Survey Results
Good Metrics Misunderstood Metrics Bad Metrics
Cost to fix defects Cost per detected defect Cost per detected defect
Defect open vs. close trends Defect counts Defect counts
Test Case Pass / Fail /
Blocked %
Pass / fail ratio Pass / Fail %
Progress – status of
execution, plan vs. actual,
project progress
Test Progress (plan vs.
executed)
Test cases executed
Effort / schedule variance
Resource productivity
Defect by developer
Defects entered by tester
Execution by tester
Cases executed by tester
Defect count by tester
Defect Age
Defect Fix %
Defect resolution time
Defect Removal Efficiency
Defect Leakage Defect Leakage
Test Coverage Test Coverage
Developer vs QA Ratio Developer vs QA Ratio
Test case count Test case count
11
11

12. Interesting Survey Quotes
Good
“Team intuition as to whether the software is fit for release”
“Any metric that fits the context, provides insight in the state of the software,
shows risks to project goals and schedules and allows the management to make
good, well-informed decisions”
“I can't think of any metric that every organization should be using. I don't think
these exist.”
“Tell stories, don't just dish out numbers!”
Misunderstood
“Cost of testing phase wise”
“Comparing pass rates between projects”
“Pretty much all of them.”
12
12

13. Interesting Survey Quotes
Bad
“Ones that do not provide value in making decisions or determining risk”
“Any metric that is not going to be used for improving defect slippage,
productivity, or Cost of Dev/testing”
“Everything other than Customer Happiness”
“All of them, except the ones used by a team to gather feedback for
themselves”
“Any metric used out of context”
“Anything related to test cases”
13
13

14. Good? Misunderstood? Bad?
14
14

15. Good? Misunderstood? Bad?
15
15

16. Good? Misunderstood? Bad?
16
16

17. Good? Misunderstood? Bad?
17
17

18. Discussion Topic #2
What tools
do you use
for metrics?
18
18

19. The Metric Tools…
19
19

20. The Metric Tools…
Excel HP QC / ALM
JIRA In-House Automated Tool
Assure TotalView MS TFS with Cube for Agile
Rally Microsoft Test Manager
Sharepoint GIT
IBM Rational Clear Quest I-Dash
Jenkins / Junit Mingle – for team reports
Site Scope SmartBear ALM
Sonar Test Director
Version One Zephyr
20
20

21. Discussion Topic #3
How are
metrics in your
organization
validated / used
as actionable
data?
21
21

22. Metrics Should Not Be One-Sided
22
22

23. 6 Steps to Creating
Metrics that Matter!
23
23

24. Getting Started
Does your current set of metrics allow you to see:
• The ability to see resource shortages and surpluses
• The ability to see funding shortages and surpluses
• The ability to see objectives being met and when
Can your metrics pass the following criteria?
• Does it have a direct/indirect impact on the revenue?
• Does it have a direct/indirect impact on costs?
• What is the action that you will take based on the result?
24
24

25. What to Measure
25
25

26. Step 6:
Communicate
Step 5: Implement
Repeatable Process
Step 4: Create KPI’s
Step 3: Link to the Organization’s
Strategy / Roadmap
Step 2: Determine how to Execute
Step 1: Define the Business Strategy
Step 1: Define the
Business Model
• Do you know the overall
strategy for your
department or
organization?
• Determine the
facilitators and business
results for each item
26
26

27. Step 6:
Communicate
Step 5: Implement
Repeatable Process
Step 4: Create KPI’s
Step 3: Link to the Organization’s
Strategy / Roadmap
Step 2: Determine how to Execute
Step 1: Define the Business Strategy
Step 2: Determine
How to Execute
• Most organizations have
multiple domains (IT,
Marketing, Operations,
etc).
• Review each area and
determine how that
domain can help meet
the organization’s
strategy.
27
27

28. Step 6:
Communicate
Step 5: Implement
Repeatable Process
Step 4: Create KPI’s
Step 3: Link to the Organization’s
Strategy / Roadmap
Step 2: Determine how to Execute
Step 1: Define the Business Strategy
Step 3: Link to the
Organization’s
Strategy / Roadmap
• Determine which items
are Customer Focused
and which are Cost
Related
28
28

29. Step 6:
Communicate
Step 5: Implement
Repeatable Process
Step 4: Create KPI’s
Step 3: Link to the Organization’s
Strategy / Roadmap
Step 2: Determine how to Execute
Step 1: Define the Business Strategy
Step 4: Create Key
Performance
Indicators (KPI’s)
• Focus on efficiency (cost
& productivity) as well as
effectiveness (quality &
value)
• Evaluate the metrics in
context to determine
which areas are doing
well
29
29

30. Step 6:
Communicate
Step 5: Implement
Repeatable Process
Step 4: Create KPI’s
Step 3: Link to the Organization’s
Strategy / Roadmap
Step 2: Determine how to Execute
Step 1: Define the Business Strategy
Step 5: Implement A
Process That Can
Be Reproduced
• Move from a manual
process to an automated
process
• Determine the velocity
(hourly, daily, weekly)
• Team can now focus on
objectives
30
30

31. Step 6:
Communicate
Step 5: Implement
Repeatable Process
Step 4: Create KPI’s
Step 3: Link to the Organization’s
Strategy / Roadmap
Step 2: Determine how to Execute
Step 1: Define the Business Strategy
Step 6: Communicate
Metrics and KPI’s
• Communicate what is
being measured, how, and
when data is collected
• Communicate
expectations on how the
metrics should be used
• Determine action steps
31
31

32. 6 Steps - Conclusion
• Verify your metrics bring value
• Verify the metrics can be reproduced
• Verify the metrics are available when you
need them
• Verify the business objectives and KPI’s have
been communicated
32
32

33. Contact Information
Jay Philips
CEO, Project Realms, Inc.
CEO, TeamQualityPro, Inc
jay@projectrealms.com
http://www.ProjectRealms.com
http://www.TeamQualityPro.com
Twitter: @jayphilips, @ProjectRealms,
@TeamQualityPro
http://www.linkedin.com/in/jayphilips
www.projectrealms.com
www.teamqualitypro.com
33
33

34. Contact Information
Mike Lyles
Sr. QA Manager
Lowe’s Companies, Inc.
mikewlyles@gmail.com
www.mikewlyles.com
http://about.me/mikelyles
Twitter: @mikelyles
http://www.linkedin.com/in/mikewlyles
www.mikewlyles.com
http://about.me/mikelyles
34
34

35. Thanks for Attending
Now for the metrics that are
important To US!!
Please fill out an evaluation form and drop it in the
collection basket located at the back of the room.
35
35