This document outlines an instructor-led presentation on software quality metrics. It introduces the instructor and his relevant experience. It then provides information from attendee feedback on previous sessions, including both positive and negative comments. The presentation agenda is then outlined, with topics like how measurement can help organizations understand, evaluate, and improve their processes. Group exercises are included to discuss defining software quality and examples of metrics used in real life. The presentation also covers best practices for using metrics, different models of software quality, and measuring quality within agile development.
3. Meet Your Instructor
• Team Lead, Data warehousing product
development
• Software product manager, BI product
• COO, large IT services company
• CEO, XBOSoft, software qa and testing services
• Relevant specialties
– Software quality process improvement
– Software quality evaluation and measurement
– Software quality in use / UX design
– Mobile User Experience and usability
Quest
2014
3
5. Some
InformaAon
• Used
1
hour
efficiently
and
covered
the
material.
It
was
interacAve
too.
• Presenter
was
comfortable
and
engaging.
Content
was
good.
• Very
interacAve
and
demanding,
commands
great
aWenAon.
• Best
content.
Provided
great
brainstorming
opportuniAes
and
gave
examples.
Great
presenter.
Quest
2014
5
6. More
informaAon
• Slides
too
low
where
people’s
heads
are.
Can’t
see.
• How
do
we
measure
quality?
You
didn’t
answer
that
at
all.
Very
disappointed.
• I
realize
interacAon
is
good,
but
found
the
discussions
to
be
distracAng.
• It
was
too
hard
to
hear
what
was
said.
• Instructor
only
focused
on
people
in
front
who’s
hands
were
up.
• Too
large
a
group.
• Slides
were
different.
Quest
2014
6
7. Session Spirit and Expectations
• Interactive given time and other context
• I won’t read the slides…
– Slides for you as a take-away
– Slightly different than your handouts
– Some new examples and ideas
– Write me and I’ll send these to you
• I’ll repeat questions (if I can remember)
7
Quest
2014
11. Don’t – Calculate metrics that
don’t help answer specific
questions
• Are you collecting measurements and
calculating metrics without thinking what
answers will I get after collecting this
information?
Quest
2014
11
12. POLL: How many metrics are
you collecting on a regular
basis within your
organization?A. 1-5
B. 6-10
C. 11-15
D. 0
13. Metrics - Benefits
• Understand
how
QA,
tesAng,
and
its
processes
and
where
the
problems
are
• Evaluate
the
process
and
the
product
in
the
right
context
• Predict
and
control
process
and
product
qualiAes
• Reuse
successful
experiences
– Feed
back
experience
to
current
and
future
projects
• Monitor
how
something
is
performing
• Analyze
the
informaAon
provided
to
drive
improvements
13
Quest
2014
14. How can measurement help us (YOU)
• Create a organizational memory – baselines of current
practices-situation
• Determine strengths and weaknesses of the current
process and product
– What types of errors are most common?
• Develop reasoning for adopting/refining techniques
– What techniques will minimize the problems?
• Assess the impact of techniques
– Does more/less manual functional testing versus
automation reduce defects?
• Evaluate the quality of the process/product
– Are we applying inspections appropriately?
– What is the reliability of the product before/after
delivery?
14
Quest
2014
15. Boil it Down-Understand,
Evaluate and Improve
• To do this… We need metrics
Can you think of other examples in our lives
where this applies? Where do you use metrics
to evaluate and improve?
15
Quest
2014
Understand
Evaluate
Improve
24. Be-‐Do-‐Have
Be
Do
Have
Quest
2014
24
Process
• Iterative (sprints)
• Daily standups
• Face to face communication
• Post mortem – end of sprint
• Delivery meeting – end of sprint
• Planning meeting – before sprint
• Self organizing
People
• Communicative
• Collaborative/Cooperative
• Flexible and willing
• Knowledgeable-multi
• Initiative/responsible
• Responsive
Results
• Speed
• Quality
Velocity
Defects
Adherence to plan (many)
Overtime
Emails
Defect Aging
Defects not fixed rate
26. Don’t – Measure the wrong thing
• Make sure your metrics help you determine if
meeting a goal
• Some sample metrics to review:
– Test Coverage = Number of units (KLOC/FP) tested /
total size of the system
– Test Density-Number of tests per unit size = Number
of test cases per KLOC/FP
– Test cases executed
– Defects per size = Defects detected / system size
– Defects found per tester
– Test cases written per day
Quest
2014
26
Don’t measure something just because you can
27. “Not everything that can be
counted counts, and not
everything that counts can be
counted.”
Albert Einstein
Don’t measure something just because you can.
29. Why we need to
measure ?
• Our bosses want us to…
• They want someone to point fingers at
• They want to fire some people and save money
• They need to report to their managers
• They
want
some
basis
on
which
to
evaluate
us
and
give
us
a
raise!
• We
need
to
figure
out
a
way
to
do
beWer!
• We
want
to
improve
our
work
and
improve
soMware
quality
29
Quest
2014
30. The Metrics Conundrum
• QA and Testing
Language
– Defects
– Execution status
– Test cases
– Pass/fail rates
– DRE…
• Business
Language
– Cost
effecAve
– ROI
– Cost
of
ownership
– Cost
of
poor
quality
– ProducAvity
– Calls
to
help
desk
– Customer
saAsfacAon
– Customer
retenAon
30
Quest
2014
31. In your organization…
• What measurements do you take in your
organization and why?
• Who uses them and for what?
31
Quest
2014
32. The Metric Reality
• Measurement and metrics are like dinner.
It takes 2-3 hours to make dinner, and 15
minutes to consume…
• But… many metrics are never reviewed or
analyzed (consumed)
• WHY?
32
Quest
2014
33. The Metric Conundrum
• Test leads and test managers rarely have the
right metrics to show or quantify value
• Metric collection and reporting are a drag
• QA metrics usually focus only on test execution
• Test tools don’t have most of the metrics we
want
• Reports generated by QA are only rarely
reviewed
• Metrics are not connected to anything of value/
meaningful for ________.
33
Quest
2014
34. Don’t – Forget to differentiate
between quality, testing and defects
• Metric becomes the goal
• Organizations concentrated on “the
metrics”, forget to understand the metric’s
relationship to the goal or objective.
• Defect counts need to be incorporated into
an overall valuation because Quality is
ultimately measured in the eyes of the end
user.
Quest
2014
34
36. Don’t – Forget about context
• Metrics don’t have consistent context so
they are unreliable – Context needs to be
defined and then maintained for
measurements to be meaningful.
• Difficult in today’s environment with
changing test platforms and other
contextual factors.
Quest
2014
36
37. What contextual factors could
there be?
• Release complexity
• Development methodology
• Software maturity
• Development team maturity and expertise
• Development team and QA integration
• Resources available
• User base
Quest
2014
37
All metrics need to be normalized for proper interpretation
38. Metrics need context to tell the
whole story
• Normalized per function point (or per LOC)
• At product delivery (first X months or first year of
operation)
• Ongoing (per year of operation)
• By level of severity
– Gross numbers don’t tell much
• By category or cause, e.g.: requirements defect,
design defect, code defect, documentation/on-
line help defect, defect introduced by fixes, etc.
– Absolute and Total numbers tell 0
– Too granular also tell 0Quest
2014
38
39. Don’t – Be sporadic or irregular
• Measurements used are not consistent –
Just as context needs to be consistent, so
do the measurements, methods, and time
intervals that you collect the
measurements and calculate the metrics.
• Just as in weighing yourself, it doesn’t
make sense to drink 2 gallons one day
and weigh in, and go jogging 10 miles the
next day and weigh in.
Quest
2014
39
44. Don’t – Collect measurements
that no one wants
• Metrics have no audience – As a corollary
to the previous factor, if there is no
question to be answered, then there will
also be no audience for the metric.
• Metrics need to have an audience in order
to have meaning.
• How many of the metrics and reports that
you generate are read?
Quest
2014
44
45. Poll: How many of you collect
metrics that you don’t need or
use?
46. Group
Exercise
Who
are
your
stakeholders?
• CEO/CTO/CIO
• Development
manager/VP
• QA
manager
• Others?
Quest
2014
46
47. Do - Collect what “they” want
• Ratios and percentages rather than
absolutes
• Comparisons over time, or by release
• Report on what leads to improvement in:
– Costs
– Time
– Quality
Quest
2014
47
48. Do - Collect what they want
Costs
(Some
potenAal
metrics
include):
• Business
losses
per
defect
that
occurs
during
operaAon
• Business
interrupAon
costs
• Costs
of
work-‐arounds
• Costs
of
reviews,
inspecAons
and
prevenAve
measures
• Costs
of
test
planning
and
preparaAon
• Costs
of
test
execuAon,
defect
tracking,
version
and
change
control
• Costs
of
test
tools
and
tool
support
• Costs
of
test
case
library
maintenance
• Costs
of
tesAng
&
QA
educaAon
associated
with
the
product
• Re-‐work
effort
(hours,
as
a
percentage
of
the
original
coding
hours)
• Lost
sales
or
goodwill
• Annual
QA
and
tesAng
cost
(per
funcAon
point)
Quest
2014
48
49. Do - Collect what they want
Time-‐Resources
(Some
potenAal
metrics
include):
• Labor
hours/defect
fix
• Turnaround
Ame
for
defect
fixes,
by
level
of
severity
• Time
for
minor
vs.
major
enhancements
– actual
vs.
planned
elapsed
Ame
• Effort
for
minor
vs.
major
enhancements
• actual
vs.
planned
effort
hours
Quest
2014
49
50. Do - Collect what they want
Quality
(Some
potenAal
metrics
include):
• Survey before, after (and ongoing) product delivery
• # system enhancement requests per year
• # maintenance fix requests per year
• User problems: call volume to customer service/Tech support
• User Satisfaction
– training time per new user, time to reach task time of x
– # errors per new user
• # product recalls or fix/patch releases/year
• # production re-runs
• Availability
(Ame
system
is
available/
Ame
the
system
is
needed
to
be
available)
Quest
2014
50
51. Do Collect what they want
• Show them in combination and relative to
each other
– Cost vs. quality
– Cost vs. time
– Quality vs. time
Quest
2014
51
52. Don’t – Make the collection
effort the end game
• Measurements are too hard to get – If you
end up designing the right metric to
answer the right question, it doesn’t matter
if it takes several man days to get the data
and do the calculations.
• Unless the value and decisions made from
these metrics have considerable value,
they’ll soon be abandoned.
Quest
2014
52
53. Poll: How many of you started to
collect metrics but then found it
was too difficult or time
consuming and quit?
54. Don’t – Forget indicators
• Metrics have no indicators so cannot
evaluate
– You collect mounds of data but then what?
– How do you determine what is ‘good’ or ‘bad’?
– Before you get started collecting and
calculating you need to put together a way to
evaluate the numbers you get with meaningful
indicators that can be used as benchmarks as
your metrics program matures.
Quest
2014
54
58. Conclusions
• Designing and implementing a software quality
metrics program requires more than defect
analysis.
• Think - who and what questions that you want to
answer or goals of using metrics.
– Many refer to this as the goal-question-metric
paradigm (GQM)
– In simple terms, what are you going to do with
the numbers once you get them?
• Most of the “Don’ts” are related to not thinking
about the objectives of the metrics and actions you
will take based on them.Quest
2014
58
59. Solutions
• Develop a stakeholders list and their goals-
objectives
• Develop a list of questions that, if answered,
would determine that the goals are met
• Develop a catalogue of metrics (that answer the
questions) that can mix and match to apply to
the goals depending on the stakeholder
• Develop and collect metrics that accompany
each part of the development process, not just
testing
– There are many “defects” not directly in dev.
Quest
2014
59
60. Measurement
Framework
Improvement
Decisions
Stakeholders
60
Quest
2014
Do Develop a Metrics Framework