Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
#AgileMetrics
@XBOSoft @philiplew
Agile	
  Metrics	
  
	
  
Not	
  Just	
  Velocity	
  
	
  
Philip	
  Lew	
  
#AgileMetrics
@XBOSoft @philiplew
Poll	
  
•  Do	
  you	
  regularly	
  collect	
  AND	
  use	
  the	
  metrics	
  
you	
 ...
#AgileMetrics
@XBOSoft @philiplew
WHAT	
  IS	
  ONE	
  OF	
  THE	
  
MOST	
  FAMOUS	
  THINGS	
  
HE	
  SAID?	
  
©	
  201...
#AgileMetrics
@XBOSoft @philiplew
A Little About Me
•  Work Experience
– Telecommunications network engineer
– Team Lead, ...
#AgileMetrics
@XBOSoft @philiplew
How	
  About	
  You	
  
•  Name	
  
•  Your	
  Company/Role	
  
•  InteresQng	
  Qdbit	
...
#AgileMetrics
@XBOSoft @philiplew
Session Spirit and Expectations
•  I won’t read the slides…
•  Slides for you as a take-...
#AgileMetrics
@XBOSoft @philiplew
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	
  Reserved.	
  
#AgileMetrics
@XBOSoft @philiplew
UlQmately	
  
What	
  We	
  Want	
  to	
  Learn	
  Today	
  
•  Mistakes	
  people	
  ma...
#AgileMetrics
@XBOSoft @philiplew
Software Quality Journey
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	
  Reserved.	
...
#AgileMetrics
@XBOSoft @philiplew
The	
  SoFware	
  Quality	
  Conundrum	
  
•  What	
  quesQons/mysteries	
  do	
  you	
 ...
#AgileMetrics
@XBOSoft @philiplew
Poll: How many of you are
collecting measurements and
calculating metrics on a
regular b...
#AgileMetrics
@XBOSoft @philiplew
Software
Quality
Evaluation
Frameworks
Goal
Question
Metric
Why and
What Metrics
Softwar...
#AgileMetrics
@XBOSoft @philiplew
CHANGES	
  IN	
  BUSINESS	
  
ENVIRONMENT	
  AND	
  PARADIGMS	
  
Why	
  is	
  SoFware	
...
#AgileMetrics
@XBOSoft @philiplew
“Software is Eating the World
And now mobile is too”
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  ...
#AgileMetrics
@XBOSoft @philiplew
Quality Gains Importance
and Shifts Meaning
Quality	
  needs	
  more	
  emphasis	
  	
  ...
#AgileMetrics
@XBOSoft @philiplew
The Problem
•  Evaluating the quality of software
applications is difficult.
•  Research...
#AgileMetrics
@XBOSoft @philiplew
Motivation
Why is QA Measurement Important
•  Quality	
  Assurance	
  measurement	
  pro...
#AgileMetrics
@XBOSoft @philiplew
Workshop Outline-Agenda
1.  Defining software quality
2.  Software quality models
3.  Me...
#AgileMetrics
@XBOSoft @philiplew
Exercises
•  Break into groups of 2-3
•  No bystanders
•  Remember the cone of learning!...
#AgileMetrics
@XBOSoft @philiplew
Exercise - Value of Metrics
Why	
  do	
  you	
  want	
  to	
  measure?	
  
1.  _________...
#AgileMetrics
@XBOSoft @philiplew
“Not everything that can be
counted counts, and not
everything that counts can be
counte...
#AgileMetrics
@XBOSoft @philiplew
Module 1: Defining Software
Quality
What is software quality?
#AgileMetrics
@XBOSoft @philiplew
What is Software Quality?
•  Hard to define and evaluate
•  Perceived quality
•  Transpa...
#AgileMetrics
@XBOSoft @philiplew
What is Software Quality?
•  The	
  meaning	
  of	
  the	
  quality	
  term	
  is	
  not...
#AgileMetrics
@XBOSoft @philiplew
YET	
  ITS	
  ABSENCE	
  IS	
  EASY	
  TO	
  SEE	
  
It	
  is	
  difficult	
  to	
  define	...
#AgileMetrics
@XBOSoft @philiplew
Exercise
•  How does your organization define
software quality?
•  In your group, come u...
#AgileMetrics
@XBOSoft @philiplew
Module 2: Software Quality
Models
Models are used to ‘organize’ our understanding
#AgileMetrics
@XBOSoft @philiplew
ISO	
  25010:	
  “DEGREE	
  TO	
  WHICH	
  THE	
  SOFTWARE	
  PRODUCT	
  
SATISFIES	
  S...
#AgileMetrics
@XBOSoft @philiplew
Quality	
  of	
  a	
  SoFware	
  Product	
  
(Three	
  Views	
  for	
  Quality)	
  
	
  ...
#AgileMetrics
@XBOSoft @philiplew
ISO 25010 Product Quality
Internal – External Quality Model
©	
  2015	
  XBOSoF,	
  Inc....
#AgileMetrics
@XBOSoft @philiplew
ISO 25010 – Quality in Use
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	
  Reserved....
#AgileMetrics
@XBOSoft @philiplew
Bigger Picture on Quality
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	
  Reserved.	...
#AgileMetrics
@XBOSoft @philiplew
Evolution of SW Quality
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	
  Reserved.	
 ...
#AgileMetrics
@XBOSoft @philiplew
SoFware	
  Quality	
  Models	
  
• What	
  good	
  are	
  they	
  to	
  us?	
  
©	
  201...
#AgileMetrics
@XBOSoft @philiplew
MORE	
  THAN	
  DEFECTS	
  
SoFware	
  quality	
  with	
  an	
  Agile	
  Mindset	
  
©	
...
#AgileMetrics
@XBOSoft @philiplew
Typical View of the Quality Life Cycle
Development	
  
Quality	
  
Assurance	
  
•  The ...
#AgileMetrics
@XBOSoft @philiplew
Building	
  a	
  Quality	
  Product	
  or	
  Service	
  
•  Using modeling concepts of i...
#AgileMetrics
@XBOSoft @philiplew
End User Perception of Quality
•  Dependent on good product, of course.
•  But also infl...
#AgileMetrics
@XBOSoft @philiplew
Total Quality View
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	
  Reserved.	
  
Wha...
#AgileMetrics
@XBOSoft @philiplew
Measuring Quality
•  Within each phase, we model to understand and
evaluate quality at t...
#AgileMetrics
@XBOSoft @philiplew
Quality Tree
Customer	
  
Perceived	
  
Quality	
  
Sales	
  Quality	
  
Requirements	
 ...
#AgileMetrics
@XBOSoft @philiplew
Product Mgmt – Requirements
Quality Definition
Product	
  
Management	
  
Quality	
  
De...
#AgileMetrics
@XBOSoft @philiplew
CSR	
  QA	
  Devl	
  Reqm’t	
  Sales	
  
PM View of the Quality LifeCycle
Customer Inter...
#AgileMetrics
@XBOSoft @philiplew
Development and QA
Quality Definition
Development	
  
Quality	
  
Code	
   Re-­‐work	
  ...
#AgileMetrics
@XBOSoft @philiplew
CSR	
  QA	
  Devl	
  Reqm’t	
  Sales	
  
Dev View of the Quality LifeCycle
Customer Inte...
#AgileMetrics
@XBOSoft @philiplew
CSR	
  QA	
  Devl	
  Reqm’t	
  Sales	
  
QA View of the Quality LifeCycle
Customer Inter...
#AgileMetrics
@XBOSoft @philiplew
Key	
  Modeling	
  Concepts	
  
Take	
  Aways	
  
•  Breakdown concept (quality) to comp...
#AgileMetrics
@XBOSoft @philiplew
Exercise
1.  Does your organization have a model for
software quality?
2.  For your smal...
#AgileMetrics
@XBOSoft @philiplew
AFTER	
  WE	
  KNOW	
  WHAT	
  IT	
  IS…	
  
How	
  to	
  Measure	
  SW	
  Quality	
  
©...
#AgileMetrics
@XBOSoft @philiplew
Module 3: Measurement and
metrics
#AgileMetrics
@XBOSoft @philiplew
Understand, Evaluate and
Improve
•  If our end goal is improvement in what?
•  To improv...
#AgileMetrics
@XBOSoft @philiplew
Metrics in real life
Food	
  Eaten	
   Weight	
   Performance	
   Race	
  Results	
  
• ...
#AgileMetrics
@XBOSoft @philiplew
Metrics - Goals
•  Understand	
  how	
  QA,	
  tesQng,	
  and	
  its	
  processes	
  
an...
#AgileMetrics
@XBOSoft @philiplew
Getting Started
•  What you choose to measure is up to your
measurement goals/objectives...
#AgileMetrics
@XBOSoft @philiplew
Measurement Helps Us Answer
Questions
•  Create a organizational memory –
baselines of c...
#AgileMetrics
@XBOSoft @philiplew
More Questions That Metrics Can
Help Answer (2/2)
•  Assess the impact of techniques
– D...
#AgileMetrics
@XBOSoft @philiplew
NEED	
  FOR	
  CONSISTENCY	
  IN	
  
TERMINOLOGY	
  AND	
  CONCEPTS	
  
Before	
  we	
  ...
#AgileMetrics
@XBOSoft @philiplew
Need for Consistent
Terminologies
•  To many organizations, errors can mean faults.
•  A...
#AgileMetrics
@XBOSoft @philiplew
Defects - Get All the Info
•  So	
  once	
  we	
  get	
  common	
  terms	
  on	
  incide...
#AgileMetrics
@XBOSoft @philiplew
Exercise
•  Collecting key measurements/information
is critical to a metrics program.
• ...
#AgileMetrics
@XBOSoft @philiplew
UNDERSTANDING	
  MEASUREMENT	
  
CONCEPTS	
  
Before	
  we	
  dig	
  deep	
  into	
  met...
#AgileMetrics
@XBOSoft @philiplew
Metric Basics
•  What is a metric?
•  What is a measurement?
•  What is an indicator?
• ...
#AgileMetrics
@XBOSoft @philiplew
Measures Versus Metrics
Measures	
  
•  To	
  be	
  executed	
  
•  Executed	
  
•  Pass...
#AgileMetrics
@XBOSoft @philiplew
© 2015 XBOSoft, Inc.- All Rights Reserved.
What is Measurement?
•  “Measurement is the p...
#AgileMetrics
@XBOSoft @philiplew
Measurement
•  First component of a measurement is
it scale, the four levels of scales a...
#AgileMetrics
@XBOSoft @philiplew
Nominal Scale
•  Classifies	
  or	
  categorizes	
  the	
  agribute	
  being	
  
measured...
#AgileMetrics
@XBOSoft @philiplew
Ordinal Scale
•  Similar	
  to	
  the	
  nominal	
  scale	
  except	
  categories	
  can...
#AgileMetrics
@XBOSoft @philiplew
Interval Scale
•  Indicates	
  exact	
  differences	
  between	
  measurement	
  
points	...
#AgileMetrics
@XBOSoft @philiplew
Ratio Scale
•  Ratio scale is the highest level of
measurement much the same as interval...
#AgileMetrics
@XBOSoft @philiplew
Measurements Quality
•  Two	
  facets	
  of	
  measurement	
  quality	
  
are	
  reliabi...
#AgileMetrics
@XBOSoft @philiplew
Metrics Abandoned for Many Reasons
•  How many of you have had these
problems?
– That me...
#AgileMetrics
@XBOSoft @philiplew
Guiding principles about metrics
•  Metrics can represent a snapshot
•  Metrics can chan...
#AgileMetrics
@XBOSoft @philiplew
Metrics can represent a
snapshot
•  Not only at a particular moment, like this
week, but...
#AgileMetrics
@XBOSoft @philiplew
Some Metrics – Let’s get started
Schedule	
  
variance	
  
Effort	
  
variance	
  
Cost	
...
#AgileMetrics
@XBOSoft @philiplew
Some Metrics – Let’s get started
•  How many of these do you know/
use?
•  What others d...
#AgileMetrics
@XBOSoft @philiplew
METRICS	
  ONE	
  BY	
  ONE	
  
Let’s	
  examine	
  a	
  bunch	
  of	
  common	
  
©	
  ...
#AgileMetrics
@XBOSoft @philiplew
Defects Found/Build
•  Common	
  Usage:	
  Progressive	
  Build	
  Improvement	
  	
  
•...
#AgileMetrics
@XBOSoft @philiplew
Defects found/Module
•  Common Usage: Determination of Defect Density
•  Discovers which...
#AgileMetrics
@XBOSoft @philiplew
Time Between Defects Found
•  Common	
  Usage:	
  Build	
  Improvement	
  	
  
•  Used	
...
#AgileMetrics
@XBOSoft @philiplew
Priority Defects Found/
•  Common	
  Usage:	
  TesQng	
  Efficacy	
  in	
  Specified	
  Tim...
#AgileMetrics
@XBOSoft @philiplew
Mean Time Between P1 Defects
•  Common	
  Usage:	
  Build	
  Improvement	
  /	
  Shippin...
#AgileMetrics
@XBOSoft @philiplew
Time to perform tests
•  Common	
  Usage:	
  TesQng	
  Efficiency	
  
•  Measurement	
  of...
#AgileMetrics
@XBOSoft @philiplew
Bugs found per test
•  Common	
  Usage:	
  Test	
  Efficacy	
  /	
  Bug	
  Density	
  
•  ...
#AgileMetrics
@XBOSoft @philiplew
Defect Removal Efficiency
•  Common	
  Usage:	
  TesQng	
  Efficiency	
  
•  DRE	
  =	
  E...
#AgileMetrics
@XBOSoft @philiplew
Some User Oriented Metrics
•  Mean	
  %me	
  to	
  failure	
  metric	
  :	
  the	
  aver...
#AgileMetrics
@XBOSoft @philiplew
QA/Development
Process Metrics
•  Defect	
  arrival	
  pa@ern	
  metric	
  :	
  combinaQ...
#AgileMetrics
@XBOSoft @philiplew
More
•  Test Coverage = Number of units (KLOC/FP) tested / total
size of the system
•  N...
#AgileMetrics
@XBOSoft @philiplew
More and More
•  Quality of Testing = No. of defects found during Testing/
(No. of defec...
#AgileMetrics
@XBOSoft @philiplew
More More and More
•  Using uninitialized or invalid memory
•  Null pointer dereferencin...
#AgileMetrics
@XBOSoft @philiplew
Even More
•  Common metrics include:
– Number of requirements to test
- by function - by...
#AgileMetrics
@XBOSoft @philiplew
Exercise
Work	
  in	
  your	
  groups	
  and	
  list	
  out:	
  
•  The	
  metrics	
  yo...
#AgileMetrics
@XBOSoft @philiplew
INDICATORS,	
  ANALYSIS,	
  AND	
  
CONTEXT	
  
EvaluaQng	
  and	
  InterpreQng	
  Metri...
#AgileMetrics
@XBOSoft @philiplew
Don’t – Forget indicators
•  Metrics without indicators cannot be
evaluated
– You collec...
#AgileMetrics
@XBOSoft @philiplew
Indicators	
  
•  Indicators	
  change	
  numbers	
  into	
  meaning	
  
•  What	
  does...
#AgileMetrics
@XBOSoft @philiplew
Indicators	
  
•  Need	
  an	
  elementary	
  indicator	
  to	
  interpret	
  the	
  
le...
#AgileMetrics
@XBOSoft @philiplew
Beware of Causality
•  The	
  cause	
  and	
  effect	
  relaQonship	
  has	
  three	
  re...
#AgileMetrics
@XBOSoft @philiplew
Mistakes in Causality
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	
  Reserved.	
  
#AgileMetrics
@XBOSoft @philiplew
Look at Trends
•  Identifying trends over time
– More
– Less
– Stable
•  Make informed d...
#AgileMetrics
@XBOSoft @philiplew
Time and Beware of Absolutes
•  Don’t use absolute numbers to determine
if a build is re...
#AgileMetrics
@XBOSoft @philiplew
Don’t – Forget about context
•  Metrics don’t have consistent context so
they are unreli...
#AgileMetrics
@XBOSoft @philiplew
What contextual factors
could there be?
•  Release complexity
•  Development methodology...
#AgileMetrics
@XBOSoft @philiplew
Exercise
•  List out contextual factors in our
organization that are needed to ensure
pr...
#AgileMetrics
@XBOSoft @philiplew
All this is good
But in reality…
#AgileMetrics
@XBOSoft @philiplew
Why We Measure?
•  Our bosses want us to…
•  They want someone to point fingers at
•  Th...
#AgileMetrics
@XBOSoft @philiplew
The Metrics Conundrum
•  QA and Testing
Language
– Defects
– Execution status
– Test cas...
#AgileMetrics
@XBOSoft @philiplew
Metrics In The Real World
•  What measurements do you take in your
organization and why?...
#AgileMetrics
@XBOSoft @philiplew
The Metric Conundrum (cont.)
•  Rarely have the right metrics to show or
quantify value
...
#AgileMetrics
@XBOSoft @philiplew
Poll: How many of you started to
collect metrics but then found it
was too difficult or ...
#AgileMetrics
@XBOSoft @philiplew
Module 4:
Goals, Metrics and
Frameworks
#AgileMetrics
@XBOSoft @philiplew
STAKEHOLDERS
UNDERSTANDING THE
CONSUMERS OF OUR METRICS
Developing	
  and	
  reporQng	
 ...
#AgileMetrics
@XBOSoft @philiplew
Stakeholders
•  Who are they?
•  What do they want to know and why?
©	
  2015	
  XBOSoF,...
#AgileMetrics
@XBOSoft @philiplew
Possible Stakeholders
Stakeholder	
   Objec]ve	
  
CIO	
  
CEO	
  
VP	
  
Director	
  -­...
#AgileMetrics
@XBOSoft @philiplew
Exercise
•  List all potential stakeholders and what
their objectives or goals are?
©	
 ...
#AgileMetrics
@XBOSoft @philiplew
GQM	
  FRAMEWORK	
  
ConnecQng	
  goals,	
  objecQves	
  to	
  quesQons	
  and	
  answer...
#AgileMetrics
@XBOSoft @philiplew
Organizing Our Thoughts
•  Putting metrics into a framework helps us
to categorize and o...
#AgileMetrics
@XBOSoft @philiplew
Measurement Frameworks
•  Many	
  soFware	
  metrics	
  programs	
  have	
  failed	
  be...
#AgileMetrics
@XBOSoft @philiplew
GQM for Software Quality
•  Overall	
  GOAL	
  is	
  to	
  evaluate	
  the	
  quality	
 ...
#AgileMetrics
@XBOSoft @philiplew
Applying Basili’s GQM
•  Many	
  soFware	
  metrics	
  programs	
  have	
  failed	
  bec...
#AgileMetrics
@XBOSoft @philiplew
Stakeholders and GQM
Goal	
  1	
   Goal	
  2	
   Goal	
  3	
  
QuesQon	
  
1	
  
QuesQon...
#AgileMetrics
@XBOSoft @philiplew
GQM - Example
•  Goals
–  Reduce total cost of development
–  Reduce total cost of testi...
#AgileMetrics
@XBOSoft @philiplew
Exercise	
  
Stakeholder	
  Workshop	
  (15	
  minutes)	
  
1.  List out stakeholders
2....
#AgileMetrics
@XBOSoft @philiplew
Metric Q&A Worksheet
Category	
   Ques]on	
   Metric	
   Tool	
  to	
  Use	
   Needed	
 ...
#AgileMetrics
@XBOSoft @philiplew
Possible Stakeholders
Stakeholder	
   Objec]ve	
   Ques]on	
   Answer	
  
CIO	
  
CEO	
 ...
#AgileMetrics
@XBOSoft @philiplew
Agile	
  ObjecQves	
  
Waterfall	
  
•  Speed	
  
•  Quality	
  
•  Cost	
  
Agile	
  
•...
#AgileMetrics
@XBOSoft @philiplew
RETURN	
  TO	
  BASICS	
  
Agile	
  Manifesto	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All...
#AgileMetrics
@XBOSoft @philiplew
Excerpts	
  from	
  the	
  Agile	
  Manifesto	
  
1.  Working	
  soFware	
  over	
  comp...
#AgileMetrics
@XBOSoft @philiplew
Agile	
  –	
  Understanding	
  Where	
  We	
  Came	
  From?	
  
©	
  2015	
  XBOSoF,	
  ...
#AgileMetrics
@XBOSoft @philiplew
What	
  We	
  Can	
  Agree	
  On	
  
•  Want	
  to	
  minimize	
  mistakes	
  and	
  doi...
#AgileMetrics
@XBOSoft @philiplew
Agile	
  –	
  How	
  Can	
  We	
  Improve?	
  
•  Some	
  say	
  no	
  metrics	
  needed...
#AgileMetrics
@XBOSoft @philiplew
What	
  are	
  the	
  key	
  characterisQcs	
  of	
  agile?	
  
Agile	
  Characteris]c	
...
#AgileMetrics
@XBOSoft @philiplew
What	
  are	
  the	
  Goals	
  of	
  Agile?	
  
Speed	
   Quality	
  
Customer-­‐User	
 ...
#AgileMetrics
@XBOSoft @philiplew
Poll	
  
•  How	
  do	
  you	
  get	
  (end)	
  customer-­‐user	
  
feedback?	
  
– Surv...
#AgileMetrics
@XBOSoft @philiplew
Matching	
  Agile	
  CharacterisQcs	
  and	
  
ObjecQves	
  
Agile	
  
Characteris]c	
  ...
#AgileMetrics
@XBOSoft @philiplew
What	
  QuesQons	
  Tell	
  Me	
  If	
  I’m	
  
Reaching	
  Those	
  ObjecQves?	
  
Spee...
#AgileMetrics
@XBOSoft @philiplew
What	
  QuesQons	
  Tell	
  Me	
  If	
  I’m	
  
Reaching	
  Those	
  ObjecQves?	
  
Spee...
#AgileMetrics
@XBOSoft @philiplew
What	
  QuesQons	
  Tell	
  Me	
  If	
  I’m	
  
Reaching	
  Those	
  ObjecQves?	
  
Qual...
#AgileMetrics
@XBOSoft @philiplew
METRICS	
  THAT	
  HELP	
  US	
  ANSWER	
  
THESE	
  QUESTIONS!	
  
ARE	
  YOU	
  READY	...
#AgileMetrics
@XBOSoft @philiplew
VELOCITY	
  
10.	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	
  Reserved.	
  
A...
#AgileMetrics
@XBOSoft @philiplew
Velocity	
  
•  Tracking	
  our	
  velocity	
  over	
  Qme	
  tells	
  us	
  our	
  prog...
#AgileMetrics
@XBOSoft @philiplew
Velocity	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	
  Reserved.	
  
-­‐25%	
 ...
#AgileMetrics
@XBOSoft @philiplew
Burn	
  Down	
  –	
  (Distance	
  over	
  Time)	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  A...
#AgileMetrics
@XBOSoft @philiplew
DEFECT	
  REMOVAL	
  EFFECTIVENESS	
  
9.	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  ...
#AgileMetrics
@XBOSoft @philiplew
Defect	
  Removal	
  EffecQveness	
  
•  We	
  don’t	
  want	
  the	
  end	
  user	
  to	...
#AgileMetrics
@XBOSoft @philiplew
What	
  Should	
  Your	
  Goal	
  Be	
  For	
  DRE	
  
•  EsQmated	
  (by	
  Capers	
  J...
#AgileMetrics
@XBOSoft @philiplew
DRE	
  DistribuQon	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	
  Reserved.	
  ...
#AgileMetrics
@XBOSoft @philiplew
ADDED	
  WORK	
  
8.	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	
  Reserved.	
...
#AgileMetrics
@XBOSoft @philiplew
Added	
  Work	
  
•  This	
  will	
  help	
  us	
  see	
  if	
  we	
  are	
  making	
  
...
#AgileMetrics
@XBOSoft @philiplew
Piling	
  On	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	
  Reserved.	
  
0	
  ...
#AgileMetrics
@XBOSoft @philiplew
WORK	
  CATEGORY	
  ALLOCATION	
  
7.	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Righ...
#AgileMetrics
@XBOSoft @philiplew
Work	
  Category	
  AllocaQon	
  
•  This	
  will	
  help	
  us	
  see	
  where	
  we	
 ...
#AgileMetrics
@XBOSoft @philiplew
Is	
  Effort	
  in	
  Alignment?	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	
  ...
#AgileMetrics
@XBOSoft @philiplew
WORK	
  SUSTAINABILITY	
  
6.	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	
  Re...
#AgileMetrics
@XBOSoft @philiplew
Work	
  Sustainability	
  
•  “Sustainable”	
  -­‐	
  We	
  can’t	
  sustain	
  our	
  e...
#AgileMetrics
@XBOSoft @philiplew
Unplanned	
  OverQme	
  %	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	
  Reserv...
#AgileMetrics
@XBOSoft @philiplew
Unplanned	
  OverQme	
  and	
  Defects	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rig...
#AgileMetrics
@XBOSoft @philiplew
DEFECT	
  REWORK	
  RATE	
  
5.	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	
  ...
#AgileMetrics
@XBOSoft @philiplew
Defect	
  Rework	
  Rate	
  
•  We	
  can’t	
  get	
  faster	
  if	
  we	
  are	
  alway...
#AgileMetrics
@XBOSoft @philiplew
Defect	
  ReWork	
  %	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	
  Reserved.	...
#AgileMetrics
@XBOSoft @philiplew
Accumulated	
  Debt	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	
  Reserved.	
 ...
#AgileMetrics
@XBOSoft @philiplew
TOTAL	
  UNEXPECTED	
  WORK	
  
4.	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	...
#AgileMetrics
@XBOSoft @philiplew
Total	
  Unexpected	
  Work	
  
•  Measurement	
  –	
  Sum	
  of	
  work	
  done	
  abov...
#AgileMetrics
@XBOSoft @philiplew
Are	
  We	
  EsQmaQng	
  Well?	
  	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	...
#AgileMetrics
@XBOSoft @philiplew
What	
  are	
  we	
  doing	
  that	
  is	
  
unplanned?	
  
©	
  2015	
  XBOSoF,	
  Inc....
#AgileMetrics
@XBOSoft @philiplew
DEFECTS	
  IN	
  PRODUCTION	
  AVG.	
  FIX	
  
TIME	
  
3.	
  
©	
  2015	
  XBOSoF,	
  I...
#AgileMetrics
@XBOSoft @philiplew
Defect	
  in	
  ProducQon	
  Avg.	
  Fix	
  Time	
  
•  If	
  defects	
  make	
  it	
  t...
#AgileMetrics
@XBOSoft @philiplew
Don’t	
  Make	
  Them	
  Wait	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	
  Re...
#AgileMetrics
@XBOSoft @philiplew
DEFECT	
  CATEGORIZATION	
  
ALLOCATION	
  %	
  
2.	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	...
#AgileMetrics
@XBOSoft @philiplew
Defect	
  CategorizaQon	
  AllocaQon	
  
•  Reduce	
  rework	
  =	
  Increase	
  speed	
...
#AgileMetrics
@XBOSoft @philiplew
Defect	
  Rework	
  AllocaQon	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	
  Re...
#AgileMetrics
@XBOSoft @philiplew
Rework-­‐Requirements	
  Errors	
  
Requirements	
  	
  
•  Change	
  in	
  requirements...
#AgileMetrics
@XBOSoft @philiplew
Poll	
  
•  What	
  actual	
  improvements	
  have	
  you	
  seen	
  from	
  
using	
  A...
#AgileMetrics
@XBOSoft @philiplew
WORKING	
  SOFTWARE	
  DELIVERY	
  
RATE	
  
1.	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  A...
#AgileMetrics
@XBOSoft @philiplew
Working	
  SoFware	
  Delivery	
  Rate	
  
•  Does	
  the	
  soFware	
  work	
  or	
  no...
#AgileMetrics
@XBOSoft @philiplew
Am	
  I	
  Ge~ng	
  Beger?	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	
  Reser...
#AgileMetrics
@XBOSoft @philiplew
Am	
  I	
  Delivering	
  So	
  I	
  can	
  Get	
  
Feedback?	
  
©	
  2015	
  XBOSoF,	
 ...
#AgileMetrics
@XBOSoft @philiplew
Be	
  
Do	
  
Have	
  
©	
  2015	
  XBOSoF,	
  Inc.-­‐	
  All	
  Rights	
  Reserved.	
  ...
#AgileMetrics
@XBOSoft @philiplew
Take	
  Aways	
  
•  Mistakes	
  people	
  make	
  in	
  (agile)	
  metrics	
  and	
  ho...
#AgileMetrics
@XBOSoft @philiplew
•  What	
  and	
  
why	
  
•  Goals	
  and	
  
objecQves	
  
Understand	
  
Ask	
  
your...
#AgileMetrics
@XBOSoft @philiplew
Post your questions on Twitter and I'll answer them @philiplew
Join us to keep updated o...
Nächste SlideShare
Wird geladen in …5
×

Agile Metrics - ASTQB Workshop by Philip Lew - XBOSoft

1.184 Aufrufe

Veröffentlicht am

When implementing software quality metrics, you need to first understand the purpose of the metric and who will be using it. Will the metric be used for measuring people, the process, illustrate the level of quality in software products, or drive towards a specific objective? QA managers typically want to deliver productivity metrics, while management may want to see metrics that support customer or user satisfaction or cost related (ROI) initiatives.

With agile development methods, we often lose sight that our primary objective is the same: quality. We’ve also added the primary objective of velocity. However, we don’t now how to measure it other than ‘velocity’ itself.

With a agile mindset, define quality for your organization with an agile looking glass. Deliver software quality metrics with actionable objectives toward increasing or improving agile’s two primary objectives, quality and velocity for working software.

You Will Learn:
-- Mistakes people make in agile metrics and how to avoid them.
-- How to consistently and systematically improve root causes of low velocity.
-- How to reduce rework.
-- How to analyze your agile process and determine meaningful metrics to present to management.

Veröffentlicht in: Software
  • Loggen Sie sich ein, um Kommentare anzuzeigen.

Agile Metrics - ASTQB Workshop by Philip Lew - XBOSoft

  1. 1. #AgileMetrics @XBOSoft @philiplew Agile  Metrics     Not  Just  Velocity     Philip  Lew  
  2. 2. #AgileMetrics @XBOSoft @philiplew Poll   •  Do  you  regularly  collect  AND  use  the  metrics   you  collect  to  monitor  and  improve  your  agile   process?   – Yes   – No   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  3. 3. #AgileMetrics @XBOSoft @philiplew WHAT  IS  ONE  OF  THE   MOST  FAMOUS  THINGS   HE  SAID?   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   “LIFE  IS  LIKE  A  BOX  OF   CHOCOLATES,  YOU  NEVER  KNOW   WHAT  YOU’RE  GOING  TO  GET”  
  4. 4. #AgileMetrics @XBOSoft @philiplew A Little About Me •  Work Experience – Telecommunications network engineer – Team Lead, Data warehousing product development – Software product manager, BI product •  Specialties – Software quality process improvement – Software quality evaluation and measurement – Software quality in use / UX design ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  5. 5. #AgileMetrics @XBOSoft @philiplew How  About  You   •  Name   •  Your  Company/Role   •  InteresQng  Qdbit  or  fact   •  ObjecQve  in  being  here   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  6. 6. #AgileMetrics @XBOSoft @philiplew Session Spirit and Expectations •  I won’t read the slides… •  Slides for you as a take-away (reference) •  Slightly different than your handouts – Always thinking of new examples and ideas, you can write me and I’ll send these to you •  Interactive – Lots of exercises and chances to learn – Lots of material •  Only 3 hours time for the workshop – May skip over material – use as reference ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  7. 7. #AgileMetrics @XBOSoft @philiplew ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  8. 8. #AgileMetrics @XBOSoft @philiplew UlQmately   What  We  Want  to  Learn  Today   •  Mistakes  people  make  in  agile  metrics  and  how  to   avoid  them.   •  How  to  consistently  and  systemaQcally  improve  root   causes  of  low  velocity.   •  How  to  reduce  rework.   •  How  to  analyze  your  agile  process  and  determine   meaningful  metrics  to  present  to  management.   •  How  to  get  to  velocity  AND  quality,  and  what  to   measure.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  9. 9. #AgileMetrics @XBOSoft @philiplew Software Quality Journey ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   SoFware   Quality   SoFware   TesQng   TesQng   Metrics   Why  and   What   Metrics   Goal   QuesQon   Metric   SoFware   Dev   Processes   SoFware   Quality   Models   Business&   Product   Processes   EvaluaQon   Frameworks   AGILE METRICS
  10. 10. #AgileMetrics @XBOSoft @philiplew The  SoFware  Quality  Conundrum   •  What  quesQons/mysteries  do  you  have  about   soFware  quality  metrics?   – SoFware  quality  is  too  nebulous  to  get  your  hands   around   – Everyone  has  a  different  idea  of  what  it  is,  so   difficult  to  measure   – What  can  we  measure  for  quality  in  agile?   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  11. 11. #AgileMetrics @XBOSoft @philiplew Poll: How many of you are collecting measurements and calculating metrics on a regular basis? a.  0 b.  1-5 c.  6-10 d.  11-15 e.  16+ 8% 44% 28% 6% 14%
  12. 12. #AgileMetrics @XBOSoft @philiplew Software Quality Evaluation Frameworks Goal Question Metric Why and What Metrics Software Quality in an Eggshell ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   Business&Product Processes/Lifecycles Software Dev Processes TesQng   Metrics   Software Testing Processes Software Quality Models Software Quality Processes Agile
  13. 13. #AgileMetrics @XBOSoft @philiplew CHANGES  IN  BUSINESS   ENVIRONMENT  AND  PARADIGMS   Why  is  SoFware  Quality  important?   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  14. 14. #AgileMetrics @XBOSoft @philiplew “Software is Eating the World And now mobile is too” ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   conventional software applications Web sites E-commerce Print advertising Telephone books Web 1.0 Fully functional applications>> Higher complexity User profiling, segmentation, personalization data collection Web 2.0 Information integration Semantic Web Web 3.0 - Big data and Mobile Web-basedapplicationsand nowmobileaccessand nativemobile New ideas, integrated platforms, more components Ubiquitous Access
  15. 15. #AgileMetrics @XBOSoft @philiplew Quality Gains Importance and Shifts Meaning Quality  needs  more  emphasis     •  ShiFs  in  user  expectaQons  and  business  models  that   enable  users  to  switch  applicaQons  quickly.     •  ApplicaQons  that  have  low  usability  or  quality  are   easily  leF  for  another     •  Many  alternaQves  available  with  another  click     – Short  trial  subscripQon     – Many  trials  are  free     – No  cost  factor  involved  in  switching  applicaQons   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   Development cycle
  16. 16. #AgileMetrics @XBOSoft @philiplew The Problem •  Evaluating the quality of software applications is difficult. •  Research and standards are not entirely applicable •  Models and frameworks lack implementation specifics so hard to use and benefit from them. •  How do we objectively measure software quality? ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  17. 17. #AgileMetrics @XBOSoft @philiplew Motivation Why is QA Measurement Important •  Quality  Assurance  measurement  provides  informaQon  for   improvement  in  your  tesQng,  your  team,  and  your  company.     –  Team   •  Is  your  QA  team  improving?   –  Process   •  Have  you  reached  the  highest  state  of  efficacy?     •  Is  your  tesQng  as  thorough  as  it  can  be?     –  Product   •  Is  soFware  ready  for  release?  Can  we  stop  tesQng  now?   •  What  parts  of  the  soFware  are  at  most  risk  ?   –  Money   •  Are  you  able  to  do  everything  you  want  within  budget?   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   What questions do you have?
  18. 18. #AgileMetrics @XBOSoft @philiplew Workshop Outline-Agenda 1.  Defining software quality 2.  Software quality models 3.  Measurement and metrics 4.  Goals, metrics and frameworks ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   Agile  
  19. 19. #AgileMetrics @XBOSoft @philiplew Exercises •  Break into groups of 2-3 •  No bystanders •  Remember the cone of learning! ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  20. 20. #AgileMetrics @XBOSoft @philiplew Exercise - Value of Metrics Why  do  you  want  to  measure?   1.  ______________________________________   2.  ______________________________________   3.  ______________________________________   4.  ______________________________________   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  21. 21. #AgileMetrics @XBOSoft @philiplew “Not everything that can be counted counts, and not everything that counts can be counted.” Albert Einstein Don’t measure something just because you can.
  22. 22. #AgileMetrics @XBOSoft @philiplew Module 1: Defining Software Quality What is software quality?
  23. 23. #AgileMetrics @XBOSoft @philiplew What is Software Quality? •  Hard to define and evaluate •  Perceived quality •  Transparent when present •  Noticeable when absent •  Many views •  User View •  Product View •  Developer/Coder View •  Installer View •  Management View ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   What other viewpoints could their be?
  24. 24. #AgileMetrics @XBOSoft @philiplew What is Software Quality? •  The  meaning  of  the  quality  term  is  not  simple  and   atomic,  but  a  mulQdimensional  and  abstract  concept.     •  Quality  can  not  be  measured  directly   –   at  least  in  a  not  very  trivial  way     •  Common  pracQce  assesses  quality  by  means  of  the   quanQficaQon  of  lower  abstracQon  concepts,  such  as   agributes  of  enQQes   •  Given  the  complexity  that  a  quality  concept  involves,  a   model  is  used  in  order  to  specify  the  quality   requirements.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  25. 25. #AgileMetrics @XBOSoft @philiplew YET  ITS  ABSENCE  IS  EASY  TO  SEE   It  is  difficult  to  define  quality…   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  26. 26. #AgileMetrics @XBOSoft @philiplew Exercise •  How does your organization define software quality? •  In your group, come up with a definition of software quality that you can agree on. – Can be one sentence to a short paragraph. •  Be prepared to share with the class. ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  27. 27. #AgileMetrics @XBOSoft @philiplew Module 2: Software Quality Models Models are used to ‘organize’ our understanding
  28. 28. #AgileMetrics @XBOSoft @philiplew ISO  25010:  “DEGREE  TO  WHICH  THE  SOFTWARE  PRODUCT   SATISFIES  STATED  AND  IMPLIED  NEEDS  WHEN  USED  UNDER   SPECIFIED  CONDITIONS”       Quality  Standards  and  Models   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  29. 29. #AgileMetrics @XBOSoft @philiplew Quality  of  a  SoFware  Product   (Three  Views  for  Quality)    Internal  Quality   •  Measure  of  the  degree  to  which  a  set  of  staQc  agributes  of  a   soFware  product  saQsfy  stated  and  implied  needs  when  the   soFware  product  is  used  under  specified  condiQons.   External  Quality   •  Measure  of  the  degree  to  which  a  soFware  product  enables   the  behavior  of  a  system  to  saQsfy  stated  and  implied  needs   when   the   system   including   the   soFware   is   used   under   specified  condiQons.   Quality  in  Use   •  Degree  to  which  a  product  used  by  specific  users  meets  their   needs  to  achieve  specific  goals  with  effecQveness,  efficiency,   flexibility,  safety  and  saQsfacQon  in  specific  contexts  of  use.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   What are some examples of quality measurements from these 3 perspectives? ISO  25010  
  30. 30. #AgileMetrics @XBOSoft @philiplew ISO 25010 Product Quality Internal – External Quality Model ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  31. 31. #AgileMetrics @XBOSoft @philiplew ISO 25010 – Quality in Use ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  32. 32. #AgileMetrics @XBOSoft @philiplew Bigger Picture on Quality ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   ISO 25010CMMI
  33. 33. #AgileMetrics @XBOSoft @philiplew Evolution of SW Quality ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   Software processes Code Process quality Software Quality (internal) Software Quality (external) Software Quality (in use) Product Usability CMMI assessment model white box testing black box testing Usability testing Usability heuristic Usability logging CMMI Assessment Company Programmer Tester End User Type of quality What is measured How measured? Who measures? ISO 9000 ISO 9241ISO 9126 ISO 25010
  34. 34. #AgileMetrics @XBOSoft @philiplew SoFware  Quality  Models   • What  good  are  they  to  us?   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  35. 35. #AgileMetrics @XBOSoft @philiplew MORE  THAN  DEFECTS   SoFware  quality  with  an  Agile  Mindset   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  36. 36. #AgileMetrics @XBOSoft @philiplew Typical View of the Quality Life Cycle Development   Quality   Assurance   •  The emphasis of most organizations is to eliminate defects in the Development and QA phases of the Software Development Life Cycle. •  Current quality models don’t model outside of development and QA Goal: Eliminating Defects ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  37. 37. #AgileMetrics @XBOSoft @philiplew Building  a  Quality  Product  or  Service   •  Using modeling concepts of influence and dependency that we covered earlier •  Quality of each phase or step has an influence on the quality of the next phase AND on the final result ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  38. 38. #AgileMetrics @XBOSoft @philiplew End User Perception of Quality •  Dependent on good product, of course. •  But also influenced by many other factors outside of development and quality assurance. •  Other parts of the organization and overall process influence the quality of the product itself and user’s perception of quality. ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  39. 39. #AgileMetrics @XBOSoft @philiplew Total Quality View ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   What does velocity measure? Does velocity measure quality?
  40. 40. #AgileMetrics @XBOSoft @philiplew Measuring Quality •  Within each phase, we model to understand and evaluate quality at that phase •  At each phase, define quality and develop quality measures •  The lower level measurements will aggregate to higher cumulative measurements •  Aggregation can be done with various weighting schemes with flexibility according to the business of the organization ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  41. 41. #AgileMetrics @XBOSoft @philiplew Quality Tree Customer   Perceived   Quality   Sales  Quality   Requirements   Quality   Product   Development   Quality   Customer   Service  Quality   Possible  to  have  a  high   quality  product,  but  with   poor  customer  perceived   quality!  Here’s  Why…   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   Each Component or Phase Has its Own Quality which aggregates and influences overall customer perception of quality What weighting scheme would you use and why?
  42. 42. #AgileMetrics @XBOSoft @philiplew Product Mgmt – Requirements Quality Definition Product   Management   Quality   Development   understanding   CommunicaQon   with   development   Complete ness   Accuracy   Timeliness   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   What other attributes would be appropriate to model product management quality?
  43. 43. #AgileMetrics @XBOSoft @philiplew CSR  QA  Devl  Reqm’t  Sales   PM View of the Quality LifeCycle Customer Interactions Defects are introduced at all phases of the lifecycle of a product. Requirements: •  Vague, ambiguity measures •  Incomplete •  Poorly Written-not clearly understood •  Lack of understanding of underlying needs-don’t match customer wants Goal: Improving Quality ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  44. 44. #AgileMetrics @XBOSoft @philiplew Development and QA Quality Definition Development   Quality   Code   Re-­‐work   Adherence   to   standards   Defects   Test  ware   ValidaQon   and   verificaQon   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   What other attributes would be appropriate to model DEV-QA quality?
  45. 45. #AgileMetrics @XBOSoft @philiplew CSR  QA  Devl  Reqm’t  Sales   Dev View of the Quality LifeCycle Customer Interactions Defects are introduced at all phases of the lifecycle of a product. Development: •  Coding errors •  Non conformance to corporate coding best practices •  Not extensible •  Poor performance •  Rework •  Recurring Defect Goal: Improving Quality ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  46. 46. #AgileMetrics @XBOSoft @philiplew CSR  QA  Devl  Reqm’t  Sales   QA View of the Quality LifeCycle Customer Interactions Defects are introduced at all phases of the lifecycle of a product. Quality Assurance: •  No test cases •  Non-standard test cases •  Inaccurate test cases, incomplete •  Defects not written understandably •  Defects unverifiable •  Defects incomplete information Goal: Improving Quality ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  47. 47. #AgileMetrics @XBOSoft @philiplew Key  Modeling  Concepts   Take  Aways   •  Breakdown concept (quality) to components (levels of abstraction) to understand, evaluate, measure, and then improve – Hierarchical requirements tree – Measurements for each factor or element (characteristic) •  Relationships and dependency – Quality at one phase influences quality at the next phase – Not only in an organization but also within a process ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  48. 48. #AgileMetrics @XBOSoft @philiplew Exercise 1.  Does your organization have a model for software quality? 2.  For your small group, develop a model for software quality. 3.  Be prepared to share with the class. a.  Why you included certain things b.  Why you left out others c.  Are all components evenly weighted? Why not? ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  49. 49. #AgileMetrics @XBOSoft @philiplew AFTER  WE  KNOW  WHAT  IT  IS…   How  to  Measure  SW  Quality   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  50. 50. #AgileMetrics @XBOSoft @philiplew Module 3: Measurement and metrics
  51. 51. #AgileMetrics @XBOSoft @philiplew Understand, Evaluate and Improve •  If our end goal is improvement in what? •  To improve, we need to evaluate •  In order to evaluate, we must understand what we are evaluating •  To do this… We need metrics Can you think of other examples in our lives where this applies? Where do you use metrics to evaluate and improve? ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  52. 52. #AgileMetrics @XBOSoft @philiplew Metrics in real life Food  Eaten   Weight   Performance   Race  Results   •  Calories •  Fat •  Carbohydrates •  Protein •  Time of day •  Vitamins •  … •  Blood pressure •  Cholesterol •  Blood glucose •  Red cell count •  White cell count •  Hematocrit •  Hemoglobin •  Body fat % •  … •  Placing •  … •  Effort/Power •  Heart rate/Watts •  Speed •  Time Intelligence Finesse Context •  Training •  Sleep ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   You don’t just measure your weight right? You measure calories!!
  53. 53. #AgileMetrics @XBOSoft @philiplew Metrics - Goals •  Understand  how  QA,  tesQng,  and  its  processes   and  where  the  problems  are   •  Evaluate  the  process  and  the  product     –  Monitor  how  something  is  performing   –  Analyze  the  informaQon  provided  to  drive   improvements   •  Improve   –  Predict  and  control  process  and  product  qualiQes   –  Reuse  successful  experiences   –  Take  correcQve  acQon   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  54. 54. #AgileMetrics @XBOSoft @philiplew Getting Started •  What you choose to measure is up to your measurement goals/objectives – First exercise in class •  You can only measure improvement in areas that you have quantified. •  Establishing a metrics and measurement framework is necessary … More later on frameworks coming up! ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  55. 55. #AgileMetrics @XBOSoft @philiplew Measurement Helps Us Answer Questions •  Create a organizational memory – baselines of current practices-situation •  Determine strengths and weaknesses of the current process and product – What types of errors are most common? •  Develop reasoning for adopting/refining techniques – What techniques will minimize the problems? ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  56. 56. #AgileMetrics @XBOSoft @philiplew More Questions That Metrics Can Help Answer (2/2) •  Assess the impact of techniques – Does more/less functional testing reduce defects? – Do less defects shorten help desk call times? •  Evaluate the quality of the process/product – Are we applying inspections appropriately? – What is the reliability of the product before/ after delivery? ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  57. 57. #AgileMetrics @XBOSoft @philiplew NEED  FOR  CONSISTENCY  IN   TERMINOLOGY  AND  CONCEPTS   Before  we  dig  deep  into  metrics…   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  58. 58. #AgileMetrics @XBOSoft @philiplew Need for Consistent Terminologies •  To many organizations, errors can mean faults. •  Anomalies usually mean a class of faults that are unlikely to cause failures but may eventually cause failures indirectly. –  anomaly is a deviation from the usual, but it is not necessary wrong. •  Defects normally refer collectively to faults and failures. –  Sometimes a defect is a particular class of fault. •  Bugs refer to faults occurring in the code. •  Crashes are a special type of incident, where the system ceases to function. •  Define terms clearly ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  59. 59. #AgileMetrics @XBOSoft @philiplew Defects - Get All the Info •  So  once  we  get  common  terms  on  incidents,  faults  and   defects,  we  want  to  record  its  key  elements,  so  that  we   can  then  invesQgate  causes  and  cures:   –  Loca%on:  Where  did  the  problem  occur?   –  Timing:  When  did  it  occur?   –  Mode:  What  was  observed?   –  Effect:  Which  consequences  resulted?   –  Mechanism:  How  did  it  occur?   –  Cause:  Why  did  it  occur?   –  Severity:  How  much  was  the  user  affected?   –  Cost:  How  much  did  it  cost?   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  60. 60. #AgileMetrics @XBOSoft @philiplew Exercise •  Collecting key measurements/information is critical to a metrics program. •  Discuss in your groups: – Define the relevant terms in collecting defects, faults, incidents, error, etc.. – List out what information are you collecting on your defects and what is lacking? ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  61. 61. #AgileMetrics @XBOSoft @philiplew UNDERSTANDING  MEASUREMENT   CONCEPTS   Before  we  dig  deep  into  metrics…   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  62. 62. #AgileMetrics @XBOSoft @philiplew Metric Basics •  What is a metric? •  What is a measurement? •  What is an indicator? •  What’s the difference? ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  63. 63. #AgileMetrics @XBOSoft @philiplew Measures Versus Metrics Measures   •  To  be  executed   •  Executed   •  Passed   •  Failed   •  Re-­‐executed   •  Total  ExecuQons   •  Total  Passes   •  Total  Failures   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   Metrics •  TC % complete •  TC % passed •  % failures •  % defects corrected •  % bad fixes •  % defect discovery
  64. 64. #AgileMetrics @XBOSoft @philiplew © 2015 XBOSoft, Inc.- All Rights Reserved. What is Measurement? •  “Measurement is the process by which numbers or symbols are assigned to attributes of entities in the real world in such a way as to characterize them according to clearly defined rules.” [1] •  “Measurement is the empirical, objective assignment of numbers, according to a rule derived from a model or theory, to attributes of objects or events with the intent of describing them.” [1] [1] Software Engineering Metrics: What Do They Measure and How Do We Know? Cem Kaner and Walter P. Bond
  65. 65. #AgileMetrics @XBOSoft @philiplew Measurement •  First component of a measurement is it scale, the four levels of scales are nominal scale, ordinal scale, interval scale, and ratio scale. •  Second component is measurement quality •  Third component is causality ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  66. 66. #AgileMetrics @XBOSoft @philiplew Nominal Scale •  Classifies  or  categorizes  the  agribute  being   measured.     •  An  example  of  this  would  be  the  classificaQons  male   and  female.     •  Doesn’t  allow  for  comparisons  to  be  made  or   mathemaQcal  operaQons  to  be  preformed.     –  So  the  expression  male  >  female  or  male  <  female  are   invalid.     –  It  is  impossible  to  subtract  a  female  from  a  male.   •  The  key  requirement  of  the  nominal  scale  is  that   each  data  item  can  fit  into  only  one  category.       ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  67. 67. #AgileMetrics @XBOSoft @philiplew Ordinal Scale •  Similar  to  the  nominal  scale  except  categories  can  be  ordered   and  comparisons  can  be  made.     •  An  example  of  this  would  be  customer  saQsfacQon  surveys   which  oFen  require  an  answer  of  completely  dissaQsfied,   somewhat  dissaQsfied,  neutral,  somewhat  saQsfied,  or   completely  saQsfied.     •  In  such  a  case,  there  is  a  natural  order  to  the  scale  unlike   nominal  scales.     •  Like  nominal  scales  mathemaQcal  operaQons  sQll  can’t  be   directly  performed.         ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  68. 68. #AgileMetrics @XBOSoft @philiplew Interval Scale •  Indicates  exact  differences  between  measurement   points  unlike  the  previous  two  scales.     •  Only  the  mathemaQcal  operaQons  add  and  subtract   can  be  applied  to  an  interval  scale  because  the  zero   point  on  the  scale  has  been  arbitrarily  decided  upon.     •  For  example  20  degrees  is  hoger  than  10  degrees  but   20  degrees  may  not  be  twice  as  hot  as  10  degrees.     •  The  key  requirement  of  the  interval  scale  is  that  units   be  clearly  defined.       ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  69. 69. #AgileMetrics @XBOSoft @philiplew Ratio Scale •  Ratio scale is the highest level of measurement much the same as interval scales •  Except zero is a non-arbitrary point allowing mathematical operations such as division. •  Examples of ratio scale measurements are weight and length. – 160 pounds is twice that of 80 pounds ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  70. 70. #AgileMetrics @XBOSoft @philiplew Measurements Quality •  Two  facets  of  measurement  quality   are  reliability  and  validity.   •  Reliability:  means  how  repeatable  the  results  are  and   low  reliability  indicates  that  many  random  errors  are   occurring  during  measurement.     •  Validity:  means  the  metric  measures  what  it  intended   to  measure.     –  Errors  in  validity,  called  systemaQc  errors,  are  usually   experienced  because  not  all  factors  were  taken  into   account  when  the  measurement  was  designed.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  71. 71. #AgileMetrics @XBOSoft @philiplew Metrics Abandoned for Many Reasons •  How many of you have had these problems? – That measurement is not valid because… – That measurement is not reliable because… ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   If no one believes (and therefore takes improvement actions) your metrics, then why bother?
  72. 72. #AgileMetrics @XBOSoft @philiplew Guiding principles about metrics •  Metrics can represent a snapshot •  Metrics can change over time •  Metrics can be simple and still very valuable/ telling •  Metrics can assist us in making decisions today •  Metrics can assist us in predicting the future •  Some metrics are critical for some while useless for others ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  73. 73. #AgileMetrics @XBOSoft @philiplew Metrics can represent a snapshot •  Not only at a particular moment, like this week, but for a process over time Code   quality   Internal   Quality   Tested   in  a   lab   External   Quality   •  What  the   end  user   sees   •  What  the   end  user   experiences   In-­‐Use   Quality   At what stage in the process do you take measurements? ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  74. 74. #AgileMetrics @XBOSoft @philiplew Some Metrics – Let’s get started Schedule   variance   Effort   variance   Cost   variance   Defect   removal   effecQveness   ProducQvity   Defect   aging   CriQcal   Defect   rate   Test   cases   executed   Test   coverage   Pass/Fail   Rate   ROI   AutomaQon   coverage   Value   Provided   %   Defects/ hour   Customer   saQsfacQon   Avg.   Time  to   Fix   Recurrence   Rate   Code   complexity   Test   Density   Defect   Find  Rate   Defect   Arrival   Rate   Require ments   VolaQlity   Require ments   Ambiguity   Planned /Actual   OverQme   Rate   CompleQon   Rate   Velocity   ExecuQon   Rate   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  75. 75. #AgileMetrics @XBOSoft @philiplew Some Metrics – Let’s get started •  How many of these do you know/ use? •  What others do you use? •  What are they used for? •  By who? •  How do you evaluate them? •  What is ‘good’, ‘acceptable’? ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  76. 76. #AgileMetrics @XBOSoft @philiplew METRICS  ONE  BY  ONE   Let’s  examine  a  bunch  of  common   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  77. 77. #AgileMetrics @XBOSoft @philiplew Defects Found/Build •  Common  Usage:  Progressive  Build  Improvement     •  One  way  to  measure  the  stability  of  builds  over  Qme  –  and  in   comparison  to  one  another     •  Number  of  defects  found  should  decrease  per  build   •  Spike  when  builds  contain  many  new  features.     •  New  features  will  result  in  higher  bug  counts,  but  over  the   course  of  the  project,  from  build  to  build,  the  number  of  bugs   found  in  each  build  should  go  down.   •  Need  baseline.     ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   Assumes that your test coverage, methods, and accuracy do not fluctuate.
  78. 78. #AgileMetrics @XBOSoft @philiplew Defects found/Module •  Common Usage: Determination of Defect Density •  Discovers which module/features functionality have the most bugs. •  Benchmarking tool to determine test efficacy and productivity. –  Used in conjunction with a measurement of how long each test (or test module) takes >> cost of testing. –  How long it takes to find a defect (average and total) in each tested area. –  Are you spending many hours testing an area that never yields bugs. –  Make decisions - where to expend your test effort. ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  79. 79. #AgileMetrics @XBOSoft @philiplew Time Between Defects Found •  Common  Usage:  Build  Improvement     •  Used  to  determine  whether  a  project  is  progressing  as  it  should.     •  As  a  project  gets  closer  to  compleQon   –  Time  that  passes  between  defects  found  should  become  greater.   –  If  no  change,  or  if  suddenly  the  Qme  between  bugs  found  gets  shorter   as  a  project  nears  its  shipping  date,  cause  for  concern.     •  If  the  Qme  between  bugs  found  does  not  increase,  then  the  project  is  not   advancing  quickly  enough.   –  Creates  the  risk  of  shipping  the  product  with  a  high  number  of  known   bugs  and/or  with  many  bugs  that  are  not  found  before  it  goes  live.   –  Or  miss  the  ship  date  in  order  to  make  sure  the  bugs  are  fixed   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  80. 80. #AgileMetrics @XBOSoft @philiplew Priority Defects Found/ •  Common  Usage:  TesQng  Efficacy  in  Specified  Timeframe     Used  to  ascertain  if  the  test  team  is  finding  the  high  Severity  and  high   Priority  defects.     •  If  the  majority  of  bugs  found  are  nit-­‐picky  low  Priority  issues,  then  either   the  release  is  low-­‐risk,  or  QA  is  missing  criQcal  issues  that  will  arise  later.   •  Important  to  know  which  tests  have  been  executed.     –  Are  the  tests  that  are  being  performed  finding  the  issues  that  are   important  to  the  company?     –  Is  the  test  team  using  its  Qme  and  resources  as  effecQvely  as  possible?   •  Not  every  defect  found  will  be  of  the  highest  priority.     –  Tests  run  should  find  issues  that  are  deemed  to  be  important  to  fix.   –  Otherwise  effort  is  being  wasted.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  81. 81. #AgileMetrics @XBOSoft @philiplew Mean Time Between P1 Defects •  Common  Usage:  Build  Improvement  /  Shipping  Readiness   •  As  soFware/build  progresses  and  nears  shipping  readiness,   the  average  Qme  between  P1  (Priority  =  1)  defects  should   become  greater.     •  Any  P1  found  will  be  a  showstopper,  so  it  becomes   important  that  the  product  progress  well  enough  for  P1s  to   stop  appearing.   •  When  the  mean  Qme  between  P1s  found  stagnates  or   lessens,  this  should  be  a  warning  sign  that  your  project  is   regressing.     •  This  is  one  of  the  clearest  quality  assurance  metrics  that   you  can  track.  It  can  carry  a  great  deal  of  weight  and   impact.  Use  it  wisely.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  82. 82. #AgileMetrics @XBOSoft @philiplew Time to perform tests •  Common  Usage:  TesQng  Efficiency   •  Measurement  of  your  team’s  tesQng  efficiency  is  how  long  it   takes  to  perform  selected  tests.     •  The  first  Qme  a  test  (or  set  of  tests)  is  executed,  should  be   higher  than  during  subsequent  execuQons.   •  As  your  team  becomes  familiar  with  each  test  and  learns  the   nuances,  test  Qme  should  decrease.     •  Some  tests  (automated  or  acQvity  dependent)  may  not  vary,   but  the  setup  and  subsequent  result  capture  should  take  less   Qme.   •  Can  idenQfy  which  tests  can  be  run  concurrently  or  in  parallel   to  gain  Qme  efficiency.   •  Can  use  to  esQmate  tesQng  future  projects     •  Used  in  conjuncQon  with  bug  density,  will  show  you  where   you  can  best  spend  your  Qme  tesQng.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  83. 83. #AgileMetrics @XBOSoft @philiplew Bugs found per test •  Common  Usage:  Test  Efficacy  /  Bug  Density   •  Evaluate  whether  your  tests  are  targeted  and  thorough  or   not.     •  Assuming  your  tesQng  does  not  fluctuate,  this  metric  will   show  you  if  you  are  tesQng  areas  that  are  yielding  bugs.   •  Knowing  which  tests  yield  how  many  bugs  –  both  as  a  total   number  and  as  a  percentage  –  allows  you  to  evaluate  your   tesQng.     •  When  measured  in  conjuncQon  with  a  “Qme  to  perform   tests”  metric,  you  can  quickly  see  if  you  are  wasQng  your   tesQng  effort  or  leveraging  your  team  effecQvely.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  84. 84. #AgileMetrics @XBOSoft @philiplew Defect Removal Efficiency •  Common  Usage:  TesQng  Efficiency   •  DRE  =  E  /  (  E  +  D  )   •  Where  E  =  No.  of  Errors  found  before  delivery  of  the  soFware   and  D  =  No.  of  Errors  found  aFer  delivery  of  the  soFware.   •  Ideal  value  of  DRE  should  be  1  which  means  no  defects  found   aFer  delivery.   •  If  you  score  low  on  DRE  it  means  to  say  you  need  to  re-­‐look  at   your  exisQng  process.     •  DRE  is  a  indicator  of  the  filtering  ability  of  quality  control  and   quality  assurance  acQvity.   •  Encourages  the  team  to  find  as  many  defects  before  they  are   passed  to  the  next  acQvity  stage.     ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  85. 85. #AgileMetrics @XBOSoft @philiplew Some User Oriented Metrics •  Mean  %me  to  failure  metric  :  the  average  Qme  the  product  runs  before   experiencing  a  crash   –  important  for  systems  like  air  traffic  control  (failure  could  be  subsQtuted   or  defined  per  your  objecQve).  Collected  during  help  desk  funcQon.   •  Customer  problem  metric  :  measure  of  problems  customers  have   encountered  with  the  product  over  the  total  usage  of  the  product.     –  Accounts  for  mulQple  instances  of  the  product  can  be  used  at  the  same   Qme,  which  effecQvely  mulQplies  the  length  of  Qme  the  product  has  been   in  operaQon  by  the  number  of  product  licenses.  Collected  during  help   desk  acQviQes.   •  Customer  sa%sfac%on  metric  :survey  asking  customers  to  rate  their   saQsfacQon  with  the  product  and/or  it’s  features  on  a  five-­‐point  scale,   usually  done  in  conjuncQon  as  part  of  a  usability  tesQng  effort.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  86. 86. #AgileMetrics @XBOSoft @philiplew QA/Development Process Metrics •  Defect  arrival  pa@ern  metric  :  combinaQon  of  the  rate  at  which   defects  are  reported  and  the  growth  of  these  rates  through  out  the   process  being  measured.   •  Backlog  management  index  metric  :  measures  the  effecQveness  of   the  process  by  comparing  the  number  of  problems  that  arrive  to   the  number  of  fixes  during  a  specified  period.   •  Fix  quality  metric  :  measures  the  number  of  defecQve  fixes   compared  to  the  number  of  successful  fixes.  A  fix  is  defecQve  when   either  it  doesn’t  repair  the  problem  reported  or  it  repairs  the   problem  but  introduces  new  defects  into  the  product.   •  Fix  response  %me  metric  :  the  Qme  between  when  a  problem  is   reported  and  the  Qme  a  fix  is  issued.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  87. 87. #AgileMetrics @XBOSoft @philiplew More •  Test Coverage = Number of units (KLOC/FP) tested / total size of the system •  Number of tests per unit size = Number of test cases per KLOC/FP •  Defects per size = Defects detected / system size •  Cost to locate defect = Cost of testing / the number of defects located •  Defects detected in testing = Defects detected in testing / total system defects •  Defects detected in production = Defects detected in production/system size ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  88. 88. #AgileMetrics @XBOSoft @philiplew More and More •  Quality of Testing = No. of defects found during Testing/ (No. of defects found during testing + No of acceptance defects found after delivery) *100 •  System complaints = Number of third party complaints / number of transactions processed •  Test Planning Productivity = No of Test cases designed / Actual Effort for Design and Documentation •  Test Execution Productivity = No of Test cycles executed / Actual Effort for testing •  Test efficiency= (number of tests required / the number of system errors) ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  89. 89. #AgileMetrics @XBOSoft @philiplew More More and More •  Using uninitialized or invalid memory •  Null pointer dereferencing •  Array and buffer overflows •  Division by zero •  Memory or resource leaks •  Dead code •  Code complexity (by module) ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  90. 90. #AgileMetrics @XBOSoft @philiplew Even More •  Common metrics include: – Number of requirements to test - by function - by priority – Number of test cases - created - executed - passed - failed – Number of defects - by severity - by status Develop a Measurements and Metric Catalog ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  91. 91. #AgileMetrics @XBOSoft @philiplew Exercise Work  in  your  groups  and  list  out:   •  The  metrics  you  currently  use   –  What  you  use  them  for   –  Who  wants  them   –  How  you  evaluate  them   –  What  decisions  they  make  based  on  them   •  The  metrics  you  don’t  currently  use  but  think  you  should   You may find some holes here. What are the common ingredients missing? ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  92. 92. #AgileMetrics @XBOSoft @philiplew INDICATORS,  ANALYSIS,  AND   CONTEXT   EvaluaQng  and  InterpreQng  Metrics   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  93. 93. #AgileMetrics @XBOSoft @philiplew Don’t – Forget indicators •  Metrics without indicators cannot be evaluated – You collect mounds of data but then what? – How do you determine what is ‘good’ or ‘bad’? – Need a way to evaluate the numbers you get with meaningful indicators that can be used as benchmarks as you/your process and organization matures ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  94. 94. #AgileMetrics @XBOSoft @philiplew Indicators   •  Indicators  change  numbers  into  meaning   •  What  does  68%  mean?   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  95. 95. #AgileMetrics @XBOSoft @philiplew Indicators   •  Need  an  elementary  indicator  to  interpret  the   level  of  saQsfacQon  met  with  decision  criteria   with  acceptability  ranges  in  a  percentage   scale:     – 0-­‐40  (unsaQsfactory  –red)  means  changes  must   take  place  with  high  priority;   – 40-­‐70  (marginal  –yellow)  indicates  a  need  for   improvement  acQons;     – 70-­‐100  indicates  a  saQsfactory  level  –green-­‐  for   the  analyzed  agribute.     ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  96. 96. #AgileMetrics @XBOSoft @philiplew Beware of Causality •  The  cause  and  effect  relaQonship  has  three  requirements:   –  cause  precedes  effect  in  Qme  or  by  logic.   –  shown  cause  and  effect  relaQonship   –  direct  constant  relaQonship   •  When  the  relaQonship  between  variables  can  be  graphed  and  a  line  of  best  fit  can  be  found   •  Be  careful  of  extreme  and  inaccurate  data  values.   •  When  defining  a  casual  relaQonship  be  sure  to  watch  for  misguided  interpreted   relaQonships.            CASE  1                  Z  causes  changes  in  X                  Z  causes  changes  in  Y                  X  is  perceived  to  cause  changes  in  Y  or  vice  versa  and  Z  is  overlooked            CASE  2                  X  causes  changes  in  Z                  Z  causes  changes  in  Y                  X  is  perceived  to  cause  changes  in  Y  and  Z  is  overlooked   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  97. 97. #AgileMetrics @XBOSoft @philiplew Mistakes in Causality ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  98. 98. #AgileMetrics @XBOSoft @philiplew Look at Trends •  Identifying trends over time – More – Less – Stable •  Make informed decisions – Identify what is happening – Determine what you want to happen – Make adjustments ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  99. 99. #AgileMetrics @XBOSoft @philiplew Time and Beware of Absolutes •  Don’t use absolute numbers to determine if a build is ready to ship. – If they have released a product for several years metrics can change over time. – What reasons can metrics change over time? – What particular elements of agile can cause metrics to change over time? ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  100. 100. #AgileMetrics @XBOSoft @philiplew Don’t – Forget about context •  Metrics don’t have consistent context so they are unreliable – Context needs to be defined and then maintained for measurements to be meaningful. •  Difficult in today’s environment with changing test platforms and other contextual factors. ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  101. 101. #AgileMetrics @XBOSoft @philiplew What contextual factors could there be? •  Release complexity •  Development methodology •  Software maturity •  Development team maturity and expertise •  Development team and QA integration •  Resources available •  User base All metrics need to be normalized for proper interpretation ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  102. 102. #AgileMetrics @XBOSoft @philiplew Exercise •  List out contextual factors in our organization that are needed to ensure proper interpretation of any metrics and measurements •  List out any past mistakes your organization has made when using and interpreting metrics ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  103. 103. #AgileMetrics @XBOSoft @philiplew All this is good But in reality…
  104. 104. #AgileMetrics @XBOSoft @philiplew Why We Measure? •  Our bosses want us to… •  They want someone to point fingers at •  They need to report to their managers •  They want some basis on which to evaluate us •  We want to improve our work and improve software quality •  How do you want to be evaluated? ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  105. 105. #AgileMetrics @XBOSoft @philiplew The Metrics Conundrum •  QA and Testing Language – Defects – Execution status – Test cases – Pass/fail rates – DRE… •  Business  Language   –  Cost  effecQve   –  ROI   –  Cost  of  ownership   –  Cost  of  poor  quality   –  ProducQvity   –  Calls  to  help  desk   –  Customer   saQsfacQon   –  Customer  retenQon   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  106. 106. #AgileMetrics @XBOSoft @philiplew Metrics In The Real World •  What measurements do you take in your organization and why? •  Who uses them and for what? •  Measurement is pie. It takes 2-3 hours to make and 15 minutes to consume… •  But… many metrics are never reviewed or analyzed (consumed) •  WHY? ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  107. 107. #AgileMetrics @XBOSoft @philiplew The Metric Conundrum (cont.) •  Rarely have the right metrics to show or quantify value •  Metric collection and reporting are a drag •  QA metrics usually focus only on test execution •  Test tools don’t have the metrics we want •  Metrics are not connected to anything of value/meaningful for ________. ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  108. 108. #AgileMetrics @XBOSoft @philiplew Poll: How many of you started to collect metrics but then found it was too difficult or time consuming and quit? Never started: Started then quit: Still collecting and reporting: 13% 30% 57%
  109. 109. #AgileMetrics @XBOSoft @philiplew Module 4: Goals, Metrics and Frameworks
  110. 110. #AgileMetrics @XBOSoft @philiplew STAKEHOLDERS UNDERSTANDING THE CONSUMERS OF OUR METRICS Developing  and  reporQng  metrics  that  mager   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  111. 111. #AgileMetrics @XBOSoft @philiplew Stakeholders •  Who are they? •  What do they want to know and why? ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  112. 112. #AgileMetrics @XBOSoft @philiplew Possible Stakeholders Stakeholder   Objec]ve   CIO   CEO   VP   Director  -­‐   Development   Owner   Auditor   Partner   ?   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  113. 113. #AgileMetrics @XBOSoft @philiplew Exercise •  List all potential stakeholders and what their objectives or goals are? ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  114. 114. #AgileMetrics @XBOSoft @philiplew GQM  FRAMEWORK   ConnecQng  goals,  objecQves  to  quesQons  and  answers  (metrics)  with   frameworks   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  115. 115. #AgileMetrics @XBOSoft @philiplew Organizing Our Thoughts •  Putting metrics into a framework helps us to categorize and organize •  Helps us to make sure we have all functions in our organization and process covered •  Helps us to develop relationships between different elements in our process ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  116. 116. #AgileMetrics @XBOSoft @philiplew Measurement Frameworks •  Many  soFware  metrics  programs  have  failed  because   they  had  poorly  defined,  or  even  non-­‐existent  objecQves.     •  Basili  developed  a  goal  oriented  approach  (GQM)  to   measurement  according  to  the  following  three  stages:   –  Set  goals  specific  to  needs  in  terms   of  purpose,  perspec%ve  and  environment.   –  Refine  the  goals  into  quanQfiable  quesQons  that  are   tractable.   –  Deduce  the  metrics  and  data  to  be  collected  (and  the   means  for  collecQng  them)  to  answer  the  quesQons.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  117. 117. #AgileMetrics @XBOSoft @philiplew GQM for Software Quality •  Overall  GOAL  is  to  evaluate  the  quality  of  the  soFware   •  To  decide  if  the  quality  is  ‘good’,  several  key  QUESTIONs  must  be  asked  to   understand  how  to  meet  the  goal  and  determine  quality   •  Once  these  quesQons  are  idenQfied,  analyze  each  quesQon  to   determine  what  must  be  measured  (and  an  associated  METRIC   determined)  in  order  to  answer  the  quesQon.     •  Generate  those  metrics  that  are  related  to  the  goal.     •  Several  measurements  may  be  needed  to  answer  a  single  quesQon   while  a  single  measurement  may  apply  to  more  than  one  quesQon.     •  GOAL  provides  the  purpose  for  collec%ng  the  data,  and  the   ques%ons  tell  you  data  needed  to  collect/calculate.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  118. 118. #AgileMetrics @XBOSoft @philiplew Applying Basili’s GQM •  Many  soFware  metrics  programs  have  failed  because  they   had  poorly  defined,  or  even  non-­‐existent  objecQves.     Goal   QuesQon  Metric   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  119. 119. #AgileMetrics @XBOSoft @philiplew Stakeholders and GQM Goal  1   Goal  2   Goal  3   QuesQon   1   QuesQon   2   QuesQon   3   QuesQon   4   QuesQon   5   Metric   1   Metric   2   Metric   3   Metric   4   Metric   5   Metric   6   Metric   7   QA  Manager  Dev  Mgr  CTO  CEO   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  120. 120. #AgileMetrics @XBOSoft @philiplew GQM - Example •  Goals –  Reduce total cost of development –  Reduce total cost of testing-effort –  Reduce the number of new feature bugs •  Questions –  Which areas have the highest re-work rates? –  Which functional areas have the most defects? –  How long does it take to repair defects? –  How complete are my requirements? –  How much time do I spend testing? ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  121. 121. #AgileMetrics @XBOSoft @philiplew Exercise   Stakeholder  Workshop  (15  minutes)   1.  List out stakeholders 2.  Brainstorm stakeholder questions (3-5 per stakeholder) 3.  Determine answers and metrics to that question 4.  Group your metrics into categories 5.  List out sources of data/information 6.  Present and discuss to the group ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  122. 122. #AgileMetrics @XBOSoft @philiplew Metric Q&A Worksheet Category   Ques]on   Metric   Tool  to  Use   Needed   Customer   SaQsfacQon   Are  we  meeQng   schedule?   Schedule   Variance   Project   Management   Tool   None   SoFware   TesQng   Efficiency   SoFware   Quality   What  kinds  of   defects  are  most   common?   Usability   Are  end  users   ge~ng  lost,   understand  our   UI   SoFware  Test   EffecQveness   ?   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  123. 123. #AgileMetrics @XBOSoft @philiplew Possible Stakeholders Stakeholder   Objec]ve   Ques]on   Answer   CIO   CEO   VP  –  QA  –   Development   Director  -­‐   Development   Owner   Auditor   Test  Lead   Partner   ?   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  124. 124. #AgileMetrics @XBOSoft @philiplew Agile  ObjecQves   Waterfall   •  Speed   •  Quality   •  Cost   Agile   •  Speed   •  Quality   •  Cost   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   Where’s the beef?
  125. 125. #AgileMetrics @XBOSoft @philiplew RETURN  TO  BASICS   Agile  Manifesto   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  126. 126. #AgileMetrics @XBOSoft @philiplew Excerpts  from  the  Agile  Manifesto   1.  Working  soFware  over  comprehensive  documentaQon   2.  Our  highest  priority  is  to  saQsfy  the  customer   through  early  and  conQnuous  delivery  of  valuable  soFware.   3.  Deliver  working  soFware  frequently,  from  a  couple  of  weeks  to  a   couple  of  months,  with  a  preference  to  the  shorter  Qmescale.   4.  Business  people  and  developers  must  work  together  daily   throughout  the  project.   5.  Working  soFware  is  the  primary  measure  of  progress.   6.  Agile  processes  promote  sustainable  development.  The  sponsors,   developers,  and  users  should  be  able  to  maintain  a  constant  pace   indefinitely.   7.  ConQnuous  agenQon  to  technical  excellence  and  good  design   enhances  agility.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  127. 127. #AgileMetrics @XBOSoft @philiplew Agile  –  Understanding  Where  We  Came  From?   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  128. 128. #AgileMetrics @XBOSoft @philiplew What  We  Can  Agree  On   •  Want  to  minimize  mistakes  and  doing  things   over  (rework)   •  Want  to  uQlize  resources  effecQvely   •  Want  to  predict  when  things  could  go  wrong   •  Want  to  do  what  we  say  we  will  do,  when  we   say  -­‐  Qmelines   •  Fix  defects  early  –  later  is  more  expensive   •  ProducQon  defects  are  bad  news     ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  129. 129. #AgileMetrics @XBOSoft @philiplew Agile  –  How  Can  We  Improve?   •  Some  say  no  metrics  needed   •  Is  velocity  all  you  track?   •  What  is  sacrificed  for  velocity?   •  How  can  we  know  what  and  how  we  are  doing   or  where  to  improve?   •  How  do  we  know  if  we’re  using  resources   effecQvely?   •  How  do  we  know  we  are  spending  our  effort/ Qme  in  the  right  place?   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  130. 130. #AgileMetrics @XBOSoft @philiplew What  are  the  key  characterisQcs  of  agile?   Agile  Characteris]c   CollaboraQve   Working  SoFware   Adapt  to  changing  requirements   ?   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  131. 131. #AgileMetrics @XBOSoft @philiplew What  are  the  Goals  of  Agile?   Speed   Quality   Customer-­‐User   Sa;sfac;on   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  132. 132. #AgileMetrics @XBOSoft @philiplew Poll   •  How  do  you  get  (end)  customer-­‐user   feedback?   – Surveys   – Defect  metrics  in  producQon   – Server  logging   – Other   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  133. 133. #AgileMetrics @XBOSoft @philiplew Matching  Agile  CharacterisQcs  and   ObjecQves   Agile   Characteris]c   Speed   Quality   Customer  and   End  User   Sa]sfac]on   Other   CollaboraQve   x   x   Working   SoFware   x   Adapt  to   changing   requirements   x   x   From the agile manifesto ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  134. 134. #AgileMetrics @XBOSoft @philiplew What  QuesQons  Tell  Me  If  I’m   Reaching  Those  ObjecQves?   Speed   Quality   Owner   ParQcipaQon   Requirements   Working   Product   Planned  versus   unplanned   Schedule   Technical  debt   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  135. 135. #AgileMetrics @XBOSoft @philiplew What  QuesQons  Tell  Me  If  I’m   Reaching  Those  ObjecQves?   Speed   Requirements   Did  I  get  them  in  Qme?   Do  I  understand  them?   Did  we  deliver  working  product?   Schedule   Am  I  on  schedule?   Can  I  be  more  producQve?   How  much  wasted  Qme?  Rework?   Unplanned   versus  planned   Did  I  esQmate  well?   Did  I  finish  what  I  said  I  would?   Do  I  make  mistakes  that  cause  more     (unplanned)  work?   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  136. 136. #AgileMetrics @XBOSoft @philiplew What  QuesQons  Tell  Me  If  I’m   Reaching  Those  ObjecQves?   Quality   Product   owner   Did    my  end  users  find  defects?   Did  product  owner  give  feedback   regularly?   Are  there  less  defects  than  before?   Requirements   Are  my  requirements  delivered  when   needed?   Did  the  delivered  product  match  the   requirement?  Or  reqt.  wrong?   Do  engineers  have  trouble   understanding  the  requirements?   Technical   Debt   Is  my  soFware  performance  ge~ng   worse  and  worse?   Am  I  prepared  if  my  lead  engineer   leaves?   Do  I  mess  things  up  when  fixing  things?   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  137. 137. #AgileMetrics @XBOSoft @philiplew METRICS  THAT  HELP  US  ANSWER   THESE  QUESTIONS!   ARE  YOU  READY  FOR  THE   COUNTDOWN?   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  138. 138. #AgileMetrics @XBOSoft @philiplew VELOCITY   10.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   Am I on schedule? Did I estimate well?
  139. 139. #AgileMetrics @XBOSoft @philiplew Velocity   •  Tracking  our  velocity  over  Qme  tells  us  our  progress   and  capacity  and  helps  us  forecast  based  on  our   capabiliQes  (now  and  in  the  future).   •  Measurement  –  Sum  of  all  the  approved  esQmates   of  the  stories   •  One  point  in  Qme  measurement  has  ligle  meaning,   so  need  to  track  over  Qme  or  wrt  another  metric   •  Metric   – Velocity  (T)-­‐(T-­‐1)/Velocity  (T)     – Velocity  (T)-­‐(T-­‐1)/TIME  UNIT   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  140. 140. #AgileMetrics @XBOSoft @philiplew Velocity   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   -­‐25%   -­‐20%   -­‐15%   -­‐10%   -­‐5%   0%   5%   10%   15%   20%   25%   30%   1   2   3   4   5   6   7   8   9   Velocity  Variance  
  141. 141. #AgileMetrics @XBOSoft @philiplew Burn  Down  –  (Distance  over  Time)   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   What information and insights could we gather here?
  142. 142. #AgileMetrics @XBOSoft @philiplew DEFECT  REMOVAL  EFFECTIVENESS   9.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   Did my end users find defects? Are there less defects than before?
  143. 143. #AgileMetrics @XBOSoft @philiplew Defect  Removal  EffecQveness   •  We  don’t  want  the  end  user  to  find  defects   •  Any  defect  found  by  the  user  influences   quality   •  Metric  –  Defects  found  in  producQon  during   (90  day)  period  post  release  (DFP)/DFP  +   Defects  found  Pre-­‐release   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  144. 144. #AgileMetrics @XBOSoft @philiplew What  Should  Your  Goal  Be  For  DRE   •  EsQmated  (by  Capers  Jones)  that  defect   removal  effecQveness  differs  by  different   levels  of  process  capability  maturity  levels:   – Level  1:      85%   – Level  2:      89%   – Level  3:      91%   – Level  4:      93%   – Level  5:      95%     ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  145. 145. #AgileMetrics @XBOSoft @philiplew DRE  DistribuQon   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   C A
  146. 146. #AgileMetrics @XBOSoft @philiplew ADDED  WORK   8.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   Am I on schedule? Am I overcommitting? Is my rate of work sustainable?
  147. 147. #AgileMetrics @XBOSoft @philiplew Added  Work   •  This  will  help  us  see  if  we  are  making   promises  that  maybe  we  should  not  –   overcommi~ng  and  unsustainable.   •  Or  Changes  that  cause  stopping  and  starQng.   •  Measurement  –  Sum  of  work  (new  stories   added)  over  and  above  the  original  plan.   •  Metric  –  New  stories  added/Original  Stories   Planned   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  148. 148. #AgileMetrics @XBOSoft @philiplew Piling  On   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   0   5   10   15   20   25   30   35   1   2   3   4   5   6   7   8   9   Revised  Planned   Original  Plan  
  149. 149. #AgileMetrics @XBOSoft @philiplew WORK  CATEGORY  ALLOCATION   7.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   Am I allocating resources optimally? Do I have trouble understanding requirements?
  150. 150. #AgileMetrics @XBOSoft @philiplew Work  Category  AllocaQon   •  This  will  help  us  see  where  we  are  spending  our  Qme     and  adjust  our  prioriQes.   •  Measurement  –  Sum  of  work  (for  each  category)   Metric  –  Divide  by  total  work  to  get  %  allocaQon   •  Categories  could  include:   –  Technical  debt  –  refactor  -­‐  redo   –  Regression  tesQng   –  New  feature  development   –  Research   –  Requirements  review   –  Defect  fixing   –  Other   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  151. 151. #AgileMetrics @XBOSoft @philiplew Is  Effort  in  Alignment?   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   0%   5%   10%   15%   20%   25%   30%   Are we spending enough time showing our stakeholders what we’ve done? Are we fixing things too much or making lots of mistakes?
  152. 152. #AgileMetrics @XBOSoft @philiplew WORK  SUSTAINABILITY   6.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   Do I have the right people on my team? Did I estimate work well? Can I attain quality sustainably?
  153. 153. #AgileMetrics @XBOSoft @philiplew Work  Sustainability   •  “Sustainable”  -­‐  We  can’t  sustain  our  effort  if  we  are   conQnuously  pu~ng  in  overQme   •  OverQme  is  necessary,  but  needs  to  be  managed  and   causes  mistakes   •  Mistakes  manifest  themselves  in  many  ways,  such  as   technical  debt  or  defects   •  OverQme  oFen  results  from  poor  work  esQmates   and  directly  proporQonal  to  technical  debt   •  Metric  –  Unplanned  OverQme  Work/Total  Work   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  154. 154. #AgileMetrics @XBOSoft @philiplew Unplanned  OverQme  %   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   0%   2%   4%   6%   8%   10%   12%   14%   16%   1   2   3   4   5   6   7   8   9   Sprint What caused this?
  155. 155. #AgileMetrics @XBOSoft @philiplew Unplanned  OverQme  and  Defects   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   0   1   2   3   4   5   6   7   1   2   3   4   5   6   7   Defects  Found   Unplanned  OT  
  156. 156. #AgileMetrics @XBOSoft @philiplew DEFECT  REWORK  RATE   5.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   Is my technical debt accumulating? Am I wasting time?
  157. 157. #AgileMetrics @XBOSoft @philiplew Defect  Rework  Rate   •  We  can’t  get  faster  if  we  are  always  fixing   things.   •  Depends  on  valid  data  whereby  people  are   logging  real  hours  to  tasks  they  take  on.     •  Metric  -­‐  Fix  Effort  RaQo  :  Time  to  fix  defects/ Total  effort  expended  (hours)   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  158. 158. #AgileMetrics @XBOSoft @philiplew Defect  ReWork  %   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   0%   5%   10%   15%   20%   25%   30%   35%   1   2   3   4   5   6   7   8   9  
  159. 159. #AgileMetrics @XBOSoft @philiplew Accumulated  Debt   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   0%   10%   20%   30%   40%   50%   60%   70%   80%   90%   100%   1   2   3   4   5   6   7   8   9   Defect  Rework%   Unfixed  %  
  160. 160. #AgileMetrics @XBOSoft @philiplew TOTAL  UNEXPECTED  WORK   4.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   What is throwing me off schedule? Am I biting off too much?
  161. 161. #AgileMetrics @XBOSoft @philiplew Total  Unexpected  Work   •  Measurement  –  Sum  of  work  done  above  the   original  esQmates  for  those  stories  in  the  plan.   •  Are  we  underesQmaQng  difficulty?   •  Many  reasons:   –  Incomplete  user  stories   –  Misunderstood  user  stories   –  Bigger  and  more  complex  than  I  thought?   –  Not  enough  Qme  for  research   –  Too  hard…   •  Metric  –  (Total  work  done  –  Planned  work)/ Planned  work   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  162. 162. #AgileMetrics @XBOSoft @philiplew Are  We  EsQmaQng  Well?     ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   0   10   20   30   40   50   60   1   2   3   4   5   6   7   8   9   Planned   Unplanned   Per Person/Week During the Sprint
  163. 163. #AgileMetrics @XBOSoft @philiplew What  are  we  doing  that  is   unplanned?   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   Research   Requirements   Coding   TesQng   Other   29% 9% 19% 23% 20%
  164. 164. #AgileMetrics @XBOSoft @philiplew DEFECTS  IN  PRODUCTION  AVG.  FIX   TIME   3.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   Are my customers/end users happy? Is the architecture prohibiting me?
  165. 165. #AgileMetrics @XBOSoft @philiplew Defect  in  ProducQon  Avg.  Fix  Time   •  If  defects  make  it  to  producQon,  that’s  bad  !   •  But  what  is  worse  if  an  important  defect  stays   unfixed.   •  Defects  that  take  a  long  Qme  to  fix  represent  either   technical  debt  or  poor  process   •  Metric  –  Time  to  fix  Defects  (released  to  producQon-­‐ usually  P1  defects  found)  /  Total  P1  Defects  (post   producQon)   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  166. 166. #AgileMetrics @XBOSoft @philiplew Don’t  Make  Them  Wait   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   0   10   20   30   40   50   60   1   2   3   4   5   6   7   8   9   #P1  Defects  Found   Average  Fix  Time  (Hrs)   Hey, we’re improving! WooHoo! What could cause long times to fix defects?
  167. 167. #AgileMetrics @XBOSoft @philiplew DEFECT  CATEGORIZATION   ALLOCATION  %   2.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   Why are things breaking? Can I get faster?
  168. 168. #AgileMetrics @XBOSoft @philiplew Defect  CategorizaQon  AllocaQon   •  Reduce  rework  =  Increase  speed   •  Discover  when  defects  are  injected     •  Reduce  and  prevent  defects  from  happening   in  the  first  place.     •  Drive  down  the  Fix  Effort  RaQo  leading  to   beger  quality  and  resulQng  in  higher  velocity.   •  Examine  defect  rates  per  category   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  169. 169. #AgileMetrics @XBOSoft @philiplew Defect  Rework  AllocaQon   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   Requirements   Coding   Environment   TesQng   38% 29% 21% 13% What’s going on here?
  170. 170. #AgileMetrics @XBOSoft @philiplew Rework-­‐Requirements  Errors   Requirements     •  Change  in  requirements   •  Don’t  understand  requirements   •  Incomplete  requirement   •  Requirement  does  not  exist   •  ?   ‘oh, I meant this…’ ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  171. 171. #AgileMetrics @XBOSoft @philiplew Poll   •  What  actual  improvements  have  you  seen  from   using  Agile?   – Accelerated  release  of  product   – Increase  in  producQvity   – Reduced  risk   – Reduced  cost   – Simplified  Dev  and  Test   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  172. 172. #AgileMetrics @XBOSoft @philiplew WORKING  SOFTWARE  DELIVERY   RATE   1.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   Am I delivering what I said I would? Am I getting feedback from the PO? Will I meet the release date?
  173. 173. #AgileMetrics @XBOSoft @philiplew Working  SoFware  Delivery  Rate   •  Does  the  soFware  work  or  not?   •  Working  does  not  mean  perfect   •  Working  means  can  show  in  order  to  get  feedback   •  Metric  -­‐  #  story  points  delivered  that  ‘work’  /  #  story   points  that  were  to  be  shown  to  the  PO  but  could   not  be  shown  to  work   –  Many  variaQons   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  174. 174. #AgileMetrics @XBOSoft @philiplew Am  I  Ge~ng  Beger?   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   0   5   10   15   20   25   1   2   3   4   5   6   7   8   9   Planning  Working  Visible   Actual  Working  
  175. 175. #AgileMetrics @XBOSoft @philiplew Am  I  Delivering  So  I  can  Get   Feedback?   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   0%   20%   40%   60%   80%   100%   1   2   3   4   5   6   7   8   9   %  Working  
  176. 176. #AgileMetrics @XBOSoft @philiplew Be   Do   Have   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.   Process •  Iterative (sprints) •  Daily standups •  Face to face communication •  Post mortem – end of sprint •  Delivery meeting – end of sprint •  Planning meeting – before sprint •  Self organizing People •  Communicative •  Collaborative/Cooperative •  Flexible and willing •  Knowledgeable-multi •  Initiative/responsible •  Responsive/adaptive Results •  Speed •  Quality Agile  Is  a  Way  of  Thinking   “Be-­‐Do-­‐Have”  
  177. 177. #AgileMetrics @XBOSoft @philiplew Take  Aways   •  Mistakes  people  make  in  (agile)  metrics  and  how  to   avoid  them.   •  How  to  examine  root  causes  of  low  velocity.   •  How  to  reduce  rework.   •  How  to  analyze  your  agile  process  and  determine   meaningful  metrics  to  present  to  management.   •  How  to  get  to  velocity  AND  quality,  and  what  to   measure.   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  178. 178. #AgileMetrics @XBOSoft @philiplew •  What  and   why   •  Goals  and   objecQves   Understand   Ask   yourself   quesQons   Evaluate   Measure   to  answer   those   quesQons   and  take   acQon   Improve   SUMMARY   ©  2015  XBOSoF,  Inc.-­‐  All  Rights  Reserved.  
  179. 179. #AgileMetrics @XBOSoft @philiplew Post your questions on Twitter and I'll answer them @philiplew Join us to keep updated on all our webinars, reports and whitepapers: facebook.com/xbosoft +xbosoft linkedin.com/company/xbosoft We post regularly on our blog – check us out! http://xbosoft.com/software-quality-blog/ Why not download our free Whitepapers, available here: http://xbosoft.com/knowledge-center/ Thanks Agile Metrics White Paper

×