Weitere ähnliche Inhalte Ähnlich wie Establishing Release Quality Levels and Release Acceptance Tests (20) Kürzlich hochgeladen (20) Establishing Release Quality Levels and Release Acceptance Tests2. www.conteneo.co
• Founder/CEO of Conteneo Inc.
• Agile Product Management consulting
• Customer needs, roadmaps, business model
• Product management mentoring and training
• Agile product guy
• VP Bus Dev (Aladdin), VP Eng & Product Dev’t
(Aurigin), VP Systems Eng (EDS Fleet Services)
• Board of Agile Alliance
• Author, speaker, blogger
• “Innovation Games”
• “Beyond Software Architecture”
• “Journey of the Software Professional”
• Agile PM blog at www.conteneo.co
About Luke Hohmann
© Copyright 2014 Conteneo, Inc. 2
3. www.conteneo.co
Goals
1. Provide a framework for thinking about
quality
2. Create better results by building the right
quality
3. Practical advice to
help you succeed
What
did I
forget?
© Copyright 2014 Conteneo, Inc. 3
4. www.conteneo.co
Agenda
• Some discussion – what’s a release?
quality?
• The iron triangle or the quality box?
• Release quality levels
• Release acceptance tests
NOT
A TOOLS
TALK
© Copyright 2014 Conteneo, Inc. 4
6. www.conteneo.co
What do we offer? And why?
• We offer things to our market ecosystem
(customers, analysts, etc.) to achieve goals
• A release is offered to generate revenue
• Lo-fi prototypes are offered to improve designs
• Betas are offered to gain pre-release feedback
(and maybe generate revenue…)
• To be offered a thing has to have a “quality”
that is suitable to helping us realize the goal
© Copyright 2014 Conteneo, Inc. 6
7. www.conteneo.co
What is Quality?
• “Quality is conformance to user requirements.”
—Phillip Crosby, Quality is Free (1980s)
• “Quality is the absence of defects that would make
software stop completely or produce unacceptable
results.”
—Capers Jones, Applied Software Measurement (1991)
• “Quality is achieving excellent levels of fitness for use,
conformance to requirements, reliability, and
maintainability.”
—Watts Humphrey, Managing the Software Process
(1980s)
• “Quality is value to some person.”
—Jerry Weinberg, Quality Software Management
(1992)
© Copyright 2014 Conteneo, Inc. 7
8. www.conteneo.co
Is that enough?
• What about the manner in which we create
“the thing”? What about the code?
• What if the code is just plain ugly?
• And if you’re using an Agile method, what
if your code doesn’t pass the green bar?
© Copyright 2014 Conteneo, Inc. 8
9. www.conteneo.co
Quality is a Relationship
• High Performance is High Quality
• to users who notice low performance.
• Elegant Coding is High Quality
• to developers who place high value on the
opinions of their peers.
• Zero Defects is High Quality
• to users who would be disturbed by those
defects.
• Lots of Features is High Quality
• to marketers who believe that features sell
products.
© Copyright 2014 Conteneo, Inc. 9
10. www.conteneo.co
The Dichotomy of Quality
• Intrinsic or Extrinsic?
• Does quality exist in the things we
observe or is it subjective, existing
only in the eye of the observer?
“You take your analytic knife, put the point directly
on the term Quality and just tap, not hard, gently,
and the whole world splits, cleaves, right in two—hip
and square, classic and romantic, technological and
humanistic—and the split is clean.”
—Robert Pirsig
Zen and the Art of Motorcycle Maintenance (1974)
© Copyright 2014 Conteneo, Inc. 10
13. www.conteneo.co
Should the triangle be a box?
Scope
Cost Schedule
The Traditional Iron Triangle
Value
(Extrinsic quality)
Quality
(Intrinsic quality)
The Quality Box
Cost Schedule
© Copyright 2014 Conteneo, Inc. 13
14. www.conteneo.co
It is a system
Scope
Cost Schedule
The Traditional Iron Triangle
Value
(Extrinsic quality)
Quality
(Intrinsic quality)
The Quality Box
Cost Schedule
© Copyright 2014 Conteneo, Inc. 14
16. www.conteneo.co
Choosing to affect the system
Value
(Extrinsic quality)
Quality
(Intrinsic quality)
The Quality Box
Cost Schedule
Let’s hold these as fixed…
Extrinsic Quality
• Which features present?
• How well do the work?
• How well have they been tested?
• Do you have enough?
• Is this release ready when I need it?
Intrinsic Quality
• Craftsmanship of the code
• Maintainability
• Level of defects (works as specified)
© Copyright 2014 Conteneo, Inc. 16
17. www.conteneo.co
Agenda
• Some discussion – what’s a release?
quality?
• The iron triangle or the quality box?
• Release quality levels
• Release acceptance tests
© Copyright 2014 Conteneo, Inc. 17
18. www.conteneo.co
Release quality level
• A predetermined quality level chosen by
product management that sets
expectations regarding the intrinsic and
extrinsic quality of a release.
• Enables organizations to make confident
decisions about their context
• Negotiated with development
© Copyright 2014 Conteneo, Inc. 18
19. www.conteneo.co
Example from VeriSign Managed Security Services
Release Level Description
Release Level 5 All functionality fully certified.
Near-zero probability of high severity errors and a low probability that medium severity errors.
Release Level 4 All new and modified functionality fully certified, except where risk is deemed minimal (e.g.
internal facing reports, low probability use cases).
Features not fully certified documented in the test plan and/or project charter. Strategic
regression testing of existing functionality performed.
Low probability that high severity errors will be identified in production. Project carries a
moderate probability that medium severity errors.
Release Level 3 All new and modified functionality at least partially certified.
Features not fully certified documented in the test plan and/or project charter.
Partial regression testing performed.
Moderate probability for high severity errors. Higher probability for medium severity errors.
Release Level 2 Most new and/or modified functionality partially certified.
Features not fully certified communicated to stakeholders.
Partial regression testing performed.
Higher probability that high and medium severity errors will be identified.
Release Level 1 New and/or modified features not certified by QA.
Regression testing may or may not occur.
Level 1 releases may be available only for demo or controlled access purposes.
High risk for high severity errors.
19
20. www.conteneo.co
Set each scale separately
Intrinsic Quality Extrinsic Quality
5 • Everything certified!
• Lots of automation!
5 • Customers will rave about this!
4 • All new stuff tested by QA
• Full regression testing
4 • Consistency very high
• Simple and sophisticated tasks
3 • All new stuff tested by QA
• Almost full regression testing
3 • Documented
• Complete for basic tasks
2 • Most new stuff tested by QA
• Partial regression tested
2 • Not documented
• Complete for basic tasks
1 • Not tested by QA
• Not regression tested
1 • Not documented
• Possibly incomplete
© Copyright 2014 Conteneo, Inc. 20
21. www.conteneo.co
Lower bound for intrinsic quality is very high
• Leading edge enterprises employ technologies that
can approach 99% cumulative defect removal rates.
• The norm for US firms is a cumulative defect
removal rate of 75%.
• A cumulative defect removal rate of 95% on a
project appears to be a nodal point where several
other benefits accrue. For projects of similar size
and type, these projects:
• have the shortest schedules.
• have the lowest quantity of effort in terms of person-
months
• have the highest levels of user satisfaction after release
—Capers Jones, Applied Software Measurement
(1991)
© Copyright 2014 Conteneo, Inc. 21
22. www.conteneo.co
Why is Intrinsic Quality so Important?
• The Impact of code quality on testing
• Error Location Dynamics
• Error Feedback Ratio
• Technical Debt
© Copyright 2014 Conteneo, Inc. 22
23. www.conteneo.co
Impact of Code Quality on Testing
Development: 10 days,
4 people, 4 KLOC,
1 d/KLOC
Development: 10 days,
4 people, 4 KLOC,
15 d/KLOC
How long to test? Assume
½ day to find & fix per defect.
Test time=
2 days
Test time=
30 days
Outcome: no time to finish testing,
technical debt increases!
© Copyright 2014 Conteneo, Inc. 23
25. www.conteneo.co
The time to finish removing errors
is critically dependent on the error
feedback ratio. The three simulations
differ only in their feedback ratios.
A 20% difference in feedback ratio
leads to an 88% difference in
completion time, but the next 10%
increase leads to a 112% increase.
ERROR FEEDBACK: Errors put into a system when attempting to correct
other faults.
ERROR FEEDBACK RATIO: The number of problems created per fix.
EFR = ERRORS CREATED / ERRORS RESOLVED
Error Feedback Ratio
© Copyright 2014 Conteneo, Inc. 25
26. www.conteneo.co
Technical Debt
• Once on far right of
curve, all choices are
hard
• If nothing is done, it just
gets worse
• In applications with high
technical debt,
estimating is nearly
impossible
• Only 3 strategies
• Do nothing, it gets worse
• Replace, high cost/risk
• Incremental refactoring,
commitment to invest1 2 3 4 5 6 7 8
Years
Technical Debt
CostofChange(CoC)
Product
Release
Actual
CoC
Optimal CoC
Customer
Responsiveness
© Copyright 2014 Conteneo, Inc. 26
27. www.conteneo.co
Agenda
• Some discussion – what’s a release?
quality?
• The iron triangle or the quality box?
• Release quality levels
• Release acceptance tests
© Copyright 2014 Conteneo, Inc. 27
28. www.conteneo.co
Lots of Testing Options!
Customer
Tests
Business Intent
(Fit, Fitnesse)
Usability
Testing
Exploratory
Testing
Programmer
Tests
Design Intent
(xUnit)
Property
Testing
Response,
Security
Scaling,…
From Brian Marick http://www.testing.com/cgi-bin/blog/2003/08/21#agile-testing-project-1
Run at least once each
development episode
Run each time a story
scenario is completed Assess each iteration
Assess each iteration
© Copyright 2014 Conteneo, Inc. 28
29. www.conteneo.co
Release Acceptance Tests
• Tests owned and written by the product
manager / product owner / customer to
verify that a story is complete and
correct.
• Should be (mostly) automated (more later)
• Popularized by Agile methods, useful
everywhere!
© Copyright 2014 Conteneo, Inc. 29
30. www.conteneo.co
Benefits of Acceptance Tests
• Increase team confidence that the system
is correct
• Help PMs think through requirements
• Informs developer estimates
• Concrete definition of completion
• Early involvement of QA
• Automated regression testing for free
© Copyright 2014 Conteneo, Inc. 30
31. www.conteneo.co
ATs test the users’ experience
• UTs should test application logic in detail
generally in isolation
• ATs should test: but not...
• Interaction & Flow Usability
• Performance Look & Feel
• Error Handling
• Security
© Copyright 2014 Conteneo, Inc. 31
33. www.conteneo.co
Acceptance Tests are “Quirky”
• Written by PMs/Customers... But executed in
code?
• ATs require a customer meta-language generic
enough to capture requirements, abstract enough to
be maintainable
• Simple enough to be easily understood
• They are interpreted by various frameworks
• Requires external systems and “spike” development
• Automation tools dictate form/format of tests
• http://www.fitnesse.org/
• http://seleniumhq.org/
• http://robotframework.org
© Copyright 2014 Conteneo, Inc. 33
36. www.conteneo.co
Writing good acceptance tests
• Tell a story of accomplishing a goal
• Goal test title
• Activities rows in your test suite
• Expect them to be stable, even if the
underlying system is going through a lot of
change
• Don’t reference domain objects
© Copyright 2014 Conteneo, Inc. 36
37. www.conteneo.co
Consider “Given-When-Then”
• Given some context
When something happens
Then some behavior
• Given an innovation games planner with a
scheduled party
When they cancel the party
Then send a cancellation email to the players
and the facilitator, a cancellation confirmation
to the planner, and ensure that the game play
URL points to the “cancelled party” page
© Copyright 2014 Conteneo, Inc. 37
38. www.conteneo.co
Overcoming common challenges
• You don’t have to automate everything
• Get started, but don’t expect quick turns
• Help your product managers by writing a
few
• Make AT writing a natural part of the
conversation
© Copyright 2014 Conteneo, Inc. 38
39. www.conteneo.co
Our Core Team
© Copyright 2014 Conteneo, Inc. 39
• Luke Hohmann, CEO & Founder
• CEO & Founder, Enthiosys
• VP Engineering, Aurigin Systems
• 1982 US National Junior Pairs
Figure Skating Champion
• Laura Richardson, VP Sales
• Managing Partner, Uptime
• Dir. Business Dev., E-Color
• Little League Ump & Tough
Mudder Competitor
• Dan O’Leary, CTO
• VP Engineering, Callidus Software
• Staff Engineer, Sun Microsystems
• Fender Stratocaster Devotee
• Tami Carter, VP Marketing
• GM Tech Web/SD Events
• Managing Editor, UBM
• Poet, Lifeguard & Coal Miner’s
Daughter
• Sue Cook, VP Prof. Services
• Owner, Spearfish Innovation
• VP Global Operations, HP
• Co-Founder Mont Blanc Ladies
Literary Guild and Trekking Society
Advisors & Investors:
• Verne Harnish, Founder & CEO, Gazelles
• Paul Gemeraad, President of Intellectual Assets, Inc.
• Chris Matts, Financial Systems Consultant
• Harry Max, Vice President of Experience Design for Rackspace
41. www.conteneo.co © Copyright 2014 Conteneo, Inc. 41
Luke Hohmann
Founder & CEO, Conteneo, Inc.
480 San Antonio Road, Suite 202
Mountain View, CA 94040
mobile: (408) 529-0319
luke.hohmann@conteneo.co