Kees did this talk for a client in 2016. It is about the basic principles behind risk based testing - also in Agile context.

1. 1© 2016

2. Risk based testing in Agile? context
Huawei, Shenzhen 2016
Kees Blokland
kees.blokland@polteq.com

3. 3© 2016
Question to you:
• What does risk based testing mean for you?

4. 4© 2016
Testers must make choices
What to test and what not to test
What to test thoroughly, what to test superficially
What to test first, what to test later
What to test once, what to test each software build
What to test by senior testers, what by junior testers
What to test scripted, what to test exploratory,
… just to name a few
We cannot test all
possible ways
through the
software

5. 5© 2016
Testers must make choices
What to test and what not to test
What to test thoroughly, what to test superficially
What to test first, what to test later
What to test once, what to test each software build
What to test by senior testers, what by junior testers
What to test scripted, what to test exploratory,
… just to name a few
We base these
choices on risk

6. 6© 2016
What is risk?

7. 7© 2016
Risk defined
WIKIPEDIA
The probability or threat of
quantifiable damage, injury,
liability, loss, or any other
negative occurrence that
is caused by external or
internal vulnerabilities, and
that may be avoided through
pre-emptive action.

8. 8© 2016
Risk defined
Michael Bolton
Some person will suffer
loss, harm, annoyance,
or reduced value
because of a problem in
the product that is
triggered by some
threat.

9. 9© 2016
Risk defined
WIKIPEDIA
The probability or threat of
quantifiable damage, injury,
liability, loss, or any other
negative occurrence that
is caused by external or
internal vulnerabilities, and
that may be avoided through
pre-emptive action.
a system failure
which is bad to a certain
extend
we don’t know for sure it
will happen
we may want to prevent it

10. 10© 2016
BMW needs to call back 10.000 cars in
China due to a software defect
source: computable.nl
Doors open
spontaneously
Gear switches from parked
state and car starts rolling by
itself from the hill

11. 11© 2016
Huge traffic jam due to system failure
source: computable.nl
Huge traffic jams
due to failing
information
boards

12. 12© 2016
Information
still usable…

13. 13© 2016
Testing: Test’s law on software failures
R = L x IRisk Likelihood Impact
NO RISK NO TEST

14. 14© 2016
Risk assessment
WIKIPEDIA
The probability or threat of
quantifiable damage, injury,
liability, loss, or any other
negative occurrence that
is caused by external or
internal vulnerabilities, and
that may be avoided through
pre-emptive action.
What failures of the
product could we
anticipate.
What impact would
these have?

15. 15© 2016
Risk assessment
WIKIPEDIA
The probability or threat of
quantifiable damage, injury,
liability, loss, or any other
negative occurrence that
is caused by external or
internal vulnerabilities, and
that may be avoided through
pre-emptive action.
What would cause faults
in the product that make it
fail like that?
What is the likelihood of
this to happen?

16. 16© 2016
Risk reduction
How can testing help to
reduce the risks
by finding these
(types of) faults so
preventing the failures to
happen in the product?

17. 17© 2016
break down break down
recognizable by
customer / user
recognizable by
development
prioritized on
business value
acceptance criteria
user scenario’s
Feature refinement, break down

18. 18© 2016
break down break down
recognizable by
customer / user
recognizable by
development
prioritized on
business value
acceptance criteria
user scenario’s
Question
• Discuss where to find these in your project

19. 19© 2016
break down break down
What failures could we anticipate.
What impact would these have?
What would cause faults in the product?
What is the likelihood of this to happen?
How can testing help to reduce the risks
by finding faults causing the failures?

20. 20© 2016
break down break down
What failures could we anticipate.
What impact would these have?
Risk assessment
Who represents
the user / client
/ business?

21. Risk Story Elements
• Some PERSON(S)
– a person, a group, the business, a bystander
• will EXPERIENCE
– bad feelings and a trigger for them
• a PROBLEM
– annoyance, loss, harm, diminished value…
• with respect to SOMETHING DESIRABLE
– capability, reliability, performance…
• that CAN BE DETECTED
– by an oracle (a means of recognizing a problem)…
• in SOME SET OF CONDITIONS
– perhaps invariably, perhaps intermittently…
• because of a VULNERABILITY
• a bug, a missing feature, an inconsistency…
• in the SYSTEM.
– some result, process, component, environment…
© James Bach and Michael Bolton, 2016

22. © James Bach and Michael Bolton, 2016
Some General Systems Risk Heuristics
• A condition in the world that interacts with your
product may happen rarely or frequently.
• A problem may be detectable or undetectable at the
time it first occurs or begins to occur (relative to our
means of detection).
• A problem may occur in a result, process, component,
or environment.
• In the realm of technology and society, problems fall
into diverse categories: usability, performance,
concurrency, legality, correctness, compatibility, etc.
• A problem may impact a user, a bystander, or the
business that produced the product.
• A problem may cause inconvenience, annoyance, or
harm.
• …

23. Examples
• sometimes (in a given situation) one of the components OF THE PRODUCT visibly FAILS WITH
RESPECT TO compatibility SUCH THAT bystander IS harmed
• sometimes (in a given situation) some process OF THE PRODUCT visibly FAILS WITH RESPECT
TO legality SUCH THAT business IS harmed
• sometimes (in a given situation) some process OF THE PRODUCT visibly FAILS WITH RESPECT
TO usability SUCH THAT user IS annoyed
• always some result OF THE PRODUCT invisibly FAILS WITH RESPECT TO concurrency SUCH
THAT bystander IS annoyed
• always some process OF THE PRODUCT invisibly FAILS WITH RESPECT TO legality SUCH THAT
bystander IS inconvenienced
• always some result OF THE PRODUCT visibly FAILS WITH RESPECT TO performance SUCH
THAT business IS annoyed
• always some result OF THE PRODUCT visibly FAILS WITH RESPECT TO compatibility SUCH
THAT bystander IS harmed
• sometimes (in a given situation) some result OF THE PRODUCT invisibly FAILS WITH RESPECT
TO correctness SUCH THAT business IS inconvenienced
• always one of the components OF THE PRODUCT invisibly FAILS WITH RESPECT TO
performance SUCH THAT user IS inconvenienced
• always one of the components OF THE PRODUCT visibly FAILS WITH RESPECT TO correctness
SUCH THAT user IS annoyed © James Bach and Michael Bolton, 2016

24. 24© 2016
Question to you: what are your examples?
and: who should we talk with?

25. 25© 2016
Risk assessment
– Discuss the impact
Losing income
Losing customers
Societal
Damage
Claims by users
Repair costs
Damage to company image
Compensation effort and cost
Development and test cost
Clients / Users /
Business perspective

26. 26© 2016
Risk assessment
User story Risk Quality Impact
User story 1
Template
Unable to install client SW, loose customer functional H
Customer does not understand how to operate
(and terminates session)
usability H
User story 2
Upload &
Preview
Preview does not function (customer
disappointed)
functional M
Unable to create a mailing, less customer value performance H
Upload over 3G is taking too long (customer
complains or stops)
functional M
User story 4 Date format incorrect functional L
User story 5 Upload does not start functional H
…

27. 27© 2016
break down break down
What would cause faults in the product?
What is the likelihood of this to happen?
Risk assessment

28. 28© 2016
Risk Analysis Heuristics
• Project factors
– Things going on in projects, among people, may lead to
bugs.
• Technology factors
– The structure and dynamics of technology itself may give
rise to bugs.
• Requirements factors
– Aspects of requirements may indicate or promote the
presence of bugs.
• Operational factors
– The circumstances and patterns of use affect the
probability and impact of bugs.
Risk Analysis Heuristics (for Digital Products)
By James Bach and Michael Bolton v. 2.1

29. 29© 2016
Example of technology factors
logic
platform
Service
boards
Main
NE
linux
lamps
cabling
subrack
…
Service
Board-1
Main b power
Service
Board-n
Service
Board-2
…
Feature map
Customer application scenario
100G extra long haul 1550nm
Protection switching
Automatic laser shutdown
Alarming
AU4-4c channel provisioning
Adding / removing wavelengths
….
Software map
Hardware mapNetwork map
Service board
Asic1
pwr
Otherstuff
Asic 2
Firmware
optics
NE NMS
Here are the
changes made

30. 30© 2016
Risk assessment
– another list for inspiration
Frequency of use
Inexperienced developers
New tools
High time pressure
Complex functions
Many changes
Many interfaces
Error prone functions

31. 31© 2016
What are your examples?

32. 32© 2016
Risk assessment
User story 1:
Template
Q Risk Fault
proba
bility
Unable to
install client
SW, loose
customer
F Install on windows; stable, predictable, hardly
bugs seen before
L
F Install on macos; NEW, no experience, high
time pressure
H
P Image becomes bigger, may cause
performance problems
M

33. 33© 2016
Risk assessment
User story 1:
Template
Q Risk Impact Fault
prob.
Risk
class
Unable to
install client
SW, loose
customer
F Install on windows; stable,
predictable, hardly bugs
seen before
H L M
F Install on macos; NEW, no
experience, high time
pressure
H H H
P Image becomes bigger,
may cause performance
problems
H M M
Impact
H M L
Fault pr.
H H M L
M M M L
L M L L
Risk class
lookup
These are most important!

34. 34© 2016
Do you remember this slide?

35. 35© 2016
Question to you:
• How to test more thorough, more deeply?

36. 36© 2016
How to map risks on tests?
Techniques
Superficial
Implicit
Thorough
Specific
Formal
Multiple
Model based
Medium risk
High risk
Low risk
Test levels
Component Test
+ System Test
+ Acceptance Test
+ System Integr. T.
Special test
All
Performance
Security
Usability
Testers
Junior
Medior
Senior
Exploratory testing
Free format
Session based
Session based by experts

37. 37© 2016
How to map risks on tests? - continued
Tested once
Automated regr. test
Test rounds Multiple
Each
Manual regression test
Review
-
Informal
Formal
Specific
Focus
Code review
-
Pairing
Pairing with testers
Testers
Junior
Medior
Senior
Experts

38. 38© 2016
Question to you:
• What techniques do you know to do more deep
testing?

39. 39© 2016
Do you remember this slide?

40. 40© 2016
What to test first and what later?
In the end execute tests on
low risk components /
features / aspects
First execute tests on
high risk components /
features / aspects
Time
Low severity
defects
undiscovered
Most serious
defects
Low severity
defects
Then execute tests on
medium risk components /
features / aspects
Less serious
defects
Project
deadline

41. 41© 2016
Determining testing strategy
Backlog item list
- Changes
- Defects
- Technical debt
Items considered for
this iteration
Iteration, e.g. 3 weeks
Iteration planning
meeting and/or
clarification /
elaboration meeting
Time 
SE
DE DV
TE
CR
Example

42. 42© 2016
Determining testing strategy
Backlog item list
- Changes
- Defects
- Technical debt
Items considered for
this iteration
Iteration, e.g. 3 weeks
Iteration planning
meeting and/or
clarification /
elaboration meeting
Time 
SE
DE DV
TE
CR
Example
• What ‘tools’ do we have?
– Pair testing
– Code review
– Automated regression testing (/checking)
– Manual regression testing
– Unit testing
– TDD
– Exploratory sessions
– Scripted testing (prepare for automated regression
checks)
– …

43. 43© 2016
Reporting about testing (risk related)
• What information have we gained about the quality of
the product?
– in general and specifically:
– regarding the potential product failures identified in the
risk assessment
• What testing has been done so far?
• What information do we have to determine further
steps?
– how testability of the product influenced testing
– what further testing could be done (related to the risks)
– the cost of further testing (effort and lead time)
Any level:
TE level, TPL level, TM level, …

44. 44© 2016
Reporting about testing (risk related)
• What information have we gained about the quality of
the product?
– in general and specifically:
– regarding the potential product failures identified in the
risk assessment
• What testing has been done so far?
• What information do we have to determine further
steps?
– how testability of the product influenced testing
– what further testing could be done (related to the risks)
– the cost of further testing (effort and lead time)
Example 1:
Software delivery delayed due to complexity problems
SO: not all planned testing could be done
AND: need EVEN MORE TESTING
because delayed delivery increases product risk
Example 2:
Found many bugs so far, which are solved and retested
BUT: more testing is likely to uncover more bugs
SO: need MORE TESTING
because many bugs found is is an indicator of high risk

45. 45© 2016
Considerations on risk based testing
• Risk based testing requires solid testing skills
• Risk based testing requires dialogue with the
client/user/customer
• Risk based testing requires dialogue with developers
• Information from testing can lower the risk level
• Information from testing can ALSO indicate a HIGHER
risk level

46. 46© 2016
Last question to you:
• Do YOU have questions?

47. 47© 2016
谢谢!
Thank you!
kees.blokland@polteq.com