Finally… reliable software!

1. Finally… Reliable
Software!
Bryan Bakker
Eindhoven 2015
bryan.bakker@sioux.eu
@Bryan_Bakker

2. Contents
 Intro
 Reliability
 Four step model
 Different steps in detail
 Conclusion
© Sioux 2015 | Confidential | 2

3. Thanks
Thanks to:
 Rob de Bie
 René van den Eertwegh
 Peter Wijnhoven
© Sioux 2015 | Confidential | 3

4. About Bryan Bakker
 Test Expert
 Certifications: ISTQB, TMap, Prince2
 Member of ISTQB Expert Level on Test Automation
 Tutor of several test related courses
 Domains: medical systems, professional security
systems, semi-industry, electron microscopy
 Specialties: test automation, integration testing, design
for testability, reliability testing
© Sioux 2015 | Confidential | 4

5. About Sioux
HERENTALS
NUENEN
EINDHOVEN
UTRECHT MOSCOW
© Sioux 2015 | Confidential | 5
DANANG

6.  Did you ever experience unreliable software?
Reliability
© Sioux 2015 | Confidential | 6

7. Reliability
 What is reliability?
“Software Reliability is the probability of failure-
free software operation in a specified
environment for a specified period of time.”
IEEE 729
In short:
 Something can be functional correct
 But is it reliable? © Sioux 2015 | Confidential | 7

8. Reliability
 What is reliability?
“Software Reliability is the probability of failure-
free software operation in a specified
environment for a specified period of time.”
IEEE 729
In short:
 Something can be functional correct
 But is it reliable? How reliable is it? © Sioux 2015 | Confidential | 8

9. Four steps
© Sioux 2015 | Confidential | 9
Example: Security and Surveillance System
 Cameras
 Recording
 Event Handling

10. Define user domain reliability targets:
 Define customer profiles
 Identify operation modes and critical
functions
 Determine reliability requirements per
operation mode / critical function
First step
© Sioux 2015 | Confidential | 10

11.  Define customer profiles
 ATM security
First step
© Sioux 2015 | Confidential | 11

12.  Define customer profiles
 ATM security
 Parking lot surveillance
First step
© Sioux 2015 | Confidential | 12

13.  Define customer profiles
 ATM security
 Parking lot surveillance
 Airport surveillance
First step
© Sioux 2015 | Confidential | 13

14.  Identify operation modes and critical
functions
 Recording mode
 Playback mode
 Auto-start (critical)
 Out-of-the-box
 User triggered
 Software reset
First step
© Sioux 2015 | Confidential | 14

15.  Determine reliability requirements per
operation mode / critical function
 Segment between 0.5s and 2s missed
 failure rate ≤1x per day
 Playback command does not function as expected
 failure rate ≤ 1x per hour of viewing
 Not auto-started  failure rate ≤ 3 * 10-7
failures/restart
First step
© Sioux 2015 | Confidential | 15

16.  Define operational profile for
user functions
 Decompose software reliability targets
 Decompose operational profile with
probabilities per component
Second step
© Sioux 2015 | Confidential | 16

17.  How will the product be used by
customers?
 Focus development resources based
on expected usage
 Operational profile: a quantitative
characterization of how a system will be used
 Developed by John Musa
 To be used for development and test
activities
 Also very useful for reliability
Operational profile
© Sioux 2015 | Confidential | 17

18.  Critical functions can be missed in
operational profiles.
 Treat them separately!
What about?
© Sioux 2015 | Confidential | 18

19.  Define operational profile for
user functions
Second step
© Sioux 2015 | Confidential | 19

20.  Decompose software reliability targets
 Not auto-started  failure rate <= 3 * 10-7
failures/restart
 Stream Handler contribution: 20%
 “Target” = 0.6 * 10-7 failures / restart
Second step
© Sioux 2015 | Confidential | 20
Focus on Stream
Handler
component

21.  Decompose operational profile with
probabilities per component
Second step
© Sioux 2015 | Confidential | 21
Playback function Occurrence Probability % Stream Handler commands
Play / Pause (toggle) 48 out of 100 48% Start / Stop Playback
Fast forward 20 out of 100 20% Start Playback
Fast reverse 20 out of 100 20% Start Playback
Setup playback
windows
10 out of 100 10% Stop / Start Playback
Search and select
event for playback
2 out of 100 2% Start Playback with time
stamp = time of selected
event

22.  Define the engineering processes
 Process steps to prevent reliability
faults
 Process steps to detect reliability faults
 Design choices to minimize effects of faults
Next steps
© Sioux 2015 | Confidential | 22

23.  Measure software reliability growth
 Design and execute reliability tests
 Based on the operational profiles
 Randomly execute test set according to the
operation profile
 Extra focus on critical functions
 Visualize reliability growths
Next steps
© Sioux 2015 | Confidential | 23

24.  Reliability growth curve
Next steps
© Sioux 2015 | Confidential | 24

25. Conclusion
 Reliability is not binary but a characteristic
that can be measured
 Based on theory of John Musa
 Practical 4 step approach
 Reliability is not reached by accident
 For a full description and worked out case
study, see next page…
© Sioux 2015 | Confidential | 25

26. Questions
© Sioux 2015 | Confidential | 26
Published February 2015:
www.amazon.com
ISBN: 978-1499226669
Rob de Bie
Bryan Bakker
René van den Eertwegh
Peter Wijnhoven

27.  Backup slides
© Sioux 2015 | Confidential | 27

28. Reliability Curves
 Bathtub curve
Hardware Reliability
 Sawtooth curve
Software Reliability
© Sioux 2015 | Confidential | 28

29. 29© Sioux 2015 | Confidential |
www.sioux.eu
bryan.bakker@sioux.eu
+31 (0)40 26 77 100