Online lecture at the School of Computer Science, University of Hertfordshire, Hatfield, UK, as part of the 10th Europe Week from 3rd to 7th March 2014.

1. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
Experiences in
Software Testing
Prof. Dr. Dagmar Monett Díaz
Computer Science Dept.
Faculty of Cooperative Studies
Berlin School of Economics and Law
dagmar@monettdiaz.com
Europe Week, 3rd – 7th March 2014

2. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 2
The waterfall Software Development
Life-Cycle in theory and practice

3. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 3
ProgrammingGeeks at Google+

4. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 4
Agenda

5. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 5
Agenda
 Where does the major content come from?
 Why is Software Testing of utter importance?
 What is Software Testing?
 Testing principles and methods
 General testing guidelines
 What to do with software errors that cause failures?
 Test-Driven Development
 Current trends in Software Testing
 Further reading, sources of inspiration, and more…

6. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 6
©

7. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
Software Engineering
Ian Sommerville
9th Edition, 792 pp.
Addison-Wesley, March 2011
ISBN-13: 978-0-137-03515-1
What I also use in my
lectures at the HWR…
7

8. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
Softwaretechnik.
Mit Fallbeispielen aus
realen
Entwicklungsprojekten
(Software Engineering.
With case studies from real-world
development projects)
Thomas Grechenig, Mario
Bernhart, Roland Breiteneder,
Karin Kappel
In German, 688 pp.
Pearson Studium, 2010
ISBN-13: 978-3-86894-007-7
What I also use in my
lectures at the HWR…
8

9. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
Further reading
 I. Sommerville (2011): “Chapter 8: Software Testing”.
With material for instructors available at
http://www.softwareengineering-9.com/
 L. Crispin, T. House (2012): “Testing extreme programming”
 R. Patton (2005): “Software Testing”
 ISTQB Exam Certification website available at
http://istqbexamcertification.com/
9

10. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 10
Why is Software Testing of utter
importance?
Image © renjith krishnan at http://www.freedigitalphotos.net/

11. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
When I searched the Web on
January 5th 2013…
11

12. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
When I searched the Web on
January 5th 2013…
12
I tried again about 30 seconds later and could
continue searching the Web…

13. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 13
But what about…

14. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 14http://news.bbc.co.uk/2/hi/3772077.stm

15. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 15
What is Software Testing?

16. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
Definition according to
IEEE 610.12, 1990
„Software testing is a formal process carried out by
a specialised testing team in which a software unit,
several integrated software units or an entire
software package are examined by running the
programs on a computer. All the associated tests are
performed according to approved test procedures on
approved test cases.“
16
Interesting issues:
- Software testing as formal process
- Testing by a specialised team
- Approved test procedures and test cases

17. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 17
Testing principles

18. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 18
Testing principles (ISTQB)
ISTQB: International Software Testing Qualifications Board

19. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 19
Testing principles (ISTQB)
Principle 1: Testing shows presence of defects
ISTQB: International Software Testing Qualifications Board

20. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 20
Testing principles (ISTQB)
Principle 1: Testing shows presence of defects
Principle 2: Exhaustive testing is impossible
ISTQB: International Software Testing Qualifications Board

21. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 21
Testing principles (ISTQB)
Principle 1: Testing shows presence of defects
Principle 2: Exhaustive testing is impossible
Principle 3: Early testing
ISTQB: International Software Testing Qualifications Board

22. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 22
Testing principles (ISTQB)
Principle 1: Testing shows presence of defects
Principle 2: Exhaustive testing is impossible
Principle 3: Early testing
Principle 4: Defect clustering
ISTQB: International Software Testing Qualifications Board

23. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 23
Testing principles (ISTQB)
Principle 1: Testing shows presence of defects
Principle 2: Exhaustive testing is impossible
Principle 3: Early testing
Principle 4: Defect clustering
Principle 5: Pesticide paradox
ISTQB: International Software Testing Qualifications Board

24. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 24
Testing principles (ISTQB)
Principle 1: Testing shows presence of defects
Principle 2: Exhaustive testing is impossible
Principle 3: Early testing
Principle 4: Defect clustering
Principle 5: Pesticide paradox
Principle 6: Testing is context dependent
ISTQB: International Software Testing Qualifications Board

25. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 25
Testing principles (ISTQB)
Principle 1: Testing shows presence of defects
Principle 2: Exhaustive testing is impossible
Principle 3: Early testing
Principle 4: Defect clustering
Principle 5: Pesticide paradox
Principle 6: Testing is context dependent
Principle 7: Absence-of-errors fallacy
ISTQB: International Software Testing Qualifications Board

26. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 26
Active learning exercise:
“Mention some concrete examples of
testing principles’ application!”
Image © renjith krishnan at http://www.freedigitalphotos.net/

27. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 27
Software testing methods

28. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
- Functional testing
(Black-box testing)
- Structural testing
(White-box testing)
- Combinations of both
(Grey-box testing)
28
Software testing methods
Images © Master isolated images and Serge Bertasius Photography @ http://www.freedigitalphotos.net/

29. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 29
Examples of Black-box testing

30. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 30
Partition testing
 Also: “Equivalence class partitioning”
 Input data and output results often fall into different
classes where all members of a class are related.

31. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 31
Partition testing
 Also: “Equivalence class partitioning”
 Input data and output results often fall into different
classes where all members of a class are related.
 Each of these classes is an equivalence partition
or domain where the program behaves in an
equivalent way for each class member.

32. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 32
Partition testing
 Also: “Equivalence class partitioning”
 Input data and output results often fall into different
classes where all members of a class are related.
 Each of these classes is an equivalence partition
or domain where the program behaves in an
equivalent way for each class member.
 Test cases should be chosen from each partition.

33. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 33
 Example:
In a system requirements documentation it is specified that
only product prices starting from 0,01 € and up to 500 € must
be entered by the user.
It can be assumed that products with a price equal to 100 €
will be properly validated and processed.
Furthermore, it can be assumed that all prices between 0,01 €
and 500 € are also correct.
Thus, it is sufficient to test only one arbitrary value from each
equivalence class in order to identify possible improper
behaviour of the system.
Partition testing

34. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 34
It would be sufficient to consider three partitions:
And then to check only one (arbitrary) representative
of each equivalence class in a test case.
Partition testing
 Example (cont.):
invalid partition 1 invalid partition 2valid partition
x 0 0 < x 500 x > 500
0 500

35. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 35
Boundary-value analysis
 Special case of the partitioning testing
 Boundary values are common locations for errors
that result in software failures

36. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 36
Boundary-value analysis
 Special case of the partitioning testing
 Boundary values are common locations for errors
that result in software failures
 Possible boundary values for the previous example:
invalid partition 1 invalid partition 2valid partition
0 500
0
0,01
500
500,01

37. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 37
Boundary-value analysis
 Or even better…

38. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 38
Boundary-value analysis
 Or even better…
invalid
partition 1
valid
partition
0 500
0
0,01
500
500,01
215
215
215+1
216
216
216+1
231
231
231+1
invalid partition 2
232
232
232+1
… …

39. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 39
Examples of White-box testing

40. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 40
Structure-based methods (i)
 Function coverage
 Statement coverage
 Decision (or branch) coverage
 Condition (or predicate) coverage
 Multiple condition coverage
 …and many other code coverage metrics that are
used in structural software testing!

41. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 41
Structure-based methods (ii)
int rst(int r, int s, int t){
if (r > 0 || s > 0) {
while (r != s) {
while (r > s) {
r = r - s;
}
r = s;
}
} else { r = t; }
return r;
}

42. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 42
Structure-based methods (iii)
int rst(int r, int s, int t){ //n0
if (r > 0 || s > 0) { //n1
while (r != s) { //n2
while (r > s) { //n3
r = r - s; //n4
}
r = s; //n5
}
} else { r = t; } //n6
return r; //n7
} //n8

43. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 43
Structure-based methods (iv)
n0
n1
n6 n2
n7
n8
n3
n4n5
Control flow
graph
int rst(int r, int s, int t){ //n0
if (r > 0 || s > 0) { //n1
while (r != s) { //n2
while (r > s) { //n3
r = r - s; //n4
}
r = s; //n5
}
} else { r = t; } //n6
return r; //n7
} //n8

44. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 44
Structure-based methods (v)
Test case: {r = 1, s = 2, t = 4}
n0
n1
n6 n2
n7
n8
n3
n4n5
Control flow
graph
int rst(int r, int s, int t){ //n0
if (r > 0 || s > 0) { //n1
while (r != s) { //n2
while (r > s) { //n3
r = r - s; //n4
}
r = s; //n5
}
} else { r = t; } //n6
return r; //n7
} //n8

45. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 45
Structure-based methods (vi)
int rst(int r, int s, int t){ //n0
if (r > 0 || s > 0) { //n1
while (r != s) { //n2
while (r > s) { //n3
r = r - s; //n4
}
r = s; //n5
}
} else { r = t; } //n6
return r; //n7
} //n8
Test case: {r = 1, s = 2, t = 4}
n0
n1
n6 n2
n7
n8
n3
n4n5
Control flow
graph
78% Statement coverage
63% Decision coverage

46. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 46
General testing guidelines

47. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
General testing guidelines
 Choose inputs that force the system to generate all
error messages
 Design inputs that cause input buffers to overflow
 Repeat the same input or series of inputs numerous
times
 Force invalid outputs to be generated
 Force computation results to be too large or too
small.
47

48. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
Test ideas and heuristics
 “Test Heuristics Cheat Sheet: Data Type Attacks & Web
Tests”
© Quality Tree Software, Inc. (2006)
Available at http://testobsessed.com/wp-
content/uploads/2011/04/testheuristicscheatsheetv1.pdf
 “How To Generate Test Ideas (A checklist). Everything you
need to trigger thoughts, discussions and actions in the
projects you are working on”
© Smart Bear and Ministry of Testing (2013)
Available at http://www.ministryoftesting.com/2013/12/checklist
-how-to-generate-test-ideas/
48

49. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 49
Active learning exercise:
“Which is then the main goal of
software testing?”
Image © renjith krishnan at http://www.freedigitalphotos.net/

50. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
Press the Bug key!
“For when you're having a bad day at work
and you can't find any bugs”
Andy Glover (@cartoontester)
http://cartoontester.blogspot.de/2012/11/press-bug-key.html
50

51. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 51
What to do with software errors
that cause failures?

52. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

53. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 53
Who should fix the failures?
How does the tester report them?

54. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 54
Managing software errors
report
Tester
re-test
rate
assign
correct
deliver
terminate
Test lead Developer Change
management

55. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 55
How should tests be conducted?
Which goals and conditions are
important?

56. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 56
Testing process (i)
 What should be considered when conducting
tests:
(1) Criteria for the success or failure of a test run.
(2) Criteria to start and finish a test run.
(3) Criteria to interrupt or continue a test.
(4) Documentation of results and communication
channels.
(5) Testing methods and depth.
(6) Definition of the test environment.
(cont.)

57. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 57
Testing process (ii)
 What should be considered when conducting
tests:
(7) Definition of the test basis (information and
stakeholders needed).
(8) Problems found and responsibilities for their
elimination.
(9) Objects to be tested.
(10) Properties to be tested.
(11) Properties that should not be tested.
(12) Economic objectives of the test.

58. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 58
Test-Driven Development (TDD)

59. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
Test-driven development
 Approach to program development in which you
inter-leave testing and code development.
 TDD was introduced as part of agile methods such
as Extreme Programming.
 Tests are written before code and ‘passing’ the tests
is the critical driver of development.
59

60. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
Test-driven development
 You develop code incrementally, along with a test
for that increment. You don’t move on to the next
increment until the code that you have developed
passes its test.
Identify new
functionality
Write test Run test
Implement
Functionality
and refactor
fail
pass
60

61. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 61
All code is guilty…
Image © http://www.printfection.com/codesmack/All-Code-Is-Guilty/_s_58243

62. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 62
Current trends in Software Testing

63. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
World Quality Report
• >1500 Respondents: Chief financial officers (CFO), chief information officers (CIO), IT directors, and
quality assurance (QA) directors worldwide (>25 countries)
• Detailed telephone interviews undertaken during April-May 2012 and April-May 2013
4th Edition 2012-2013
5th Edition 2013-2014
Available at
http://www.capgemini.com/insights-and-resources/
by-publication/world-quality-report-2012--2013/
Available at
http://www.capgemini.com/thought-leadership/
world-quality-report-2013-14
63

64. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
Change in testing budgets
Figure 1, pp. 13 (2012), Gapgemini, Sogeti and HP
64
2012-13

65. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
Greatest increase in spend
65
Figure 3, pp. 16 (2012), Gapgemini, Sogeti and HP
2012-13

66. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
Increase in mobile testing
Figure 10, pp. 25 (2013), Gapgemini, Sogeti and HP
66
2013-14

67. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
Challenges to mobile testing
Figure 7, pp. 25 (2012), Gapgemini, Sogeti and HP
67
2012-13

68. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
Challenges to mobile testing
Figure 11, pp. 27 (2013), Gapgemini, Sogeti and HP
68
2013-14

69. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
Germany – Top trends
pp. 64 (2012), Gapgemini, Sogeti and HP
69
 „There is a significant increase in move towards
development of a Testing Centre of Excellence
(TCOE).“
 „A cautious migration of applications towards the
cloud has become evident.“
 „More German organizations are testing mobile
applications and devices than in any other country
surveyed.“
2012-13

70. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
UK – Top trends
pp. 72 (2012), Gapgemini, Sogeti and HP
70
 „19% of UK organizations currently test mobile
applications or devices.“
 „More than 83% of organizations intend to use
Testing as a Service (TaaS) offered by a third
party.“
 „Mobile application and device testing is a growing
challenge.“
2012-13

71. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 71
Homework:
“Search for testing tools and use
them in your current course
projects!”
Image © renjith krishnan at http://www.freedigitalphotos.net/

72. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 72
To take away…

73. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 73
Fehlermanagement (i)

74. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 74
Assessment
Image © renjith krishnan at http://www.freedigitalphotos.net/

75. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
Question
75
(Taken from the public Practice Exam © ISTQB)
Which of the following statements BEST describes one
of the seven key principles of software testing?
(A) Automated tests are better than manual tests for
avoiding the Exhaustive Testing.
(B) Exhaustive testing is, with sufficient effort and tool
support, feasible for all software.
(C) It is normally impossible to test all input / output
combinations for a software system.
(D) The purpose of testing is to demonstrate the absence
of defects.
https://docs.google.com/forms/d/1r338sJJzg_DwpWTouEiEeD7KUBI0VD6k8AdbNI7LcfU/viewform
PLEASE ANSWER AT:

76. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 76
Last active learning exercise
Image © renjith krishnan at http://www.freedigitalphotos.net/

77. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 77
The content

78. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 78
References

79. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
Others…
79

80. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
Further reading and sites…
 ISTQB - International Software Testing Qualifications Board
http://www.istqb.org/
 Testing Experience magazine
http://www.testingexperience.com/
 Agile Testing Days 2014, “Europe’s greatest testing event”,
Potsdam, Germany
http://www.agiletestingdays.com/call_for_proposals.php
 IEEE International Conference on Software Testing,
Verification and Validation (ICST’14), Cleveland, Ohio
https://sites.google.com/site/icst2014/
80

81. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
With special thanks to…
81

82. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 82
Slides of the talk per request:
dagmar@monettdiaz.com
Prof. Dr. Dagmar Monett Díaz
monettdiaz
@dmonett
http://monettdiaz.com