Notes on teaching software testing

Notes on Teaching Software
Testing

Alpen-Adria-Universität, Klagenfurt, 28 March 2013

Ladislav Samuelis, Technical University of Košice, Slovakia

Motivation
• For years I was notoriously curious about the latest
updates in program development.
• The main question:
– How a program “is born”. Later changed to “what is software
evolution”.
• Challenge: grant of the Slovak Ministry of Education:
“Implementation of Current Software Engineering
Courses for Computer Science Programs …”

Ladislav Samuelis: Notes on Teaching
2 of 23
Software Testing

Dawn of software testing
• P. Naur, 1969: “If you develop programs
systematically, you can dispense with testing
altogether.”
• This hope to achieve the goal is still present.
• However, in practice testing cannot be avoided by
any other means, whether it is the most careful
design, construction, or formal analysis of the
requirements.
• Elimination of testing from the development
process just postpones the test with reality until
the product is put into production.
3 of 23
Software Testing

Lehman’s law
• E-type software, software implementing real
world computer applications, must be
continually evolved to maintain user
satisfaction, generally over several years and
releases.
• It is likely that the responsible organisation
will, sooner or later, consider its replacement
through redevelopment or acquisition. (1974)

4 of 23
Software Testing

Specific of software testing
• Testing is a part of the development
process, not the manufacturing process.
• This testing is different from many other
engineering products, where the
emphasis is on testing the correct
reproduction.

5 of 23
Software Testing

What are the questions?

• What are we teaching in
software testing courses and
why?
• What are the scientific
foundations of testing?

6 of 23
Software Testing

Why should we have courses on
software testing?
• Developers spend 50% of their time testing,
and reliability requirements are steadily
increasing.
• All programmers spend a significant amount
of time to trial their software.
• It gets negligible significance in most software
development curricula.
• Accidents, ARIANE …
7 of 23
Software Testing

What are we teaching in software
testing courses?
• Techniques, measurements, management …
– Plenty of comprehensive books …
• ISTQB (International Software Testing
Qualification Board)
– Effective for undergraduate informatics study-
programmes
– Required by the industry

8 of 23
Software Testing

Notes on obstacles in building
software testing course
• How to avoid the phenomenon of “bag
of tricks”?
• It is not easy to design a specific
curriculum with adequate labs and
assessment procedures.
• The border between the program
analysis and testing is fuzzy.
9 of 23
Software Testing

How do we cooperate with the
industry?
• RWE IT local company provides selected
lectures on:
– testing of large projects
– testing techniques
– test cases by examples
• Diploma theses and internships
• Experimental data published in papers
• Workshop biannually for prospective
employees
10 of 23
Software Testing

Dijkstra’s “law”
• “Program testing can be used to show the presence
of bugs, but never to show their absence”. (1969)
• Donald Knuth:
– Beware of bugs in the above code; I have only
proved it correct, not tried it.
This law is expresses the limitations of black-box
testing.
Question:
How do we explain the theory of this “law” to
students?
11 of 23
Software Testing

On the limits of formal methods
(D.L.Parnas)
Simplification (model) is achieved by ignoring
certain facts such as the limits in the sizes of
data elements and the errors in arithmetic
operations. Unfortunately, these are exactly the
type of details that can cause faults and lead to
failures.
No formal analysis of such a model that leaves
out critical limits can reveal faults attributable
to those limits.

12 of 23
Software Testing

The theory of program testing
• The idea of program testing is as old as
programming.
• 1970 – new field of research called testing theory
emerged – efficiency and effectivity
• What are the main questions?
– How to design test cases from different sources:
specification, source code, input and output domains
of programs
– How to select a subset of test cases from the set of all
possible test cases?

13 of 23
Software Testing

A scheme of testing
(a mathematical approach)

14 of 23
Software Testing

The theory of Goodenough and
Gerhart
(very simplified explanation of the ideal test)

P “runs” and gives B number of bugs.
We create new set Tk until Bk ≥ B. Then
it is an ideal test.

• D = input domain, T≤D, d є D
• P= program, result is P(d)
• d is true, if output P(d) is acceptable
15 of 23
Software Testing

Other testing theories
• Theory of Weyuker and Ostrand, Gourlay
• Dines Bjørner: On a Triptych of Software
Development
– Domain engineering,
– Requirements engineering,
– Design
• SEMAT initiative

Question:
How much testing theory do we have to teach?
16 of 23
Software Testing

Power law
The notion of power laws has been around for
more than a century.

P(x) ~ c.x−k

Question:
What does it mean in software testing?

17 of 23
Software Testing

Power law and software testing
• Object-oriented software ~ network of
objects.
• It is worth to test less complex classes than
many trivial classes.

Note:
This is an empirical “law” or concept , rarely
taught in testing courses.
18 of 23
Software Testing

Gödel’s incompleteness theorem
• No computer program is capable of
proving all programs.
• For any consistent system of axioms
there will always be statements that
are improvable within the system.

19 of 23
Software Testing

Teaching of Gödel's incompleteness
theorems?
• Elaine Weyuker - Evaluating Software Test
Data Adequacy (1988)

Question:
Where are the places of Gödel’s theorems in
software engineering courses?

20 of 23
Software Testing

Note on testing and deductive
reasoning
• Deductive reasoning;
– Major premise – initial set of facts
– Minor premise – goal fact
• For example, PROLOG program –
the inference engine gives the verdict.
• Not-PROLOG program – no inference engine,
program is executed and tester “gives a
verdict” against requirements.
21 of 23
Software Testing

Questions on the role of observation in
software testing

Questions:
Why do we select observations?
Where do we perform the observation?
In which phase of the product lifecycle do
we perform observations?

22 of 23
Software Testing

Notes on the history of testing
concurrent programs
• Before object-oriented programming:
– mechanisms for ensuring mutual exclusion
(Dijkstra),
– monitors (Hoare) and
– critical regions (Brinch-Hansen)

Question:
What topics to teach in concurrent program
testing?
23 of 23
Software Testing

Notes on teaching software testing

Notes on teaching software testing

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Notes on teaching software testing

Similar to Notes on teaching software testing (20)

More from Förderverein Technische Fakultät

More from Förderverein Technische Fakultät (20)

Recently uploaded

Recently uploaded (20)

Notes on teaching software testing