1. HISTORYOFSOFTWARETESTING
How to Learn
The History of Software Testing
Keizo Tatsumi
2010-12-19
Translated into English on July 2014
If it wasn't useful, it wouldn't be learned.
If it couldn't ever be interesting, it wouldn't
deserve to be learned.
WACATE 2010 Winter
2. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Agenda
Prehistory
The History of Computers and Software Engineering
The Growth of Software Testing
The History of Testing Techniques
The History of Testing in Japan
The Forefront of Software Testing Research
Concluding Remarks
(C) K. Tatsumi 20142
3. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Prehistory
The First Computer
Charles Babbage’s Analytical Engine
• Conceived in 1837 (unfinished)
• The input via punched cards
• The output via a printer, a curve plotter and a
bell
The First Programmer
Ada Byron, Lady Lovelace
• Worked on the Analytical Engine (1843)
– Ada’s notes on the Analytical Engine were
recognized as a description of a computer and
software
(1/2)
(C) K. Tatsumi 20143
4. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Prehistory
Who was The First Tester ?
(2/2)
The first programmer, Lady Lovelace, who coded for
Charles Babbage's wonderful but unfinished
computer in the nineteenth century, I'm sure often
said, "Just one more week, Mr. Babbage, and it'll be
done." Lucky for her the hardware was never finished,
so she never did have to go beyond desk checking.
(B. Beizer, Software system testing and quality assurance, 1984, p.277)
Boris Beizer wrote:
• “The first discussion of testing and debugging predates
computers by almost a century, in the memoirs of lady
Ada Lovelace who wrote software for Babbage’s (never
completed) mechanical computer.”
(posted to comp.software.testing on Sep. 18 2007)
(C) K. Tatsumi 20144
5. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Agenda
Prehistory
The History of Computers and Software
Engineering
The Growth of Software Testing
The History of Testing Techniques
The History of Testing in Japan
The Forefront of Software Testing Research
Concluding Remarks
(C) K. Tatsumi 20145
7. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Victor Basili and John Musa,
The Future Engineering of Software:
A Management Perspective, 1991
1960s: The Functional Era
• IT penetrated institutions
• How to exploit IT to meet institutional needs
1970s: The Schedule Era
• Software crisis
NATO Software Engineering Conference, 1968
• How to develop software in a timely, planned, and controlled
fashion
• Life-cycle models and schedule tracking
ex. Royce waterfall model, Quality models
1950 1980 19901960 1970 2000 2010
Schedule Era Cost Era Quality EraFunctional Era
Historical Division by Management Perspective (1/2)
(C) K. Tatsumi 20147
8. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
1980s: The Cost Era
• Hardware costs decreased, PC created a mass market
• The importance of productivity in software development
increased
• Various cost models
ex. Function point, Putnam model, COCOMO
1990s: The Quality Era
• The increased dependence of institutions on information
processing
• The consumer mass market increases the demands on quality
1950 1980 19901960 1970 2000 2010
Schedule Era Cost Era Quality EraFunctional Era
Historical Division by Management Perspective (2/2)
(C) K. Tatsumi 20148
9. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Software Engineering Evolution
Barry Boehm,
A Hegelian View of Software Engineering Evolution,
2006
Autonomy;
Bio-
Computing
1990’s 2010’s2000’s1970’s 1980’s1960’s1950’s
COTS
Software
as Craft
Theses
Syntheses
Antitheses
Formality,
Waterfall
Productivity;
Reuse;
Objects;
Peopleware
Plan-
Driven
Software
Maturity
Models
Agile
Methods
Engineer
Software
like
Hardware
Risk-Based
Agile/Plan-
Driven Hybrids;
Model-Driven
Development
Integrated
Sw-systems
Engineering
Value-Based
Methods;
Collaboration;
Global
Development;
Enterprise
Architectures
Software
Differences,
Engineer
Shortages
Many defects
Scalability,
Risk Mgmt.
Prototyping
Time to Market,
Rapid Change
Scalability
Domain Engr.
Risk Mgmt.
Compliance
Process Overhead
Software
Value-Add
Soft
SysE
Global
Systems
of
Systems
B. Boehm, A View of 20th and 21st Century Software Engineering, 2006
(C) K. Tatsumi 20149
10. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Agenda
Prehistory
The History of Computers and Software Engineering
The Growth of Software Testing
The History of Testing Techniques
The History of Testing in Japan
The Forefront of Software Testing Research
Concluding Remarks
(C) K. Tatsumi 201410
11. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
The Growth of Software Testing
D. Gelperin and W. Hetzel,
The Growth of Software Testing, 1988
Five evolutionary periods divided
by landmark literatures
1. The Debugging-Oriented Period ( -1956)
2. The Demonstration-Oriented Period (1957-1978)
3. The Destruction-Oriented Period (1979-1982)
4. The Evaluation-Oriented Period (1983-1987)
5. The Prevention-Oriented Period (1988- )
(C) K. Tatsumi 201411
12. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
The Debugging-Oriented Period
Testing was not differentiated from debugging
Evalua
tion
Demonstration
Destru
ction
Debugging Prevention
1950 1980 19901960 1970 2000 2010
S. Gill, The diagnosis of mistakes in programmes on the EDSAC, p.539, 1951
" Software problems were submerged in the concern
for hardware reliability."
"The difficulty lies not in detecting the presence of a
mistake, but in diagnosing it."
You "wrote" a program and then you "checked it out."
• Program checkout, debugging, and testing were not
clearly differentiated.
McCracken, Digital Computer Programming, 1957
(The earliest programming text)
• Techniques referred to as today‘s debugging and testing
are described in "Chapter 13 Program Checkout".
(C) K. Tatsumi 201412
13. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
The Demonstration-Oriented Period
Testing to make sure that the software satisfies its
requirements
C. Baker, Review of "Digital Computer Programming", 1957
• "McCracken fails to distinguish the two phases of program checkout."
– Debugging: The process of making sure that the program does what the
coder meant it to do.
– Testing: The process of making sure that the program solves the problem
it is intended to solve.
Big Projects (late 1950s-1960s)
• SAGE (US air defense control system), SABRE (Computer reservation system),
NASA's Mercury, Gemini, and Apollo projects
• IBM OS/360 ref. F. Brooks, "Mythical Man-Month"
"Software Engineering"
• NATO Software Engineering Conferences (1968, 1969)
– "Testing shows the presence, not the absence of bugs." (Dijkstra,1969)
(1/2)
DemonstrationDebugging
Evalua
tion
Destru
ction
Prevention
1950 1980 19901960 1970 2000 2010
(C) K. Tatsumi 201413
14. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
The Demonstration-Oriented Period
Growth of Professionalism
• Debugging techniques
– Debugging Techniques in Large Systems Symposium, 1970
• The first formal conference on software testing
– The Computer Program Test Methods Symposium, 1972
• The first book on software testing
– Hetzel (Ed.), "Program Test Methods," 1973
Growth of Software Testing Research
• The fundamental theorem of software testing
– Goodenough and Gerhart, "Toward a Theory of Test Data Selection," 1975
• The first workshop on software testing
– Software Testing and Test Documentation Workshop, 1978
The origin of ISSTA (International Symposium on Software Testing and Analysis)
cf. W. Wulf, Concise definitions of software quality attributes, 1973
B. Boehm, et al., Software quality characteristics, 1973
DemonstrationDebugging
Evalua
tion
Destru
ction
Prevention
1950 1980 19901960 1970 2000 2010
(2/2)
(C) K. Tatsumi 201414
15. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
The Destruction-Oriented Period
Testing to detect implementation faults
G. Myers, The Art of Software Testing, 1979
• "Testing is the process of executing a program with the
intent of finding errors"
• "Since exhaustive testing is out of the question, the
objective should be to maximize the yield of the testing
investment by maximizing the number of errors found
by a finite number of test cases."
• "Test-case design is so important because complete
testing is impossible; a test of any program must be
necessarily incomplete. The obvious strategy, then, is to
try to make tests as complete as possible."
1950 1980 19901960 1970 2000 2010
DemonstrationDebugging
Evalua
tion
Destru
ction
Prevention
(C) K. Tatsumi 201415
16. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
The Evaluation-Oriented Period
Testing to evaluate products during the software
lifecycle
FIPS 101, Guideline for Lifecycle Validation, Verification, and
Testing of Computer Software, 1983
1950 1980 19901960 1970 2000 2010
DemonstrationDebugging
Evalua
tion
Destru
ction
Prevention
• "A VV&T methodology is a procedure of review,
analysis, and testing employed throughout the
software lifecycle from software planning through the
end of software use to ensure the production and
maintenance of quality software."
• "No single VV&T technique can guarantee correct,
error-free software. However, a carefully chosen set of
techniques for a specific project can help to ensure the
development and maintenance of quality software for
that project."
(C) K. Tatsumi 201416
17. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
The Prevention-Oriented Period
Testing to prevent faults in requirements, design, and
implementation
W. Hetzel, The Complete Guide to Software Testing (2nd Ed.), 1988
1950 1980 19901960 1970 2000 2010
DemonstrationDebugging
Evalua
tion
Destru
ction
Prevention
• STEP (Systematic Test and Evaluation Process)
A life cycle prevention model that sees testing parallel to
development with an activity sequence containing
planning,
analysis (setting test requirements or objectives),
design (specifying an architecture for the set of tests and details
of individual cases and procedures),
implementation (acquiring or developing test data, procedures,
and test support software),
execution (running and rerunning tests and determining the
results), and
maintenance (saving and updating the tests as the software
changes).
cf. W-Model
(C) K. Tatsumi 201417
18. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Phases in a Tester‘s Mental Life
Phase 0 - There‘s no difference between testing and
debugging. Other than in support of debugging,
testing has no purpose.
Phase 1 - The purpose of testing is to show that the
software works.
Phase 2 - The purpose of testing is to show that the
software doesn't work.
Phase 3 - The purpose of testing is not to prove anything,
but to reduce the perceived risk of not working
to an acceptable value.
Phase 4 - Testing is not an act. It is a mental discipline
that results in low-risk software without much
testing effort.
B. Beizer, Software Testing Techniques, 2nd Ed., 1990
(C) K. Tatsumi 201418
19. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Agenda
Prehistory
The History of Computers and Software Engineering
The Growth of Software Testing
The History of Testing Techniques
The History of Testing in Japan
The Forefront of Software Testing Research
Concluding Remarks
(C) K. Tatsumi 201419
20. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Testing Techniques : Beginnings
The state of the art in software testing before 1970s
< The early 1960s >
Published papers (cumulative)
• -1969: 57 papers (*1) cf. -1973: 200 papers (*2), -1977: 400 papers+ (*3)
• There are few papers on software testing techniques.
• Many papers only address testing in a peripheral manner such as test
procedures or automation.
< The late 1960s >
IBM OS/360 Project (F. Brooks, "Mythical Man-Month")
• Released in 1966. MVT, multi tasking OS, was released in 1967.
• A lot of software testing work must have been carried out.
• Much know-how about test process, test techniques, and test
management must have been accumulated.
(1/2)
(*1) W. Elmendorf, "Program Testing – A Bibliography of Published Literature, 1962–1968", 1969
(*2) W. Hetzel, Number of references in "Program Test Methods", 1973
(*3) E. Miller, Number of references in "Tutorial: Program Testing Techniques", COMPSAC '77, 1977
(C) K. Tatsumi 201420
21. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Testing Techniques : Beginnings
The test control process, IBM
W. Elmendorf, "Controlling the functional testing of
an operating system," 1969
(2/2)
• Elmendorf started to work on software testing
for OS/360 in 1965.
• A disciplined test control process
- From the "laissez-faire approach" to a
disciplined approach
- From the "testing-is-an-art" approach to a
scientific approach
A pioneering work of a preventive test
approach like W-Model
(C) K. Tatsumi 201421
22. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Equivalence Partitioning / Boundary-Value Analysis
<History>
1979: The names and concepts of EP and BVA were
first introduced by G. Myers of IBM in his "The Art of
Software Testing."
Similar concepts had already appeared in Elmendorf‘s
paper in 1967.
• External Interaction Variations
"... If the actions taken by the control program are
predictable, acceptable ranges of field values, field sizes,
field repetitions, etc., will be identified as variations at, and
just beyond, the extremes of each range. There are external
limits which, if exceeded, cause unpredictable actions."
W. Elmendorf, Evaluation of the Functional Testing of Control Programs, 1967
(C) K. Tatsumi 201422
23. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Cause-Effect Graphing
<History>
1970: W. Elmendorf developed Cause-Effect Graphing
by adopting a hardware logic testing method.
• Extending the concepts to cover software specific issues
such as constraints on the inputs
• Development of a test case generation tool
– TELDAP (TEst Library Design Automation Program) was
developed for generating test cases from a cause-effect graph,
based on the techniques for generating test patterns for the
logic circuits in hardware.
1979: Cause-Effect Graphing became widely known
after publication of Myers‘ book which introduced this
technique.
W. Elmendorf, Automated Design of Program Test Libraries, IBM-TR-00.2089, 1970
(C) K. Tatsumi 201423
24. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Decision Table Testing
<History>
Circa 1958: Decision tables were developed by General Electric
and Sutherland corp. individually
• To express the logic of product design, operation planning,
management decision rules, etc. .
1960: GE developed TABSOL (Tabular systems oriented
language) which generated program codes from decision tables.
1965: The early paper to apply DT to software testing
• Test categories and test parameters are entered to condition fields.
Consequences and test actions are entered to action fields.
1975: Condition table method by Goodenough & Gerhart
• A technique derived from decision table techniques for developing
and describing test predicates
B. Grad, Tabular form in decision logic, DATAMATION, July 1961
B. Scheff, An application of decision tables as the source language for automatic testing, 1965
J. Goodenough and S. Gerhart, Toward a theory of test data selection, 1975
T. Kavanagh, TABSOL A fundamental concept for system-oriented languages, 1960
(C) K. Tatsumi 201424
25. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Orthogonal Array / Pairwise Testing
<History>
1920s-30s: Design of Experiments (DoE), R. Fisher
1940s: Concepts of orthogonal arrays as DoE, C. Rao
1950s: Quality engineering (Taguchi Methods)
1980s: Applying DoE to software testing
• 1984: Operating system testing, Fujitsu Ltd., Japan
• 1985: Ada compiler testing, R. Mandl, US
1990s: Spreading in the US from the mid 1990s
• 1992: OATS, AT&T Bell Lab.
• 1994: CATS, AT&T
• 1994: AETG, commercial tool, Belcore
S. Sato and H. Shimokawa, Methods for setting software test parameters using the DoE, 1984
R. Mandl, Orthogonal Latin squares: an application of experiment design to compiler testing, 1985
R. Brownlie, J. Prowse, M. Phadke, Robust testing of AT&T PMX/Starmail using OATS, 1992
G. Sherwood, Effective testing of factor combinations, 1994
D. Cohen et al., The automatic efficient test generator (AETG) system, 1994
(C) K. Tatsumi 201425
27. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Control Flow Testing (1/2)
<History>
1960s: Applying graph theory to software development
• 1960: Compiler development, Karp
• 1963: Applying to software test design, Miller & Maloney
• 1976: Cyclomatic complexity and Basis path testing, McCabe
1960s: Measuring test coverage
• Code coverage tool development in IBM Poughkeepsie
– C. Warner Jr., Evaluation of program testing, 1964
Earliest known use of a hardware instruction coverage monitor:
COBOL and FORTRAN source. (from Beizer)
– I. Hirsh, MEMMAP/360, 1967
Earliest known description of a software statement and branch
coverage analyzer. (from Beizer)
R. Karp, A note on the application of graph theory to digital computer programming, 1960
J. Miller and C. Maloney, Systematic mistake analysis of digital computer programs, 1963
T. McCabe, A Complexity Measure, 1976
(C) K. Tatsumi 201427
28. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Control Flow Testing (2/2)
Early 1970s: Dynamic analysis systems (Coverage measurement tools)
• 1972: PACE (Product Assurance Confidence Evaluator), TRW
• 1972: PET (Program Evaluator and Tester), McDonnell Douglas
• 1974: RXVP (Requirement Evaluation & Verification Package), General Research
J. Brown, et al., Automated Software Quality Assurance: A Case Study of Three Systems, 1972
L. Stucki, Automatic Generation of Self Metric Software, 1972
E. Miller, et al., Structurally based automatic program testing, 1974
Early 1970s: Coverage criteria
• 1972: TER (Test Effectiveness Ratio) [Brown]
• 1975: C0, C1, C2, … [Miller]
(1975-1977) C0 : Programmer‘s intuition, C1 : Every statement in a
program exercised at least once, C2 : Every program predicate
outcome exercised at least once, ...
(1977-) C0 : Every statement executed at least once, C1 : Every segment
executed at least once, C1
p
: Every predicate term executed for each
outcome, C2 : C1 + interior and boundary tests for each iteration, ...
J. Brown, Practical applications of automated software tools, 1972
E. Miller, The Art and the Theory of Program Testing, 1975
E. Miller, Coverage levels, in "Infotech State of the Art Report: Software Testing," Vol. 1, 1979
(C) K. Tatsumi 201428
29. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Data Flow Testing
<History>
Late 1960s: Data flow analysis techniques were developed in the
field of optimizing compilers. F. Allen( IBM)
1974: Applying data flow analysis techniques to software testing,
Osterweil & Fosdick (Univ. of Colorado)
1982-: Data flow criteria, Rapps & Weyuker
Coverage criteria subsumption including control flow
criteria and data flow criteria
F. Allen and J. Cocke, A program data flow analysis procedure, 1976
L. Osterweil and L. Fosdick, Data Flow Analysis as an Aid in Documentation, Assertion Generation,
Validation and Error Detection, 1974
S. Rapps and E. Weyuker, Data Flow Analysis Techniques for Test Data Selection, 1982
(C) K. Tatsumi 201429
30. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
State Transition Testing
<History>
Originally, state transition diagrams and state transition tables
were developed to represent finite state machines (FSM).
1956: Moore‘s "Gedanken-experiments on sequential machines"
• The origin of FSM model based testing
1978: Coverage criteria "n-switch cover" proposed by Chow
E. Moore, Gedanken-experiments on sequential machines, 1956
T. Chow, Testing software designs modeled by finite-state machines, 1978
(C) K. Tatsumi 201430
31. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Agenda
Prehistory
The History of Computers and Software Engineering
The Growth of Software Testing
The History of Testing Techniques
The History of Testing in Japan
The Forefront of Software Testing Research
Concluding Remarks
(C) K. Tatsumi 201431
32. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
The History of Testing in Japan (1/3)
1958, The Earliest paper on software testing
Fujiwara, A report on program testing on the IBM 704 large
scale electronic computer (in Japanese), 1958
• Preliminary program testing for a numerical weather prediction
system of the Japan Meteorological Agency
1964, Discussion on the program inspection
Information Processing Society of Japan, Special features :
Software ,IPSJ Magazine, 1964
• Necessity of independent section for software inspection
• Role of software quality assurance section
• Process improvement through the feedback from QA section
1969, Hitachi’s Software Works (Factory)
Establishing the function of software quality assurance in the
software factory system
1971: Also, Fujitsu created a formal product inspection
procedures for release to the customer.
(C) K. Tatsumi 201432
33. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
The History of Testing in Japan (2/3)
1972, An Approach for Software Inspection
A. Kanno, An Approach for Software Inspection, 1972
• Concepts, activities and techniques in software quality assurance department
in Hitachi software factory
1974, Development of QA techniques
K. Sakata(Hitachi), Formulation for predictive methods in
software production control, 1974
• Static prediction and failure rate transition model
Quality prediction by reliability growth curve or Gompertz curve
• Dynamic prediction: quality probe
Quality prediction by "quality probe" that comprises a small percentage of
the complete regular test
(C) K. Tatsumi 201433
34. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
The History of Testing in Japan (3/3)
1980-, R&D of testing techniques and tools
1980: AGENT(Automated GENeration system for Test cases), Hitachi
• Test case generation tool based on cause-effect graphing
1984: AGENT Function Diagram, Hitachi
• Representation of functional specification (dynamic part -> state transition
diagram, static part -> decision table or cause-effect graph)
1984: Application of DoE to software testing, Fujitsu
• Test case generation technique and tool for application of DoE
1988: CFD(Case Flow Diagram or Cause Flow Diagram), NEC
• External and internal specifications are organized into Cause Flow Diagram,
then decision table is generated from the diagram.
1991, Japan's Software Factories
M. Cusumano, Japan's Software Factories: A Challenge
to U.S. Management, 1991
• A management research work about Japanese software
development and a survey of the making of Japan‘s computer
industry, Hitachi, Toshiba, NEC, and Fujitsu etc.
(C) K. Tatsumi 201434
35. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Agenda
Prehistory
The History of Computers and Software Engineering
The Growth of Software Testing
The History of Testing Techniques
The History of Testing in Japan
The Forefront of Software Testing Research
Concluding Remarks
(C) K. Tatsumi 201435
36. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
The Forefront of Software Testing
A. Bertolino, "Software Testing Research:
Achievements, Challenges, Dreams," 2007
•29th ICSE, Future of Software Engineering track
•Bertolino is the KA co-leader for Software Testing in
the IEEE Guide to the SWEBOK
The software testing research roadmap
• Achievements
The most notable achievements from past research
• Dreams
The desired destination
• Challenges
The challenges faced by current and future testing research
(C) K. Tatsumi 201436
37. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
The Many Faces of Software Testing
Software testing:
Observing a sample of executions,
and giving a verdict over them
6 aspects of software testing
WHY: test objective
looking for faults?, the product can
be released?, evaluate the usability?
HOW: test selection
ad hoc, at random, or systematic way
HOW MUCH: test adequacy
coverage analysis, reliability
measures
WHAT: levels of testing
unit test, component/subsystem test,
integration test
WHERE
in house, simulated environment, the
target final context
WHEN
when is it in the product lifecycle that
we perform the observations?
(C) K. Tatsumi 201437
38. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Software Testing Research Roadmap
A. Bertolino, “Software Testing Research: Achievements, Challenges, Dreams,” FOSE’07, p.85-103, 2007, Figure 1
Testing
process
Reliability
testing
WHY How How much What Where When
Protocol
testing
Test
criteria
Comparison among
test criteria
Component-basedtesting
Object-orientedtesting
Education of software testers
Testing patterns
Controlling evolution
Leveraging user population
and resources
Understanding the
costs of testing
Test input
generation
On-line testing
Test
oracles
Model-based testing
Anti-model-based testing
Explicit test
hypotheses
Test effectiveness
Empirical body of evidence
Compositional
Testing
Domain-specific
test approaches
Achievements Challenges Dreams
Efficacy-
maximized
test engineering
100% automatic
testing
Test-based
modeling
Universal
test theory
(C) K. Tatsumi 201438
40. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Dreams
(1) Universal test theory
A comprehensive theory which is useful to backup and nourish
test technology
(2) Test-based modeling
Test-based modeling is closely related to the old idea of
“Design-for-testability.”
(3) 100% automatic testing
Advanced techniques for generating the test inputs, innovative
support procedures to automate the testing process
(4) Efficacy-maximized test engineering
Practical testing methods, tools and processes for development
of high quality software
(C) K. Tatsumi 201440
41. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Dream: Universal test theory
<Challenges>
Explicit test hypotheses
• Making explicit for each technique which are its underlying
assumptions
Test effectiveness
• Assessing the effectiveness of existing and novel test criteria
Compositional testing
• Reusing the test results observed in the separate testing of the
individual levels
• A foundational theory for component-based software reliability
Empirical body of evidence
• An empirical body of knowledge which is at the basis for
building and evolving the theory for testing
(C) K. Tatsumi 201441
42. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Dreams: Test-based Modeling
<Challenges>
Model-based testing
• Combining different styles of modeling (transition-based,
pre/post condition-based and scenario-based)
• Integrating model-based testing practice into current software
processes
Anti-model-based testing
• A model is derived a posteriori via testing for the cases in which
the models do not exist or are not accessible, such as for COTS
or legacy components.
Test oracles
(C) K. Tatsumi 201442
43. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Dream: 100% automatic testing
<Challenges>
Test input generation
• Model-based test generation, random test generation, search-
based test generation
Domain-specific test approaches
On-line testing
• Monitoring a system‘s behavior in real life operation using
dynamic analysis and self-test techniques
(C) K. Tatsumi 201443
45. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Transversal challenges
<Challenges>
Testing within the emerging development
paradigm
• Testing of service-oriented applications in the Service-oriented
Computing
• A special importance for testing of services (monitoring the real-
world execution is the only way to observe the application
behavior)
Coherent testing of functional and extra-
functional properties
• Conventional functionality testing does not provide for any
notion of time nor tackle resource usage and workloads.
• For the model-based approach, we need effective ways to
enhance models with desired extra-functional constraints.
(C) K. Tatsumi 201445
46. HISTORYOFSOFTWARETESTING
HISTORYOFSOFTWARETESTING
Concluding Remarks
The Analects - Chinese Text Project http://ctext.org/analects
• The Master said, "If a man keeps cherishing his old knowledge,
so as continually to be acquiring new, he may be a teacher of
others."
• The Master said, "Is it not pleasant to learn with a constant perseverance
and application? Is it not delightful to have friends coming from distant
quarters? Is he not a man of complete virtue, who feels no discomposure
though men may take no note of him?"
• The Master said, "Learning without thought is labor lost; thought without
learning is perilous."
• The Master said, "You, shall I teach you what knowledge is? When you
know a thing, to hold that you know it; and when you do not know a
thing, to allow that you do not know it - this is knowledge."
(C) K. Tatsumi 201446