Test engineering foundation (v2.01)

Test Engineering Foundation
Essential Knowledge for Test Professionals
Rex Black
RBCS, Inc.
31520 Beck Road
Bulverde, TX 78163 USA
Phone: +1 (830) 438-4830
Fax: +1 (830) 438-4831
www.rexblackconsulting.com
rex_black@rexblackconsulting.com

Introductions, Objectives,
and Overview

Rex Black Consulting Services, Inc.
Technical and Project Management Expertise for Quality
Test Engr Foundation (4D-R2.01)Copyright (c) Rex Black 1996-2005Page 3
The Course
Who is this presenter?
What is this course?
What are the hours? breaks?
Must I do exercises?
Do I have homework?
Can I get certified?
May I use my cell phone or read e-mail
during the course?
Any others???

The Materials
These slides following the Foundation Syllabus 2005
Terms come from the ISTQB Glossary (v1.1)
In practice terminological variation is wide
Not all terms in syllabus are defined in glossary
Reference materials, glossary, and study aids for
those pursuing certification
For use in exercises
Omninet Marketing Requirements Document
Omninet System Requirements Document
Solutions for the exercises
A notepad on which to do the exercises

ISTQB Foundation Syllabus 2005
Developed by a team of eight authors spanning seven
countries: Thomas Müller (chair), Rex Black, Sigrid
Eldh, Dorothy Graham, Klaus Olsen, Maaret
Pyhäjärvi, Geoff Thompson and Erik van
Veendendaal
Forty primary reviewers spanning nine countries
Final review and approval by 18 National Boards
Distills over 1,000 person-years of experience
The ISTQB and the authors are the source of the
syllabus (copyright © 2005 by the authors and ISTQB)
which is used by permission as the basis for this
course

The Main Topics Covered
1. Fundamentals of testing
2. Testing throughout the software
lifecycle
3. Static techniques
4. Test design techniques
5. Test management
6. Tool support for testing

The Exercises
In many of the exercises, you are working as
a tester on the Omninet project
Omninet is a project to deploy a network of
public access Internet kiosks in places like
malls, theaters, and other public places
On this realistic project, you will have a
chance to apply many of the techniques we
discuss
Other exercises have been added to illustrate
specific points, too

Your Course Objectives
This material is designed to help you
become a more effective and efficient
test professional, and to help you
obtain ISTQB certification
Please spend the next few minutes
writing down what you’d like to get
out of the course
Use the following page for your
objectives
Don’t let the class end with an
objective unfulfilled!

Course Objectives

It’s Your Course
Please join in
 Ask questions
 Make comments
 Share experiences
 Second opinions and
disagreements welcome
Why are you here?
What do you want to
learn, discuss, and
teach?
…as well as this.
The best sessions have a lot of this...

Chapter 1:
Fundamentals of Testing

1. Fundamentals of Testing
1. Why is testing necessary?
2. What is testing?
3. General testing principles
4. Fundamental test process
5. The psychology of testing

Chapter 1:
Section 1:
Why is testing necessary?

Why is Testing Necessary?
Key concepts
How bugs can cause harm
Bugs and their effects
The necessity of testing
The role of testing in quality assurance
Terms to remember

The Threat of Bugs
Company
Damaged reputation
for quality
High or
unpredictable
maintenance costs
Unexpected delays
in release cycles
Lack of confidence in
system
Lawsuits
Environment
Pollution
Waste
People, societies,
and states
Lost jobs
Lost lives
Lost rights
Lost missions
Lost wars

Where Bugs Come From and What Bugs Do
People put bugs (defects) into the system
Requirements and design specifications
Code (business logic and user interface)
Documentation (electronic and hard copy)
When the implementation of these bugs are
executed, failures occur
If these failures are visible to customers,
users, or other stakeholders, dissatisfaction
with system quality results

From Whence the Bugs and Failures?
Bugs occur due to…
programmer, analyst, and other individual
contributor (including tester) fallibility
time pressure
complexity of the code, infrastructure, or problem
to be solved
changing and meshing technologies
many system interactions.
Failures occur due to bugs and…
environmental conditions
misuse (deliberate and accidental)

Testing to Manage Quality Risks
Risks and constraints for a software project
Features: Right set
Schedule: Quickly enough
Budget: Acceptably cheap
Quality: Ready for customers/release/next step
Testing provides the information to guide the
project, reduce and manage the risks, and
repair the important problems
Testing may also address compliance,
contractual, and regulatory needs

What Does “Quality” Mean to You?
“Fitness for use” vs. “Conformance to requirements”
Testing and quality
Tests gives confidence where they find few bugs
Passing tests reduce the level of quality risk
Failing tests provide a chance to improve quality
The test set gives an assessment of quality
What are the important quality characteristics for
your system? Are you testing them (enough)?
Testing, quality assurance, and quality improvement
Ideally, testing is part of a larger quality assurance strategy
for a project
For future projects, analyze the root causes of defects found
on current projects and take steps to reduce their incidence

Terms to Remember
Bug, defect, or fault
Error or mistake
Failure
Quality
Risk
Software
Test
Test case

Chapter 1:
Section 1:
A diversion

What Happens When Quality is Lacking

Chapter 1:
Section 2:
What is testing?

What is Testing?
Key concepts
Common objectives of testing
The purpose of testing…
•in software development, maintenance, and
operations…
•…to find defects, provide confidence and
information, and prevent defects
Terms to remember

Testing Objectives
Typical general testing objectives
Finding bugs and providing programmers with
the information they need to fix important bugs
Gaining confidence about the level of quality of
the system
Preventing defects (through early involvement in
reviews and advanced test design)
Provide information about the most important
aspects of the quality of the system under test
Help management understand system quality
Can you think of others for your projects?

Unit/Component Test
Find bugs in the individual pieces of the system under test before the pieces
are fully integrated into the system
Integration/String Test
Find bugs in the relationships and interfaces between pairs and groups of
components in the system under test as the pieces come together
System Test
Find bugs in the overall and particular behaviors, functions, and responses of
the system under test as a whole
Acceptance/Pilot Test
Demonstrate that the product is ready for deployment/release or to assess
quality and give information on the risk of deployment/release
Maintenance Test
Check for errors introduced during development of the changes
Operational Test
Assess non-functional system characteristics such as reliability or availability
Test Phases and Objectives

Effectiveness and Efficiency
Effective: Producing a
decided, decisive, or
desired result;
impressive
To be effective testers,
we must select the
appropriate objective
and desired results
Efficient: Productive of
desired effect;
especially productive
without waste
To be efficient testers,
we must allocate
resources (time and
money) appropriately
These terms are most meaningful in the context of the entire
development or maintenance process, not just the testing process.

Testing vs. Debugging
Testing finds failures that are caused by bugs
Debugging…
…identifies the root cause of a bug…
…repairs the code…
…and checks that the defect is fixed correctly
Confirmation testing ensures the fix resolves
the observed failure
Different responsibilities:
Testers test
Programmers debug

Find-Debug-Confirm

Beyond Test Execution
Testing is not just running tests against a running
system
Other test activities, before and after execution,
include…
Planning and control
Choosing test conditions
Design test cases
Checking test results
Evaluating exit criteria
Test result reporting
Closure/end-of-test tasks
We will revisit this later

Terms to Remember
Code
Debugging
Software development
Review
Requirement
Test case
Test objective
Testing
Test basis

Chapter 1:
Section 2:
Exercise

Exercise: The Triangle Tests
A program accepts three integers representing the
lengths of a triangle’s sides.
It outputs “scalene” (no equal sides), “isosceles” (two
equal sides), or “equilateral” (three equal sides).
Write an effective (finds common bugs) and efficient
(as few tests as possible) set of test cases. Usually, a
test case consists of tester action, data, and expected
result, but here the action is generally “input data”.
Discuss.

Chapter 1:
Section 3:
General testing principles

General Testing Principles
Key concepts
Testing reveals the presence of bugs
Impossibility of exhaustive testing
Benefits of early testing
Lumpiness of bugs: defect clustering
Pesticide paradox
Testing should adapt to specific needs
Absence-of-errors fallacy
Terms to remember

Testing Reveals Presence of Bugs: A Parable
You have a beautiful vegetable garden, but
one day you see eaten leaves on the tomatoes
“Oh no,” you think, “I have hornworms!”
You know you have bugs in your garden
If you had not seen the symptoms, could you
be sure you had no bugs?
Some bugs are easy to spot, others aren’t
Testing can reveal the presence
of bugs, but cannot prove their
absence

Mission Impossible: Exhaustive Testing
“Just make sure the software works before we
ship it…”
This charter is demonstrably impossible
The execution paths in non-trivial software are
almost infinite
Large dataflows separated across space (features)
and time (static data)
Slight changes can cause regressions which are not
linear to the size of the change
Myriad usage profiles and field configurations,
some unknown and some unknowable

Defusing Exhaustive Testing Expectations
Exhaustive testing as a way to prove the
software works is a common (mis)expectation
Bad expectations create problems for test
professionals and test teams
Unachievable high demands on test group
Perception of incompetence when these demands
aren’t met
Testers must be ready to communicate (in
words the project stakeholders will
understand) how testing can contribute

Benefits of Early QA and Testing
The cost of a bug tends to increase as the
project continues
Most of the costs associated with pre-release
bugs tend to be associated with the effort
required to remove them, so the higher cost
means longer schedules
The more bugs enter a quality assurance or
test activity, the more bugs will escape from
that activity

Click here to see
the answers

Defect Clustering
Studies have long shown and continue to
show the unequal distribution of bugs
MVS: 38% of field bugs in 4% of modules
IMS: 57% of field bugs in 7% of modules
Capers Jones reports that the excessive
presence of error-prone modules causes a
50% reduction of productivity in software
maintenance

Pesticide Paradox
Return to your vegetable garden
You spray pesticide on your garden, and the
hornworms die, but the pesticide is not
effective against all bugs
Just as pesticides become less effective, so do
tests
Functional tests can’t find
performance bugs
Try new test techniques

These two pictures
compare efficiencies of
the top 11 bug-finding
test suites (out of 27).
The top graph shows the
first time the set of test
suites was run. The
bottom graph shows the
results for the fifth time
that same set of test suites
was run. The average
test suite efficiency in the
first pass, 0.4, is used as
the axis crossing point in
both graphs.
With one exception, the
same test suites are in the
top 11 test suites, which
shows bug clustering
again. However, all test
suites are less effective
after five executions.

Absence-of-Errors Fallacy
Finding and fixing many bugs does not
guarantee user, customer, and/or
stakeholder satisfaction
Many low-defect products have failed
in the market place
Successful projects balance competing
forces in terms of features, schedule,
budget, and quality

Testing Should Adapt to Needs
Different projects, organizations, and
products have different testing needs
Best testing practices exist (and are
discussed in this course) but you need
to tailor them to your project
Failure to adapt the test team and its
methods to these needs is a common
result of dissolution of test teams

Terms to Remember
Exhaustive testing

Chapter 1:
Section 3:
Exercise

Exercise: Test Principles Observed (and Not)
Think back on a recent project.
Note which of the principles that you can recall were
observed.
How about principles that you can recall were not
observed (i.e., violated)?
Discuss.

Chapter 1:
Section 4:
Fundamental test process

Fundamental Test Process
Key concepts
Plan, prepare, perform, perfect
Analysis and design
Implementation and execution
Evaluating test exit criteria and reporting
Test closure activities
Terms to remember

Test Processes: Critical Testing Processes
The test engineer’s role focuses on some of these activities, not all,
depending on how roles are defined. However, the effective and
efficient test engineer must understand how the test process works
and how it fits into the overall project from a big picture perspective.

ISTQB Fundamental Test Process
Another way to think of the test process is in
terms of the following steps
Analysis and design
Implementation and execution
Evaluating test exit criteria and reporting
Test closure activities
These steps may overlap or take place
concurrently

Planning and Control
Planning
Determine test scope, risks,
objectives, strategies
Determine required test
resources
Implement the test strategies
Schedule test analysis and
design
Schedule implementation,
execution and evaluation of
tests
Determine the test exit
criteria
Control
Measure and analyze results
Monitor and document
progress, coverage and test
exit criteria
Initiate corrective actions
Make decisions

Analysis and Design
Analysis
Review the test basis (e.g.,
requirements or design
specifications, network/
system architecture, quality
risks)
Identify test conditions, test
requirements, or test
objectives and required test
data based on analysis of test
items, its specification,
behavior and structure
Design
Select specific combinations
of test data, actions, and
expected results to cover the
test basis
Evaluate testability of the
requirements and system
Design the test environment
Identify any required
infrastructure and tools

Implementation and Execution
Implementation
Develop and prioritize test
cases, create test data, write
test procedures
Prepare test harnesses and
write automated test scripts
Organize test suites and
sequences of test cases for
efficient test execution
Verify that the test
environment has been set up
correctly
Execution
Execute test cases (manual
or automated)
Log test results, and the
versions of the software
under test, test tools and the
testware
Compare actual and
expected results
Report and analyze incidents
Repeat corrected and/or
updated tests
Run confirmation and/or
regression tests

Exit Criteria, Reporting, and Closure
Exit and reporting
Check test logs against the
test exit criteria specified in
test planning
Assess if more tests are
needed or if the exit criteria
specified should be changed
Write a test summary report
for stakeholders
Closure
Confirm test deliverables,
final resolution or deferral of
bug reports, and the
acceptance of the system
Finalize and archive
testware, test environment
and test infrastructure
Deliver testware to the
maintenance organization
Perform a retrospective to
capture improvements for
future releases, projects, and
test processes

Terms to Remember
Testware
Test plan
Test strategy
Test execution
Test basis
Test log
(Test) exit criteria
Test summary report
Test coverage
Test condition

Terms to Remember
Test data
Test input
Confirmation testing
Regression testing
Incident

Chapter 1:
Section 4:
Exercise

Exercise: Test Steps and Tasks Performed
Think back on a recent project.
Note which of the steps and tasks of the ISTQB test
process were carried out.
Note which of the steps and tasks of the ISTQB test
process were not carried out.
Note whether some steps overlapped or were
completely parallel.
Discuss.

Chapter 1:
Section 5:
The psychology of testing

Psychology of Testing
Key concepts
Psychological factors for testing success
Clear objectives
Self-testing and independent testing
Respectful communication
Programmer and tester outlooks for
success
Terms to remember

Certainty Versus Progress
Thorough research can lead to indisputable bug
reports, unquestionably correct test cases, etc.
Such certainty is intellectually satisfying and reduces
rejected and irreproducible bug reports
But certainty also can consume too much time and
effort for the payoff and delay forward progress
Test projects adapt to schedule pressures or fail
Testing computers is test engineering
Not a search for truth—that’s science
Engineering is making useful objects for customers

Test Result Misinterpretation
On the one hand we could
report correct behavior as a
bug
assign excessively high
severity or priority
otherwise overstate bug
significance
On the other hand we could
fail to detect or report
incorrect behavior
assign excessively low
severity or priority
otherwise understate bug
significance
Some tips to avoid these errors
1. Have testers take breaks so they don’t miss important events
2. Automate where practical
3. Define expected results as clearly as possible
4. Assign the right testers to each test execution task
5. Use peer reviews for test execution and bug reports
However…recognize that perfect test execution takes too long.
Define “good enough” for testing and live with that decision

Professional Pessimism
Explore depressing possibilities of failure
Anticipate the worst possibilities in order to
achieve best obtainable product quality
Not adversarial, but a different outlook than
the programmers
Remember: to assume nothing will fail
during testing denies the entire history
of computing
Caveat: not a license to offend
 Don’t target programmers with reports
or take glee in failure
Challenge: to be positive, pleasant, and
the bearer of bad news, all at once
Pierre has the pessimism, but
perhaps not professionalism?

Bad News == Bad Guy?
Testers are sometimes on receiving end of emotions
brought on by news of project problems
The key is professional pessimism
Don’t take on the role of “Lone Champion of Quality”
• Product quality is a business decision
• Quality risks are weighed against other risks
Don’t second-guess development’s bug-fix efforts
Never gloat, even if you were right and everyone else
was wrong
Juan de Mariana: “The greatest of follies is to
exert oneself in vain, and to weary oneself
without winning anything but hatred.”
Grim
reaper or
friendly
guide?

Balanced Curiosity
Balance need for thoroughness in any one
area with need to cover many areas in a short
time
Effective and efficient test engineers have a
talent for spending time where the bugs are
Effective and efficient test engineers can do
thorough bug isolation quickly
Ineffective and inefficient test engineers
Write tests to search for unlikely, low-impact bugs
Spend hours researching trivial bugs

Two types of focus problems
Pursuing issues narrow-mindedly,
losing sight of more important
priorities
Getting distracted from key tasks
Balance and re-evaluate
priorities every so often
Stay focused on the goals of
the test project
Focus
A seasoned test engineer
can find his way towards
test project goals, with
clear signposts from his
test manager

Defining Tester Skills
Reading
Specifications, e-mails, test cases, etc.
Writing
Test cases, bug reports, test documentation, etc.
Not “native language” dependent
Statistics and other mathematics
Pertinent technology, project, and testing skills
Technology: Programming languages and more, like
operating systems, networking, HTML/Web, etc.
Application domain: banking, human factors, office
applications, etc.
Testing: scripting, exploring and attacking the system,
automation, performance modeling, etc.

Balancing the Skills
Effective and efficient tester have right
mix of skills for tasks and activities
Application domain expert
Understands intended behavior
Skilled tester
Knows quality risks and test techniques
Technical guru
Aware of technical issues and limitations
What is the right mix for…
…Internet appliance testing?
…nuclear medicine testing?
…your project?
The appropriate depth and
length of each arrow in the
figure depends on the
project, process, and product

Terms to Remember
Independent testing

Chapter 1:
Section 5:
Exercise

Exercise: Psychology in Action
Reflect on the attitudes and behaviors of the most
successful testers you know.
To what extent do they display the psychological
aspects discussed in this section?
What other elements of their personalities and
skillsets do you think lead to success for them?
Discuss.

Chapter 2:
Testing throughout
the Software Lifecycle

2. Testing Throughout the Software Lifecycle
1. Software development models
2. Test levels or phases
3. Test types or targets
4. Maintenance testing

Chapter 2:
Testing throughout
Section 1:
Software development models

Software Development Models
Key concepts
The relationship between development and
test activities
Adapting software development models to
the context of the project and product
Reasons for different levels of testing
Terms to remember

The “V” Model in Waterfall Projects
Usually schedule- and
budget-risk-driven
Do ever-deeper levels
of design, then build, then
test
Intuitive and familiar model
Beats chaos!
It’s hard to plan that far in
advance!
When plans fail, test--at the
end--suffers!

Evolutionary and Incremental Models
Schedule-risk-driven to hit market window or delivery date
Feature set grown around core functionality
Can ship (something) any time once the core functionality is ready
Becoming a popular approach
Ranges in formality from Extreme Programming to RAD and RUP
 Still a temptation to ship a system with buggy features
 In the “agile” world, the role of testing still evolving

System Integration
Many projects involve
integrating components
Risk-mitigation options
Integrate, track, and manage
vendor testing in distributed
test effort
Trust vendor component testing
Fix vendor testing/quality
Disregard and replace their
testing (ouch!)
Watch politics in last two
options
Plan on integration and system
testing yourself
Coupling: Strong interaction or consequence
of failure between component and system
Irreplaceability: Few similar components
available
Essential: Key features in system unavailable
if component does not work properly
Vendor quality problems: Increased likelihood
of a bad component

Verification & Validation
Verification
Look for
bugs in phase
deliverables
“Are we building
the system right?”
Validation
Looking for bugs
in system, based
on phase
deliverables
“Are we building the
right system?”

IEEE 12207 Standard
1. Scope
Purpose, application, tailoring, compliance, limitations
1. Normative references
2. Definitions
3. Application
Lifecycle processes, tailoring, and fitting to organization
1. Primary life cycle processes
Acquisition, supply, development, operation,
maintenance
1. Supporting processes
Tech pubs, CM, QA, IV&V, audits, problem resolution
1. Organizational life cycle processes
Management, infrastructure, improvement, training

CMMI Process Maturity
Capability Maturity Model Integration: a five-level
model from the Software Engineering Institute
Initial: unpredictable, poorly controlled, reactive
Managed: process established at process level, often reactive
Defined: process established across organization, usually
proactive
Quantitatively managed: process measured and controlled
at organization
Optimizing: focus on continuous improvement, usually
driven by data
Testing has been underemphasized in CMM, leading
to test-specific model (e.g., Critical Testing Processes,
Test Process Improvement, Testing Maturity Model)

Regardless of the Models…
General characteristics of good testing
Testing activity for each development activity
(e.g., unit test and implementation)
Test levels have focused objectives, with
coordination to avoid gaps, overlap
Test analysis, design begins early, prevents bugs
Testers involved in any reviews they are qualified
to attend, bringing their unique perspective
You can combine or reorganize test levels
provided you keep these characteristics in
mind

Terms to Remember
Test level
Verification
Validation
V-Model
Incremental development model
Commercial Off The Shelf (COTS)

Chapter 2:
Testing throughout
Section 1:
Exercise 1

Exercise: Models and Reality
Famed quality expert W.E. Deming said, “All models
are wrong; some are useful.”
Indicate which lifecycle model in this section applied
most closely to your past project (or, if there was no
organizing model, indicate “code-and-fix”).
To what extent do you think the model was useful?
To what extent, if any, was it harmful?
Discuss.

Chapter 2:
Testing throughout
Section 1:
Exercise 2

Exercise: Omninet Lifecycle Considerations
Read the Omninet Marketing Requirements
Document.
Is a V-model (sequential) or incremental model more
appropriate for this project? Why?
Are maintenance, integration, or verification and
validation important for this project? Why?
Discuss.

Chapter 2:
Testing throughout
Section 2:
Test levels or phases

Test Phases or Levels
Key concepts
Major objectives of testing for each level
Typical testing objects for each level
Typical targets of testing for each level
Testing work products for each level
Test participants for each level
Types of defects and failures for each level
Terms to remember

Component (Unit) Test
Objective: Find bugs in the individual pieces of the
system under test prior to system integration
Basis: Code, database, reqs/design, quality risks
Test types: Functionality, resource use, performance,
structural
Item Under Test (IUT): Varies. Smallest
independently testable item (function or class) or a
distinct component providing services to others
Harnesses and tools: API level (drivers and stubs),
freeware and commercial
Responsible: Usually programmers, but the level of
proficiency and degree of execution varies.

Unit/Component Test Process
Unit/component testing typically…
…involves access to the code
…is run in a development environment
…requires drivers, stubs, and/or harnesses
…is done by the programmer who wrote the code
Often, bugs are fixed upon being found without any
reporting, which reduces the transparency of the
development process with respect to quality
Test-first/test-driven development
Develop a set of unit tests
Build and integrate code
Run the tests and debug until the tests pass

Drivers and Stubs
During unit, component, and integration testing--and
for testing APIs--it’s often necessary to simulate parts of
call flow reachable from module(s) under test
Data setup sometimes necessary, also
Driver: function(s)
that call module(s)
under test
Stub: function(s)
called--directly or
indirectly--by
module(s) under test

Integration Test
Objective: Find bugs in the relationships and
interfaces between pairs and groups of components
in the system under test as the pieces come together
Basis: Design, architecture, schemas, dataflows,
quality risks
Test types: Functionality, resource use, performance
Item Under Test: Builds or backbones
Harnesses and tools: API and CLI level, freeware and
commercial
Responsible: Ideally both testers and programmers,
but often no one

Integration Techniques
Big bang
Take all tested modules; put
them all together; test
Quick, but where’s the bug?
Why wait until all code is
written to start integration?
Bottom up
Start with bottom layer
modules; use appropriate
drivers; test
Repeat process, replacing
drivers with modules, until
done
Good bug isolation, but what if
nasty problems are at the top?
Top down
Like bottom up, but start from
top and use stubs
Good bug isolation, but what if
nasty problems are at the
bottom?
Backbone
Start with critical modules;
build initial backbone; use
drivers and stubs; test
Repeat process, replacing stubs
and drivers with modules in
risk order
Good bug isolation and finds
integration bugs in risk order

Backbone Integration Technique
Backbone 0 (BB0)
Consider the following example
(based on a real project)
We start with a basic backbone
Communication APIs
Basic networking architecture
Test basic functionality, error handling
and recovery, reliability, and
performance
Quality risk: Is the underlying system
architecture untenable?
Backbone 0: Testing the basic
communication software and
network architecture

Backbone 1 (BB1)
Add the modules that implement
the most critical core operations and
services
Again, test basic functionality, error
handling and recovery, reliability,
and performance
Quality risk: Do the core operations
and functions integrate with the
transport layer?
Continue process with next level
of quality risk…. Backbone 1: Testing some core
operations and services through the
communication API and network

Backbone N (BBn)
As the last step, we are testing
(in this case, using automated
test drivers) end-to-end through
the GUIs on the host systems
Quality risk: Does the fully
integrated system work?
The final backbone for
integration testing is also the first
fully integrated build for system
test
Backbone N: Testing the entire,
completely integrated system end-
to-end. When this works, we’re
ready for System Test

Integration Test Levels
There may be more than one level of integration
testing on a project
Component integration testing: look for bugs in interactions
between units or components following unit/component
test
System integration test: look for bugs in interaction between
entire systems following system test
System integration testing is complex
Multiple organizations controlling the systems’ interfaces,
making change dangerous
Business processes may span systems
Hardware/system compatibility issues can arise

System Test
Objective: Find bugs in the overall and particular
behaviors, functions, and responses of the system
under test as a whole
Basis: Requirements, high-level design, use cases,
quality risks, experience, checklists, environments
Test types: Functionality, security, performance,
reliability, usability, portability, etc.
Item Under Test: Whole system, in as realistic-as-
possible test environment
Harnesses and tools: API, CLI, or GUI, freeware and
commercial
Responsible: Typically independent testers

Acceptance Test
Objective: Demonstrate that the product is ready for
deployment/release
Basis: Requirements, contracts, experience
Test types: Functional, portability, performance
Item Under Test: Whole system, sometimes in the
production or customer environment
Harnesses and tools: GUI usually
Responsible: Often users or customers, but also
independent testers

Variations in Acceptance Testing
User acceptance testing: Business users verify fitness
for functional purposes.
Operational testing: Acceptance by system
administrators (e.g., backup-restore, disaster
recovery, user management, maintenance, security)
Contract and regulation testing: Verification of
conformance to contractually-agreed or legally
mandated requirements, regulations, or standards.
Alpha, Beta, and field testing: Testing and
confidence-building by potential or existing
customers. Beta testing and field testing are
performed in the actual environment(s).

Pervasive Testing: Early, Cross-Functional
Test
execution
activities
for test
phases
Project
timeline

Why Pervasive Testing?
Different participants can test different granularities
Different skills for different granularities
Different granularities emphasized in each phase
Unit testing: primarily structural
System testing: primarily behavioral
Acceptance testing: primarily live
It’s important to be flexible, though
Test techniques of various granularities can be useful in all
the test execution phases
Phase overlap a function of entry and exit criteria
Not all test phases occur on all projects

When Testing Pervades Projects
Test tasks occur throughout the development effort
Test execution is planned with multiple cycles to allow for fix time
Features will be dropped or slipped into later iterations rather than
slipping test phase entry dates or entering before ready

Terms to Remember
Alpha testing
Beta or field testing
Business process-based testing
Component testing
Contract acceptance testing
Daily build
Drivers
Functional requirements
Integration
Integration testing

Terms to Remember
Non-functional requirements
Operational (acceptance) testing
Regulation testing
Requirements-based testing
Robustness testing
Stub
System testing
Test-driven development
Test environment
Testware
User acceptance testing

Chapter 2:
Testing throughout
Section 2:
Exercise

Exercise: Omninet Test Levels
Read the Omninet System Requirements Document.
If you were managing all the testing for Omninet,
which levels or phases of testing would you plan?
Why?
What would the major goals of each level or phase of
testing be?
What kind of acceptance test, if any, would you plan?
Why?
How do these test levels relate to and affect the
lifecycle model you selected in the previous exercise?
Discuss.

Chapter 2:
Testing throughout
Section 3:
Test types or targets

Test Types or Targets
Key concepts
Major software test types or targets
Functional and non-functional tests
Structural tests
Confirmation and regression tests
Use of functional, non-functional, and
structural tests at various levels
Terms to remember

Functionality
Reasonable or required action not provided,
inaccessible, or seriously impaired
No add function on a calculator
Add function implemented, “+” key doesn’t work
Can only add integers, not real numbers
Right action, wrong result
Add function: 2+2=5?
Right action, right result, wrong side-effect
Divide function: 2/2=I (Roman numeral format)
System, subsystem or component functionality is described in
documents like requirements specification, use cases, or a functional
specification (sometimes). However, often some functions remain
undocumented, and testers must understand “reasonable behavior”.

Security
Security threats include
Virii
Cracking into servers
Denial of service
Common bad assumptions
Encryption (HTTPS) on Web
server solves security problems
Buying a firewall solves problems
Unskilled network or system
administrators can solve
problems
Specialized field of test
expertise
The determined--or bored--cracker might
break into any of your servers. One
“exploit” can lead the cracker to other
vulnerabilities--and to valuable data.

Performance And Reliability
Performance
Too slow throughout
performance curve
Unacceptable “knee”
in performance curve
Unacceptable performance
degradation over time
Reliability
System fails to complete normal functions
System functions normally, but randomly crashes
or hangs

Stress, Capacity, and Volume
Stress: extreme conditions cause failure
Combinations of error, capacity, and volume tests
Capacity: functionality, performance, or
reliability problems due to resource depletion
Fill hard drive or memory to 80+%
Volume: functionality, performance, or
reliability problems due to rate of data flows
Run 80+% of rated transactions per minute,
number of simultaneous users, etc.

Maintenance And Maintainability
Maintenance
Update and patch
install and deinstall
processes don’t work
Configurations can’t be
changed appropriately
(e.g., plug-and-play, hot
plugging, adding disk
space, etc.)
Maintainability
Software itself (source
code) not maintainable
Databases not
upgradeable
Databases grow
monotonically
Software not efficiently
testable during
maintenance; e.g.,
excessive regression

Usability and User Interface
A system can function properly but be
unusable by the intended customer
Cumbersome interfaces that do not follow
workflows
Inaccessible functionality
Inappropriately difficult for the users to learn
Instructional, help, and error messages that
are misleading, confusing, or misspelled

Configuration, Compatibility, Portability
A single platform may be configured in many
different ways in software
A family of platforms may support various
hardware configurations
Are configuration changes handled?
Add disk space or other storage
Add memory
Upgrade or add CPU
Interoperability with programs, OS, database
Portability to various environments

Other Functional/Non-Functional Tests
Localization (user
interface)
Localization
(operational)
Standards and
regulatory compliance
Error handling and
recovery
Disaster recovery
Networked/internet-
worked or distributed
Timing and
coordination
Data quality
Data conversion
Operations
Installation
De-installation
Date and time handling
Documentation
And many others…

Structural Tests
Tests based on how the system is built
Code
Data
Design
Structural (white box) coverage can be
measured after functional and non-functional
(black box) tests are run to check for
omissions
This topic will be covered in more depth
later…

Regression and Confirmation
Regression testing checks the effects of changes, since
even small, localized, isolated changes, don’t always
have small, localized, or isolated effects
Regression strategies are covered in the next section
Confirmation testing confirms that…
Changes made to the system are present
Bug fixes introduced in the system solve the observed
symptoms
Repeatability of tests helps with regression and
confirmation testing
Automation, covered in depth later, is also helpful

Comparing Test Types
Static Structural Func./Non-func.
What Test without running Test how system works Test what system does
Why Identify bugs before
they’re built
Identify bugs in
functions or interfaces
Identify bugs in system
results and behavior
Who All stakeholders Mostly developers (and
knowledgeable testers)
Mostly testers (and
knowledgeable
developers)
When During specification
and development
Mostly during unit,
component, integration
Mostly integration,
system, and acceptance
How Analyze reqs, design,
data, code, tests, etc.
Derive tests from code,
data, design
Derive tests from reqs,
design, quality risks
Tools, data, and cases can be shared: encourage cross-pollination of techniques
We’ll discuss techniques for each type of testing in upcoming sections

ISO 9126 Quality
Characteristics/Subcharacteristics
Functionality: suitability, accuracy, interoperability,
security, compliance
Reliability: maturity (robustness), fault-tolerance,
recoverability, compliance
Usability: understandability, learnability, operability,
attractiveness, compliance
Efficiency: time behavior, resource utilization,
compliance
Maintainability: analyzability, changeability,
stability, testability, compliance
Portability: adaptability, installability, co-existence,
replaceability, compliance

Terms to Remember
Automation
Behavior-based testing
Black box or specification-based testing
Code-based, white box, or structural testing
Code coverage
Confirmation testing
Functional test
Interoperability testing
Load testing
Maintainability testing
Performance testing

Terms to Remember
Portability testing
Regression test
Reliability testing
Security testing
Stress testing
Test suite
Usability testing
Use case testing

Chapter 2:
Testing throughout
Section 3:
Exercise

Exercise: Omninet Test Types
Referring back to the levels you selected in the
previous exercise, list for each level the test targets or
types you would include in each level.
How does the lifecycle model you selected in a
previous exercise affect the amount of regression
testing?
How does the lifecycle model you selected in a
previous exercise affect the amount of confirmation
testing?
Discuss.

Chapter 2:
Testing throughout
Section 4:
Maintenance testing

Maintenance Testing
Key concepts
Reasons for maintenance testing
Maintenance testing versus new
application testing
Role of regression testing and impact
analysis
Terms to remember

Reasons for Maintenance
Three typical triggers for maintenance and
maintenance testing
Modification: enhancements, bug fixes,
operational environment changes, patches
Migration: a new supported environment
Retirement: end-of-life of a subsystem or entire
system triggers replacement
Maintenance testing addresses the change
itself--and what wasn’t changed and
shouldn’t change

Testing Maintenance Releases
Regression is the big risk for maintenance releases
Some organizations try to put a major release worth
of features into a short maintenance release
Time to develop new tests is scarce
Large project test estimation rules-of-thumb fail

Regression
Regression (misbehavior of a previously correct
function, attribute, or feature due to a change) types:
Local (fix creates new bug)
Exposed (fix reveals existing bug)
Remote (fix in one area breaks something in another area)
Regression can affect new and existing features

Regression Strategy 1: Repeat All Tests
If our tests are aligned with quality, then
repeating all tests should find most important
regressions
Automation is the only practical means for
large complex system
We’ll cover automation more thoroughly in a
later section

Regression Strategy 2: Repeat Some Tests
Often, full automation
is impossible
Select some tests to
repeat
Traceability
Change analysis
Risk analysis
Use cross-functional
tests for “accidental
regression testing”
Can use code coverage
to assess level of risk

Three Other Regression Strategies
Release more slowly
Release every six months rather than every month
Partial or even full repetition can increase
coverage on bigger releases
Combine emergency patches with slower
release process to allow for flexibility while
keeping regression risk low
Use customer or user testing
Beta for mass-market software
Pilot, staged or phase, parallel release for IT

Terms to Remember
Maintenance testing
Impact analysis
Modifications
Migration
Retirement

Chapter 2:
Testing throughout
Section 4:
Exercise

Exercise: Omninet Maintenance
Assume you can fully automate the testing of the
Omninet kiosk and call center. If a post-release
change is made to the call-center user interface that
does not affect functionality, what would you retest?
Assume you have not automated the testing of the
Omninet kiosk and call center. If the same change is
made, what would you retest? What if the change
did affect functionality?
Discuss.

Chapter 3:
Static Techniques

3. Static Techniques
1. Reviews and the test process
2. Review process
3. Static analysis by tools

Chapter 3:
Static Techniques
Section 1:
Reviews and the test process

Reviews and the Test Process
Key concepts
Software work products and static
techniques
The importance and value of static
techniques
The difference between static and dynamic
techniques
Terms to remember

Static Testing
Reviews and tools
Reviews range from informal to very formal
Tools can perform some types of static tests
Static techniques can be used for requirements and designs,
plus code, database schemas, documentation, tests…
Models and prototypes
A diagram of a complex system can often reveal design
problems that can hide in words
An ugly diagram means lots of bugs
Test cases and data
Test analysis and design based on requirements and design
specs is a form of structured review
Test analysis and design often reveals problems

Static Tools
Static analysis
Problematic wording: Spell/grammar checkers
Dangerous programming: J-Test, Safer C, lint…
Measurement: Complexity analysis
System simulations
General Purpose System Simulator
Performance modeling/operations research tools
Spreadsheets

Reviews: Costs and Benefits
Costs
Time required to perform reviews
Effort required to gather and analyze metrics
Process improvement
Benefits
Shorter schedules (due to efficient bug removal)
Shorter testing periods and lower testing costs
Developer productivity
Improved quality of product (which reduces downstream
costs)
Bottom line: Reviews of all kinds are proven, high-
return techniques for improving quality

Relating Static and Dynamic Testing
Similarities
Seek to identify defects
Work best when a
broad cross-section of
stakeholders are
involved
Save the company
money and time
Differences
Each technique can find
different types of
defects more effectively
and efficiently
Static techniques find
defects rather than
failures

Terms to Remember
Reviews
Dynamic testing
Static analysis

Chapter 3:
Static Techniques
Section 1:
Exercise

Exercise: Omninet Static Testing
Do you see reviews and static analysis as useful for
the Omninet project?
If so, what kinds of problems do you think these
reviews and static analyses would locate?
What kinds of problems might they not locate?
Discuss.

Chapter 3:
Static Techniques
Section 2:
Review process

Review Process
Key concepts
Phases, roles and responsibilities of a
typical formal review
The differences between different types of
review
The factors for successful reviews
Terms to remember

Types of Reviews
Informal: no real process (hallway chats, buddy tests,
pair programming), yet useful, cheap, popular
Peer: documented and defined defect removal
process, involving peers and technical experts but not
managers
Walkthroughs: author “walks” peers “through” the
document or code
Inspections: a trained moderator (other than the
author) leads the inspection team (with defined roles)
through a formal inspection process (rules, checklists,
entry and exit criteria), which includes gathering
defect removal metrics

Incompleteness and
ambiguity can hide the
real meaning of the
specifications
Agreement and uniform
understanding of the
specifications
Consensus and Understanding
Long before any code exists, the
specification must be handed to an
outside testing group to be scrutinized
for completeness and clarity. As [V.A.]
Vyssotsky [of Bell Lab’s Safeguard
Project] says, the developers themselves
cannot do this: “They won’t tell you they
don’t understand it; they will happily
invent their way through the gaps and
obscurities.”
– Fred Brooks
The Mythical Man-Month
1975

A Generic Review Process
1. Planning
2. Kick-off
3. Preparation
4. Review meeting
5. Rework/repair
6. Follow-up
The details of the review
process depend on the specific
review type used on the project
Includes estimating and planning,
training participants, etc.
Follow-up includes on individual
items as well as overall process
improvement analysis, evaluation of
defect (bug) removal at phase exit
reviews (exit meetings), etc.
These steps of the process repeat per
each item reviewed. Preparation is
usually one to two hours alone.
Meeting is one to two hours together.
Rework/repair is fixing the bugs
found.

Roles and Responsibilities
Moderator: Lead the review meetings
Scribe or secretary: Gather information on findings
Author: Describe, explain, answer questions on item
Reviewer/inspector: Find defects (bugs) in item
Manager: Plan, arrange resources and training,
support, analyze process metrics
In some cases, one person may play multiple roles
Authors sometimes act as moderators
One of the reviewers can act as the secretary
The specifics are determined by the type of review

Suggestions for Successful Reviews
Provide training
Review the product, not
the producer
Set and follow agenda
and objectives
Limit debate
Focusing on finding,
not fixing, problems
Take written notes
Limit and carefully
select participants
Insist on preparation
(e.g., by having people
submit notes)
Develop a checklist for
each type of item that is
reviewed
Review the reviews
Use the right techniques
Ensure management
support
Learn and get better!

Common Requirements and Design Bugs
Ambiguities: What exactly does that mean?
E.g.: System shall allow user to read ISP e-mail
What ISPs? What size e-mails? Attachments?
Incompleteness: Okay, and then what?
E.g.: Upon three invalid passwords, system shall lock user’s
account…
For how long? How to unlock? Who can unlock?
Untestability: How can I check this item?
E.g.: System shall provide 100% availability
No known test technique to demonstrate perfect availability
Excessive dependencies, coupling and complexity
Look for ugly design diagrams and confusing requirements

IEEE 1028 Standard for Software Reviews
1. Overview
Purpose, scope, conformance, organization, application
1. References
2. Definitions
3. Management reviews
Responsibilities, inputs/outputs, entry/exit criteria,
procedures
1. Technical reviews
procedures

IEEE 1028 Standard for Software Reviews
6. Inspections
procedures, data collection, process improvement
6. Walkthroughs
procedures, data collection, process improvement
6. Audits
procedures

Terms to Remember
Review process
Review meeting
Moderator/ inspection leader
Reviewer
Scribe (or secretary)
Inspection
Entry criteria
Exit criteria
Formal review

Terms to Remember
Walkthrough
Informal review
Peer or technical review
Metrics
Kick-off

Chapter 3:
Static Techniques
Section 2:
Exercise

Exercise: Omninet Requirements Review
Break into teams of three to five people.
Select a review technique from among those
discussed. If your selected technique involves specific
roles, assign the roles.
Review the Omninet Marketing Requirements
Document.
Discuss your team’s findings.

Chapter 3:
Static Techniques
Section 3:
Static analysis by tools

Static Analysis by Tools
Key concepts
The objective of static analysis versus
dynamic testing
Typical defects and errors identified by
static analysis
Typical benefits of static analysis
List typical code and design defects
identified by static analysis tools
Terms to remember

Static Analysis and Dynamic Testing
Like dynamic testing, static analysis looks for defects
in software source code and software models
Unlike dynamic testing, static analysis is performed
without actually executing the system
Static analysis involves analysis of the system or its
components by a tool, while dynamic testing does
not always involve tools
Static analysis can find defects that are hard to find or
isolate in dynamic testing
Examples include maintainability issues, unsafe pointer use
Isolation is easier because you find the bug, not the
symptom

What Can We Analyze?
Program code (e.g. control flow and data
flow)
Models of the program (e.g., simulations)
Generated output such as HTML and XML
Requirements and design documents

Benefits of Static Analysis
Early and cheaper detection of bugs (before test
execution starts)
Warnings about where bug clusters might exist, due
to dangerous programming, high complexity, etc.
Location of bugs dynamic testing might miss
Detection of dependencies and inconsistencies in
software models (e.g., such as link problems in Web
pages)
Improved maintainability of code and design
Prevention of defects based on metrics gathered and
lessons learned from analysis

Typical Static Analysis Bugs
Referencing a variable with an undefined
value
Inconsistent interface between modules and
components
Variables that are never used
Unreachable (dead) code
Programming standards violations
Security vulnerabilities
Syntax violations of code and software
models

Using Static Analysis Tools
Typical users are…
Programmers, often during component and
integration testing
Designers and system architects during design
During initial introduction against an existing
system, static analysis tools may produce a
large number of warning messages
Compilers do some static analysis, but many
sophisticated tools are available

Terms to Remember
Static analysis
Compiler
Complexity
Control flow
Data flow

Chapter 3:
Static Techniques
Section 3:
Exercise

Exercise: Omninet Static Analysis
What kind of static analysis would you suggest for
Omninet?
Would you use static analysis in areas that would be
subject to later testing?
Discuss.

Chapter 4:
Test Design Techniques

4. Test Design Techniques
1. Identifying and designing test cases
2. Categories of test design techniques
3. Specification-based or black-box
techniques
4. Structure-based or white-box
techniques
5. Experience-based techniques
6. Choosing test techniques

Chapter 4:
Section 1:
Identifying and designing test cases

Identifying and Designing Test Cases
Key concepts
• Base testing on risk analysis
• Determine level of risk with likelihood and
impact
Specify test designs, cases, procedures
Write test cases
Relate test cases and test procedures
Develop test execution schedule
Terms to remember

Phases of Test Development
Test development often proceeds in phases
(Quality risk) analysis
High-level test design
Low-level test design (implementation)
External inputs used to create internal deliverables
(testware)

Analysis

High-Level Design
These arrows show the
relationships between
the risk categories and
the test suites. We’ll
capture this relationship
information in the form
of what’s called
traceability.

Low-Level Design (Implementation)
Each risk to be mitigated
via testing will have one or
more test cases associated
with it. (I’ve not shown all
traces, to avoid clutter.)

What’s a Quality or Product Risk?
Risk
The possibility of a negative or undesirable
outcome
The likelihood of a risk becoming an outcome is…
>0, <1 in the future…
0 or 1 in the past
Quality or product risk
The possibility that the system will fail to satisfy
customers, users, or other stakeholders
A family of possible bugs are behind quality risks

How Can We Analyze Quality Risks?
Informal ISO 9126
Failure Mode and Effect
Analysis
Start with the classic
quality risk categories
Start with six main
quality characteristics
Start with categories, char-
acteristics, or subsystems
Functionality, states and
transactions, capacity and
volume, data quality, error
handling and recovery,
performance, standards and
localization, usability, etc.
Functionality, Reliability,
Usability, Efficiency,
Maintainability, Portability
(FRUEMP), then decompose
into key subcharacteristics
for your system
Key stakeholders list possible
failure modes, predict their
effects on system, user, society,
etc., assign severity, priority,
and likelihood, then calculate
risk priority number (RPN)
Set priority for testing
each quality risk with key
stakeholders
Set priority for testing
each subcharacteristic
with key stakeholders
Stakeholders use RPN to
guide appropriate depth
and breadth for testing
Regardless of technique the keys are cross-functional stakeholder
participation, consensus, and a best-possible-outcome outlook

Quality Risk
Tech.
Risk
Bus.
Risk
Risk
Pri. #
Extent of
Testing Tracing
Risk Category 1
Risk 1
Risk 2
Risk n
Quality risks are potential system problems which could reduce user satisfaction
A hierarchy of
risk categories
can help
organize the
list and jog
your memory.
1 = Very high
2 = High
3 = Medium
4 = Low
5 = Very low
Technical risk: Likelihood of the problem
Business (operational) risk: Impact of the problem
1-5 = Extensive
6-10 = Broad
11-15 = Cursory
16-20 = Opportunity
21-25 = Report bugs
Risk priority number: Aggregate measure of problem risk
The product of
technical and
business risk,
from 1-25.
Tracing
information back to
requirements,
design, or other
risk bases

Tips for Risk Analysis
Use a cross-functional brainstorming team
Identify the risk items, then assign the level of risk
Only separate risk items when necessary to
distinguish between different levels of risk
Consider technical and business risk
Technical risk: likelihood of problem, impact of problem on
system
Business risk: likelihood of usage, impact of problem on
users
Follow up and re-align risk analysis, testing, and the
project at key project milestones

Chapter 4:
Section 1:
Exercise 1

Exercise: Omninet Risk Analysis
Based on your reading of the Omninet Marketing
Requirements Document, the Omninet System
Requirements Document, and your experience with
testing and bugs, perform a risk analysis for Omninet
Discuss.

IEEE 829 Test Design Specification
The test design specification describes a collection of
test cases at a high level, and includes the following
sections
Test design specification identifier
Features to be tested (in this test suite)
Approach refinements (specific techniques, tools, etc.)
Test identification (tracing to test cases in suite)
Feature pass/fail criteria (e.g., test oracle, test basis, legacy
systems, etc.)
This collection of test cases often called a test suite
Sequencing (test suites and cases within suites) often
driven by risk and business priority and affected by
project constraints, resources, and progress

IEEE 829 Test Case Specification
A test case specification describes the details of a test
case, and includes the following sections
Test case specification identifier
Test items (what is to be delivered and tested)
Input specifications (user inputs, files, etc.)
Output specifications (expected results, including screens,
files, timing, etc.)
Environmental needs (hardware, software, people, props…)
Special procedural requirements (operator intervention,
permissions, etc.)
Intercase dependencies (if needed to set up preconditions)
In practice, test cases vary significantly in effort,
duration, and number of test conditions covered

IEEE 829 Test Procedure Specification
A test procedure specification describes how to run
one or more test cases, and includes the following
sections
Test procedure specification identifier
Purpose (e.g., which tests are run)
Special requirements (skills, permissions, environment, etc.)
Procedure steps (logging, set up, start, proceed [steps
themselves], measurement of results,
shutdown/suspension, restart [if needed], stop, wrap
up/tear down, contingencies)
Test procedures are often embedded in test cases
A test procedure is sometimes referred to as a test
script, and may be manual or automated

Test Case Coverage (Traceability)
Measure or correlate tests
against areas of concern
Used as a way to measure
and enhance the tests
Practical types
Requirements specifications
Design specifications
Functional areas
Quality risks
Configurations
This is a commonly-used
technique to ensure
thorough test coverage
One technique
Use spreadsheet
List test cases and coverage
area to measure
0: none; 1: indirect; 2: direct
Test Case
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
2.1
2.2
2.3
2.4
2.5
Total
Spec
1.1 1 1 1 1 2 6
1.2 2 2 1 1 2 8
1.3 0
1.4 2 2 4
1.5 1 1 1 1 4
1.6 2 2
1.7 2 1 2 2 7
1.8 2 1 3
1.9 1 1 2 4
1.10 1 1 1 1 4
Total 5 4 2 5 4 3 3 4 2 2 7 1 0

Terms to Remember
Priority
Risk
Risk identification
Risk analysis
Risk control
Risk-based testing
Product risk
Test case
Test condition
Test data

Terms to Remember
Test case specification
Test design specification
Test procedure specification
Test script
Traceability

Chapter 4:
Section 1:
Exercise 2

Exercise: Omninet Test Design Specification
Based on your quality risk analysis for Omninet,
outline a set of test suites to address the important
areas of risk.
For each test suite, list briefly:
What the test would cover
How you would recognize passing or failing tests
Establish the relationship between your test suites
and the risks they will cover.
Based on your risk analysis, how would you
sequence the test suites? What other considerations
would affect the sequencing of test suites?
Discuss.

Chapter 4:
Section 2:
Categories of test design techniques

Categories of Test Design Techniques
Key concepts
Reasons for specification-based (black box),
structure-based (white box), and
experience-based tests
Common black box and white box
techniques
Characteristics and differences between
specification-based testing, structure-based
testing, and experience-based testing
Terms to remember

Three Types of Test Design Techniques
Specification-based (black box), functional and non-
functional
Create tests primarily by analysis of the test basis
Look for bugs in the way the system behaves
Structure-based (white box)
Create tests primarily by analysis of the structure of the
component or system
Look for bugs in the way the system is built
Experience-based (attacks, checklists, exploratory)
Create tests primarily based on understanding of the
system, past experience, and educated guesses about bugs
Look for bugs in the places other systems have bugs

Specification-Based or Black Box
Common elements include:
Formal or informal models used to specify the
problem to be solved, the software or its
components
Test cases derived systematically from these
models
Examples include:
Equivalence partitioning and boundary value
analysis
State transition diagrams
Decision tables

Structure-Based or White Box
System structure (e.g., code, design, etc.) used to
derive the test cases, for example code and design
The extent of structural coverage can be measured
for existing other test cases
Further test cases can be derived systematically to
increase coverage
Examples include:
Statement coverage
Branch coverage

Experience-Based
The knowledge and experience used to derive test
cases
Can consider knowledge and experience of the
software, its usage and its environment or…
…knowledge about historical and likely defects
and their distribution
Examples include:
Attacks
Checklists
Exploratory

Terms to Remember
Black box or specification-based techniques
Experience-based techniques
White box or structure-based techniques

Chapter 4:
Section 2:
Exercise

Exercise: Omninet Test Techniques
Refer to your outline of test suites for Omninet.
For each test suite, identify whether one, two, or all
three of the following categories of test techniques
would be useful in designing test cases:
Specification-based (black box)
Structure-based (white box)
Experience-based
Discuss.

Chapter 4:
Section 3:
Specification-based or black box
techniques

Specification-based Techniques
Key concepts
Writing test cases from given software
models using equivalence partitioning,
boundary value analysis, decision tables,
and state transition diagrams
The main purpose of each technique and
how coverage may be measured
Use case testing
Terms to remember

Equivalence Partitioning
Divide the inputs,
outputs, behaviors, and
environments into
classes you think will be
handled equivalently
Define at least one test
case in each partition, or
use boundary values in
partitions that are
ranges
Can use marketing info
to favor classes
Ex: Testing supported
printers
Physical interface:
Parallel, serial, USB 1.1,
USB 2.0, infrared,
Firewire, SCSI, others?
Logical interface:
Postscript, HPPL, ASCII,
others…
Image application: Laser
jet, ink jets, bubble jets,
dot matrix, line printers,
letter quality, pen
plotters, others…

Boundary Value Analysis
A refinement of equivalence partitioning that selects
the edges or end-points of each partition for testing
Equivalence partitioning looks for bugs in the code that
handles each equivalent class
Boundary values also look for bugs in the definitions of the
edges
Can only be used when the elements of the
equivalence partition are ordered
Non-functional boundaries (capacity, volume, etc.)
can be used for non-functional testing, too

Integer
“How many items would you like to order?”

Test engineering foundation (v2.01)

Test engineering foundation (v2.01)

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie Test engineering foundation (v2.01)

Ähnlich wie Test engineering foundation (v2.01) (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Test engineering foundation (v2.01)