Using pairwise test case design to reduce the test cases needed for maximum test coverage.

1. Pairwise
Tes*ng
Strategic
Test
Case
Design
www.xboso6.com

2. • XBOSo&
– Quality
and
Tes4ng
Process
Improvement
and
So&ware
Tes4ng
Experts
– San
Francisco,
Beijing,
Oslo,
Amsterdam

3. Pairwise
Tes*ng
The
Problem:
• Many
defects
only
occur
when
a
combina4on
of
inputs
or
events
occur
that
interact
with
each
other.
• But
to
test
every
combina4on
or
permuta4on
is
impossible.
The
Solu4on:
• Pairwise
tes4ng
normally
begins
by
selec4ng
values
for
the
system’s
input
variables.
• These
individual
values
are
o&en
selected
using
domain
equivalence
par44oning.
• The
values
are
then
permuted
to
achieve
coverage
of
all
the
pairings.
3

4. Pairwise
Tes*ng
• The
reasoning
behind
all-­‐pairs
tes4ng:
the
simplest
bugs
in
a
program
are
generally
triggered
by
a
single
input
parameter.
• The
next
simplest
category
of
bugs
consists
of
those
dependent
on
interac4ons
between
pairs
of
parameters,
which
can
be
caught
with
all-­‐pairs
tes4ng.
• Bugs
involving
interac4ons
between
three
or
more
parameters
are
progressively
less
common,
while
at
the
same
4me
being
progressively
more
expensive
to
find
by
exhaus4ve
tes4ng,
which
has
as
its
limit
the
exhaus4ve
tes4ng
of
all
possible
inputs.
4

5. Pairwise
Tes*ng
Problem:
Test
System
S
Inputs
X
=
A,B
Y
=
C,D
Z
=
E,F
There
are
23
=
8
possible
combina4ons
5
System S
X Y Z

6. Pairwise
Tes*ng
6
8 Test Cases
You can get rid of Test Case 2 (T2)
• {A, -, E} is covered in T1
• { , -} is covered in T4
• { -, } is covered in T6
Problem:
• How many test cases can you get rid of?
• List the Test cases you would keep.
• Time yourself.
Test
ID
X
Y
Z
T1
A
C
E
T2
A
D
E
T3
A
C
F
T4
A
D
F
T5
B
C
E
T6
B
D
E
T7
B
C
F
T8
B
D
F
A, D
D, E

7. Pairwise
Tes*ng
Test
ID
X
Y
Z
T1
A
C
E
T2
A
D
E
T3
A
C
F
T4
A
D
F
T5
B
C
E
T6
B
D
E
T7
B
C
F
T8
B
D
F
7
Test
ID
X
Y
Z
T1
A
C
E
T4
A
D
F
T6
B
D
E
T7
B
C
F
8 Test Cases Pairwise testing
4 Test Cases

8. Example
8
How many combinations are there for twelve inputs?
- Hexawise.com

9. Pairwise
Tes*ng
9
- Hexawise.com

10. Pairwise
Tes*ng
10
- Hexawise.com

11. Pairwise
Tes*ng
11

12. Pairwise
Tes*ng
Tools
To
generate
the
20
test
cases
in
the
previous
slide
from
the
possible
3.7M
test
cases
would
be
extremely
4me
consuming.
Just
a
few
of
the
many
SW
tools
to
generate
the
tes4ng
pairs.
• Hexawise
• AllPairs
by
James
Bach,
free
• PICT
(Pairwise
Independent
Combinatorial
Tes4ng)
tool
• Automa4c
Efficient
Test
Generator
or
AETG™
12

13. Limita*ons
of
Pairwise
Tes*ng
Common
tes*ng
problems
are
made
worse:
• Selec*ng
the
wrong
input:
The
problems
of
equivalence
par44oning
are
made
worse
with
a
pair
of
wrong
inputs.
• The
‘Oracle
Problem’:
Is
a
defect
properly
revealed
and
do
we
recognized
it
when
it
is
revealed?
• The
input
values
to
the
system
under
test
are
not
discrete.
Problems
of
Pairwise
tes*ng:
• High
probability
combina4ons
get
too
lihle
ahen4on.
• Not
knowing
the
interdependencies
between
variables.
13

14. Pairwise
Tes*ng
Works
well
when:
• Number
of
possible
parameter
values
is
large
• You
can
effec4vely
separate
parameter
values
into
equivalence
classes.
14

15. Pairwise
Tes*ng:
Cau*on
• Pairwise
tes4ng
is
a
tool,
it
cannot
replace
a
skilled
experienced
tester.
• As
a
tool,
it
is
effec4ve
only
as
much
as
the
tester
is
effec4ve.
• Responsibility
for
good
tes4ng
remains
with
the
tester,
not
the
tool.
15

16. Pairwise
Tes*ng
Conclusion:
• When
used
properly,
pairwise
test
set
genera4on
is
an
important
technique
that
can
help
you
produce
beher
so&ware
systems.
16