5. Software Testing: definition [SWEBOK v3]
“So>ware
tes:ng
consists
of
the
dynamic
verifica:on
that
a
program
provides
expected
behaviours
on
a
finite
set
of
test
cases,
suitably
selected
from
the
usually
infinite
execu:on
domain.”
2015
-‐
2016
xavier.devroey@unamur.be
5
6. Testing
2015
-‐
2016
xavier.devroey@unamur.be
6
Req.
Test
engineer
Test
case
defines
Pgm.
executed
on
7. Dynamic verification
• Tes:ng
implies
execu6ng
the
program
under
test
• Contrary
to
sta6c
analysis
– Formal,
semiformal,
or
informal
– Examples:
• compiler
warnings
(code
analysis),
• javadoc
warnings
(documenta:on
analysis),
• design
inspec:on
(model
analysis),
• theorem
proving
(e.g.,
Event-‐B),
• code
analysis
– Code
review
– IDE
warnings
– FindBugs
h`p://findbugs.sourceforge.net
2015
-‐
2016
xavier.devroey@unamur.be
7
8. Expected behaviours
• It
must
be
possible
to
decide
whether
the
observed
outcomes
for
a
given
input
is
acceptable
or
not
• The
oracle
problem
– “An
oracle
is
any
human
or
mechanical
agent
that
decides
whether
a
program
behaved
correctly
in
a
given
test
and
accordingly
results
in
a
verdict
of
“pass”
or
“fail”.”
[SWEBOK
v3]
2015
-‐
2016
xavier.devroey@unamur.be
8
9. Finite set of test cases
• Exhaus:ve
tes:ng
is
not
possible
in
prac:ce
– Infinite
number
of
test
cases
– Select
a
subset
of
all
possible
tests
– Trade-‐off
between
limited
resources,
schedules
and
unlimited
test
requirements
• Priori:za:on
based
on
risk,
usage,
cost,
…
“Program
tes6ng
can
be
used
to
show
the
presence
of
bugs,
but
never
to
show
their
absence!”
Dijkstra
(1970)
2015
-‐
2016
xavier.devroey@unamur.be
9
10. Suitably selected
• How
to
select
test-‐cases
?
• Selec:on
criteria
– Input/output
domains
coverage
(black
box
tes:ng)
• Domain
par::oning
techniques
• Based
on
specifica:on
documents
• No
access
to
the
source
code
– Code
coverage:
statement
coverage,
decision
coverage,
path
coverage,
etc.
(white
box
tes:ng)
• Access
to
source
code
– Test
engineer
exper:se
2015
-‐
2016
xavier.devroey@unamur.be
10
12. Types of testing
• Source
of
test
genera:on
– Black
box
tes:ng
• Based
on
specifica:on
documents
• No
access
to
the
source
code
– White
box
tes:ng
• Access
to
the
source
code
– Model-‐based
tes:ng
• Characteris:c
being
tested
– Func:onal
tes:ng:
is
the
output
correct
for
a
given
input
?
– Non-‐func:onal
tes:ng
• Performance,
robustness,
security,
usability,
etc.
2015
-‐
2016
xavier.devroey@unamur.be
12
14. Testing in the waterfall model
2015
-‐
2016
xavier.devroey@unamur.be
14
Requirements
specification
Design
Coding and
unit testing
Integration testing
System testing
Acceptance
testing
Release
Maintenance and
regression testing
15. Testing in the V-model
2015
-‐
2016
xavier.devroey@unamur.be
15
Requirements
specification
Design
Coding and
unit testing
Integration/
System
testing
Acceptance
testing
Release
Integration/
System
test plan
Acceptance
test plan
Maintenance and
regression testing
16. Testing in an Agile process
1. Include
tes:ng
ac:vi:es
from
the
beginning
2. Specify
requirements
in
terms
of
tests
3. Testers
and
developers
are
not
enemies
4. Test
o>en
and
in
small
chunks
2015
-‐
2016
xavier.devroey@unamur.be
16
17. Test Driven Development (TDD)
• Red
– Write
automated
tests
(examples)
from
user
stories
– Tests
fail
(as
the
func:onality
is
not
implemented
yet)
• Green
– Developers
design
and
write
code
to
make
the
tests
pass
– Tests
are
run
frequently
• Refactor
– Refactor
code
without
changing
interfaces
or
behaviour
(tests
must
pass)
Red
Green
Refactor
2015
-‐
2016
xavier.devroey@unamur.be
17
19. Agile process implementation: TSSG
• From
Dowling,
P.,
&
McGrath,
K.
(2015).
Using
free
and
open
source
tools
to
manage
soGware
quality.
Communica:ons
of
the
ACM,
58(7),
51–55.
doi:
10.1145/2755503
• Open
Source
tools
• Support
team’s
ac:vi:es
management
2015
-‐
2016
xavier.devroey@unamur.be
19
20. 2015
-‐
2016
xavier.devroey@unamur.be
20
Development environment
Project Management
Requirements Management
Issues Tracking
Building
Code
Repository
Unit testing
Code
coverage
Cobertura
Continuous Integration
Test environment
Functional
Testing
Test Case
Management
Load Testing
Security
Testing
21. 2015
-‐
2016
xavier.devroey@unamur.be
21
Development environment
Project Management
Requirements Management
Issues Tracking
Building
Code
Repository
Unit testing
Code
coverage
Cobertura
Continuous Integration
Test environment
Functional
Testing
Test Case
Management
Load Testing
Security
Testing
22. 2015
-‐
2016
xavier.devroey@unamur.be
22
Development environment
Project Management
Requirements Management
Issues Tracking
Building
Code
Repository
Unit testing
Code
coverage
Cobertura
Continuous Integration
Test environment
Functional
Testing
Test Case
Management
Load Testing
Security
Testing
23. Workflow
• Developer
forks
the
code
base
• Implements
changes
• Integrates
possible
changes
from
code
repo.
• Runs
a
complete
build
– Including
Unit
tests
• Submits
changes
to
code
repo.
• CI
server
performs
integra:on
tests
2015
-‐
2016
xavier.devroey@unamur.be
23
h`ps://www.flickr.com/photos/b3c>/6713516627/
Do
NOT
break
the
build!
24. 2015
-‐
2016
xavier.devroey@unamur.be
24
Development environment
Project Management
Requirements Management
Issues Tracking
Building
Code
Repository
Unit testing
Code
coverage
Cobertura
Continuous Integration
Test environment
Functional
Testing
Test Case
Management
Load Testing
Security
Testing
25. 2015
-‐
2016
xavier.devroey@unamur.be
25
Development environment
Project Management
Requirements Management
Issues Tracking
Building
Code
Repository
Unit testing
Code
coverage
Cobertura
Continuous Integration
Test environment
Functional
Testing
Test Case
Management
Load Testing
Security
Testing
Hudson
26. 2015
-‐
2016
xavier.devroey@unamur.be
26
Development environment
Project Management
Requirements Management
Issues Tracking
Building
Code
Repository
Unit testing
Code
coverage
Cobertura
Continuous Integration
Test environment
Functional
Testing
Test Case
Management
Load Testing
Security
Testing
Production
Deployment
32. Maven project structure
• myfirstmavenproject/
– pom.xml
– src/
• main/
– java/
» Here
goes
your
Java
source
code
– resources/
» Here
goes
other
resources
(files,
etc.)
included
in
the
.jar/.war
• test/
– java/
» Here
goes
the
JUnit
tests
– resources/
» Here
goes
other
resources
used
by
the
tests
– sub-‐module-‐1/
• pom.xml
• src/
– sub-‐module-‐2/
• …
2015
-‐
2016
xavier.devroey@unamur.be
32
34. Pom.xml
2015
-‐
2016
xavier.devroey@unamur.be
34
Iden9fier
• Maven
iden:fiers
have
to
be
unique
• Format:
groupId:ar6factId:version
• GroupId
should
iden:fy
the
organisa:on
using
Java
package
naming
conven:on
• Ar:factId
should
iden:fy
the
project
• Version
should
be
encoded
on
3
number
XX.XX.XX
• “-‐SNAPSHOT”
is
used
to
indicate
the
currently
under
development
version
35. Pom.xml
2015
-‐
2016
xavier.devroey@unamur.be
35
Type
of
the
artefact
• Packaging
indicates
to
Maven
the
type
of
ar:fact
produced
during
build
• “jar”
indicates
a
.jar
file
(executable
or
not)
• “pom”
indicates
a
maven
project
without
compiled
code
(used
by
a
parent
project)
• “war”
indicates
a
.war
file
that
will
be
deployed
on
a
web
server
36. Pom.xml
2015
-‐
2016
xavier.devroey@unamur.be
36
Proper9es
used
by
project
and
sub-‐projects
• Proper:es
are
used
as
global
constants
• Implicit
proper:es
• ${basedir}
current
project
root
directory
• ${project.version}
current
project
version
• ${project.groupId}
current
project
groupId
• Declared
proper:es
• Should
contain
dependencies
versions
37. Sub-modules
• Parent
project
(should
have
pom
packaging)
• Sub-‐module
has
to
declare
parent
project
• In
this
case:
sub-‐module-‐1
version
is
inherited
2015
-‐
2016
xavier.devroey@unamur.be
37
<project
…>
…
<packaging>pom</packaging>
<modules>
<module>sub-‐module-‐1</module>
</modules>
</project>
<project
…>
<parent>
<groupId>be.unamur.info</groupId>
<artifactId>myfirstmavenproject</artifactId>
<version>0.0.1-‐SNAPSHOT</version>
</parent>
<artifactId>sub-‐module-‐1</artifactId>
…
</project>
38. Dependencies
• Maven
central
repository
– h`p://repo.maven.apache.org/maven2
– h`p://search.maven.org
(to
search
an
ar:fact)
• Declare
dependcy
in
a
Maven
project
2015
-‐
2016
xavier.devroey@unamur.be
38
<project
…>
…
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
39. mvn <phase>
• compile
– Compile
source
files
• test
– Launch
JUnit
tests
in
src/test/java/
• package
– Create
.jar
or
.war
or
…
file
in
target/
• install
– Install
artefact
in
local
Maven
repository
• deploy
– Deploy
artefact
using
deployment
configura:on
2015
-‐
2016
xavier.devroey@unamur.be
39
41. JUnit: Introduction
• Collec:on
of
classes
to
perform
unit
tes:ng
• Uses
annota:ons
(e.g.,
@Test)
2015
-‐
2016
xavier.devroey@unamur.be
41
import
static
org.junit.Assert.assertEquals;
import
org.junit.Test;
public
class
CalculatorTest
{
@Test
public
void
evaluatesExpression()
{
Calculator
calculator
=
new
Calculator();
int
sum
=
calculator.evaluate("1+2+3");
assertEquals(6,
sum);
}
}
java
-‐cp
.;junit-‐4.12.jar;hamcrest-‐core-‐1.3.jar
org.junit.runner.JUnitCore
CalculatorTest
42. Anatomy of a Junit test class
import
…
public
class
MyClassTest{
private
sta:c
final
Logger
logger
=
LoggerFactory.getLogger(JHConwayLifeCellTest.class);
@Rule
public
TestRule
watcher
=
new
TestWatcher()
{
@Override
protected
void
star:ng(Descrip:on
descrip:on)
{
logger.info(String.format("Star:ng
test:
%s()...”,
descrip:on.getMethodName()));};};
@BeforeClass
public
sta:c
void
setUpClass()
{logger.info("Se|ng
up
before
class");}
@A>erClass
public
sta:c
void
tearDownClass()
{logger.info("Tearing
down
a>er
class");}
@Before
public
void
setUp()
{logger.info("Se|ng
up
before
test");}
@A>er
public
void
tearDown()
{logger.info("Tearing
down
a>er
test");}
@Test
public
void
testMyMethod()
{assertTrue(true);}
}
2015
-‐
2016
xavier.devroey@unamur.be
42
44. Hamcrest common matchers
• Core
– anything
-‐
always
matches,
useful
if
you
don't
care
what
the
object
under
test
is
• Logical
– allOf
-‐
matches
if
all
matchers
match,
short
circuits
(like
Java
&&)
– anyOf
-‐
matches
if
any
matchers
match,
short
circuits
(like
Java
||)
– not
-‐
matches
if
the
wrapped
matcher
doesn't
match
and
vice
versa
• Object
– equalTo
-‐
test
object
equality
using
Object.equals
– hasToString
-‐
test
Object.toString
– instanceOf,
isCompa:bleType
-‐
test
type
– notNullValue,
nullValue
-‐
test
for
null
– sameInstance
-‐
test
object
iden:ty
• Collec:ons
– hasEntry,
hasKey,
hasValue
-‐
test
a
map
contains
an
entry,
key
or
value
– hasItem,
hasItems
-‐
test
a
collec:on
contains
elements
– hasItemInArray
-‐
test
an
array
contains
an
element
• Number
– closeTo
-‐
test
floa:ng
point
values
are
close
to
a
given
value
– greaterThan,
greaterThanOrEqualTo,
lessThan,
lessThanOrEqualTo
-‐
test
ordering
• Text
– equalToIgnoringCase
-‐
test
string
equality
ignoring
case
– equalToIgnoringWhiteSpace
-‐
test
string
equality
ignoring
differences
in
runs
of
whitespace
– containsString,
endsWith,
startsWith
-‐
test
string
matching
2015
-‐
2016
xavier.devroey@unamur.be
44
45. Hands-on: Maven, JUnit and Hamcrest
• be.unamur.info:mavencrashcourse-‐cellularautomata
– From
“Travaux
pra:ques
de
concep:on
et
programma:on
orientée
objet
:
Automate
cellulaire”
G.
Saval,
2008
• Classes
to
test
– JHConwayLifeCell
– Grid
– CellGrid
• Test
coverage
using
Cobertura
– mvn
cobertura:cobertura
• Code
analysis
using
FindBugs
– mvn
findbugs:findbugs
– mvn
findbugs:gui
2015
-‐
2016
xavier.devroey@unamur.be
45
49. Selenium Commands
• open
– opens
a
page
using
a
URL
• click/clickAndWait
– performs
a
click
opera:on,
and
op:onally
waits
for
a
new
page
to
load
• verifyTitle/assertTitle
– verifies
an
expected
page
:tle
• verifyTextPresent
– verifies
expected
text
is
somewhere
on
the
page
• verifyElementPresent
– verifies
an
expected
UI
element,
as
defined
by
its
HTML
tag,
is
present
on
the
page
• verifyText
– verifies
expected
text
and
its
corresponding
HTML
tag
are
present
on
the
page
• verifyTable
– verifies
a
table’s
expected
contents
• waitForPageToLoad
– pauses
execu:on
un:l
an
expected
new
page
loads.
Called
automa:cally
when
clickAndWait
is
used
• waitForElementPresent
– pauses
execu:on
un:l
an
expected
UI
element,
as
defined
by
its
HTML
tag,
is
present
on
the
page
2015
-‐
2016
xavier.devroey@unamur.be
49
50. Hands-on: Selenium IDE
• Let’s
Chat
applica:on
– h`ps://sdelements.github.io/lets-‐chat/
• Test
server
– h`p://rollier.info.fundp.ac.be:5000
• Test
Cases
– Create
chat
room,
post
some
message,
exit
room
– Edit
profile,
change
display
name,
first
name
and
last
name
– Create/Enter
room,
edit
room
name
and
descrip:on
2015
-‐
2016
xavier.devroey@unamur.be
50
51. References
• Mathur,
A.
P.
(2008).
Founda6ons
of
soGware
tes6ng.
Pearson
Educa:on.
• U|ng,
M.,
&
Legeard,
B.
(2007).
Prac6cal
Model-‐Based
Tes6ng:
A
Tools
Approach.
Morgan
Kaufmann.
• Bourque,
P.,
&
Fairley,
R.
E.
(2014).
Guide
to
the
SoGware
Engineering
Body
of
Knowledge
(SWEBOK(R)):
Version
3.0.
IEEE
Computer
Society.
• Dowling,
P.,
&
McGrath,
K.
(2015).
Using
free
and
open
source
tools
to
manage
soGware
quality.
Communica:ons
of
the
ACM,
58(7),
51–55.
doi:
10.1145/2755503
• Tim
O'Brien
et
al.
Maven
by
Example.
Sonatype,
h`p://www.sonatype.com/books/mvnex-‐book/reference/public-‐book.html
• h`p://junit.org
• h`p://hamcrest.org
• h`p://www.seleniumhq.org
2015
-‐
2016
xavier.devroey@unamur.be
51
Images
credits:
h`p://hedbonstudios.deviantart.com;
h`p://commons.wikimedia.org/;
h`p://wh1pla5h.deviantart.com;
h`p://wh1pla5h.deviantart.com;
h`p://
darkangelaw1986.deviantart.com;
h`p://phantomcameron.deviantart.com;
h`p://newyorkkid618.deviantart.com;
h`p://doloresmine`e.deviantart.com;
h`ps://
www.flickr.com/photos/tbuser
;
h`p://fullmetaldevil.deviantart.com/
;
h`ps://www.flickr.com/photos/fastjack/
;
h`ps://www.flickr.com/photos/eli:stczar/