1. www.unamur.be
Software Testing: an Introduction
Xavier Devroey <xavier.devroey@unamur.be>
INFO M110 IngĂŠnierie du logiciel
Namur, Belgique
2. Plan
⢠Introduction
⢠Software testing typology
â Source of test generation
â Characteristic being tested
â Lifecycle phase
⢠Agile process implementation: the TSSG case study
⢠Unit testing using Maven and JUnit
⢠Code review
⢠Test-case assessment
21/10/2016 xavier.devroey@unamur.be 2
6. Error, fault, failure, and incident
Error
Fault Exec.
Failure (bug)
Reports
Incident
Bug Tracker
21/10/2016 xavier.devroey@unamur.be 6
Dev.
User
7. Software testing: definition [SWEBOK v3]
âSoftware testing consists of the dynamic verification
that a program provides expected behaviours
on a finite set of test cases, suitably selected from the
usually infinite execution domain.â
21/10/2016 xavier.devroey@unamur.be 7
12. Finite set of test cases
⢠Exhaustive testing is not possible in practice
â Infinite number of test cases
â Select a subset of all possible tests
â Trade-off between limited resources, schedules and unlimited
test requirements
⢠Prioritization based on risk, usage, cost, âŚ
âProgram testing can be used to show the presence of bugs,
but never to show their absenceâ Dijkstra (1970)
21/10/2016 xavier.devroey@unamur.be 12
13. Suitably selected
⢠How to select test-cases ?
⢠Selection criteria
â Input/output domains coverage (black box testing)
⢠Domain partitioning techniques
⢠Based on specification documents
⢠No access to the source code
â Code coverage: statement coverage, decision coverage, path
coverage, etc. (white box testing)
⢠Access to source code
â Test engineer expertise
21/10/2016 xavier.devroey@unamur.be 13
15. Types of testing
⢠Source of test generation
â Black box testing
⢠Based on specification documents
⢠No access to the source code
â White box testing
⢠Access to the source code
â Model-based testing
⢠Characteristic being tested
â Functional testing: is the output correct for a given input ?
â Non-functional testing: Performance, robustness, security,
usability, etc.
21/10/2016 xavier.devroey@unamur.be 15
16. Types of testing
⢠Lifecycle phase
â Coding
⢠Unit testing
⢠Component testing
â Integration
⢠Integration testing
â System integration
⢠System testing
â Release
⢠Acceptance/Beta testing
â Maintenance
⢠Regression testing
21/10/2016 xavier.devroey@unamur.be 16
17. Testing in the waterfall model
21/10/2016 xavier.devroey@unamur.be 17
Requirements
speciďŹcation
Design
Coding and
unit testing
Integration testing
System testing
Acceptance
testing
Release
Maintenance and
regression testing
18. Testing in the V-model
21/10/2016 xavier.devroey@unamur.be 18
Requirements
speciďŹcation
Design
Coding and
unit testing
Integration/
System
testing
Acceptance
testing
Release
Integration/
System
test plan
Acceptance
test plan
Maintenance and
regression testing
19. Testing in an Agile process
1. Include testing activities from the beginning
2. Specify requirements in terms of tests
3. Testers and developers are not enemies
4. Test often and in small chunks
21/10/2016 xavier.devroey@unamur.be 19
20. Test Driven Development (TDD)
⢠Red
â Write automated tests
(examples) from user stories
â Tests fail (as the functionality
is not implemented yet)
⢠Green
â Developers design and write
code to make the tests pass
â Tests are run frequently
⢠Refactor
â Refactor code without
changing interfaces or
behaviour (tests must pass)
Red
GreenRefactor
21/10/2016 xavier.devroey@unamur.be 20
22. Agile process implementation: TSSG
⢠From Dowling, P., & McGrath, K. (2015). Using free and
open source tools to manage software quality.
Communications of the ACM, 58(7), 51â55.
doi:10.1145/2755503
⢠Open Source tools
⢠Support teamâs activities management
⢠Language: Java
21/10/2016 xavier.devroey@unamur.be 22
24. 21/10/2016 xavier.devroey@unamur.be 24
Project Management
Requirements Management
Issues Tracking
Development environment
Building
Unit testing
Code
coverage
Cobertura
Continuous Integration
Code
Repository
25. 21/10/2016 xavier.devroey@unamur.be 25
Project Management
Requirements Management
Issues Tracking
Development environment
Building
Unit testing
Code
coverage
Cobertura
Continuous Integration
Code
Repository
Test environment
Functional
Testing
Test Case
Management
Load Testing
Security
Testing
26. Workflow
⢠Developer forks the code
base
⢠Implements changes
⢠Integrates possible
changes from code repo.
⢠Runs a complete build
â Including Unit tests
⢠Submits changes to code
repo.
⢠CI server performs
integration tests
21/10/2016 xavier.devroey@unamur.be 26
Do NOT break the build!
27. 21/10/2016 xavier.devroey@unamur.be 27
Project Management
Requirements Management
Issues Tracking
Development environment
Building
Unit testing
Code
coverage
Cobertura
Continuous Integration
Code
Repository
Test environment
Functional
Testing
Test Case
Management
Load Testing
Security
Testing
28. 21/10/2016 xavier.devroey@unamur.be 28
Project Management
Requirements Management
Issues Tracking
Development environment
Building
Unit testing
Code
coverage
Cobertura
Continuous Integration
Code
Repository
Test environment
Functional
Testing
Test Case
Management
Load Testing
Security
Testing
29. 21/10/2016 xavier.devroey@unamur.be 29
Project Management
Requirements Management
Issues Tracking
Development environment
Building
Unit testing
Code
coverage
Cobertura
Continuous Integration
Code
Repository
Test environment
Functional
Testing
Test Case
Management
Load Testing
Security
Testing
30. 21/10/2016 xavier.devroey@unamur.be 30
Project Management
Requirements Management
Issues Tracking
Development environment
Building
Unit testing
Code
coverage
Cobertura
Continuous Integration
Code
Repository
Test environment
Functional
Testing
Test Case
Management
Load Testing
Security
Testing
36. Maven project structure
⢠myfirstmavenproject/
â pom.xml
â src/
⢠main/
â java/
Âť Here goes your Java source code
â resources/
Âť Here goes other resources (files, etc.) included in the .jar/.war
⢠test/
â java/
Âť Here goes the JUnit tests
â resources/
Âť Here goes other resources used by the tests
â sub-module-1/
⢠pom.xml
⢠src/
â sub-module-2/
⢠âŚ
21/10/2016 xavier.devroey@unamur.be 36
38. Pom.xml
21/10/2016 xavier.devroey@unamur.be 38
Identifier
⢠Maven identifiers have to be unique
⢠Format: groupId:artifactId:version
⢠GroupId should identify the organisation using Java package naming
convention
⢠ArtifactId should identify the project
⢠Version should be encoded on 3 number XX.XX.XX
⢠â-SNAPSHOTâ is used to indicate the currently under
development version
39. Pom.xml
21/10/2016 xavier.devroey@unamur.be 39
Type of the artefact
⢠Packaging indicates to Maven the type of artifact produced during
build
⢠âjarâ indicates a .jar file (executable or not)
⢠âpomâ indicates a maven project without compiled code (used by a
parent project)
⢠âwarâ indicates a .war file that will be deployed on a web server
45. JUnit: introduction
⢠Collection of classes to perform unit testing
⢠Uses annotations (e.g., @Test)
21/10/2016 xavier.devroey@unamur.be 45
import static org.junit.Assert.assertEquals;
import org.junit.Test;
public class CalculatorTest {
@Test
public void evaluatesExpression() {
Calculator calculator = new Calculator();
int sum = calculator.evaluate("1+2+3");
assertEquals(6, sum);
}
}
java -cp .;junit-4.12.jar;hamcrest-core-1.3.jar org.junit.runner.JUnitCore CalculatorTest
46. Anatomy of a Junit test class
import âŚ
public class MyClassTest{
private static final Logger logger = LoggerFactory.getLogger(JHConwayLifeCellTest.class);
@Rule
public TestRule watcher = new TestWatcher() {
@Override protected void starting(Description description) {
logger.info(String.format("Starting test: %s()...â, description.getMethodName()));};};
@BeforeClass
public static void setUpClass() {logger.info("Setting up before class");}
@AfterClass
public static void tearDownClass() {logger.info("Tearing down after class");}
@Before
public void setUp() {logger.info("Setting up before test");}
@After
public void tearDown() {logger.info("Tearing down after test");}
@Test
public void testMyMethod() {assertTrue(true);}
}
21/10/2016 xavier.devroey@unamur.be 46
48. Hamcrest common matchers
⢠Core
â anything - always matches, useful if you don't care what the object under test is
⢠Logical
â allOf - matches if all matchers match, short circuits (like Java &&)
â anyOf - matches if any matchers match, short circuits (like Java ||)
â not - matches if the wrapped matcher doesn't match and vice versa
⢠Object
â equalTo - test object equality using Object.equals
â hasToString - test Object.toString
â instanceOf, isCompatibleType - test type
â notNullValue, nullValue - test for null
â sameInstance - test object identity
⢠Collections
â hasEntry, hasKey, hasValue - test a map contains an entry, key or value
â hasItem, hasItems - test a collection contains elements
â hasItemInArray - test an array contains an element
⢠Number
â closeTo - test floating point values are close to a given value
â greaterThan, greaterThanOrEqualTo, lessThan, lessThanOrEqualTo - test ordering
⢠Text
â equalToIgnoringCase - test string equality ignoring case
â equalToIgnoringWhiteSpace - test string equality ignoring differences in runs of whitespace
â containsString, endsWith, startsWith - test string matching
21/10/2016 xavier.devroey@unamur.be 48
49. Hands-on: Maven, JUnit and Hamcrest
⢠be.unamur:myfirstwebapp
⢠Class to test: be.unamur.myfirstwebapp.Message
⢠Test coverage using Cobertura
â mvn cobertura:cobertura
â Report in target/site/cobertura/index.html
21/10/2016 xavier.devroey@unamur.be 49
50. Hands-on: test a data access object
⢠Using a test database
â Derby (https://db.apache.org/derby/)
â H2 (http://www.h2database.com/)
â DB server test instance (MySQL, Oracle, etc.)
⢠Code designed to
â Externalize DB access information (e.g., using a property file)
â Manage DB connections (using a DataSource object)
⢠Connection pooling using commonds-dbcp
(https://commons.apache.org/proper/commons-dbcp/)
⢠In this case SQL execution using commonds-dbutils
(https://commons.apache.org/proper/commons-dbutils/)
⢠Exercice: Add test to cover âifâ branch in
MessageDAO.update
21/10/2016 xavier.devroey@unamur.be 50
51. Hands-on: test a servlet
⢠Using mock objects to emulate servlet environment
⢠Code designed to
â Strictly separate model (Beans + DAOs), view (JSPs), and controller
(Servlets)
⢠Mockito: tasty mocking framework for unit tests in Java
â http://site.mockito.org
⢠Create the mock
â HttpServletRequest request = mock(HttpServletRequest.class);
⢠Stub method calls
â when(request.getParameter("author")).thenReturn("Albert");
⢠Check methods were called with given arguments
â verify(context).getRequestDispatcher("/WEB-INF/messages.jsp");
21/10/2016 xavier.devroey@unamur.be 51
54. Code review
⢠Over-the-shoulder reviewing
â sometimes called stepmother reviewing
⢠Pair programming
⢠Pull request
21/10/2016 xavier.devroey@unamur.be 54
Dev.
Team
Project
master
Creates the
feature in a
dedicated branch
in local repository
Pushes the
branch to central
rep. and ďŹlls a
pull request
Reviews the
code, discusses it,
and alters it
Merges the
feature into the
oďŹcial rep. and
closes the pull
request
56. Code analysis: FindBugs
⢠http://findbugs.sourceforge.net
⢠Called after Maven test phase
â mvn test findbugs:findbugs findbugs:gui
21/10/2016 xavier.devroey@unamur.be 56
60. Selenium Commands
⢠open
â opens a page using a URL
⢠click/clickAndWait
â performs a click operation, and optionally waits for a new page to load
⢠verifyTitle/assertTitle
â verifies an expected page title
⢠verifyTextPresent
â verifies expected text is somewhere on the page
⢠verifyElementPresent
â verifies an expected UI element, as defined by its HTML tag, is present on the page
⢠verifyText
â verifies expected text and its corresponding HTML tag are present on the page
⢠verifyTable
â verifies a tableâs expected contents
⢠waitForPageToLoad
â pauses execution until an expected new page loads. Called automatically when clickAndWait is used
⢠waitForElementPresent
â pauses execution until an expected UI element, as defined by its HTML tag, is present on the page
21/10/2016 xavier.devroey@unamur.be 60
61. EVALUATION OF THE TESTS
PERFORMED
Are my test cases suitably selected?
64. Mutation testing
⢠Mimic typical errors by introducing small modifications in
the program
⢠Based on 2 hypothesis:
â Competent programmer hypothesis
â Coupling effect
⢠Mutation operators introduce small syntactic errors
21/10/2016 xavier.devroey@unamur.be 64
Original program:
if (a && x < 5) {
foo();
} else {
fee();
}
Mutant 1:
if (a || x < 5) {
foo();
} else {
fee();
}
Mutant 2:
if (a && x < 5) {
fee();
} else {
fee();
}
69. Hands-on: test cases assessment
⢠http://pitest.org
⢠Called after Maven test phase
â mvn org.pitest:pitest-maven:mutationCoverage
21/10/2016 xavier.devroey@unamur.be 69
71. Symbolic execution
⢠Execute the program with symbolic values
â Constraints on the variables to follow defined path
⢠http://klee.github.io (C)
⢠Pex (C#); CodeHunt (C# and Java)
21/10/2016 xavier.devroey@unamur.be 71
int foo(int x){
if(x < 0){
return -1/x;
}
return 1/x;
}
72. Symbolic execution
⢠Execute the program with symbolic values
â Constraints on the variables to follow defined path
⢠http://klee.github.io (C)
⢠Pex (C#); CodeHunt (C# and Java)
21/10/2016 xavier.devroey@unamur.be 72
{x : int}
{x : int | x < 0}
{x : int | x < 0}
int foo(int x){
if(x < 0){
return -1/x;
}
return 1/x;
}
73. Symbolic execution
⢠Execute the program with symbolic values
â Constraints on the variables to follow defined path
⢠http://klee.github.io (C)
⢠Pex (C#); CodeHunt (C# and Java)
21/10/2016 xavier.devroey@unamur.be 73
{x : int}
{x : int | x > 0} v {x : int | x = 0}
int foo(int x){
if(x < 0){
return -1/x;
}
return 1/x;
}
74. Search-based software testing
⢠Use metaheuristics (e.g., evolutionary algorithms) to
â generate test cases (or test data);
â prioritize test cases;
â minimize test suites; etc.
⢠For instance, we want to generate input data for a given
function, we need:
â A search algorithm: hill climbing or genetic algorithm
â A representation of the program and input values
â Search operators: find the neighbours of the input values
â A fitness function: decision coverage
⢠This requires instrumentation of the program (to measure coverage
when executing tests)
21/10/2016 xavier.devroey@unamur.be 74
75. Hands-on: test cases generation
⢠http://www.evosuite.org/
⢠Generates tests in .evosuite/ folder
⢠mvn compile evosuite:generate
⢠mvn evosuite:export -DtargetFolder=target/generated-
test-sources/evosuite
⢠mvn test
21/10/2016 xavier.devroey@unamur.be 75
77. ⢠Mathur, A. P. (2008). Foundations of software testing. Pearson Education.
⢠Utting, M., & Legeard, B. (2007). Practical Model-Based Testing: A Tools Approach.
Morgan Kaufmann.
⢠Bourque, P., & Fairley, R. E. (2014). Guide to the Software Engineering Body of
Knowledge (SWEBOK(R)): Version 3.0. IEEE Computer Society.
⢠Dowling, P., & McGrath, K. (2015). Using free and open source tools to manage
software quality. Communications of the ACM, 58(7), 51â55. doi:10.1145/2755503
⢠Tim O'Brien et al. Maven by Example. Sonatype,
http://www.sonatype.com/books/mvnex-book/reference/public-book.html
⢠http://junit.org
⢠http://hamcrest.org
⢠http://site.mockito.org
⢠http://www.seleniumhq.org
⢠http://pitest.org
⢠http://klee.github.io
⢠https://www.microsoft.com/en-us/research/project/pex-and-moles-isolation-and-
white-box-unit-testing-for-net/
⢠http://www.evosuite.org/
⢠https://openclassrooms.com/courses/creez-votre-application-web-avec-java-ee
21/10/2016 xavier.devroey@unamur.be 77