1. Software Testing Strategy
What kind of testing approach we should take to apply
the already developed test cases ?
13 January 2012 Made By Utpal Ray 1
2. Software Testing Strategy
Strategic Approach
It is about applying designed test cases strategically to uncover
more number of errors in a shorter span of time.
Deciding a suitable test strategy calls for a suitable test plan and a
test procedure document.
Testing should begin at the component level ( unit level ) and move
‘outward’ towards integration of the entire computer based system.
Different testing techniques are required at different point of
testing span.
Testing should be performed by an Independent Test Group, who
did not contribute towards the design and development of the
software.
Any testing strategy must incorporate test planning, test case
design, test execution, test result collection, test result evaluation
etc.
13 January 2012 Made By Utpal Ray 2
3. Software Testing Strategy
A typical Software Test Strategy
Considering the fact that testing is a big task, the entire
testing cycle can be divided into four different phase where
different test strategy is employed in different phase; also,
different test techniques are used in different phase.
The test target and the test focus also changes along this
four different phases of testing cycles.
The most interesting part is how error finding rate changes
as testing progresses along this four different phases and so
as the cost associated with fixing all those errors.
It is to be noted that test strategy for a newly developed
software ( which was developed from scratch ) will be
different than the test strategy for a moderately modified
software.
13 January 2012 Made By Utpal Ray 3
4. Software Testing Strategy
A typical Software Test Strategy for a newly developed software
Test Phase Early Phase Integration Validation Final Phase
( Phase – I ) Phase Phase ( Phase – IV )
Scenario ( Phase – II ) ( Phase – III )
Test Unit Integration Validation System
Strategy Testing Testing Testing Testing
Test Mainly White White Box Black Box Black Box
Technique Box And Black Testing Testing
Testing Box Testing
Test Individual A group of The Whole The entire
Target Module Closely Software s/w in the
related operational
Module environment
13 January 2012 Made By Utpal Ray 4
5. Software Testing Strategy
A typical Software Test Strategy for a newly developed software
( Contd. )
Test Phase Early Phase Integration Validation Final Phase
Scenario ( Phase – I ) Phase Phase ( Phase–IV )
( Phase – II ) ( Phase-III )
Testing Coding Design and Functional The s/w along
Focus the s/w Requirement with the other
architecture and other system elements
requirement
Error Finding
Rate
13 January 2012 Made By Utpal Ray 5
6. Software Testing Strategy
A typical Software Test Strategy for a newly developed software (
Contd. )
Test Phase Early Phase Integration Validation Final Phase
Scenario ( Phase – I ) Phase Phase ( Phase–IV )
( Phase – II ) ( Phase-III )
Difficulty and
cost of fixing
those errors
The
responsible
persons for Individual Project Team The whole The whole
the errors Department Organization
13 January 2012 Made By Utpal Ray 6
7. Software Testing Strategy
Unit Testing
It is done on the individual module level.
Component level design is used as a guide.
It is important to fix the scope of the unit testing.
The module interface needs to be examined and tested.
The boundary conditions need to be examined ( some sort of
black box testing ).
The basis path testing ( white box testing ) need to be
performed.
The error conditions, the error handlings and the error
messages – all these criteria need to be tested.
13 January 2012 Made By Utpal Ray 7
8. Software Testing Strategy
Unit Testing ( contd. )
If it is necessary driver and suitable stubs need to be
developed and integrated with the modules ( illustrated
below ).
DRIVER
Test Cases Test Results
Module To be
Tested
STUB STUB
13 January 2012 Made By Utpal Ray 8
9. Software Testing Strategy
Integration Testing
This is the testing phase after unit testing. This phase
deals with the combination of different modules.
Unless each of the module is unit tested
satisfactorily, one cannot enter this testing phase.
The best approach for integration testing is the
incremental integration; where modules are added one
after another. Integration testing is performed every
time after each of the module is added. This process of
testing is good enough to find out which module is
causing the error.
There are two distinct methods of incremental
integration – Top-Down integration and Bottom-Up
integration.
13 January 2012 Made By Utpal Ray 9
10. Software Testing Strategy
Top-Down Integration
Modules are added by moving downward through the
control hierarchy, beginning with the main control
module ( top most executive module ) first.
The subordinate modules ( to the main module ) can
be added either in a breadth-first manner or in a
depth-first manner.
For the Top-Down integration, you may need stub to
effectively test the effect of module addition.
13 January 2012 Made By Utpal Ray 10
11. Software Testing Strategy
Top-Down Integration ( contd. )
Breadth-First
M1
M2 M3 M4
M5 M6 M7 Sequence of Module Addition :-
M1, M2, M3, M4, M5, M6, M7, M8
M8
13 January 2012 Made By Utpal Ray 11
12. Software Testing Strategy
Top-Down Integration ( contd. )
Depth-First
M1
M2 M3 M4
M5 M6 M7
Sequence of Module Addition :-
M1, M2, M5, M8, M6, M3, M7, M4
M8
13 January 2012 Made By Utpal Ray 12
13. Software Testing Strategy
Bottom-Up Integration
Modules are added from the bottom; which means
modules from the lower most level of the ‘call and
return’ architecture are added first. Then the addition
process moves upwards.
The low level components are combined into multiple
clusters and each of this cluster is tested first using a
driver. While testing each of the cluster; modules are
added from the lower most level for each of this
cluster.
Once the clusters have been successfully tested; the
clusters are attached to the main branch ( after
removing the drivers ) and addition of modules
process continues. This way the integration testing is
performed on the whole ‘call and return’ arch.
13 January 2012 Made By Utpal Ray 13
14. Software Testing Strategy
Bottom-Up Integration ( contd. )
An example of a ‘Call and Return’ Architecture
MA
MB MC MD
M11 M12 M21 M22 M31
M13 M23 M24 M32
M14 M25 M26
CLUSTER 1 CLUSTER 2 CLUSTER 3
13 January 2012 Made By Utpal Ray 14
15. Software Testing Strategy
Bottom-Up Integration ( contd. )
Testing Cluster 1
DRIVER1
Sequence of Module Addition :-
M14, M13, M12, M11
M11 M12
M13
M14
13 January 2012 Made By Utpal Ray 15
16. Software Testing Strategy
Bottom-Up Integration ( contd. )
Testing Cluster 2
DRIVER2
Sequence of Module Addition :-
M26, M25, M24, M22, M23, M21
M21 M22
M23 M24
M25 M26
13 January 2012 Made By Utpal Ray 16
17. Software Testing Strategy
Bottom-Up Integration ( contd. )
Testing Cluster 3
DRIVER3
Sequence of Module Addition :-
M32, M31
M31
M32
13 January 2012 Made By Utpal Ray 17
18. Software Testing Strategy
Validation Testing
This phase is entered after the successful completion
of the integration testing.
The software validation is achieved through a series of
black box tests that demonstrate conformity with the
requirements.
One needs a test plan to identify a series of tests which
need to be used as a validation test. One needs a valid
test procedure also, regarding executing all those
individual tests.
13 January 2012 Made By Utpal Ray 18
19. Software Testing Strategy
Validation Testing ( contd. )
The validation test ensures that the following criteria are
satisfied successfully :-
- All Functional Requirements
- All Behavioral characteristics
- All Performance Requirements
- All Documentation Correctness
- All Other Requirements ( Transportability,
Compatibility, Error Recoverability,
Maintainability etc. )
If some of the criteria mentioned above do not meet the
requirements, a deficiency list need to be prepared
identifying which one is deviating from the normal
reference.
13 January 2012 Made By Utpal Ray 19
20. Software Testing Strategy
System Testing
It is the final phase of the testing cycle.
It is about integrating the software with hardware,
people and information.
The primary purpose of the system testing is to fully
exercise the computer based system.
The system test is really effective if it can be done in the
customer’s premises. Otherwise the customer’s
environment is re-created in-house.
13 January 2012 Made By Utpal Ray 20
21. Software Testing Strategy
System Testing ( contd. )
There are quite a few varieties of system testing as
described below :-
- Recovery Testing [ How system recovers
from Faults? ]
- Security Testing [ How secure the system is? ]
- Stress Testing [ How much load the system can
take? ]
- Performance Testing [ How system performs in
normal condition and stresses condition ]
13 January 2012 Made By Utpal Ray 21
22. Software Testing Strategy
Verification & Validation ( V&V )
One has to Validate the software to ensure that all the
functional requirements are met by the software.
One has to verify that one particular functionality has
been correctly implemented in the software.
Verification : Are we building the product right ?
Validation : Are we building the right product ?
Validation comes nearly at the end of the testing cycle
Verification may come throughout the testing cycle.
13 January 2012 Made By Utpal Ray 22
23. Software Testing Strategy
Entry and Exit Criteria for testing
When to exit testing from the current test phase and
enter the next testing phase ?
Since the quote ‘You are never done with testing, the
burden simply shifts from the engineer to the
customer’, is always true; the decision to end testing
mainly depends upon the resources available and also
depends upon the confidence of the testing personnel.
Usually, the rate of finding errors drops as testing
progresses. When one cannot find any more errors;
that could be the time to exit the current phase and
enter the next phase.
13 January 2012 Made By Utpal Ray 23
24. Software Testing Strategy
Sample Test Plan Template
1. Document Control
- Distribution
- Approvers/Reviewers
- Change History
2. Overview
- Project Summary
- Overall Test Goals and Objectives
3. Test Environment
- Hardware and Software Configuration
4. Test Tools To Be Used
5. Administration
- Test Assumptions and Dependencies
- Entry and Exit Criteria
- Status Tracking Approach
- Problem Reporting and Tracking Approach
- Maintenance Strategy
- Deliverables
6. Schedules
7. Test Matrices and Scenario
13 January 2012 Made By Utpal Ray 24
25. Software Testing Strategy
Test plan template, IEEE 829 format
1.0 References
2.0 Introduction
3.0 Test Items
4.0 Features to be Tested
5.0 Features not to be Tested
6.0 Approach
7.0 Item Pass/Fail Criteria
8.0 Suspension Criteria and Resumption Requirements
9.0 Test Deliverables
10.0 Remaining Test Tasks
11.0 Environmental Needs
12.0 Staffing and Training Needs
13.0 Responsibilities
14.0 Schedule
15.0 Planning Risks and Contingencies
16.0 Approvals
17.0 Glossary
13 January 2012 Made By Utpal Ray 25
26. Software Testing Strategy
Other Types of Testing Scenario
1. Alpha Testing
2. Beta Testing
3. Gamma Testing
4. Smoke Testing
5. Regression Testing
6. Automated Testing
7. Fuzz Testing
8. Static Testing
13 January 2012 Made By Utpal Ray 26
27. Software Testing Strategy
Other Types of Testing Scenario ( contd. )
Alpha Testing
Different Software Organization has a different view
regarding definition of this test. This may be also
called ‘Engineering’ Test. Usually this test is attempted
after the integration testing is over. This test is
conducted in-house. This is a some kind of validation
test in which different in-house department may take
participation.
13 January 2012 Made By Utpal Ray 27
28. Software Testing Strategy
Other Types of Testing Scenario ( contd. )
Beta Testing
This testing is attempted after the alpha testing is over.
This test is always conducted in the customer
premises; that’s why it is a case of ‘live’ testing. Usually
developers are not present during this kind of testing.
The customer records all problems that were
encountered during beta testing and reports this to
the developers at regular intervals.
13 January 2012 Made By Utpal Ray 28
29. Software Testing Strategy
Other Types of Testing Scenario ( contd. )
Gamma Testing
Gamma testing is a little-known informal phrase that
refers derisively to the release of "buggy" (defect-
ridden) software. It is not a term of art among
testers, but rather an example of referential humor.
Cynics have referred to all software releases as "gamma
testing" since defects are always found in the customer
premises.
13 January 2012 Made By Utpal Ray 29
30. Software Testing Strategy
Other Types of Testing Scenario ( contd. )
Smoke Testing
It is a ritual of daily testing.
It need not be an exhaustive testing rather it is an end
to end testing.
The test cases are designed keeping the frequently
performing functions in mind.
This is a one type of integration testing, where the
entire software is smoke tested daily.
13 January 2012 Made By Utpal Ray 30
31. Software Testing Strategy
Other Types of Testing Scenario ( contd. )
Regression Testing
It is about running a subset of old test suite, to
make sure that the newly added changes did not
break the earlier functionality.
This is very much necessary when software moves
from one version to another version upwardly.
One has to run a regression test suite to make sure
the functionality of the earlier version did not
change even if the new functions has been added.
13 January 2012 Made By Utpal Ray 31
32. Software Testing Strategy
Other Types of Testing Scenario ( contd. )
Automated Testing
Considering the fact that testing is a repetitive
process, most of the testing execution process can be
automated.
In it’s simplest form, an automated test suite does not wait
for user’s input; rather it takes the input from a already
prepared file. At the same time it does not leave the test
results for user’s interpretation but compares it with an
already prepared sample output file.
It is comparatively easier to write automated test suite
when input/output involves with keyboard and monitor. In
case of GUI testing, a different method of automated
testing should be employed.
13 January 2012 Made By Utpal Ray 32
33. Software Testing Strategy
Other Types of Testing Scenario ( contd. )
Fuzz Testing
The basic idea is to attach the inputs of a program to a
source of random data ("fuzz"). If the program fails
(for example, by crashing, or by failing built-in code
assertions), then there are defects to correct.
13 January 2012 Made By Utpal Ray 33
34. Software Testing Strategy
Other Types of Testing Scenario ( contd. )
Static Testing
Static Testing (also known as "Dry Run Testing") is a
form of software testing where the software isn't
actually used. Syntax checking and manually reading
the code to find errors are methods of static testing.
This type of testing is mostly used by the developer
himself (who designed or code the module). Static
testing is usually the first type of testing done on any
system.
13 January 2012 Made By Utpal Ray 34
35. Software Testing Strategy
Still More Varieties of Testing
Proposal Testing; Requirement Testing; Design Testing;
Big Bang Testing; Sandwich Testing; Complexity Testing;
Compatibility Testing; Security Testing; Performance Testing;
Volume Testing; Stress Testing; Recovery Testing; Installation Testing;
Error Handling Testing; Manual Support Testing; Intersystem Testing;
Control Testing; Sanity Testing; Adhoc Testing (Monkey Testing,
Exploratory Testing, Random Testing); Execution Testing;
Operations Testing; Compliance Testing; Usability Testing;
Decision Table Testing (Axiom Testing); Documentation Testing;
Training Testing; Rapid Testing; ‘COTS’ Testing; Client-Server Testing;
Web Application Testing; Mobile Application Testing;
eBusiness/eCommerce Testing; Agile Development Testing;
Data Warehouse Testing;
[ Courtesy; Software Testing – M G Limaye; McGraw-Hill
13 January 2012 Made By Utpal Ray 35
36. Software Testing Strategy
Home Task
1. Why is a highly coupled module difficult to unit test
?
2. Develop an integration testing strategy for the
problems which you have already got the
architectural design.
3. Who should perform the validation test – the
software developer, an independent tester or the
software user ? Justify your answer.
13 January 2012 Made By Utpal Ray 36