Software Testing fundamentals

1. Testing Fundamentals
 Topics
–
–
–
–
covered
What is black-box and white-box testing
How static and dynamic testing differ
What high-level techniques can be used for
reviewing a product specification
What specific problems should look for when
reviewing a product specification in detail

2. Product Specification


The product specification is a written document using
words and pictures to describe the intended product.
Produce spec might read something like this for
Windows Calculator:
–
–
The Edit menu will have two selections: Copy and Paste. These can be chosen by one of
three methods: pointing and clicking to the menu items with the mouse, using accesskeys (Alt+E and then C for Copy and P for Paste), or using the standard Windows
shortcut keys of Ctrl+C for Copy and Ctrl+V for Paste.
The Copy function will copy the current entry displayed in the number text box into the
Windows Clipboard. The Paste function will paste the value stored in the Windows
Clipboard into the number text box.

3. Black-Box Testing



Black Box Testing is testing without knowledge of the
internal workings of the item being tested. For
example, when black box testing is applied to software
engineering, the tester would only know the "legal"
inputs and what the expected outputs should be, but
not how the program actually arrives at those outputs.
Black-box testing is sometimes referred to as
functional testing or behavioral testing.
The test planning for black-box can begin as soon as
the specifications are written.

4. Advantages of Black-Box Testing






more effective on larger units of code than glass box
testing
tester needs no knowledge of implementation,
including specific programming languages
tester and programmer are independent of each other
tests are done from a user's point of view
will help to expose any ambiguities or inconsistencies
in the specifications
test cases can be designed as soon as the
specifications are complete

5. Disadvantages of Black Box
Testing






only a small number of possible inputs can actually be tested, to
test every possible input stream would take nearly forever
without clear and concise specifications, test cases are hard to
design
there may be unnecessary repetition of test inputs if the tester is
not informed of test cases the programmer has already tried
may leave many program paths untested
cannot be directed toward specific segments of code which may
be very complex (and therefore more error prone)
most testing related research has been directed toward glass box
testing

6. Tests under Black-Box Testing


Equivalence partitioning: Equivalence partitioning is designed to
minimize the number of test cases by dividing tests in such a way
that the system is expected to act the same way for all tests of
each equivalence partition. Test inputs would be selected from
each partition.
Boundary value Analysis is a form of black box testing in which
input values at the boundaries of the input domain are tested. It
has been widely recognized that input values at the extreme ends
of, and just outside of, input domains tend to cause errors in
system functionality.

7. Tests under Black-Box Testing
(Cont..)
 Smoke
Test: A smoke test is a cursory
examination of all of the basic components of a
software system to ensure that they work.
Typically, smoke testing is conducted
immediately after a software build is made.
 The term comes from electrical engineering,
where in order to test electronic equipment,
power is applied and the tester ensures that
the product does not spark or smoke.

8. White-Box Testing




White-box testing is testing that takes into account the internal
mechanism of a system or component (IEEE, 1990).
White-box testing is also known as structural testing, clear box
testing, and glass box testing (Beizer, 1995).
In order to implement white box testing, the tester has to deal with
the code and hence is needed to possess knowledge of coding
and logic i.e. internal working of the code.
White box test also needs the tester to look into the code and find
out which unit/statement/chunk of the code is malfunctioning.

9. Advantages of White-Box Testing
 As
the knowledge of internal coding structure is
prerequisite, it becomes very easy to find out
which type of input/data can help in testing the
application effectively.
 The other advantage of white box testing is
that it helps in optimizing the code
 It helps in removing the extra lines of code,
which can bring in hidden defects.

10. Disadvantages of White-Box
Testing
 As
knowledge of code and internal structure is
a prerequisite, a skilled tester is needed to
carry out this type of testing, which increases
the cost.
 And it is nearly impossible to look into every bit
of code to find out hidden errors, which may
create problems, resulting in failure of the
application.

11. Types of testing under White/Glass
Box Testing Strategy



Unit Testing: The developer carries out unit testing in
order to check if the particular module or unit of code is
working fine.
Static and dynamic Analysis: Static analysis involves
going through the code in order to find out any possible
defect in the code. Dynamic analysis involves
executing the code and analyzing the output.
Statement Coverage: In this type of testing the code is
executed in such a manner that every statement of the
application is executed at least once. It helps in
assuring that all the statements execute without any
side effect.

12. Types of testing under White/Glass
Box Testing Strategy (cont..)



Branch Coverage: Branch coverage testing helps in
validating of all the branches in the code and making
sure that no branching leads to abnormal behavior of
the application.
Security Testing: Security Testing is carried out in
order to find out how well the system can protect itself
from unauthorized access, hacking – cracking, any
code damage etc.
Mutation Testing: A kind of testing in which, the
application is tested for the code that was modified
after fixing a particular bug/defect.

13. Performing a High-Level Review





Defining the Software Product is a difficult process.
The process is an inexact science and is prone to
having problems.
Examine the specification for large fundamental
problems.
It will give better understanding of what the software
should do.
Better understanding of the whys and hows behind
the specifications.

14. Pretend to be the Customer
A
tester is to pretend to be the customer when
received a specification for review.
 Do some research about who the customer will
be.
 Know about the end user.
 Find out who will be using the software and talk
to them.

15. Pretend to be the Customer (cont..)
 It
is important to understand the customer’s
expectations.
 Remember that the definition of quality
means “meeting the customer’s needs”.
 Tester must understand those needs to test.
 Tester is not necessary to be a domain expert
 Gaining some familiarity with the field the
software applies to would be a great help.

16. Pretend to be the Customer (cont..)
 During
the review if you don’t understand it,
don’t assume that it’s correct and go on.
 Eventually you will have to use specification to
design your test cases
 Finding bugs along the way is better.
 Don’t forget about software security.
 Security must be specified in details.

17. Research Existing Standards and
Guidelines
 Difference
between standards and guidelines
is a matter of degree.
 A standard is much more firm than a guideline.
 Standards should be strictly adhered to.
 Guidelines are optional but should be followed.
 As a tester your job is not to define what
guidelines and standards should be applied.
This lies with Project Manager.

18. Several examples of standards and
guidelines
 Corporate
–
If the software is tailored for a specific company, it
should adopt the common terms and conventions
used by the employees of that company.
 Industry
–
Terminology and Conventions:
Requirements:
Medical, pharmaceutical, industrial, and financial
industries have very strict standards that their
software must follow.

19. Several examples of standards and
guidelines



Government Standards:
– The government, especially the military, has strict
standards.
Graphical User Interface (GUI):
– There are published standards and guidelines of
Microsoft and Apple Macintosh Operating Systems
– How the software should look and fee to a user.
Security: Software and its interfaces and protocols
may need to meet certain security standards or levels.

20. Review and Test Similar Software


Some things to look for when reviewing competitive
products include:
Scale:
–
–
–

Fewer or greater features
Less or more code
Will that size difference matter in your testing?
Complexity:
–
–
More or less complex
Will this impact your testing?

21. Review and Test Similar Software
 Testability:
–
Will you have the resources, time, and expertise to
test software such as this?
 Quality/Reliability:
–
–
Is this software representative of the overall quality
planned for your software?
Will your software be more or less reliable?
 Security:
–
How does the competitor’s software security?

22. Review and Test Similar Software
 There
–
–
–
is no substitute for hands-on experience.
Use the similar software
Bang on it.
Demonstrate it.
 Gain
a lot of experience and it will help during
the review of specification.

23. Review and Test Similar Software
 Don’t
forget about reading online and printed
software reviews and articles about the
competitive software.
 This can be especially helpful for security
issues
 They will be well know in the press.

24. Low-Level Specification Test Techniques:
Specification Attributes Checklist




Complete. Is anything missing or forgotten? Is it
thorough? Does it include everything necessary to
make it stand alone?
Accurate. Is the proposed solution correct? Does it
properly define the goal? Are there any errors?
Precise, Unambiguous, and Clear. Is the description
exact and not vague? Is there a single interpretation?
Is it easy to read and understand?
Consistent. Is the description of the feature written so
that it doesn't conflict with itself or other items in the
specification?

25. Specification Attributes Checklist




Relevant. Is the statement necessary to specify the feature? Is it
extra information that should be left out? Is the feature traceable
to an original customer need?
Feasible. Can the feature be implemented with the available
personnel, tools, and resources within the specified budget and
schedule?
Code-free. Does the specification stick with defining the product
and not the underlying software design, architecture, and code?
Testable. Can the feature be tested? Is enough information
provided that a tester could create tests to verify its operation?

26. Specification Terminology Checklist



Always, Every, All, None, Never. If you see words
such as these that denote something as certain or
absolute, make sure that it is, indeed, certain. Put on
your tester's hat and think of cases that violate them.
Certainly, Therefore, Clearly, Obviously, Evidently.
These words tend to persuade you into accepting
something as a given. Don't fall into the trap.
Some, Sometimes, Often, Usually, Ordinarily,
Customarily, Most, Mostly. These words are too
vague. It's impossible to test a feature that operates
"sometimes."

27. Specification Terminology Checklist




Etc., And So Forth, And So On, Such As. Lists that finish with
words such as these aren't testable. Lists need to be absolute or
explained so that there's no confusion as to how the series is
generated and what appears next in the list.
Good, Fast, Cheap, Efficient, Small, Stable. These are
unquantifiable terms. They aren't testable. If they appear in a
specification, they must be further defined to explain exactly what
they mean.
Handled, Processed, Rejected, Skipped, Eliminated. These
terms can hide large amounts of functionality that need to be
specified.
If…Then…(but missing Else). Look for statements that have
"If…Then" clauses but don't have a matching "Else." Ask yourself
what will happen if the "if" doesn't happen.