1. Unit 9: Web Application Testing
 Testing is the activity conducted to evaluate the quality of a
product and to improve it by finding errors.
Testing
dsbw 2011/2012 q1 1

2. Testing Terminology
 An error is “the difference between a computed, observed, or
measured value or condition and the true, specified, or
theoretically correct value or condition” (IEEE standard 610.12-
1990).
 This “true, specified, or theoretically correct value or condition”
comes from
 A well-defined requirements model, if available and complete
 An incomplete set of fuzzy and contradictory goals, concerns,
and expectations of the stakeholders
 A test is a set of test cases for a specific object under test: the
whole Web application, components of a Web application, a
system that runs a Web application, etc.
 A single test case describes a set of inputs, execution conditions,
and expected results, which are used to test a specific aspect of the
object under test
dsbw 2011/2012 q1 2

3. Testing [and] Quality
 Testing should address compliance not only to functional
requirements but also to quality requirements, i.e., the kinds of
quality characteristics expected by stakeholders.
 ISO/IEC 9126-1 [Software] Quality Model:
dsbw 2011/2012 q1 3

4. Goals of Testing
 The main goal of testing is to find errors but not to prove
their absence
 A test run is successful if errors are detected. Otherwise, it is
unsuccessful and “a waste of time”.
 Testing should adopt a risk-based approach:
 Test first and with the greatest effort those critical parts of an
application where the most dangerous errors are still
undetected
 A further aim of testing is to bring risks to light, not simply to
demonstrate conformance to stated requirements.
 Test as early as possible at the beginning of a project: errors
happened in early development phases are harder to localize
and more expensive to fix in later phases.
dsbw 2011/2012 q1 4

5. Test Levels (1/2)
 Unit tests
 Test the smallest testable units (classes, Web pages, etc.)
independently of one another.
 Performed by the developer during implementation.
 Integration tests
 Evaluate the interaction between distinct and separately tested
units once they have been integrated.
 Performed by a tester, a developer, or both jointly.
 System tests
 Test the complete, integrated system.
 Typically performed by a specialized test team.
dsbw 2011/2012 q1 5

6. Test Levels (2/2)
 Acceptance tests
 Evaluate the system with the client in an “realistic”
environment, i.e. with real conditions and real data.
 Beta tests
 Let friendly users work with early versions of a product to get
early feedback.
 Beta tests are unsystematic tests which rely on the number and
“malevolence” of potential users.
dsbw 2011/2012 q1 6

7. Fitting Testing in the Development Process
 Planning: Defines the quality goals, the general testing strategy, the test
plans for all test levels, the metrics and measuring methods, and the test
environment.
 Preparing: Involves selecting the testing techniques and tools and
specifying the test cases (including the test data).
 Performing: Prepares the test infrastructure, runs the test cases, and then
documents and evaluates the results.
 Reporting: Summarizes the test results and produces the test reports.
dsbw 2011/2012 q1 7

8. Web Testing: A Road Map
Content Interface
Testing Testing Usability Testing
user
Navigation
Testing
Component
Testing
Configuration
Testing
Performance Security
technology Testing Testing
dsbw 2011/2012 q1 8

9. Usability
 Usability is a quality attribute that assesses how easy user
interfaces are to use. Also refers to methods for improving
ease-of-use during the design process.
 Usability is defined by five quality components:
 Learnability: How easy is it for users to accomplish basic tasks
the first time they encounter the design?
 Efficiency: Once users have learned the design, how quickly
can they perform tasks?
 Memorability: When users return to the design after a period
of not using it, how easily can they reestablish proficiency?
 Errors: How many errors do users make, how severe are these
errors, and how easily can they recover from the errors?
 Satisfaction: How pleasant is it to use the design?
dsbw 2011/2012 q1 9

10. Why Usability matters*
 62% of web shoppers gave up looking for an item. (Zona
study)
 50% of web sales are lost because visitors can’t easily find
content. (Gartner Group)
 40% of repeat visitors do not return due to a negative
experience. (Zona study)
 85% of visitors abandon a new site due to poor design.
(cPulse)
 Only 51% of sites complied with simple web usability
principles. (Forrester study of 20 major sites)
(*) data from www.usabilitynet.org/management/c_cost.htm
dsbw 2011/2012 q1 10

11. Why people fail
Search
Findability (IA, Category
names, Navigation, Links)
Page design (Readability,
Layout, Graphics, Amateur,
Scrolling)
Information (Content,
Product info, Corporate info,
Prices)
Task support (Workflow,
Privacy, Forms, Comparison,
Inflexible)
Fancy design (Multimedia,
Back button, PDF/Printing,
New window, Sound)
Other (Bugs, Presence on
Web, Ads, New site,
Usability problems weighted by how frequently they Metaphors)
caused users to fail a task [NL06]
dsbw 2011/2012 q1 11

12. Top Ten (Usability) Mistakes in Web Design
1. Bad search
2. Pdf files for online reading
3. Not changing the color of visited links
4. Non-scannable text
5. Fixed font size
6. Page titles with low search engine visibility
7. Anything that looks like an advertisement
8. Violating design conventions
9. Opening new browser windows
10. Not answering users' questions
dsbw 2011/2012 q1 12

13. Assessing Usability
 Two major types of assessing methods:
 Usability evaluations:
 Evaluators and no users
 Techniques: surveys/questionnaires, observational
evaluations, guideline based reviews, cognitive
walkthroughs, expert reviews, heuristic evaluations
 Usability tests: focus on users working with the product
 Usability testing is the only way to know if the Web site
actually has problems that keep people from having a
successful and satisfying experience.
dsbw 2011/2012 q1 13

14. Usability Testing
 Usability testing is a methodology that employs potential
users to evaluate the degree to which a website/software
meets predefined usability criteria.
 Basic Process:
1. Watch Customers
2. They Perform Tasks
3. Note Their Problems
4. Make Recommendations
5. Iterate
dsbw 2011/2012 q1 14

15. Measures of Usability
 Effectiveness (Ability to successfully accomplish tasks)
 Percentage of goals/tasks achieved (success rate)
 Number of errors
 Efficiency (Ability to accomplish tasks with speed and ease)
 Time to complete a task
 Frequency of requests for help
 Number of times facilitator provides assistance
 Number of times user gives up
dsbw 2011/2012 q1 15

16. Measures of Usability
 Satisfaction (Pleasing to users)
 Positive and negative ratings on a satisfaction scale
 Percent of favorable comments to unfavorable comments
 Number of good vs. bad features recalled after test
 Number of users who would use the system again
 Number of times users express dissatisfaction or frustration
 Learnability (Ability to learn how to use site and remember it)
 Ratio of successes to failures
 Number of features that can be recalled after the test
dsbw 2011/2012 q1 16

17. Usability Testing Roles
 Facilitator:
 Oversees the entire test process
 Plan, test, and report.
 Participant:
 Actual or potential customer.
 Representative users (marketing, designers) avoided.
 Observer (optional):
 Records events as they occur.
 Limits interaction with the customer.
 Does contribute to the report.
dsbw 2011/2012 q1 17

18. Usability Testing Process
Step 1: Planning The Usability Test
 Define what to test
 Define which customers should be tested
 Define what tasks should be tested
 Write usability scenarios and tasks
 Select participants
Step 2: Conducting The Usability Test
 Conduct a test
 Collect data
Step 3: Analyzing and Reporting The Usability Test
 Compile results
 Make recommendations
dsbw 2011/2012 q1 18

19. People – Context – Activities
Step 1: Planning The Usability Test
 Define what to test
 → Activities (Use Cases)
 Define which customers (user profiles) to be tested
 → People (Actors)
 Provide a background for the activities to test
 → Context
dsbw 2011/2012 q1 19

20. Usability Scenarios and Tasks
 Provide the participant with motivation and context to make
the situation more realistic
 Include several tasks:
 Make the first task simple
 Give a goal, without describing steps
 Set some success criteria, examples:
 N% of test participants will be able to complete x% of tasks in
the time allotted.
 Participants will be able to complete x% of tasks with no more
than one error per task.
 N% of test participants will rate the system as highly usable on
a scale of x to x.
dsbw 2011/2012 q1 20

21. Example of Scenario with Tasks
 Context:
 You want to book a sailing on Royal Caribbean International for
next June with your church group. The group is called “Saint
Francis Summer 2010”. The group is selling out fast, so you
want to book a cabin, which is close to an elevator because
your leg hurts from a recent injury.
 Tasks to perform:
1. Open your browser
2. Click the link labeled “Royal Caribbean”
3. Tell me the available cabins in the “Saint Francis Summer
2010” group
4. Tell me a cabin number closest to an elevator
5. Book the cabin the best suits your needs
dsbw 2011/2012 q1 21

22. Selecting Participants
 Recruit participants
 In-house
 recruitment firms, databases, conferences
 Match participants with user profiles
 Numbers: of participants, floaters
 Schedule test sessions
 Incentives:
 Gift checks ($100 per session)
 Food or gift cards
dsbw 2011/2012 q1 22

23. How Many Test Participants Are Required?
 The number of usability problems found in a usability test
with n participants is:
N(1-(1-L)n)
 N : total number of usability problems in the design
 L : the proportion of usability problems discovered while testing
a single participant.
For L = 31%
dsbw 2011/2012 q1 23

24. How Many Test Participants Are Required?
 It seems that you need to test with at least 15 participants to
discover all the usability problems
 However, is better to perform 3 tests with 5 participants than
to perform one with 15 participants:
 After the first test with 5 participants has found 85% of the
usability problems, you will want to fix them in a redesign.
 After creating the new design, you need to test again.
 The second test with 5 users will discover most of the
remaining 15% of the original usability problems that were not
found in the first test (and some new one).
 The new test will be able to uncover structural usability
problems that were obscured in initial studies as users were
stumped by surface-level usability problems.
 Fix the new problems, and test …
dsbw 2011/2012 q1 24

25. Usability Labs … Not Necessary
The testing room contains office The observer side contains a
furniture, video tape equipment, a powerful computer to collect the
microphone and a computer with usability data and analyze it. A one-
appropriate software. way mirror separates the rooms.
dsbw 2011/2012 q1 25

26. Test Side-by-Side
dsbw 2011/2012 q1 26

27. Conducting Tests: Facilitator’s Role
 Start with an easy task to build confidence
 Sit beside the person not behind the glass
 Use “think-out-loud” protocol
 Give participants time to think it through
 Offer appropriate encouragement
 Lead participants, don’t answer questions (being an enabler)
 Don’t act knowledgeable (treat them as the experts)
 Don’t get too involved in data collection
 Don’t jump to conclusions
 Don’t solve their problems immediately
dsbw 2011/2012 q1 27

28. Collecting Data
 Performance
 Objective (what actually happened)
 Usually Quantitative
 Time to complete a task
 Time to recover from an error
 Number of errors
 Percentage of tasks completed successfully
 Number of clicks
 Pathway information
 Preference
 Subjective (what participants say/thought)
 Usually Qualitative
 Preference of versions
 Suggestions and comments
 Ratings or rankings (can be quantitative)
dsbw 2011/2012 q1 28

29. Report findings and recommendations
 Make report usable for your users
 Include quantitative data (success rates, times, etc.)
 Avoid words like “few, many, several”. Include counts
 Use quotes
 Use screenshots
 Mention positive findings
 Do not use participant names, use P1, P2, P3, etc.
 Include recommendations
 Make it short
dsbw 2011/2012 q1 29

30. Component Testing
 Focuses on a set of tests that attempt to uncover errors in
WebApp functions
 Conventional black-box and white-box test case design
methods can be used at each architectural layer
(presentation, domain, data access)
 Form data can be exploited systematically to find errors:
 Missing/incomplete data
 Type conversion problems
 Value boundary violations
 Fake data
 Etc.
 Database testing is often an integral part of the component-
testing regime
dsbw 2011/2012 q1 30

31. Configuration Testing: Server-Side Issues
 Is the WebApp fully compatible with the server OS?
 Are system files, directories, and related system data created correctly
when the WebApp is operational?
 Do system security measures (e.g., firewalls or encryption) allow the
WebApp to execute and service users without interference or
performance degradation?
 Has the WebApp been tested with the distributed server configuration (if
one exists) that has been chosen?
 Is the WebApp properly integrated with database software? Is the
WebApp sensitive to different versions of database software?
 Do server-side WebApp scripts execute properly?
 Have system administrator errors been examined for their affect on
WebApp operations?
 If proxy servers are used, have differences in their configuration been
addressed with on-site testing?
dsbw 2011/2012 q1 31

32. Configuration Testing: Client-Side Issues
 Hardware—CPU, memory, storage and printing devices
 Operating systems—Linux, Macintosh OS, Microsoft
Windows, a mobile-based OS
 Browser software—Internet Explorer, Mozilla/Netscape,
Opera, Safari, and others
 User interface components—Active X, Java applets and
others
 Plug-ins—QuickTime, RealPlayer, and many others
 Connectivity—cable, DSL, regular modem, T1
dsbw 2011/2012 q1 32

33. Security Testing
 Designed to probe vulnerabilities
 of the client-side environment,
 the network communications that occur as data are passed
from client to server and back again,
 and the server-side environment
 On the client-side, vulnerabilities can often be traced to pre-
existing bugs in browsers, e-mail programs, or
communication software.
 On the network infrastructure
 On the server-side,
Review the DSBW Unit on
 At host level WebApp Security
 At WebApp level
dsbw 2011/2012 q1 33

34. Performance Testing: Main Questions
 Does the server response time degrade to a point where it is
noticeable and unacceptable?
 At what point (in terms of users, transactions or data loading) does
performance become unacceptable?
 What system components are responsible for performance
degradation?
 What is the average response time for users under a variety of
loading conditions?
 Does performance degradation have an impact on system security?
 Is WebApp reliability or accuracy affected as the load on the
system grows?
 What happens when loads that are greater than maximum server
capacity are applied?
dsbw 2011/2012 q1 34

35. Performance Testing: Load Tests
 A load test verifies whether or not the system meets the
required response times and the required throughput.
 Steps:
1. Determine load profiles (what access types, how many visits per day,
at what peak times, how many visits per session, how many
transactions per session, etc.) and the transaction mix (which
functions shall be executed with which percentage).
2. Determine the target values for response times and throughput (in
normal operation and at peak times, for simple or complex accesses,
with minimum, maximum, and average values).
3. Run the tests, generating the workload with the transaction mix
defined in the load profile, and measure the response times and the
throughput.
4. The results are evaluated, and potential bottlenecks are identified.
dsbw 2011/2012 q1 35

36. Performance Testing: Stress Tests
 A stress test verifies whether or not the system reacts in a
controlled way in “stress situations”, which are simulated by
applying extreme conditions, such as unrealistic overload, or
heavily fluctuating load.
 The test is aimed at answering the questions:
 Does the server degrade ‘gently’ or does it shut down as
capacity is exceeded?
 Does server software generate “server not available”
messages? More generally, are users aware that they cannot
reach the server?
 Are transactions lost as capacity is exceeded?
 Is data integrity affected as capacity is exceeded?
dsbw 2011/2012 q1 36

37. Performance Testing: Stress Tests (cont.)
 Under what load conditions the server environment fails? How
does failure manifest itself? Are automated notifications sent to
technical support staff at the server site?
 If the system does fail, how long will it take to come back on-
line?
 Are certain WebApp functions (e.g., compute intensive
functionality, data streaming capabilities) discontinued as
capacity reaches the 80 or 90% level?
dsbw 2011/2012 q1 37

38. Performance Testing: Interpreting Graphics
 Load: the number of
requests that arrive at
the system per time unit
 Throughput: the number
of requests served per
time unit.
 SLA: Service Level
Agreement
dsbw 2011/2012 q1 38

39. Test Automation
 Automation can significantly increase the efficiency of testing and
enables new types of tests that also increase the scope (e.g.
different test objects and quality characteristics) and depth of
testing (e.g. large amounts and combinations of input data).
 Test automation brings the following benefits:
 Running automated regression tests on new versions of a
WebApp allows to detect defects caused by side-effects to
unchanged functionality.
 Various test methods and techniques would be difficult or
impossible to perform manually. For example, load and stress
testing requires to simulate a large number of concurrent users.
 Automation allows to run more tests in less time and, thus, to
run the tests more often leading to greater confidence in the
system under test.
 Web Site Test Tools: http://www.softwareqatest.com/qatweb1.html
dsbw 2011/2012 q1 39

40. References
 R. G. Pressman, D. Lowe: Web Engineering. A Practitioner’s
Approach. McGraw Hill, 2008. Chapter 15.
 KAPPEL, Gerti et al. Web Engineering, John Wiley & Sons,
2006. Chapter 7.
 [NH06] NIELSEN, J. and LORANGER, H. 2006 Prioritizing Web
Usability. New Riders Publishing.
 www.useit.com (Jakob Nielsen)
 www.usability.gov
dsbw 2011/2012 q1 40