2. Automated Test Tools Evaluation Criteria Version 1.02 (1/18/07)
Table of Contents
1. INTRODUCTION 1
1.1 Author’s Background ...........................................................................................................1
1.2 Allocate Reasonable Resources and Talent..........................................................................1
1.3 Establish Reasonable Expectations ......................................................................................2
2. RECOMMENDED EVALUATION CRITERIA 3
2.1 GUI Object Recognition.......................................................................................................3
2.2 Platform Support ..................................................................................................................3
2.3 Recording Browser Objects..................................................................................................3
2.4 Cross-browser Playback .......................................................................................................3
2.5 Recording Java Objects ........................................................................................................4
2.6 Java Playback .......................................................................................................................4
2.7 Visual Testcase Recording ...................................................................................................4
2.8 Scripting Language...............................................................................................................4
2.9 Recovery System ..................................................................................................................4
2.10 Custom Objects ....................................................................................................................5
2.11 Technical Support.................................................................................................................5
2.12 Internationalization Support .................................................................................................5
2.13 Reports..................................................................................................................................5
2.14 Training & Hiring Issues ......................................................................................................5
2.15 Multiple Test Suite Execution ..............................................................................................5
2.16 Testcase Management...........................................................................................................6
2.17 Debugging Support...............................................................................................................6
2.18 User Audience ......................................................................................................................6
ii Terry Horwath
3. Automated Test Tools Evaluation Criteria Version 1.02 (1/18/07)
1. INTRODUCTION
This document provides a list of evaluation criteria which has proven useful to me when
evaluating automated test tools like Mercury Interactive’s QuickTest Professional, WinRunner
and Segue’s Silk over the last several years for a variety of clients. Hopefully some readers will
find this information useful, such that it reduces your evaluation effort.
The specific criteria used for each project differs based on the client’s:
• testing environment, and
• test engineers’ programming backgrounds and skill sets, and
• type of software being tested [especially the software develop tool, such as Visual Basic,
PowerBuilder, Java, browser based applications, etc.], and
• application(s) testing requirements.
The remainder of this chapter provides a variety of miscellaneous thoughts I have on automating
the testing process, while Chapter 2 contains my list of potential evaluation criteria. Note that
some of the Chapter 2 evaluation criteria is Java and web application testing oriented. Substitute
your application development tool—for example Visual Basic or PowerBuilder—in the Java
related evaluation criteria items.
1.1 Author’s Background
I have designed custom frameworks as well as hundreds of test cases using Silk/QaPartner from
1994 (version 1.0) through 2004 (version 5.5). with WinRunner (version 5) and Test Director in
1999 and 2000 and with QuickTest Professional since 2006 (versions 8 and 9).
1.2 Allocate Reasonable Resources and Talent
Most software testing projects do not fail because of the selected test tools—virtually all of top
automated testing tools on the market can be used to do an adequate job, even when the test tool
is not well matched with the software development environment. Rather I believe that most
failures are due to a combination of the following reasons:
1. Test engineers fail to treat the effort to develop a large number of complex test cases and test
suites as a large software development project—it is crucial to apply good software
development methodology to produce a test product, which includes defining requirements,
developing a schedule, implementing each test suite using a shared custom framework of well
known libraries and guidelines, as well as using a software version control system.
2. Sufficient manpower and time are not allocated early enough in the application development
cycle. Along with incomplete testing this also leads to the phenomenon of test automation
targeted for use with Release N actually being delivered and used with Release N+1.
3. Test technicians with improper skills are assigned to use these automated test tools. Users of
these tools must have strong test mentalities and in all but a few situations they must also
possess solid programming skills with the automation tool’s scripting language.
1 Terry Horwath
4. Automated Test Tools Evaluation Criteria Version 1.02 (1/18/07)
1.3 Establish Reasonable Expectations
Through their promotional literature automated test tool vendors often establish unrealistic
expectations in one or more of the following areas:
• What application features and functions can truly be tested with the tool.
• The skill level required to effectively use the tool.
• How useful the tool’s automatic recording capabilities are.
• How quickly effective testcases can be produced.
This is unfortunate because in the hands of test engineers possessing the proper skill set all of the
top automated test tools can be used to test significant portions of virtually any GUI-centric
application. Use the following assumptions when reviewing this document and planning your
evaluation effort:
1. Even when a test tool is well matched with a software development tool, the test tool will still
only be able to recognize a subset of the application’s objects—windows, buttons, controls,
etc.—without taking special programming actions. This subset will be large when the
development engineers create window objects using the development tool’s standard class
libraries. The related issue of cross-browser playback also rears it head when testing web
applications.
2. If the test engineer wants to unleash the full power of the test tool they will need to have, or
develop, solid programming skills with the tool’s scripting language.
3. With few exceptions recording utilities—those tools which capture user interaction and insert
validation functions—are only effective in roughing out a testcase. Thereafter captured
sequences will most often need be cleaned up and/or generalized using the scripting
language.
4. If an application has functionality which can’t be tested through the GUI you will need to:
(a) use the tool’s ability to interface to DLLs—for Windows based applications;
(b) use its SDK (software developer kit) or API if it supports one of these mechanisms;
(c) use optional tools—at an additional cost—offered by the test tool vendor;
(d) use other 3rd party non-GUI test tools more appropriate to the testing task.
5. If you are currently manually testing the application to be automated you will need to initially
increase the size of the test team by a minimum of 1 or 2 test engineers—who possess good
programming backgrounds. After a significant portion of testcases have been written and
debugged you can start removing some of the manual test engineers. Pay back comes at the
end of the automation effort, not during the initial implementation.
6. If the test team does not contain at least one member previously involved with automating the
test process, coming up to speed is no small task—no matter which tool is selected. Budget
dollars and time for training classes and consulting offered by the tool vendor to get your test
team up and running.
7. Budget 80 hours of time to do a detailed evaluation of each vendor’s automated test tool
against your selected evaluation criteria, using one of your applications. While you might
initially recoil from this significant investment in time, keep in mind that the selected tool
will likely be part of your department’s testing effort for many years—selecting the wrong
tool will reduce productivity many times over 80 hours.
2 Terry Horwath
5. Automated Test Tools Evaluation Criteria Version 1.02 (1/18/07)
2. RECOMMENDED EVALUATION CRITERIA
2.1 GUI Object Recognition
Does the tool:
(a) Provide the ability to record each object in a window—or on a browser page—such that a
logical object identifier, used in the script, is definable independent of the operating system
dependent property [or properties] used by the tool to access that object at runtime.
(b) (1) Provide the ability to associate (i.e. map) the logical object identifier with more than one
operating system dependent property. And, (2) does the tool offer some technique to support
a property definition technique which supports internationalization [if language localization
is a testing requirement]?
(c) provide the ability to record—and deal effectively with—dynamically generated objects
[often encountered when testing web applications].
2.2 Platform Support
Are all of the required platforms [i.e. NT 4.0, Windows XP, Windows Vista, etc.] supported for:
(a) testcase playback?
(b) testcase recording?
(c) testcase development [programming without recording support]?
2.3 Recording Browser Objects
Does the tool provide the ability to record against web applications under test, correctly
recognizing all browser page HTML objects, using the following browsers:
(a) IE7?
(b) IE6?
(c) Firefox?
2.4 Cross-browser Playback
Does the tool provide the ability to reliably and repeatedly playback test scripts against the
browsers which were not used during object capture and testcase creation, with little or no:
(a) Changes to the GUI map (WinRunner), GUI declarations (Silk) or the equivalent in other
tools?
(b) Changes to testcase code?
(c) Does the tool provide some type of generic capability [without using sleep–like commands
in the code] to deal with “browser not ready” to correctly synchronize code execution—such
as an access to a web page over a slow internet connection?
3 Terry Horwath
6. Automated Test Tools Evaluation Criteria Version 1.02 (1/18/07)
2.5 Recording Java Objects
Does the tool:
(a) Provide the ability to record objects against, and see all standard Swing, AWT and JFC 1.1.8
and 1.2 objects, when running the Java application under test?
(b) Provide the ability to record objects against [and interact with] non-standard Java classes
required by the Java application under test (i.e. for example the KLGroup’s 3rd party controls,
when the application under test uses that 3rd party toolset)?
(c) Require that the platform’s static classpath environment variable be set with tool specific
classes, or can this be set within the tool on a test suite by test suite basis?
2.6 Java Playback
Does the tool:
(a) Reliably and repeatedly play back the evaluation testcases?
(b) Provide some type of generic capability [without using sleep –like commands in the code]
to deal with “application not ready” to correctly synchronize code execution? [This may or
may not be an issue, depending on the application being tested].
2.7 Visual Testcase Recording
Does the tool:
(a) Provide the ability to visually record testcases by interacting with the application under test as
a real user would?
(b) Provide the ability, while visually recording a testcase, to interactively insert—without
resorting to programming—validation statements?
(c) Provide the ability, while interactively inserting a validation statements, to
visually/interactively select validation properties (i.e. contents of a text field, focus on a
control, control enabled, etc.)?
2.8 Scripting Language
Is the test tool’s underlying scripting language:
(a) object-oriented?
(b) Proprietary?
2.9 Recovery System
Does the tool support some type of built-in recovery system, which the programmer can
control/define, that drives the application under test back to a know state? (Especially in the case
where modal dialogs were left open when a testcase failure occurred)?
4 Terry Horwath
7. Automated Test Tools Evaluation Criteria Version 1.02 (1/18/07)
2.10 Custom Objects
What capabilities does the tool provide to deal with unrecognized objects in a window or on a
browser page? [Spend a fair amount of time evaluating this capability, as it is quite important].
2.11 Technical Support
What was the quality and timeliness of technical support received during product evaluation?
[Remember—it won’t get any better after you purchase the product, but it might get worse].
2.12 Internationalization Support
Evaluate the support for internationalization [also referred to as language localization] in the
following areas [if this is a testing requirement]:
(a) Object recognition
(b) Object content (such as text fields, text labels, etc.).
(c) Evaluate and highlight any built–in or add-on multi–language support offered by the vendor.
2.13 Reports
What type of reporting and logging capabilities does the tool provide?
2.14 Training & Hiring Issues
(a) What is your [not the vendor’s] estimated learning curve to be competent (i.e. can write
useful test scripts which may need to be rewritten later);
(b) What is your estimated learning curve to become an skilled (i.e. can write test scripts which
rarely need to be rewritten).
(c) What is your estimated learning curve to become an expert (i.e. can design frameworks).
(d) What is your estimated availability of potential (i) employees, and (ii) expert consultants,
skilled with this tool in your geographic area.
2.15 Multiple Test Suite Execution
(a) Can multiple test suites be driven completely from the tool [or from a command line
interface] thereby allowing X number of unrelated suites/projects to be executed under a
cron-like job or shell? (For true unattended operation).
(b) …including the ability to save the results log, as text, prior to or during termination/exit?
(c) …including the ability to return a reliable pass/fail status on termination/exit?
5 Terry Horwath
8. Automated Test Tools Evaluation Criteria Version 1.02 (1/18/07)
2.16 Testcase Management
Does the tool support some type of test case management facility (either built-in or as an add-on)
that allows each test engineer to execute any combination of tests out of the full test suite for a
given project? How difficult is it to integrate manual testing results with automated test results?
2.17 Debugging Support
What type of debugging capabilities does the tool support to help isolate scripting and/or runtime
errors?
2.18 User Audience
Which of the following groups of users does the tool primarily target?
• Test technicians possess good test mentalities, but often lack much if any background in
programming or software development methodologies. They are the backbone of many test
groups and have often spent years developing and executing manual testcases.
• Test developers possess all of the test technician’s skill set, plus they have had some formal
training in programming and limited experience working with on a software development
project and/or automated testcases.
• Test architects possess all of the test developer’s skill set, plus they have had many years of
experience developing and maintaining automated test cases, as well as experience defining
and implementing the test framework under which multiple automated test suites are
developed. They are a recognized expert with at least one automated tool.
6 Terry Horwath
