Randy Rice presented on lessons learned from user acceptance testing (UAT) on four different projects. The first project involved a new laboratory testing system that had severe performance issues and required three redeployments. The second project with the same company was more successful due to improved testing practices. The third project involved designing many tests based on business scenarios before knowing the user interface. The last project involved a complex legal system where system testing found most defects and UAT was limited due to user capabilities. Key lessons included not relying solely on UAT, having contingency plans, and flexibility in testing plans.

1. T1
Test Management
5/8/2014 9:45:00 AM
A Funny Thing Happened on the
Way to User Acceptance Testing
Presented by:
Randy Rice
Rice Consulting Services, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Randy Rice
Rice Consulting Services, Inc.
A leading author, speaker, and consultant with more than thirty years of experience in the field
of software testing and software quality, Randy Rice has worked with organizations worldwide to
improve the quality of their information systems and optimize their testing processes. He is
coauthor (with William E. Perry) of Surviving the Top Ten Challenges of Software Testing and
Testing Dirty Systems. Randy is an officer of the American Software Testing Qualifications
Board (ASTQB). Founder, principal consultant, and trainer at Rice Consulting Services, Randy
can be contacted at riceconsulting.com where he publishes articles, newsletters, and other
content about software testing and software quality. Visit Randy’s blog.

3. 4/26/2014
1
A FUNNY THING
HAPPENED ON THE
WAY TO THE
ACCEPTANCE TEST
RANDALL W. RICE, CTAL
RICE CONSULTING SERVICES, INC.
WWW.RICECONSULTING.COM
2
THIS PRESENTATION
• The account of four different
acceptance tests, in three
organizations.
• The names have been withheld and
the data generalized to protect
privacy.
• One project was in-house developed
and the other three were vendor-
developed systems.
• So, a more traditional UAT approach
was taken.

4. 4/26/2014
2
3
A COMMON
PERCEPTION OF UAT
• UAT is often seen as that last golden moment or phase of
testing, where
• Users give feedback/acceptance
• Minor problems are identified and fixed
• The project is implemented on time
• High fives, all around
4
IN REALITY…
• UAT is one of the most risky and
explosive levels of testing.
• UAT is greatly needed, but happens at
the worst time to find major defects –
at the end of the project.
• Users may be unfriendly to the new
system
• They like the current one just fine,
thank you.
• Much of your UAT planning may be
ignored.
• People tend to underestimate how
many cycles of regression testing are
needed.

5. 4/26/2014
3
5
THERE ARE MANY
QUESTIONS ABOUT UAT
• Who plans it?
• Who performs it?
• Should it only be manual in nature?
• What is the basis for test design and
evaluation?
• When should it be performed?
• Where should it be performed?
• Who leads it?
• How much weight should be given to it?
6
PROJECT #1
• Medical laboratory testing business
that closely resembles a manufacturing
environment.
• New Technology for the company and
for the industry.
• The previous project had failed
• The company almost went out of
business because of it!
• Very high growth in terms of both
business and employees.
• Company at risk of failure.
• This project was truly mission-critical.

6. 4/26/2014
4
7
PROJECT #1 – CONT’D.
• Few functional requirements.
• 8 pages for over 400 modules
• Test team had little knowledge of:
• subject matter,
• test design,
• or testing at all.
• Very little unit or integration testing
being done by developers.
• Some system testing was done.
• UAT was the focus.
8
DEFECT DISCOVERY
AND BACKLOG
System Test UAT 1st Deploy
2nd Deploy
3rd Deploy
4 weeks 3 weeks

7. 4/26/2014
5
9
PROJECT #1 RESULTS
• Very high defect levels in testing.
• Many were resolved before implementation.
• Severe performance problems.
• Old system could process 8,000 units/day
• New system could process 400 units/day
• Many problems due to the new technology being used
• “bleeding edge” issues
• “Deadline or else” attitude
• The business was under extreme pressure to deploy due to
increased processing volumes.
• System was de-installed/re-installed 3 times before
performance was acceptable to deploy.
10
WHAT WE LEARNED
• Requirements are important.
• Even if you have to create some form of
them after the software has been
written.
• Early testing is important.
• That would have caught early
performance bottlenecks.
• Teamwork is critical.
• Things got so bad we had to have a “do
over.”
• The deadline is a bad criteria for
implementation.
• Always have a “Plan B”.

8. 4/26/2014
6
11
UAT LESSONS
• Build good relationships with subject
matter experts.
• They often determine acceptance
• Listen to the end-users.
• Understand what’s important
• Don’t rely on UAT for defect detection.
• Interesting factoid
• A similar project with the exact same
technology failed due to performance
errors two years later for a city water
utility. $1.8 million lawsuit lost by the
vendor.
12
PROJECT #2
• Same company as before, but two
years later
• Integration of a vendor-developed
and customized accounting
system
• Lots of defects in the vendor
system
• Implemented two months late with
practically zero defects.

9. 4/26/2014
7
13
WHAT MADE THE
DIFFERENCE?
• Same people – testers, IT manager, developer
• Different project manager who was a big
supporter of testing
• More experience with the technology
• Better understanding of testing and test design
• A repeatable process
• Less pressure to implement
• Having a contingency plan
• Having the courage to delay deployment in favor
of quality.
• The financials had to be right.
14
PROJECT #3
• New government-sponsored entity.
• Everything was new – building, people,
systems
• System was a vendor-developed
workers compensation system.
• Some customization
• Little documentation
• Designed all tests based on business
scenarios.
• We had no idea of the UI design.

10. 4/26/2014
8
15
KEY FACTORS
• No end-users in place at first to help with
any UAT planning.
• In fact, we had to train the end-users in
the system and the business.
• Lots of test planning was involved
• 50% or more effort in planning and
optimizing tests.
• This paid off big in test execution and
training
16
RESULTS
• Tested 700 modules with 250 business
scenario tests.
• We had designed over 300 tests
• The management and test team felt
confident after 250 tests we had covered
enough of the system.
• Found many defects in a system that
had been in use in other companies for
years.
• Reused a lot of the testware as training
aids.
• Successful launch of the organization
and system.

11. 4/26/2014
9
17
HARD LESSONS
LEARNED
• “You don’t know what you don’t know”
AND “You sometimes don’t know what you
think you know.”
• Newly hired SME with over 30 years
workers comp experience provided
information that was different (correct)
than what we had been told during test
design.
• We had to assign two people for two weeks
to create new tests.
• These were complex financial functions –
we couldn’t make it up on the fly.
18
HARD LESSONS
LEARNED (2)
• Real users are needed for UAT.
• Sometimes the heavy lifting of test
design may be done by other testers,
but users need heavy involvement.

12. 4/26/2014
10
19
PROJECT #4
• State government, Legal application
• Vendor-developed and customized
• Highly complex system purchased to replace two co-
existing systems.
• Half of the counties in the state used one system, the other
half used another.
• Usability factors were low on the new system
• Data conversion correctness was critical
20
THE GOOD SIDE
• Well-defined project processes
• Highly engaged management and
stakeholders
• Good project planning and tracking
• Incremental implementation strategy
• The entire system was implemented,
only one county at a time.
• Heavy system testing
• Good team attitude

13. 4/26/2014
11
21
THE CHALLENGES
• The system’s learning curve was very high.
• The key stakeholders set a high bar for acceptance.
• The actual users were few in number and were only able to
perform a few of the planned tests.
• Very high defect levels.
22
LEADING UP TO
VENDOR SELECTION
• Over 2 years of meeting with users and
stakeholders to determine business
needs.
• Included:
• JAD sessions
• Creation of “as-is” and “to-be” use
cases
• Set of acceptance criteria
(approximately 350 acceptance criteria
items)

14. 4/26/2014
12
23
THE STRATEGY
• Create test scenarios that described
the trail to follow in testing a task,
but not to the level of keystrokes.
• Based on use cases.
• The problem turned out to be that
even the BAs and trainers had
difficulty in performing the scenarios.
• System complexity was high.
• Training had not been conducted.
• Usability was low
24
DEFECT DISCOVERY
AND BACKLOG
System Test UAT 1st Deploy
10 weeks 4 weeks
750
250

15. 4/26/2014
13
25
WHAT WAS
VALIDATED
• The precise “point and click” scripts provided
by the vendor were long and difficult to
perform.
• Each one took days.
• Plus, there were errors in the scripts and
differences between what the script indicated
and what the system did.
26
THE BIG SURPRISES
• We planned the system test to be a practice run for UAT.
• It turned out to be the most productive phase of testing in
terms of finding defects.
• We planned for a 10 week UAT effort with 10 users
• It turned out to be a 2 week effort with 4 users.
• First sense of trouble: initial users were exhausted after 3
days of a pre-test effort.

16. 4/26/2014
14
27
THE BIG SURPRISES (2)
• We used none of the planned tests (around 350 scenarios)
in UAT.
• Instead, it was a guided “happy path” walkthrough, noting
problems along the way.
• Defects were found, but the majority of defects had been
found in system test.
28
LESSONS LEARNED
• The early system test was invaluable in
finding defects.
• Learning the system is critical for users in
new systems before they are able to test it.
• The test documentation is not enough to
provide context of how the system works.
• It took a lot of flexibility on the part of
everyone (client, vendor, testers, users,
stakeholders) to make it to the first
implementation.
• Sometimes actual users just aren’t able to
perform a rigorous test.

17. 4/26/2014
15
29
WHAT CAN WE LEARN FROM
ALL THESE PROJECTS?
• UAT is a much-needed test, but happens at the worst
possible time – just before implementation.
• You can take some of the late defect impact away with
system testing and reviews.
• You can lessen the risk of deployment by implementing to
a smaller and lower risk user base first.
• Actual end-users are good for performing UAT, but much
depends on what you are testing and the capabilities of
the users.
• The reality is the users are going to have to use the system
in real-life anyway.
• However, not all users are good testers!
30
WHAT CAN WE
LEARN? (2)
• Be careful how much time and effort
you invest in planning for UAT before
the capabilities are truly known.
• That is, senior management may want
actual users to test for 8 weeks, but if
the people aren’t available or can’t
handle the load, then it probably isn’t
going to happen.
• Don’t place all the weight of testing on
UAT.
• In project #4 our system testing found
a majority of the defects.

18. 4/26/2014
16
31
WHAT CAN WE
LEARN? (3)
• UAT test planning isn’t bad, just expect
changes.
• People, software, business, timelines –
they all change.
• Try to optimize and prioritize.
• Example: If you have 500 points of
acceptance criteria, can they be
validated with 200 tests?
• Which of the acceptance criteria are
critical, needed and “nice to have”?
32

19. 4/26/2014
17
33
BIO - RANDALL W. RICE
• Over 35 years experience in building and testing
information systems in a variety of industries
and technical environments
• ASTQB Certified Tester – Foundation level,
Advanced level (Full)
• Director, American Software Testing Qualification
Board (ASTQB)
• Chairperson, 1995 - 2000 QAI’’’’s annual software
testing conference
• Co-author with William E. Perry, Surviving the
Top Ten Challenges of Software Testing and
Testing Dirty Systems
• Principal Consultant and Trainer, Rice
Consulting Services, Inc.
34
CONTACT INFORMATION
Randall W. Rice, CTAL
Rice Consulting Services, Inc.
P.O. Box 892003
Oklahoma City, OK 73170
Ph: 405-691-8075
Fax: 405-691-1441
Web site: www.riceconsulting.com
e-mail: rrice@riceconsulting.com