The most successful Web sites are those that put the user's needs, rather than the organization's, at the center of its design. But without User Research and Usability Testing, how can you have a user-centered design? In this presentation to the Potomac Forum, Director of User Research, Toral Contractor, and Information Architect, Kirsten Miller, outline the following major methods of user research and usability testing, providing case studies for each method with tips and important discoveries. And help you understand how each method fits in to the project lifecycle.

1. User Researchofand Experience
Architects the User
Usability Testing:
Techniques and Tips for Optimizing Government Web Sites
Toral Contractor, Director of User Research
Kirsten Miller, Information Architect
Presented to Potomac Forum Symposium
Building Better Government Web Sites: From Strategy to Fulfillment
March 30, 2010

2. Potomac Forum | March 30, 2010
Agenda
User Experience Design
User Research
• Methods
• Case Studies & Tips
Usability Testing
• Methods
• Case Studies & Tips
Project Lifecycle
Wrap-Up
NavigationArts, LLC 2

3. Potomac Forum | March 30, 2010
User Experience Design
NavigationArts, LLC 3

4. User Experience Design Potomac Forum | March 30, 2010
User Experience (UX) Design
User-centered design, which puts users rather than organizational or system-related
considerations at the center of the design process, is the foundation of our UX
practice.
Information Architecture
• Structure information and functionality for easy access
• Design effective navigational systems and taxonomies
Interaction Design
• Design complex online operations so they are intuitive for users
• Establish clear priorities for user action
Creative Interface Design
• Create an engaging, attractive and brand-building visual interface
• Create a visual hierarchy to facilitate browsing and scannability
NavigationArts, LLC 4

5. User Experience Design Potomac Forum | March 30, 2010
UX Design Inputs
Usability Expertise
• Expert heuristic assessment
Business Goals
• Stakeholder interviews
Our focus today: User Research
• Direct vs. indirect
Usability Testing
• Live site vs. prototype
• Moderated vs. unmoderated
• In-person vs. remote
NavigationArts, LLC 5

6. Potomac Forum | March 30, 2010
User Research
NavigationArts, LLC 6

7. User Research Potomac Forum | March 30, 2010
Direct Methods
Contextual inquiry In a nutshell:
• Interview and observe users in their own environment as
they use the site and perform relevant tasks What do
Intercept interviews YOU think?
• “Hi. Would you like to help us improve your web site by
answering a few questions?”
Focus groups
• Structured group interviews used to collect opinions and
insights into user needs and perceptions
User surveys
• Collect a range of qualitative and quantitative data on user
demographics, behavior and opinions
Card sorts
• Users sort, group and label site topics
NavigationArts, LLC 7

8. User Research: Direct Methods Potomac Forum | March 30, 2010
Case Study: Contextual Inquiry
Department of State – Bureau of Consular Affairs
Intranet redesign
• Shadowed employees at work
• Inventoried common user tasks both on and off web site
• Asked about browser bookmarks and paper documents
kept close at hand
• Observed ways of browsing and using information
• Inventoried likes and dislikes about current site, wishlist
for redesign
Benefits:
• Understand the environment they work in. Are they
under a lot of stress while working? Do they have a
short amount of time to search for something?
• Opportunity to observe documents they have printed out
at their desks for easy access and reference for
common tasks.
DEVELOPMENT
UX DESIGN DEPLOYMENT SUPPORT
INTEGRATION
NavigationArts, LLC 8

9. Potomac Forum | March 30, 2010
Contextual Inquiry Tip:
Pay Attention to the Sticky Notes!
NavigationArts, LLC 9

10. User Research: Direct Methods Potomac Forum | March 30, 2010
Case Studies: Intercept Interviews
M.D. Anderson Cancer Center
• Interviewed patients and family members in
the Learning Center who were seeking
information or passing through on their way to
New Patient Orientation.
• Discovered two modes: Research mode (e.g.,
seeking information about a disease, the
organization, or a doctor) and Task mode (e.g.,
quickly print a map or schedule).
George Washington University
• Asked students about their primary reasons for
applying, attending or leaving GW
• Gained understanding of current student
perspectives and expectations of GW
Benefits
• Quick and economical way to sample a target
population
DEVELOPMENT
UX DESIGN DEPLOYMENT SUPPORT
INTEGRATION
NavigationArts, LLC 10

11. Potomac Forum | March 30, 2010
NavigationArts, LLC 11

12. User Research: Direct Methods Potomac Forum | March 30, 2010
Case Study: Focus Group
Nemours
• Interviewed parents of child patients
• Learned of two main types of needs with patient
families: Long-term (when a child has cancer),
and Short-term (when a child has a minor
condition).
• Learned that families weren’t expecting the
hospital to provide information about diseases,
but did want hospital-recommended or approved
sources of information.
Benefits
• Able to interview a large group at once for their
opinions, evaluation or formulation of a concept.
• Group discussion triggers conversations and
topics that may not occur in one-on-one
interviews.
• Pick up on common user language, terminology
DEVELOPMENT
UX DESIGN DEPLOYMENT SUPPORT
INTEGRATION
NavigationArts, LLC 12

13. Potomac Forum | March 30, 2010
Focus Group Tip:
Get written feedback from
participants before starting
group discussion.
NavigationArts, LLC 13

14. User Research: Direct Methods Potomac Forum | March 30, 2010
Case Study: User Survey
Department of State – Bureau of Consular Affairs
• ForeSee Survey implemented
• Questions about user roles, goals, satisfaction,
problems encountered, and suggestions for
improvement
• ACSI model questions allow customer satisfaction
benchmarking against other ForeSee clients
Benefits
• Large sample of quantitative data and qualitative
feedback to guide and support design decisions
• Customer segmentation helps refine and prioritize user
profiles
NavigationArts, LLC 14

15. Potomac Forum | March 30, 2010
User Survey Tips:
• Form questions around user tasks
• Use survey logic to ask targeted
questions to various audiences
• Include some open-ended
questions
NavigationArts, LLC 15

16. User Research: Direct Methods Potomac Forum | March 30, 2010
Case Study: Card Sort
The George Washington University
• Open sort: High school students and
prospective undergraduate students grouped
and labeled undergraduate majors
• Closed sort: Freshman evaluated main
navigation of the website
Nemours
• Open sort: Groups of patient families
categorized Nemours content to build the main
navigation.
• Learned that all the participants wanted a “My
Account” area for easy access to payment
options, history of patient activity, appointment
information, and an easy way to communicate
with the doctor.
Benefits
• Formulate and evaluate navigation that aligns to
users’ mental model of the information
DEVELOPMENT
UX DESIGN DEPLOYMENT SUPPORT
INTEGRATION
NavigationArts, LLC 16

17. Potomac Forum | March 30, 2010
Card Sort Tips:
• Conduct open card sort first
• Then conduct closed sort to
validate design
NavigationArts, LLC 17

18. User Research Potomac Forum | March 30, 2010
Indirect Methods
Web Analytics In a nutshell:
• Analyze site traffic and user behavior
Call center interviews and log analysis
What are
• Interview call center agents and review call THEY doing?
logs, if available
Social media sites
• Explore groups that form around common user
interests; review user behavior on your own
social media sites/pages
• Can also use for direct user research
NavigationArts, LLC 18

19. User Research: Indirect Methods Potomac Forum | March 30, 2010
Case Study: Web Analytics
Department of State – Bureau of Consular Affairs
intranet redesign
• Number of visitors
• Most popular pages and directories
• Most popular entry and exit pages
• Top referrers/inlinks
• Most popular search terms
• Time on site
• Pages per visit
Benefits
• Understand user behaviors and priorities
• Set success metrics/benchmarks to evaluate post-
launch
• Identify areas for future qualitative research
DEVELOPMENT
UX DESIGN DEPLOYMENT SUPPORT
INTEGRATION
NavigationArts, LLC 19

20. Potomac Forum | March 30, 2010
Web Analytics Tip:
Look at exit pages for areas to focus on for
further qualitative research
NavigationArts, LLC 20

21. User Research: Indirect Methods Potomac Forum | March 30, 2010
Case Study: Call Center Analytics
NHTSA
• Interviewed call center personnel about
common questions and emails from the public
• Gained insight into customer needs not
currently being met by the web site
Benefits
• Call center agents are the front line for direct
interaction with the public
• Agents often walk customers through the web
site and know where the pain points are
DEVELOPMENT
UX DESIGN DEPLOYMENT SUPPORT
INTEGRATION
NavigationArts, LLC 21

22. Potomac Forum | March 30, 2010
Call Center Analytics Tip:
Ask agents about browser
bookmarks and FAQ lists.
NavigationArts, LLC 22

23. User Research: Direct & Indirect Methods Potomac Forum | March 30, 2010
Case Studies: Social Media Sites
Indirect: Council of Residential Specialists
• Networking is very important to realtors and
therefore, social networking sites was a key source
for information
• Researched social networking sites to see what
users are discussing, gain insight into their needs
and interests
Direct: Smithsonian
• Help imagine a completely new and more engaging
way for people from around the world to interact with
the Smithsonian Institute online
• Used YouTube to solicit feedback in forms of videos
to understand what users wanted to see from
Smithsonian
DEVELOPMENT
UX DESIGN DEPLOYMENT SUPPORT
INTEGRATION
NavigationArts, LLC 23

24. Potomac Forum | March 30, 2010
Social Media Tip:
Sign up for Google alerts so
you know what people are
saying online about your
organization.
NavigationArts, LLC 24

25. Potomac Forum | March 30, 2010
Usability Testing
NavigationArts, LLC 25

26. Usability Testing Potomac Forum | March 30, 2010
Usability Testing Techniques
Live site vs. Prototype testing
• Live site: Test participants interact with currently existing site
• Prototype: Test participants interact with proposed design; can be
low or high fidelity, paper or electronic
Moderated vs. Unmoderated
• Moderated: Test participants interact with system/prototype and
with researcher/facilitator
• Unmoderated: Test participants interact only with Mix and match!
system/prototype; researcher may observe behavior and/or
analyze data collected by system
Remote vs. In-person
• Remote: Test participant and researcher/facilitator in separate
physical locations
• In-person: Test participant and researcher/facilitator in same
physical location
NavigationArts, LLC 26

27. Usability Testing Potomac Forum | March 30, 2010
Case Study: Unmoderated, Remote, Prototype Test
Department of State – travel.state.gov
• Testing navigation menu prototypes using an
online heatmap generator
Benefits
• Participants complete test in their own
environment and whenever they have time
• No need for travel or appointments
• Inexpensive
• Many tools available now for creating remote,
unmoderated usability tests and analyzing
data generated
PLANNING & DEVELOPMENT
DEPLOYMENT SUPPORT
DISCOVERY INTEGRATION
NavigationArts, LLC 27

28. Usability Testing Potomac Forum | March 30, 2010
Case Studies: Moderated, Live Site Test
Remote: World Bank
• Tested search feature on live public site with
domestic and international users within the
international development community and media
• Conducted free exploration test in which users
formulated their own scenarios true to the way they
use the site
In-person: Department of State - travel.state.gov
• Set specific tasks with correct and incorrect
outcomes; scored and analyzed results
Benefits
• Directly observe user behavior on the site; identify
trouble spots
• Draw out user opinions and insights about
performing tasks while they are doing it
DEVELOPMENT
UX DESIGN DEPLOYMENT SUPPORT
INTEGRATION
NavigationArts, LLC 28

29. Potomac Forum | March 30, 2010
Moderated Live Site Test Tip:
International users?
Plan to work odd hours!
NavigationArts, LLC 29

30. Usability Testing Potomac Forum | March 30, 2010
Case Study: Moderated, Remote, Prototype Test
KidsHealth.org
• Tested a low-fidelity, clickable prototype with
children, teens, parents and teachers to
evaluate design at wireframe stage
Benefits
• Users test in their environment or the context
in which they would use the site
• No need to travel
• Lower participant incentives
• Many tools on the market now to facilitate
remote user testing
PLANNING & DEVELOPMENT
DEPLOYMENT SUPPORT
DISCOVERY INTEGRATION
NavigationArts, LLC 30

31. Potomac Forum | March 30, 2010
Moderated
Remote Test Tip:
Be prepared for
distractions
NavigationArts, LLC 31

32. Potomac Forum | March 30, 2010
Project Lifecycle
NavigationArts, LLC 32

33. Project Lifecycle Potomac Forum | March 30, 2010
When?
User Research
Contextual inquiry
Intercept interviews
Focus groups
User surveys
Card sorts
Web analytics
Call center interviews, log analysis
Social media sites
Usability Testing
Live site testing
Prototype testing
NavigationArts, LLC 33

34. Project Lifecycle Potomac Forum | March 30, 2010
When?
User Research
Contextual inquiry 
Intercept interviews 
Focus groups 
User surveys     
Card sorts  
Web analytics     
Call center interviews, log analysis  
Social media sites  
Usability Testing
Live site testing  
Prototype testing  
NavigationArts, LLC 34

35. User Researchofand Experience
Architects the User
Usability Testing:
Questions?
Toral Contractor, Director of User Research
Kirsten Miller, Information Architect
Presented to Potomac Forum Symposium
Building Better Government Web Sites: From Strategy to Fulfillment
March 30, 2010