The document provided will describe our research group’s findings pertaining to our usability test of a newly created iteration of the IAKM public facing web site. The testing completed was performed utilizing the ClearView eye tracking software to accurately monitor the user’s interaction on multiple dimensions. The facility allowed us to work directly with the user, and monitor interaction via click tracking, eye movement, audio and video. Combining those attributes, based on a time scale, our research team was able to more accurately identify where the site changes were successful and where they were not.

1. iakm-dev2.slis.kent.edu/
Usability Testing
Cathy Carr
Holly C. Murphy
Matthew Pahls
December 8, 2009

2. Table of Contents
Statement of Work . . . . . . . . . . . . . . 3
Characterizations of Tasks . . . . . . . . . . 4-5
Data Collection . . . . . . . . . . . . . . . 6
Characterization of Participants. . . . . . . . . 7
Conclusions. . . . . . . . . . . . . . . . 8
Recommendations for Redesign . . . . . . . . 9
Raw Data. . . . . . . . . . . . . . . . 10-22
Page 2

3. Statement of Work
The document provided will describe our research group’s findings pertaining to our usability test of
a newly created iteration of the IAKM public facing web site. The testing completed was performed
utilizing the ClearView eye tracking software to accurately monitor the user’s interaction on multiple
dimensions. The facility allowed us to work directly with the user, and monitor interaction via click
tracking, eye movement, audio and video. Combining those attributes, based on a time scale, our
research team was able to more accurately identify where the site changes were successful and where
they were not.
The Information Architecture and Knowledge Management web site, like many other programs in a
large university’s environment, utilizes its web presence to interact with a worldly audience. Commonly
the web site is first and foremost, a first impression for a prospective student. So having a strong web
presence is paramount to sustaining positive enrollment and retention. The IAKM program at Kent
State University roots itself in a rapidly evolving field that has strong ties to the web. As the field
establishes itself, IAKM must be able to accurately portray the field in a manor such that people with
little preexisting knowledge could identify with it.
The site recently underwent some structural changes, and our team was tasked with conducting a
full usability study on the product. Since the changes are not dramatic, we needed to know if these
slight alterations would have a positive or negative impact on the overall experience. The hierarchical
changes were based around the main navigation. The intent of the changes was to improve the overall
findability of information within the site. Our group’s investigation would prove or disprove the
effectiveness of the changes.
The IAKM office tasked our team with focusing our efforts on two particular audience segments; the
prospective student and the current student. We used a short screener developed in Qualtrics to identify
our four user participants that fit either of the two audience segments. From there, we had each of the
users come in to the usability lab individually and we conducted a series of task based analysis tests.
These tests were targeted at particular tasks that the audience segments should have the capabilities of
easily performing. We quickly discovered that the tasks we perceived as relatively simple turned out to
be rather difficult, and the users were having issues completing the tasks.
Page 3

4. Characterization of Tasks
Two sets of tasks were developed to test the site with two sets of users. The first set of tasks tested the
site for prospective students. These tasks were standard site activities that a user researching or preparing
for enrollment would conduct. The second set of tasks tested the site for current students. These tasks
were intended for students already in the program. The activities supported navigating the site for
content that the student would need during the program cycle.
Prospective IAKM student
Task 1: You are a prospective KM certificate student. What are the core-required courses?
Rationale: This is a basic task that a student would perform to locate information vital to
proceeding within the program.
Task 2: You are a prospective UXD master’s student. How would you apply to the program?
Rationale: For many students this may be their first experience with the site. It is imperative
that this task be easy to find and perform.
Task 3: You are interested in a graduate assistantship. Find information to apply for this position.
Rationale: All prospective students may not attempt this task. But, it is an important item for
graduate schools that should be easily located on their sites.
Current IAKM student
Task 1: You are interested in a graduate assistantship. Find information to apply for this position.
Rationale: Again, this may not be a task attempted by all current students. But, it is possible
for current students to become graduate assistants.
Task 2: It is time to schedule for the Spring 2010 semester. Use your current student standing and
find offered classes using the IAKM site.
Rationale: The IAKM site lists classes offered for every semester. This is very helpful for
students when preparing for registration.
Task 3: You are approaching graduation and beginning to think about the master’s project. Find
information that will help you.
Rationale: Every student will need to locate this information, not only to begin the project,
but also to submit the appropriate forms.
Page 4

5. Task 4: You wish to take a workshop next semester. Find the form you will need to ensure you
receive credit for the workshop.
Rationale: This specific task may not be performed by every student. The ease of locating this
particular information assures that the user is able to navigate to the forms page of the site,
where other vital program information is located.
Task Administration
The testing took place in the Kent State University Usability Lab. The above tasks were typed, printed,
and read by the facilitator to each participant. The facilitator either sat or stood to the side of the
participant while the session was in progress. Each task began from the home page. The participant was
able to ask for clarification or restatement of the task. Some guidance was given if the participant strayed
off track. The participant was asked to think-aloud during the tasks. ClearView was used to capture
audio, video, and eye tracking. These recordings were analyzed at a later time for user comments,
difficulties, and successes.
Page 5

6. Data Collection
Information was collected from our selected participants on the dimensions of auditory, visually, and
click tracking. Each axis is powerful in its own right, but combining them and being able to analyze
them collaboratively allows us to produce truly accurate analysis of the participant. While tasks were
in progress, we elicited verbal feedback so we could record what the user was thinking. As they were
talking out the tasks we were able to understand why the user was doing the task and seeing why they
had their own understanding on how the site should function. The lab is also equipped with a computer
monitor that has the capability to track eye movements (“gaze”). These recordings are accomplished by
having the monitor use an infrared signal projected onto the back on the participants eye and the signal
then reflects off the eye and recorded at that point. As the user interacts with the site, the monitor tracks
the movement and combines that information with the current page of interaction. This allows us to see
exactly what the user us looking at in real time. The recorded instances of the visual attribute can be
exported to two rather interesting and rather valuable outputs. Gaze plots and heat maps allow us to take
the visual recording and map them to the site so we can visually see where the user focused and how
long they focused for.
Now we add the fourth and final component to our recording attributes and this is time. Combined
with the auditory, visual, and click actions, we can accurately track all user interaction in retrospect to
the tasks assigned to the user throughout the entire test. These elements empower the research team to
effectively capture the data necessary for proper data analysis, and more importantly see how effective
the site alterations are on our target user segments.
Page 6

7. Characterization of Participants
Not all of the study participants fit the user profiles established in the requirements analysis. The
established profiles are as follows:
Prospective IAKM student
• Age: 26 – 40
• Sex: Female
• State: Ohio
• Visual impairment: None
• Is not involved with the program
• Has not used the current IAKM site
• Is looking for more information about the IAKM program
• Computer experience: average to above average
• Uses a PC every day
Current IAKM student
• Age: 26 – 40
• Sex: Male
• State: Ohio
• Visual impairment: None
• Is involved in the program
• Learned of the program from a former or current student
• Computer experience: above average to expert
• Uses a PC every day
For the study, both a male and female prospective student were selected to participate. The female wore
glasses. One of the current student participants also wore glasses. The other current student uses a Mac.
Page 7

8. Conclusions
• Vocabulary confusion. Two of the participants were confused with the site terminology. More
interesting is that these were the current student participants that have used the previous IAKM
site.
• Student standing. The participants that were chosen as prospective IAKM students consider
themselves as current students because they are already enrolled in KSU.
• GA task. All four participants were given the task of locating information to apply for a graduate
assistantship. Each individual chose a different path to locate this information, and verbalized a
different rationale for their path.
• Approach. The two current students performed very different approaches to locate the task
information. One performed the hunt and peck method, while the other took a few seconds to
plot out a course of action. The prospective students conducted trial and error approaches.
• Searching Issues. When the users tried to perform a search the results were somewhat
inaccurate.
• Using a Crutch. When users are confused they turned to the FAQ as a crutch for locating the
information. This proves the user is becoming quickly frustrated with the site and tries to find
the quickest path to the proper knowledge. All users tested were personally vested in finding
the information because they were being tested on it. The general user might not have that
investment in finding the information so they will tend to use the path of least resistance.
Page 8

9. Recommendations for Redesign
• Evaluate the navigation labels and organization. This was a stumbling block for the participants.
The user had to decide where he or she thought the information was located and was incorrect
several times. We recommend performing a card sorting exercise on the labels that are already
defined, because the information that is present in the site is accurate, it just lacks proper
organization.
• Relocate the graduate assistantship information. This task was very insightful from both the
current and prospective student perspective. This information may warrant multiple locations
within the site.
• More prominent display for end of study requirements. The Master’s Project information was
located by two different methods. When it was located, it was not where the prospective student
would have looked, under forms.
• Improve the accuracy of the search. Improving the search will elicit a quick fix. The new search
will allow the sites user population to find information while they become accommodated to any
heuristic alterations.
Page 9

10. Raw Data:
Each task and user is broken down into significant actions. The actions are then grouped into navigation
paths, with each path illustrated below the table.
User 1: Current IAKM student
User 1/Task 1:
Illustration User 1/Task 1
Begin Task
Looks at side menu
1.1
Clicks on Programs
1.2
Clicks on Current Students
Clicks on Employment Opportunities
1.3
Clicks on FAQs Advising
1.4
Tries Search
Found target
Total task time
User 1/Task 1
Time
:35
1:03
1:22
1:46
2:57
3:27
2:53
IAKM Home Page
Side Menu
1.4 Search
Target
1.1 Programs
1.2 Current Students
1.3 FAQs
Employment
Opportunities
Advising
Page 10

11. User 1/Task 2:
Illustration User 1/Task 2
none
Clicked on Courses and found target
4:30
:05
User 1/Task 3:
Illustration User 1/Task 3
3.1
3.2
Looks at side menu
Looks at concentrations in center of
page and clicks
Found target
User 1/Task 3
5:05
5:22
5:30
:25
IAKM Home Page
3.1 Side Menu
3,2 UXD Concentration
Master’s Program link
top center of
home page
Target
Page 11

12. User 1/Task4:
Illustration User 1/Task 4
Begin Task
4.1
Clicks on Courses
Looks at side menu. Subject seems
lost and says, “why don’t they have a
button labeled forms?”
4.2
Goes to search
Clicks on forms
Found target
User 1/Task 4
Time
6:20
6:30
6:50
7:20
7:35
7:50
1:30
IAKM Home Page
Side Menu
4.1 Courses
4.2 Search
Target
Page 12

13. User 2: Current IAKM student
User 2/Task 1:
User 2/Task 1
Illustration User 2/Task 1
Begin Task
1.1
Looked at side menu
Clicked on Prospective Students
Looked at sub-menu
Clicked on Financial Aid
Found target
Total time
Time
:17
:29
:36
:38
:41
:24
IAKM Home Page
1.1 Side Menu
Prospective Students
Financial Aid
Target
Page 13

14. User 2/Task 2:
Illustration User 2/Task 2
Begin Task
2.1
Looked at side menu
Clicked on Courses
Clicked on Schedules
Found target
Total time
User 2/Task 2
Time
1:26
1:30
1:43
:17
IAKM Home Page
2.1 Side Menu
Courses
Schedules
Target
Page 14

15. User 2/ Task 3
Illustration User 2/Task 3
Begin Task
Looked at side menu
3.1
Clicked on Programs
3.2
Clicked on Courses
3.3
Clicked on Current Student
Clicked on Forms
Found Target
Total time
User 2/Task 3
Time
2:30
2:35
2:37
2:46
3:01
3:18
:48
IAKM Home Page
Side Menu
3.1 Programs
3.2 Courses
3.3 Current Students
Forms
Target
Page 15

16. User 2/ Task 4:
User 2/Task 4
Illustration User 2/Task 4
Begin Task
Looked at side menu
3.1
Clicked on Current Students
3.2
Clicked on Forms
Found Target
Total time
Time
4:10
4:12
4:17
:07
IAKM Home Page
1.1 Side Menu
Current Students
Forms
Target
Page 16

17. User 3: Prospective IAKM student
User 3/Task 1:
Illustration User 3/Task 1
Begin Task
Looked at center of page
3.1
Clicked on KM Certificate
Found Target
Total time
User 3/Task 1
Time
5:04
5:07
5:15
:11
IAKM Home Page
1.1 KM Concentration
Certificate Program link
top center of
home page
Target
Page 17

18. User 3/Task 2:
Illustration User 3/Task 2
Begin Task
Looked at center of page
2.1
Clicked on UXD Master’s link
2.2
Looked at side menu
Clicked on Apply Now
Found target
Total time
Time
5:45
5:48
5:55
6:00
:15
User 2/Task 3
IAKM Home Page
2.1 UXD Certificate
Program link
top center of
home page
2.2 Apply Now
Target
Page 18

19. User 3/Task 3:
Illustration User 3/Task 3
Begin Task
Looked at side menu
3.1
Clicked on Current Students
Time
7:02
3.2
7:49
Clicked on Employment
Opportunities
Looked at Side Menu
Clicked FAQs
Clicked on Financial Aid
Found target
Total time
User 2/Task 3
7:15
7:34
7:52
:50
IAKM Home Page
Side Menu
3.1 Current Students
Employment
Opportunities
3.2 FAQs
Financial Aid
Target
Page 19

20. User 4: Prospective IAKM student
User 4/Task 1:
Illustration User 4/Task 1
Begin Task
Looked at side menu
1.1
Clicked on Courses
Time
4:25
4:29
4:40
1.2
1.3
1.4
5:05
User 4/Task 1
Clicked on Catalog
Clicked on Prospective Students
Programs
Looks at concentrations in center of
page and clicks on KM
Found target
Total time
4:50
:40
IAKM Home Page
Side Menu
1.1 Courses
Catalog
1.2 Prospective
Students
1.2 Prospective
Students
1.3 KM Concntration
Certificate Program
link top center of
home page
Target
Page 20

21. User 4/Task 2:
Illustration User 4/Task 2
Begin Task
2.1
In Courses sub menu
Clicked on Curriculum in page
Found target
Total time
User 4/Task 2
Time
5:55
6:03
:08
IAKM Home Page
Side Menu
2.1 Courses
Clicked on KM
curriculum on page
Target
Page 21

22. User 4/Task 3:
Illustration User 4/Task 3
Begin Task
Looked at side menu
3.1
Clicked Alumni and Professionals
Time
3.3
9:14
9:23
9:32
1:42
3.2
Clicked on Current Students
Clicked on Employment
Opportunities
Clicked on Prospective Students
Clicked on Financial Aid
Found target
Total time
User 4/Task 3
7:50
7:58
8:17
8:19
IAKM Home Page
Side Menu
3.1 Alumni and
Professionals
3.3 Prospective
Students
3.2 Current Students
Forms
Employment
Opportunities
Target
Page 22