Methods and results in a small scale usability study of a university website.

1. Getting Them There: A Small-Scale
Usability Test of a University Library
Website
Jason C. Simon
Technology & Serials Librarian
Fitchburg State University
jsimon6@fitchburgstate.edu
Twitter: @jason_c_simon

2. Prior Situation

2011: Old text-heavy site needed a redesign

University introduces new CMS requiring all
university sites to migrate in a short time-frame.

Quickly discovered that all advanced code/includes
would be stripped

Some discussion amongst librarians over layout
of content, but time was limited.

3. Old old site

4. Migration to CMS

5. Need to Redesign

Consensus existed in the library that there were problems

Needed to do some testing before making changes

Identified areas of focus:

What the library provides:
− Services
− Resources

What Students want/need:
− How do I...?

6. Testing Models

Some standardized question areas were identified based on
prior research (Letnikova, 2008)

Format Type:

Form for user to fill out
− Pros: no user anxiety
− Cons: problem with accuracy of data (users may treat it
like test, and think they have to “get it right”)

Observation
− Pros:

Can observe actions

Can provide clear instructions
− Cons: user might be nervous with observer

IRB Approval needed

7. Test Model: Observation

Test subject seated at computer

Monitor shows mouse movement of the test subject

Two observers enables one to catch things that another might miss.

8. Sample

Who to use

Student Workers

Rationale: point of study is to identify initial problems of layout. These
students are theoretically knowledgeable.
− If they have trouble → clear indication of problems

Sample size

Wanted 10. Used 7.

While larger studies may be more granular (VandeCreek, 2005) Small
numbers can work for this type of test.

Testing one user is better than none.

9. Nielsen model
“where N is the total number of usability problems in the design and L is the proportion of
usability problems discovered while testing a single user. The typical value of L is
31%, averaged across a large number of projects we studied. Plotting the curve for L
=31% gives the following result:” (Nielsen, 2000).

10. Instrument

Test required no name, a number was given to each test
subject.

Each observer had the associated number on their result form

Instructions

Read aloud to subject

Demographics

User fills out

Content:

Action statements, not questions.

Results observed and recorded by observers

User-told to indicate when done with each action (or told to move on if it was
clear that a solution was not being found)

11. Sample Questions

Find if the library has a copy of Huckleberry Finn by Mark
Twain.
− Find whether it is checked out or not.
− Find instructions on how to locate it on the shelves

How can you find scholarly articles on humor, child
development and education?
− Assume you have been unable to find anything satisfactory. Find a guide or
a person to help you in your research

Find out if the library has the item referenced in the citation
below:
− Dozois, D. A., Martin, R. A., & Faulkner, B. (2013). Early maladaptive
schemas, styles of humor and aggression. Humor: International Journal Of
Humor Research, 26(1), 97-116. doi:10.1515/humor-2013-0006

12. Analysis

Since the sample was small, mostly qualitative

Able to group some answers together

Could tie together some results based on
various demographic characteristics

Not looking for statistical data

Looking for any cases of confusion

13. Results

Some Expected

Some not

No student spent more than 20 minutes

No subject wanted to spend more than 3 minutes,
most less

Minor problems with Testing Instrument

Instructional wording, briefer, more explicit

Convert to Action Statements

Sampling issues

14. Results (cont)

No trouble finding catalog

(note: sample were student workers who should have been familiar with this)

Keyword search seemed okay

Trouble with title or author searching

Need for less “library language”

e.g. “Interlibrary Loan”

“Research Help”

Concept of “Research Guides” seemed alien

Needed clearer access to citation help

15. Results (cont)

Need for content which did not exist:
− Tutorials; guides

Students did not understand what “Research Guides” were
− More explanatory terminology
− New navigation systems

Overall navigation troubles

Artifacts of CMS (e.g. left navigation)

Users unwilling to go more than one click deep

Need to have better system of content
promotion

16. Current Site

17. Current site (2)

18. Current site (2a)

19. Current site (3)

20. Conclusions/Discussion

Potential problems with redesign

Evidence students still have trouble understanding some concepts

No significant change in usage stats

Problems may be due to
− Usage may be tied to instruction
− Trouble with overall site (outside of library control)
but

Tool can be re-used

Larger sample, different groups:
− Undergrads
− Grad Students
− Faculty

21. References
Letnikova, G. (2008). Developing a standardized list of questions for the usability testing
of an academic library web site. Journal of Web Librarianship, 2(2), 381-415.
doi:10.1080/19322900802205817
Nielsen, J. 2000. “Why you only need to test with 5 users.” Nielsen-Norman Group.
http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ (Last
Accessed May 19, 2014)
VandeCreek, L. M. (2005). Usability analysis of northern illinois university libraries'
website: A case study. OCLC Systems & Services, 21(3), 181-192.
doi:10.1108/10650750510612380

22. Jason Simon
Twitter: @jason_c_simon
jsimon@designingmindsconsulting.com
http://www.designingmindsconsulting.com
@2designingminds