Presented to Cleveland Clinic Medical Informatics Grand Rounds meeting, Aug 17, 2012

1. Medical Informatics Grand Rounds
Aug 17, 2012
Cleveland Clinic's Usability Study
Establish Standard Sitemap for consistent user experience
Kaitlan Chu | Information Architect | Cleveland Clinic

2. Medical Informatics Grand Rounds
Aug 17, 2012
Connect with Me
Email
ChuK@ccf.org
Twitter
@KaitlanChu
LinkedIn
linkedin.com/in/KaitlanChu
SlideShare
slideshare.net/KaitlanChu

3. Medical Informatics Grand Rounds
Aug 17, 2012
Cross reference + Category labels
New Methods to Analyze Quantitative
Open Card Sort:
Cleveland Clinic's First UX Study
Formal, full-cycle, by in-house staff
Kaitlan Chu | Information Architect | Cleveland Clinic

4. Medical Informatics Grand Rounds
Aug 17, 2012
New Methods to Analyze Quantitative
Open Card Sort:
Cleveland Clinic's First UX Study
Designed to maximize stakeholder buy-in
Kaitlan Chu | Information Architect | Cleveland Clinic

5. Background Planning Data Collection Analysis Impact
About
• Cleveland Clinic
One of Top 4 Hospital in US
41,000+ Employees
•Digital Marketing
ClevelandClinic.org
13,000 Pages
120,000 visits/day
Most visited US hospital site
$5/visit in contribution margin

6. Background Planning Data Collection Analysis Impact
About
•Sole IA for ClevelandClinic.org
•Thought leadership: Usability / Navigation
•Sign off on site changes
•First to conduct UX studies in house
•Academic research background
•Analytics & front-end development, Fortune 100

7. Background Planning Data Collection Analysis Impact
Influence Decision Makers
• Who they are
• How they think

8. Background Planning Data Collection Analysis Impact
Influence Decision Makers
• Who they are
• How they think

9. Background Planning Data Collection Analysis Impact
Influence Decision Makers
• Who they are
• How they think
Quantitative Studies Authoritative Sources

10. Background Planning Data Collection Analysis Impact
State of Site Structure
• Inconsistent navigation
What is navigation?

11. Background Planning Data Collection Analysis Impact
A website is…

12. Background Planning Data Collection Analysis Impact
A website is… Site Map
Navigation Menu

13. Background Planning Data Collection Analysis Impact
State of Site Structure
• Inconsistent navigation

14. Background Planning Data Collection Analysis Impact
Goal:
Consistent User Experience across ccf.org
Solution:
Standard Institute Sitemap

15. Background Planning Data Collection Analysis Impact
State of Sitea standard
Needs Structure site map
applicable to all departments
• Inconsistent navigation

16. Background Planning Data Collection Analysis Impact
Settle Design Debate
• Started out as a design dilemma
• Real problem – inconsistent site structures
• Solution – standard sitemap

17. Background Planning Data Collection Analysis Impact
Plan the study
• Quantitative Corporate culture
• Open card sort No presumption
• Online, remote Content inventory
• Put together a team Participant Recruiting: Panel
• Budget $300 Statistical Analysis
• Analysis guidelines Hard to find; Invent our own

18. Background Planning Data Collection Analysis Impact
Research Questions
• Elements of a sitemap
•Number of categories
•Content within categories
•Cross-reference ?
•Category label ?

19. Background Planning Data Collection Analysis Impact
Generate Cards
• Full Content Inventory
• 71 cards

20. Background Planning Data Collection Analysis Impact
Online open
card sort
• Sort all 71 cards
• See card label &
description
• Name each
category

21. Background Planning Data Collection Analysis Impact
Cross reference + Category labels
New Methods to Analyze Quantitative
Open Card Sort:
Cleveland Clinic's First UX Study

22. Background Planning Data Collection Analysis Impact
Statistical Analysis Methods
• 179 completes, 71 cards => 12710 lines in excel New Method
Cluster (Dendrogram) Multidimensional Scaling Cell Plot

23. Background Planning Data Collection Analysis Impact
In-house Cluster Analysis

24. Background Planning Data Collection Analysis Impact
Statistical Analysis Methods
Cluster (Dendrogram) Multidimensional Scaling Cell Plot

25. Background Planning Data Collection Analysis Impact
In-house Multidimensional Scaling

26. Background Planning Data Collection Analysis Impact
Statistical Analysis Methods
Cluster (Dendrogram) Multidimensional Scaling Cell Plot
• Elements of a sitemap
•Number of categories
•Content within categories
•Cross-reference ?
•Category label ?

27. Background Planning Data Collection Analysis Impact
Cell Plot Analysis
Cross reference

28. Background Planning Data Collection Analysis Impact
Category Labels
• Elements of a sitemap
• What keywords to use •Number of categories
•Content within categories
• What NOT to use •Cross-reference
•Category label ?

29. Background Planning Data Collection Analysis Impact
Keyword Frequency
? Participant #1 #2, #3, #4………… Participant #179
words words
Patient 1378
Info 632
General 152
Care 81
Appointment 80
Resource 60
Become 17

30. Background Planning Data Collection Analysis Impact
Impact of the study
• Established a standard sitemap
Accepted by all departments
• Stakeholders see the value
Greatly increased budget line
• Increase perceived value of my
team & UX studies

31. Background Planning Data Collection Analysis Impact
Application of Standard Sitemap

32. Background Planning Data Collection Analysis Impact
Conclusions & Discussions
• Tailor UX study to corporate culture
• Analyze data in as many methods as possible
• Cluster
• Multidimensional Scaling 3D
• Cell Plot
• Keyword Frequency
New study findings:
• Pre-test online study in-person
Treatment Outcomes
• - Mix qualitative & quantitative
• - Test the usability of online study
• Pilot test with participants represent real users
• Provide decent participant incentives
• Number of cards?

33. Background Planning Data Collection Analysis Impact
Treatment Outcomes
• Qualitative
• Before product launch
- Opportunity to improve
• One-on-one, in-person, lab-based study
Camera
Participant
Moderator

34. Background Planning Data Collection Analysis Impact
Treatment Outcomes
Scenario
Task

35. Background Planning Data Collection Analysis Impact
Treatment Outcomes
Website
Task

36. Background Planning Data Collection Analysis Impact
Task: Evaluate quality of care
Task Description:
You’re comparing a few hospitals to take your uncle to treat his epilepsy. Avoid
keyword
Where do you go to evaluate the quality of medical services and expertise the “Outcomes”
center provide?
For example,
• How many patients did the epilepsy center successfully treat last year?
• Is the epilepsy center using up-to-date technology for diagnosis and treatments?
• Does the center have leading experts in the field?
Your Task: Where do you go to evaluate the quality of medical services and
expertise the center provide? E.g. Number of successfully treated patients; up-to-
date technology; and leading experts in the field.
Research questions:
•Does an average patient understand “outcomes”?
•Do they read Outcome book?
•We should place it in “For Medical Professionals”

37. Background Planning Data Collection Analysis Impact
Evaluate quality of care
Diagnosis
Research
4 Outcomes
Outcome Book Outcome Book
1 About Us Outcomes
•“Outcomes” is associated with quality of medical care.
•Does an average patient understand “outcomes”? Yes
•Do they read Outcome book? Yes, at least 2 read cover to cover
•We should place it in “For Medical Professionals”. Patients never
navigate to that section.

38. Background Planning Data Collection Analysis Impact
Outcome Book
On task “evaluate qualify of medical care”:
Participant:
“The site [outcome book pdf] itself is sort of like you’re reading a
textbook. So I blew right over that the first time.
The comparison chart with national data is a great chart. Once you
caught my attention with that then I would spend more time reading
the other stuff [charts and data].”

39. Medical Informatics Grand Rounds
Aug 17, 2012
References
•Capra, M. (2005) Factor Analysis of Card Sort Data: An Alternative to Hierarchical Cluster Analysis. Human Factors and Ergonomics Society Annual
Meeting Proceedings, Computer Systems , pp. 691-695(5). Human Factors and Ergonomics Society.
•Hinkle, V. (2008, October) Card-Sorting: What You Need to Know about analyzing and Interpreting Card Sorting Results. Retrieved on February 15
2011 from Usability News October 2008, Vol. 10 Issue 2 (http://www.surl.org/usabilitynews/102/cardsort.asp).
•Krystal M. Lewis, Peter Hepburn, (2010) "Open card sorting and factor analysis: a usability case study", Electronic Library, The, Vol. 28 Iss: 3, pp.401 –
416
•Paul, Celeste Lyn (2008) “A modified Delphi Approach to a New Card Sorting Methodology” Journal of Usability Studies 4 (1), 7-30.
•Puskala, A. (Nov 17 2009) Card Sort cluster analysis tool 1.1. Retrieved on February 15 2010 from User Point (
http://www.userpoint.fi/tools/card_sort_cluster_analysis_tool/).
•Spencer, D. (2009) Card Sorting: Designing Usable Categories. New York: Rosenfeld Media.
•Spencer, D. (2007, June 7) Card sort analysis spreadsheet. Retrieved on February 15 2011 from RosenfeldMedia.com (
http://rosenfeldmedia.com/books/cardsorting/blog/card_sort_analysis_spreadsheet).
•Tullis, T. & Albert, B. (2008) Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Burlington, MA: Morgan
Kaufmann Publishers.
•Tullis, T. & Wood, L.E. (2005). How Can You Do a Card-sorting Study with LOTS of Cards? Poster presented at the Annual Meeting of the Usability
Professionals Association, June 24 - July 1, Montreal, QB, Canada.
•Free tool to generate cluster analysis for large open card sort http://www.userpoint.fi/tools/card_sort_cluster_analysis_tool/

40. Medical Informatics Grand Rounds
Aug 17, 2012
Special Thanks to
•Amy Sternad
•Michelle Tackla Wallace
•Amanda Beacher
•Jieying Jane Chen
•Zhijie Li
•Scott Linabarger
•Lynn Pecl

41. Background Planning Data Collection Analysis Impact