1. Preference and Desirability Testing: Measuring Emotional Response to Guide Design Michael Hawley Chief Design Officer, Mad*Pow Paul Doncaster Senior User Experience Designer, Thomson Reuters

2. Agenda Why we should care Why it’s not always as simple as asking:“Which option do you prefer?” Methods to consider Case Study: Greenwich Hospital Case Study: WestlawNext Summary/Comparison

3. Why we should care 3

4. Impressions Count

5. An important role of visual design is to lead users through the hierarchy of a design as we intend For interactive applications, a sense of organization can affect perceived usability and, ultimately, users’ overall satisfaction with the product Functional Desirability

6. Visceral Emotions

8. Hope/fear

9. Acceptance/rejectionhttp://www.behaviormodel.org/

10. 8 Positioning Desirability Studies http://www.xdstrategy.com/2008/10/28/desirability_studies/

11. “Which do you prefer?” 9

12. Quantity, granularity breed apathy

13. Poor articulation “It reminds me of…” “It’s nice and clean.” “There’s just something about it . . .” “I ordinarily don’t like red, but for some reason it works here . . .” “It’s better than the other ones.”

14. What Stakeholders Should Care About “We should go with design C over A and B, because I feel it evokes the right kind of emotional response in our audience that is closer to our most important brand attributes.”

15. Methods to Consider 13

16. Present three different concepts or ideas to participants, and ask them to identify how two of them are different from the third and why. 14 Triading

18. QUIS (Questionnaire for User Interface Satisfaction)

19. WAMMI (Website Analysis and Measurement Inventory)15 Qualitative Questionnaires

21. www.fivesecondtest.com16 Attention designers: You have 50 milliseconds to make a good first impression Quick Exposure Memory Tests

24. Blood Volume Pressure (BVP): Arousal

25. Pupil Dilation: Arousal and Mental Workload

26. Respiration: Negative Valence or Arousal 17 Physiological and Neurological

27. 18 PrEmo Emotional Measurement Dr. Pieter Desmet, Technical University of Delft http://www.premo-online.com

28. 19 http://www.microsoft.com/usability/uepostings/desirabilitytoolkit.doc Product Reaction Cards

29. Case Study: Greenwich Hospital 20

30. Determine intended brand attributes (and their opposites) 21 Product Reaction Cards: Before You Begin Leverage existing marketing/brand materials Alternatively, stakeholder brainstorm to identify key brand attributes/descriptors using full list of product reaction cards as a start Tip: “If the brand was a person, how would it speak to your customers?”

31. Methodology Include 60/40 split of positive and negative words Target 60 words, optimized to test brand Simple question: “Which of the following words do you feel best describe the site/design/product (please select 5):” One comp per participant, or multiple comps per participant (no more than 3) Participants Qualitative: Paired with usability testing Quantitative: Target minimum of 30 per option if possible 22 Product Reaction Cards: Conducting

32. 23 Process - Analyzing Calculate percentage of positive and negative attributes per design Visualize overall sentiment of feedback using “word clouds” (see wordle.net) 68% Positive 32% Negative

34. Update the site after nearly 10 years

35. Counter impressions that Greenwich is more than just maternity and elder care

36. Communicate that they are long-standing members of the community24 Case Study: Greenwich Hospital Website Redesign

38. 50 people reacted to each comp (quantitative) via survey

39. Additional feedback obtained via participant interviews (qualitative)Survey Questions Hello, I am requesting feedback on a website I am working on. Your answers let me know if the site is conveying the right feel. 1. What are your initial reactions to the web site? 2. Which of the following words best do you feel best describe the site (select 5):

40. 26 Three Different Visual Designs

41. 27 Results: Concept 1 12% Negative 88% Positive

42. 28 Results: Concept 2 87% Positive 13% Negative

43. 29 Results: Concept 3 5% Negative 95% Positive

45. Qualitative helps provide color to the results

46. Quantitative resonates with stakeholders and executives

47. Position results as one form of input to decision-making process, not declaring a “winner”

48. Simple, cost-efficient way to assess audience’s emotional response to a design30 Lessons Learned

49. Case Study: WestlawNext UX Research Team: Drew Drentlaw Shannon O’Brien Bill Quie November Samnee 31

52. Isolate preference trends for specific page design aspects

53. Determine tolerance for manipulation of the site “brand”

54. Maintain tight securityGoals

55. Sessions were held in 4 cities over 5 days Seattle Denver Memphis Minneapolis-St. Paul 4 sessions were held per day, with a maximum of 25 participants per session 1.5 hours allotted per study, most participants finished in less than 1 hour 319 participants successfully completed their sessions Phase 1: Logistics & Execution

56. Participants completed the study at individual workstations at their own pace All workstations included a 20” monitor, at 1024x768 resolution Phase 1: Logistics & Execution Memphis, TN, May 2009

57. Brief review of Westlaw critical screens Positive/negative word selection to describe Westlaw 36 Positive/negative product descriptors

58. Each set of Element variations were viewed in full screen Participant selects “top choice” by dragging a thumbnail image to a drop area 37 Homepage: Design Elements

59. 38

60. Homepage: Design Elements (1) All options viewed in full screen Participant selects “top choice” by dragging a thumbnail image to a drop area

63. Visual Weight (6 options) Use of Imagery (8 options) Components (4 options) Search Area (4 options) Palette (10 options) Homepage: Design Elements

64. 19 HP designs viewed in full screen (randomized) All 19 options are presented again; participant assigns a rating using a 10-point slider. Top 5 and Bottom 2 choices are positioned in order of rating values on one long, scrollable page. Next to each design displayed, rates key aspects for each design on a 5-point scale Homepage: Design Gallery

71. Column Collapsing (4 options)

72. Column Separation (2 options)

73. Theme/Color (8 options)

74. Design Gallery

75. 14 Results Lists designs (randomized)

76. Key Aspects Rated

77. Color scheme

78. Global Header

79. Summary and Excerpt (list contents)

80. Filters design (left column)

82. Tabs vs. Links (4 options)

83. Background Separation (4 options)

84. Margin Width (3 options)

85. Font Size (12 options)

86. Locate (2 options)

87. Design Gallery

88. 9 Results Lists designs (randomized)

89. Key Aspects Rated

90. Color scheme

91. Layout of content

92. Text formatting

95. Age

96. Testing Location

97. Years of Experience

98. Hours per Week Researching

99. Organization Size

100. Role (decision-maker status) The top picks were surprisingly consistent across all of the ‘Top 5’ lists analyzed High-level Results

101. 53 Top Homepage Designs by Job Title Job Title. Top 5 out of 19 possible. Overall (319) Associate (189) Librarian (37) Partner (81) Solo Practitioner (5) 1 1 1 1 1 HP16 HP1 HP16 HP5 HP15 2 HP8 HP15 HP15 HP6 HP16 2 2 2 2 3 3 3 3 3 HP8 HP8 HP16 HP8 HP10 4 HP15 HP1 4 4 4 4 HP5 HP7 HP5 5 5 5 5 5 HP10 HP13 HP8 HP19 HP14

102. Home Page (19) HP16 & HP15 designs consistently placed in the Top 5 across all filters Results List (14) RL4 consistently placed in the Top 3 across all sample filters, and was the #1 choice for 80% of all participants Document Display (9) DD3 placed in the Top 5 across all sample filters and was the #1 choice for 77% of all participants Phase 1: High-level Results

103. Note, participants were asked to describe the current Westlaw before being shown the new designs. 55 Phase 1: Word Selection Results

105. “Clean and Fresh”

106. “Contrast is Important”

107. “Prefer Westlaw Blue”

108. “No Big Fonts Please”The study narrowed the list of potential designs, and we better understood what design elements that Westlaw users liked and disliked. Phase 1: High-level Results

109. 57 Phase 2: September 2009 Kansas City, MO, Sept 2009

111. Understand users personal reasons for liking their preferred choices

112. Get closure on other design options for online and printed content

114. Method View, Rate, and Pick Top Choice for Homepage (3 options) Result List (2 options) Document Display (2 options) “Why?” Simple preference selection for two unresolved UI design issues Citing References: Grid display or List display? Out of Plan Indication design (6 options) Type formatting preferences for 3 different content types Font Face Font Size Margin Width Phase 2: September 2009

115. Logistics 3 cities (Philadelphia, Kansas City, Los Angeles) 1 Day 226 participants Analysis Filters (8 categories) were used to score the designs for each visual preference Results Clear choices for top designs in each of all categories “Why” feedback shed new light on designs under consideration and helped focus “homestretch” design activities Phase 2: September 2009

116. Home Page (3) HP3 ranked #1in 94% of filter groups (54% of total participants) Results List (2) RL5 ranked #1in 97% of filter groups (58% of total participants) Document Display (2) DD7 ranked #1in 94% of filter groups (61% of total participants) Phase 2: High-level Results

117. The main concerns regarding Homepage Design HP3 Search Box Too small How do I do a Terms-and-Connectors search? Browse Section How do I specify multiple or specific search content? Poor organization Poor label Need access to “often-used” content Need better access to help 62 Participant Comments: Homepage

118. Goals Get feedback on branding options from decision makers and those who influence purchase of the product Get closure on final outstanding design issues Tool Same as in Rounds 1 & 2, with some minor revisions to accommodate specialized input Phase 3: December 2010

119. Method Wordmark/Branding View wordmark color combinations and design elements against different backgrounds, pick top choice and provide comments Make a final “Top Choice” from all selections Simple preference selection for outstanding UI design issues Header Space: Tile or No Tile? Notes Design Location: Inline or Column? State: Open or Closed? Headnote Icon design (4 variations) Phase 3: December 2010

120. What color combination do you prefer? Please rank the 4 combinations below according to your preferences. To rank, click and drag an item from the left to a box on the right. Your Most Liked 1 2 3 4 Your Least Liked

121. Logistics 3 cities (Seattle, Denver, Boston) 1 Day 214 participants Analysis Simple preference, no advanced filters Results Decision-makers confirmed that critical brand elements should be retained Phase 3: December 2010

122. Decision Makers’ Picks (1 of 2)

123. 68

125. “Slicing” across identifiable filters

126. Emphasis on “gut-level” reactions

127. Intolerance for manipulation of product brand

128. Rapid turnaround of data to all stakeholders

129. Executive

130. Design

133. If we had asked why in the first round, we might have

134. avoided some of internal design battles

135. gotten more granular ammunition for communicating the design vision to stakeholders

136. “Need for speed” attained at the cost of detailed analysisRetrospective

138. Get a good technical vendor for on-site

139. Report results in as close to real-time as possible on a wiki or other web-pageRetrospective

140. Summary/Comparison 73

141. Both groups valued support in design decision making Align methodology with needs of the project Research-inspired, not research-decided 74 Summary/Comparison

142. Additional Reading and Tools 75

143. Benedek, Joey and Trish Miner. “Measuring Desirability: New Methods for Evaluating Desirability in a Usability Lab Setting.” Proceedings of UPA 2002 Conference, Orlando, FL, July 8–12, 2002. http://www.microsoft.com/usability/uepostings/desirabilitytoolkit.doc Lindgaard, Gitte, Gary Fernandes, Cathy Dudek, and J. Brown. "Attention Web Designers: You Have 50 Milliseconds to Make a Good First Impression!" Behaviour and Information Technology, 2006. http://www.imagescape.com/library/whitepapers/first-impression.pdf Rohrer, Christian. “Desirability Studies: Measuring Aesthetic Response to Visual Designs.” xdStrategy.com, October 28, 2008. Retrieved February 10, 2010. http://www.xdstrategy.com/2008/10/28/desirability_studies 76 Additional Reading

144. User Focus. "Measuring satisfaction: Beyond the Usability Questionnaire." Retrieved February 10, 2010. http://www.userfocus.co.uk/articles/satisfaction.html UserEffect. "Guide to Low-Cost Usability Tools." Retrieved May 12, 2010.http://www.usereffect.com/topic/guide-to-low-cost-usability-tools Tullis, Thomas and Jacqueline Stetson. “A Comparison of Questionnaires for Assessing Website Usability.” Usability Professionals’ Association Conference, 2004.home.comcast.net/~tomtullis/publications/UPA2004TullisStetson.pdf Westerman, S. J., E. Sutherland, L. Robinson, H. Powell, and G. Tuck. “A Multi-method Approach to the Assessment of Web Page Designs.” Proceedings of the 2nd international conference on Affective Computing and Intelligent Interaction, 2007.http:// portal.acm.org/citation.cfm?id=1422200 77 Additional Reading

145. Five Second Testhttp://fivesecondtest.com/ Feedback Army http://www.feedbackarmy.com Wordlehttp://www.wordle.net PrEmohttp://www.premo-online.com 78 Additional Tools