How does one pair online usability testing with lab usability testing? On May 6th, 2010, UserZoom joined forces with Bentley University’s Design and Usability Center to present how online, unmoderated usability testing can be combined with lab usability testing to produce rich, actionable results. We examined Amazon’s e-commerce grocery store and compared the results of each method to reveal the secrets and lessons we have learned.

1. Webinar: Combining Lab and Online Usability Testing: Lessons Learned May 2010 Dial: 312-878-0211 Access Code: 599-198-841 Audio PIN: Shown after joining the Webinar Webinar ID: 309-304-048

2. First, the introductions… Alfonso de la Nuez Partner & Chief Marketing Officer at UserZoom Kim Oslob Research & Product Strategy Director at UserZoom Bill Albert Director of the Design & Usability Center, Bentley University Alfonso

4. What makes us proud? Customers in 10 countries Alfonso

8. Brief introduction to both research methods Usability testing in the Lab  Moderation, think-aloud… Bill

9. Brief introduction to both research methods Usability testing in the Lab  Observing, recording…

10. Brief introduction to both research methods Usability testing in the Lab  Eye Tracking…

12. Brief introduction to both research methods Online Usability Testing (a.k.a. Unmoderated Remote Usability Testing) Kim

14. Brief introduction to both research methods Online usability testing  Build the study first using a software solution

17. Brief introduction to both research methods Online usability testing  This is what it looks like…

18. Brief introduction to both research methods

20. Online usability testing  Analyzing results Brief introduction to both research methods

23. Our UX Toolkit

25. Seeing the Big Picture

26. Which one goes first? Lab first, then Online Online first, then Lab Identify/fix “low hanging fruit”, then focus on remaining tasks with large sample size Identify the most significant issues online through metrics, then use lab study to gather deeper qualitative understanding of those issues Generate new concepts, ideas, questions through lab testing, then test/validate online Collect video clips or more quotes of users to help bring metrics to life Validate attitudes/preferences observed in lab testing Gather all the metrics to validate design – if it tests well, then no need to bring users into the lab

44. Case study: Parallel studies Highlighted findings from the Online Usability Study: Kim

47. Study Design Initial Questionnaire: How frequently do you visit Amazon.com? Where do you usually buy your groceries? Why have you considered purchasing groceries online? Post Task: Validation by url or question (Success, Error, Abandon ratios collected). Ease of Use. How intuitive links and menus were and how easy it was to start the search. Issues and problems experienced while completing tasks. Final Questionaire: Overall satisfaction Satisfaction with navigation, product search, information offered and look and feel of the site. Likelihood to purchase groceries from Amazon.com. Likelihood to recomment to a friend.

48. T1: Locating Cascade Complete Success Non-Success Participants did feel particularly lost on the site (most likely due to experience using Amazon.com. However they did not feel the menus and links were extremely intuitive and weren’t always sure were to start their search.

49. T1: Locating Cascade Complete Over half (51%) of the participants defaulted to using search indicating the menus and navigation of the site were not very intuitive. Fifteen percent started in Health and Personal while 5% looked in promotional deals. Highlights dominant path *3% insignificance removed

50. T1: Locating Cascade Complete What participants found difficult when attempting the task: Exact Quotes: “ Hard to navigate to find desired product. Too long and was getting frustrating.” “ There were a lot of results so you have to type it in exactly or spend a couple minutes searching.” “ I didn’t see a soap category and didn’t realize how the search function worked.” “ I didn’t know where to look.”

51. T2: Finding a discount code Success Non-Success Forty-seven percent of the participants thought it was very difficult to locate the discount code. They thought it was very unclear how to start searching, it was not where they expected it to be and were somewhat lost during the search process. Extremely Easy Extremely Difficult

52. T2: Finding a discount code

53. T2: Finding a discount code Error participants (55%) could not locate the code, thought it was too difficult to find the code or the peanut butter and stated: “ I was expecting the discount in the results page, or the product page. I ended up looking for it under ’specials’, but I don’t know if that’s the best place for it.” “ Too much searching and clicking.” “ Looked under special offers having firstly used search, had to notice the offer code above product within the offer description.”

54. T3: Schedule Grocery Delivery Success Non-Success Seventy-five percent of participants could not complete the task. They did not think it was clear how to start searching and felt lost during the search process. Forty percent of participants who were successful, felt it was very difficult to complete the task. Extremely Easy Extremely Difficult

55. T3: Schedule Grocery Delivery Abandon participants (38%) could not find the information, thought the menus were difficult to understand and the task was taking too long. “ It was not obvious to me that I needed to click on the link at the top right of the screen to get this information until I had spent quite a bit of time looking all over the rest of the screen.” “ I wasn’t certain where to look for the information.” “ Nothing about this process has been user friendly.”

56. Final Questionnaire Fifty-three percent of participants thought the navigation was very difficult, 39% thought it was difficult to search for a product, forty-seven percent rated the overall look and feel of the site as very poor.

57. Final Questionnaire Fifty-five percent of participants stated they would not purchase their groceries in the future from Amazon.com. Net Promoter Score (NPS) calculation: 0 to 6 = Detractors 7 to 8 = Passive 9 to 10 = Promoters Net Promoter Score (NPS) = % of Promoters - % of Detractors

59. Conclusions: Looking at the ‘big picture’

62. º USA 440 N. Wolfe Rd. Sunnyvale, CA 94085 USA Phone: +1 (408) 524 – 74 45 Contact: Alfonso de la Nuez [email_address] UK Aylesbury House, 17 – 18 Aylesbury Street, London ECIR 0DB, UK Phone: +44 (0) 20 7193 2171 Cell: +44 (0) 7900 472 920 Contact: Arthur Moan [email_address] SPAIN Av. Diagonal 419 3º 2ª 08008 Barcelona, Spain Phone: +34 93 414 7554 Cell: +34 93 368 4295 Contact: Javier Darriba [email_address] Thanks so much for your time! Oh! Don’t forget: The slides and the video of this webinar will be available in a few days in our blog www.userzoom.com