UserZoom worked in partnership with UX Alliance members; Telono, User Intelligence and Xperience Consulting to benchmark the User Experience across 4 countries and 4 mobile operators (Orange, Vodafone, T-mobile and Swisscom)
15. What is UX benchmarking and why is it important? OK……. So how do we define UX benchmarking?
16.
17. The process of comparing the business processes and performance metrics including cost, cycle time, productivity, or quality to another that is widely considered to be an industry standard benchmark or best practice.
18.
19. It’s about how a person feels about using a system. It highlights the experiential, affective, meaningful and valuable aspects of HCI and product ownership, but it also covers a person’s perceptions of the practical aspects such as utility, ease of use and efficiency of the system.
20.
21.
22.
23. What is UX benchmarking and why is it important? ‘You can’t manage what you can’t measure’ Attributed to Peter Drucker, Mgt Guru The question is: How are you actually measuring UX and defining your benchmark?
85. Orange Spain (SP) had the lowest success rate across tasks 24%, the lowest participants satisfaction scores, and substantially lower preference (25%) vs. Vodafone (75%).
86. All mobile providers tested in each country received low usability satisfaction scores by participants.
87. The lowest system usability scale scores (SUS) were received by Vodafone Spain (51) and Orange Spain (40).
88.
89. Top level overview of Task 1 across all countries with deep dive into Switzerland & UK results “Find the monthly cost to surf the internet daily with an iPad”
109. Successful Participants spent an average of 5 minutes 24 seconds and 19 clicks completing the task. Task Effectiveness
110. Orange > iPad: UK The majority of participants were able to find the correct price for the plan (75%). Successful Participants spent an average of 1 minute 11 seconds and 7 clicks completing the task. Task Effectiveness
111. Top level overview of Task 2 across all countries with deep dive into Netherlands & Spain’s results “Locate a store address/fax number”
114. The T-mobile site provided the best results regarding this task, 77% of the participants was successful.
115. On average it took slightly over one minute to find the right information, with an average of only 4 page views and 8 clicks.
116. On the Vodafone website 52% was successful in achieving this information.
117.
118. The Vodafone website was less successful, but only 28% of the participants regarded it to be difficult to acquire the needed information.
119. On the T-mobile website, people were far more successful, but also less satisfied with the level of difficulty to get to the right URL: 42% of the participants thought this was (very) hard to achieve.
122. Page not loading: Especially the Vodafone website was unable to load the required page. Remarkable is that people wait up to 5 minutes!
123. Information not in expected location: Both Vodafone and T-mobile present information in an unexpected manner. Participants have certain expectations on where and how to find the required information. In case these expectations are not met, people easily abandon the website.QUOTES “I was looking for it under Customer service and expected to be able to fill in my location and that it would automatically give the closest store location.” “ I was unable to open the page.”
124.
125. They regard it as easy to use for inexperienced users and there is no need for extra help.
135. As it has been seen in the previous task users again had problems with the information provided. I have entered province “Madrid”, city “Madrid” and no results appeared with this search options. I have tried with different options and it always happened the same.
136.
137. Users explained that all the site is focusing on the iphone whereas ipad information is quite difficult to find.
138. The system could be improve using an accessible and easy to find search engine. There is no way to understand what is going on here. There is no way to find anything. What a fuss!
139.
140. The links to the information were not clear enough.
141. The site is very focused on selling products or services instead of giving clients the service that they need. I don’t have anything against Orange but its website is a disaster
142. Final Questionnaire – all countries: Which mobile operator do you prefer? Netherlands Switzerland Spain UK
143. Metrics: SUS Score The System Usability Scale (SUS) is a set of 10 questions that is used to measure the perception of a systems overall usability. Participants rate each question with a 1 to 5 scale (5 being strongly agree). 1. I think that I would like to use this system frequently. 2. I found the system unnecessarily complex. 3. I thought the system was easy to use. 4. I think that I would need the support of a technical person to be able to use this system. 5. I found the various functions in this system were well integrated. 6. I thought there was too much inconsistency in this system. 7. I would imagine that most people would learn to use this system very quickly. 8. I found the system very cumbersome to use. 9. I felt very confident using the system. 10. I needed to learn a lot of things before I could get going with this system.
144. Metrics: SUS Score After completing the tasks on each mobile provider website participants were asked to complete the SUS questionnaire. The averages for each question are used to calculate an overall score (0-100). SUS Score Guide*: 35=poor, 50=OK, 70=good, 85=excellent Bangor, A. (2009, May). Determining What Individual SUS Scores Mean: Adding and Adjective Rating Scale. Journal of Usability Studies, 4(3), 114-123.
145. Net Promoter Score (NPS®) After completing the four tasks, participants were asked how likely they would be to recommend each mobile provider’s website. Promoters Detractors - = % of Promoters (9s and 10s) Detractors (0 through 6) NPS
146.
147.
148. UK - although participants had notably higher task success rates on the Orange site (75% and 71%) vs. the Vodafone site (50% and 49%), when asked for site preference at the end of the study, the Orange website was selected only 8% more than the Vodafone site (54% vs 46%) respectively. Out of the Orange customers (n=17) more than half (9) preferred the Vodafone website.
158. Offices in Amsterdam and Helsinki, additional service offering in Sweden and Norway
159.
160. We help companies implement a systematic user-centred design process, from user experience strategy and user research to interaction design and usability evaluations.
166. 150 clients (15 countries)Usability and user experience experts, we create interfaces that suit user needs and objectives of your company. Intuitive and easy to use webs.
167.
168. Back in 2005 the leading user experience companies from France, Germany, Spain, UK and US met in Barcelona with a single common goal in mind: how to provide effective international user experience services to our clients.
169. Having worked together on international projects for many years, we recognised that we were ideally placed to lead international user experience research. Thus, the UXalliance was born… www.uxalliance.com
In this webinar, we will show you how you can remove face to face moderation, have hundred of users participate simultaneously from different locations in a cost effective and fast way. Userzoom is an on demand research service that will automatically capture and gather both customers attitudes and behaviors.After our demonstration, we will open up to Questions, but let’s start now by diving into UserZoom software as a service.
In this webinar, we will show you how you can remove face to face moderation, have hundred of users participate simultaneously from different locations in a cost effective and fast way. Userzoom is an on demand research service that will automatically capture and gather both customers attitudes and behaviors.After our demonstration, we will open up to Questions, but let’s start now by diving into UserZoom software as a service.
In this webinar, we will show you how you can remove face to face moderation, have hundred of users participate simultaneously from different locations in a cost effective and fast way. Userzoom is an on demand research service that will automatically capture and gather both customers attitudes and behaviors.After our demonstration, we will open up to Questions, but let’s start now by diving into UserZoom software as a service.