Presentation by Tim Bosenick, SirValUse, Germany for the Usability Marathon 2009

1. Measuring User Experience
Combining the “Qualitative” with the “Quantitative”
Some Examples from the Web
Tim Bosenick, SirValUse Consulting GmbH
Bild

2. What do I know about my site and the users?
I have some data of Sometimes a
users given during apply surveys
I know that some order. on my site.
users order.
I have a Web
Analytics System
I know the and collect loads
serverlogs, PIs and of data.
sessions.
I run regularely
But... qualitative User
Experience Tests.

3. But still there are many questions…
Who are my most
important Are my competitos
competitors? better? Why?
Do I reach my (scope, CR, ..)
target groups?
How does the
Who exactly are my business
users? Do they act competition
effectively on my develop?
website?
Where is my target group on Does my marketing
the way in internet? reach the people I want
Where can I appeal to them? to reach?

4. Conclusion:
User experience, usability
and marketing (the whole
“experience chain”) come
closer together
We need a 360
degree view on
our costumers
We often know the
“what happens”, we
sometimes know the
“why” – a combination
would be great

5. The Idea
WebWerte 5

6. Combining qualitative and quantitative
observational and survey data.
Further requirements:
Quantitative Qualitative » Single source data
collection for high
validity
Detailed Observation
User Tracking analysis of » Scalable to large
user sessions numbers of participants
» Possibility of true-
intent and task-based
Standardised Free Survey Data
user experience tests,
Surveys User Feedback also experiments
» Measurement without
cooperation of website
for one specific website,
owners or publishers
site centric,
short time frame
(only Add-On)

7. Whole internet, user centric
Features of the LEOtrace® Browser Add-Ons
Features: Deliverable data:
» Single source measurement of reactive » Non-reactive data ("Tracking")
and non-reactive data » Session information (e.g. duration)
» Visited URL’s
» Data collection without cooperation of
» Search queries (Google, ...)
website owners or publishers
» Precise ad impressions
» Allows data collection on third party sites
» Screenshots
» Flexible setup » Clicks (and mouse movements)
» Customising of design and branding
» Reactive data
» Remote control of all functions
» Standardised surveys (also event-
» Extended research designs possible: triggered) with any survey software
» Task-based designs » Free ad-hoc feedback
» Experimental designs (manipulation of » Client information
client-side HTML code)
» Operating system, web browser, …
» Data collection in SSL-encrypted areas
» User structure data provided from
possible (usually disabled)
access panel

8. Whole internet, user centric
Installation process
Download
1
Add-On
Optional: Identification
on every
browser start
4
Install
2
Add-On
Easy setup – just
like any other Optional:
browser add-on 3 Registration on
first browser start

9. Specific website, site centric
The user are invited on your website …
» To elevate the problems occuring while using the website,
the LEOtrace® Remote Test has been developed:
Preliminary interview Follow-Up Interview
Invitation by a layer Remote Session
Online questionnaire Online questionnaire
• "What is your today main • "Please surf the website as you • "Did you achieve everything
reason of visit?" would do it normally." today you planned to?"
• "Please evaluate the website
on the basis of following
• User-Behaviour items."
• Clickway-
Analysis
• User-Feedback
… and use the website according to your natural performance.

10. Specific website, site centric
The Technology
Participants Website
2 1
Utility target-website Content enquieries
Proxy-Server through
Give free and scaled
3 feedback Recalls monitored actions of
participants
Usability Expert
Surveyserver Recalls comments and evaluations
of participants
Participants and website-providers don´t need to install software .
Only the Invitationlayer has to be linked. Adjustments have only to be
neccessary in the LEOtrace®-System itself

11. Specific website, site centric
The Feedback-Bar
Call up feedback-
To the finaly survey
sessions
Via feedback-buttons the users are always able to call up a short survey or to
give a positive or negative feedback .

12. Types of Studies
WebWerte 12

13. Typical Studies
Tasked based Benchmarking
2 3
Initial survey Task
(e.g. favourite search with detailed recording of
engines, …) user behaviour (clicks,
screenshots) as well as free
feedback and event-triggered
questionnaires
1
4
Recruitment
Final survey
from online access panel
(e.g. rating of websites used);
(screening according to
Deinstallation of Add-On
target group criteria);
Installation of LEOtrace®
Browser-Add-On

14. Typical Studies
Tasked based Benchmarking & Usage Monitoring
2 3
Initial survey Task
(e.g. favourite search with detailed recording of
engines, …) user behaviour (clicks,
3a screenshots) as well as free
feedback and event-triggered
questionnaires
Non-reactive
monitoring of online usage
during several weeks
(before receiving task)
1
4
Recruitment
Final survey
from online access panel
(e.g. rating of websites used);
(screening according to
Deinstallation of Add-On
target group criteria);
Installation of LEOtrace®
Browser-Add-On

15. Typical Studies
Digital Behaviour Studies
2
Initial survey
(e.g. favourite search
engines, …)
3
Non-reactive
monitoring of online usage
during several weeks
(before receiving task)
1
4
Recruitment
Final survey
from online access panel
(e.g. rating of websites used);
(screening according to
Deinstallation of Add-On
target group criteria);
Installation of LEOtrace®
Browser-Add-On

16. Typical Studies
Digital Behavior Studies with Event based Surveys
2
Initial survey Event based surveys,
(e.g. favourite search e.g. having used a
engines, …) certain function,
3 leaving a certain website,
leaving a certain page
Non-reactive
monitoring of online usage
during several weeks
(before receiving task)
1
4
Recruitment
Final survey
from online access panel
(e.g. rating of websites used);
(screening according to
Deinstallation of Add-On
target group criteria);
Installation of LEOtrace®
Browser-Add-On

17. Typical Studies
Web Efficiency Panel
Generating single
Fusion &
source consumer &
Browser Add-On sends analysis
Panellist
Internet usage data:
 URLs
 Ad contacts scans
 Search queries purchases » Purchases
(Consumer Scan)
» Internet usage
Internet user buys online
 PageImpressions
Internet user
Internet user buys offline  Visits
in WEP Offliner  AdImpressions
buys
offline  AdClicks
 Queries with relevant
search engines
» Structural data
Offliner
in GfK Consumer Scan panel

18. Typical Studies
Industry Benchmarking
» We use the data from the Web Efficiency Panel to calculate benchmarking
KPIs for certain industries (e.g. e-commerce, automobile, …)
» Possible analyses:
 Target group
 Cross usage
 Previous and following websites
 Usage of functions / areas
 Conversion rates
 …

19. Typical Studies
True Intent Experience Reports
2
Initial survey
(e.g. reason for
the visit, …)
3
Non-reactive
monitoring of website usage
during this session
optional: free feedback
1
4
Recruitment
Final survey
online (real users)
(e.g. rating of website,
(screening according to
satisfaction, reason for
target group criteria);
leaving, …)
no installation needed

20. The Importance of Measurements

21. Quantitative studies are an ideal enhancement of
qualitative studies
Qualitative Approach Quantitative Approach
Focus on “usability” Focus on “user experience”
Derive recommendations Measure key performance indicators
Focus on product management / UX Focus on market research / general
departments management
Sometimes “artificial” lab situation, “True intent” studies possible,
no “real” behavior can be observed observation of “real behavior”
Combination with qualitative Combination with marketing KPIs
marketing research possible possible
Testing of “offline” applications Test object must be “online”
possible

22. Why is quantitative measurement important?
» Users tend to misjudge their behaviour
 Social request, known brands are over-rated, problem to correctly quantify their own behavior
 Example: User were asked: What kind of websites did you use while looking for a product?
Answer: 88% Google. In the actual measurement phase, only 68% really used Google.
» “User Experience” gets more and more important
 “Usability” still is an important factor of the overall user experience, but e.g. “joy of use” and
“design” come into the focus of testing – also in the early development phase.
 With this, classical market research methodologies and “real measurement” become central
for the usability testing practice – or else market research companies will gain bigger parts of
the market.
» “Usability” and Marketing come closer together
 It is nowadays not only important to get as much traffic as possible to a website (online
marketing) and to optimize the conversion rate (usability), it is also important to combine both
views so that the whole shopper experience can be optimized.

23. Automobile Industry Monitor 2009
WebWerte 23

24. Methods
To satisfactorily address the questions at hand, we have employed all of the following
methods:
» Permanent data collection regarding all surfing
behaviour by means of the Web Efficiency Panel (WEP)
designed by GfK during Q1 2009 (01/01 - 31/03).
Internet Tracking » Quantitative evaluation of internet use with regard to
relevant automotive websites.
» Determination of indicators and modelling of
navigation behaviour.
» Surveys completed by the WEP panelists regarding the
Surveys topics automotive ownership, automotive purchase,
brand affinity and advertising awareness.
» Expert analysis of the most important manufacturer
Market Analysis websites with regard to user experience.
 Identification of best practices.

25. In Q1 2009, one-fifth of internet users visited at
least one automotive manufacturer website.
Internet users in Germany: 42,540,000
Total users of "Auto-Websites": 16,380,824
Users of manufacturers' websites: 8,308,536
Users of sales portals: 7,692,277
Users info portals: 5,423,271
Users of auto club websites: 4,153,947
Users of online community websites: 2,678,939
Users of media websites: 2,352,151
Users of auto group websites: 223,034 1%
Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009

26. In the first quarter, manufacturer websites were
visited around 41 million times, and on average,
9.5 PIs were generated.
9.5 11.1 5.0 8.9 5.9 3.8 4.6 PIs per Visit
(in mil.)
Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009

27. The VW website has by far the most users,
followed by Opel, Ford and Audi.
Users of
manufacturer
websites
(in thousands)
Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009 Page 1 of 2

28. The Mercedes and VW websites have the highest
rate of repeat visits.
Mercedes 1.9
VW
Opel
Audi
Skoda
Fiat
Citroën
Toyota
Mitsubishi
Mazda
Seat
Porsche Visits per user of the given
Renault manufacturer websites (mean)
Jaguar
Suzuki
Smart
Honda
Dacia
Ford
Peugeot
BMW
Nissan
Alfa Romeo
Daihatsu
Hyundai
Subaru
Kia
Volvo
Data base: Internet users in Chevrolet
WEP designed by GfK
between 01/01.–31/03/2009

29. Overview of the most significant indicators for
manufacturer websites
User Proportion
User Proportion Proportion of User Proportion User Proportion User Proportion
Unique Users PIs per Visits per for
for Showroom ended for Financing Financing for Dealer
(in thousands) Visit User Configurator
use configurations pages Calculator Search
use
Mean Value* 899 9.6 2.2 52% 28% 36% 11% 6% 13%
Audi 1,051 10.6 2.3 52% 23% 45% 1% 1% 0%
BMW 615 6.1 1.5 43% 26% n. a. 5% 5% 4%
Chevrolet 306 15.7 1.3 70% 23% 37% 14% n. a. 16%
Citroën 692 9.0 2.0 62% 29% 59% 8% 0% 8%
Dacia 756 9.3 1.6 89% n. a. n. a. 29% n. a. 31%
Fiat 545 14.9 2.1 40% 45% 45% 0% 13% 13%
Ford 1,053 6.4 1.6 51% 27% n. a. 9% 3% 20%
Honda 895 6.4 1.6 24% 12% 50% 4% 0% 11%
Mercedes 771 7.8 5.0 39% 24% 24% 5% 8% 31%
Mitsubishi 274 5.7 2.0 54% 27% 6% 0% 6% 18%
Opel 1,109 16.6 2.5 59% 48% 50% 3% 16% 10%
Peugeot 923 5.0 1.5 37% 14% n. a. 27% n. a. 11%
Renault 882 9.4 1.8 61% 43% 27% 3% 1% 10%
Skoda 722 14.6 2.2 40% 36% n. a. 4% 5% 4%
Toyota 860 6.5 2.0 75% 24% n. a. 10% 12% 16%
VW 2,933 9.2 3.7 34% 20% 12% 59% 2% 5%
* Refers to the 16 manufacturers represented
Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009 Info: The best three providers are marked in green, the worst three in red.

30. Showrooms, configurators and information about
financing are of particular interest to users who
intend to purchase – the dealer search is somewhat
less used by those interested in purchasing.
Info: In the calculation of the mean,
only those manufacturer websites
are represented for which use data
are available for the given area.
Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009

31. Fewer than half of manufacturer website users
visit showrooms – if one is visited, then others are
likely to be also.
Showroom users (Q1): 3,680,249
1 Showroom visited: 1,373,005
2 Showrooms visited: 626,044
3 Showrooms visited: 438,454
4 Showrooms visited: 330,935
5 Showrooms visited: 190,848
More than 5 Showrooms visited: 720,963
Number of showrooms visited for all
users of manufacturer websites (mean):
1.0
Number of showrooms visited for all
users of showrooms (mean):
4.2
Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009

32. Overview of the most significant indicators for
manufacturer websites
User Proportion
User Proportion Proportion of User Proportion User Proportion User Proportion
Unique Users PIs per Visits per for
for Showroom ended for Financing Financing for Dealer
(in thousands) Visit User Configurator
use configurations pages Calculator Search
use
Mean Value* 899 9.6 2.2 52% 28% 36% 11% 6% 13%
Audi 1,051 10.6 2.3 52% 23% 45% 1% 1% 0%
BMW 615 6.1 1.5 43% 26% n. a. 5% 5% 4%
Chevrolet 306 15.7 1.3 70% 23% 37% 14% n. a. 16%
Citroën 692 9.0 2.0 62% 29% 59% 8% 0% 8%
Dacia 756 9.3 1.6 89% n. a. n. a. 29% n. a. 31%
Fiat 545 14.9 2.1 40% 45% 45% 0% 13% 13%
Ford 1,053 6.4 1.6 51% 27% n. a. 9% 3% 20%
Honda 895 6.4 1.6 24% 12% 50% 4% 0% 11%
Mercedes 771 7.8 5.0 39% 24% 24% 5% 8% 31%
Mitsubishi 274 5.7 2.0 54% 27% 6% 0% 6% 18%
Opel 1,109 16.6 2.5 59% 48% 50% 3% 16% 10%
Peugeot 923 5.0 1.5 37% 14% n. a. 27% n. a. 11%
Renault 882 9.4 1.8 61% 43% 27% 3% 1% 10%
Skoda 722 14.6 2.2 40% 36% n. a. 4% 5% 4%
Toyota 860 6.5 2.0 75% 24% n. a. 10% 12% 16%
VW 2,933 9.2 3.7 34% 20% 12% 59% 2% 5%
* Refers to the 16 manufacturers represented
Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009 Info: The best three providers are marked in green, the worst three in red.

33. The showrooms on the websites of Dacia, Toyota
and Chevrolet are visited especially often.
Number of users of a
website who visited the
showroom on that
website.
Proportion of total users who
visited the showroom on any 52%
manufacturer website (mean):
Data base: Internet users in WEP designed by GfK between 01/01.–31/03/2009

34. Images of the models displayed on the start page
spur users on to visit the showrooms.
» The first-ranked websites of Dacia, Toyota
and Chevrolet feature links to the
showroom directly on the start page,
including images of the models.
Toyota
Dacia
» Websites that only offer a text link
(e.g. Peugeot, VW) create less interest
for showrooms.
 If the links to the models are not prominently placed,
the showrooms are used significantly less often (see Honda).
Honda
Showrooms with a high degree of multimedia concentration often may not
exploit their potential because they are less able to be located.

35. eCommerce Branchenmonitor 2008
WebWerte 35

36. Analysis question
» Why does a website sells good – and better than the competition?
» Why do customers decide for one special online-shop?
» Where (also online/offline) and how do customers inform themselves?
» Which products are bought out of what reasons?
Focus: Consuming goods in low to middle pricesegment, which are bought
regularly

37. Method- Mixing
1. LEOtrace® Behaviour Tracking of online-shopper
» Multi-Method Study for wholistic mapping and analysis of the
onlineshopping-process, from wording the shoppingintention until delivery.
» Monitoring of users during productresearch, measuring of detailled data-
characteristics and different interviews before, during and after the
monitoring phase.
2. Expert assessment by the SirValUse E-Commerce-Team
» Focus: Best Practice-Analysis.

38. Fact sheet
LEOtrace® Behaviour Tracking
» Several weeks long Add-On-Study for Internet Explorer and
Method Firefox.
» 440 Users were recruited via an Online-Access-Panel for the
study.
– All participants planned to purchase online-products during
Sample field time.
– 59% Women, 41% Men.
– 52% under 30 years, 26% 30-39 years, 22% 40 years & older.
» Testing of all websites which could be important for research-
and shoppingprocess.
Tested websites – Focus on onlineshops.
– Furthermore: price comparing-websites, searchengines and
portals, manufacturer-websites and community-websites.
Field time » From Juli 1st until September 8th 2008.

39. Expiry of Behaviour Trackings for Users
Final survey:
First survey: Field time:  Afterexploratio
 Screening  Handle the n
 Installation of AddOn shoppinglist  Imagemeasure
 Questions to online-  Monitoring of ment of
shopping surf behaviour onlineshops
 Event-Interviews  Deinstallation
of AddOn
2 weeks 7 weeks 1 week

40. Technologie
Browser Extension LEOtrace® AddOn

41. The Shoppinglist
» Origin- and endpoint of all
productresearch.
» The Users were told to keep
following aspects up to date:
– Productcategory
– Productname
– (contemplated) price of
product
– State of research
» Furthermore they were told
before a research session, what
product had to been
researched.

42. Devolution of a research-session
Eventinterview
"general"
Eventinterview
"detailled product
website
Shoppinglist: Shoppinglist:
Post survey:
Instruction for If so matching of
Research-session Evaluation of
product to be price and
research-session
researched for research-status
Eventinterview
"searchfunction
Eventinterview
"ordering process"

43. Example: Research- & Shoppingsession (1)
Product for which exists a buying interest:
Nintendo Wii
Research-sessions:
 July 20th (21:12 - 21:57)
User: 244756  July 21st (18:20 - 18:41)
male,
29 years On July 20th following websites were selected:
 Geizkragen.de
 Search for "Nintendo Wii"
 cyberport.com
 discount24.de
 amazon.de

44. Example: Research- & Shoppingsession (2)
On July 21st amazon.de was accessed:
 search for "Wii"
 Game console put into shopping basket
 Search for "Mark Medlock"
 CD "Cloud Dancer" by Mark Medlock put into shopping basket
User: 244756
male,
29 years Analysis of research:
Both products were purchased online at amazon.de
Comment about order:
"Great that I could bought the CD here as well"

45. Research-Status at end of field time
Cases
Cases Quota
per user
Still in research-phase 800 36% 1,8
Bought online 920 42% 2,1
Bought offline 189 9% 0,4
No shopping interest
289 13% 0,7
anymore
Total 2.198 5,0
Basis: Total (N=440).

46. Used Onlineshops during field time
Basic: Total (N=440).

47. Rating of Onlineshops
Amazon (N=342)
eBay (N=281)
Bonprix (N=50)
Weltbild (N=57)
Neckermann (N=53)
Tchibo (N=65) Question: How helpful were
following Onlineshops during
research and purchase of the varied
Buecher.de (N=46) products?
Esprit (N=35) Basic = User of each website
Rating on a scale of 6:
Ikea (N=33) 1=Very helpful
6=Not helpful at all.
Quelle (N=80) Only a section of the answering scale is
shown .
Otto (N=81)
Buch.de (N=42)
H&M (N=36)

48. Used shops per product following
productcategories
Videogames 2,3
Tickets
Sports goods / -tools
CDs / DVDs
housewares / -tools
Consumer electronics
Fashion / Shoes
Toys
Jewellery/Watches/Accessoiries
Fitments
Perfume / Cosmetiques
Telecommunicationproducts
Healthproducts
Books
Computerhardware / -equipment
Other products
Basic: products being researched for (N=1.766).

49. Rating of experiences during research-phase
How do you rate the experiences you made during the last 6 weeks according to research and shopping in internet?
Incomplete delivery Couldn´t find anything fitting
Unsufficient productdescription/-mapping
No wanted payment options
no response on E-Mail-request Complicated navigation
Und
Delivery problems pricing not transparent
zwar:
Product not available unsufficient searchfunction Oversupply
Speedtransfer Too expensive
Tag Cloud View
Basic: Total (N=440).

50. True Intent Study
WebWerte 50

51. Area: Shop – 'Handys & Shop'
Success
in percent
successful not successful
Buy something
(N=50) 37 63
Buy something
online (N=50) 52 48
Question 10040: Did you succeed in what you wanted to do on the website today?

52. Area: Shop – 'Handys & Shop'
Structure of navigation
in percent
"Tarife &
Optionen" *
10 30
8 28 12 6
8
Handset "Handys & 9 "Aktions- "Vertrags- 24
details Datengeräte" Angebote" Verlängerung"
24
5
Login area "Prepaid-
27
("https") Angebote" * First page of the area
(Overview page)
Navigation
Cancel of remote-session
Results generated from log data.
Participants who started from the page "Handys & Shop": N=100. Only navigation steps with N > 1.

53. Area: Shop – 'Handys & Shop'
Structure of navigation – Buyers
in percent
"Handys &
11
Shop" *
37 23 Homepage
25
"Handys & "Aktions-
18 18
Datengeräte" angebote"
9
Price 35
pop-up 20
Handset Login-area
20 20
details ("https")
20 23
* First page of the area
(Overview page)
Navigation
Cancel of visit
Results generated from log data.
Participants with buying intention: N=100. Only navigation steps with N > 3.

54. Area: Shop – 'Handys & Shop'
General
 The homepage of the section 'Handys & Shop' was praised as clearly arranged and
pleasantly designed.
 The range of devices in 'Handys & Shop' was well liked.
 The choice of mobile phones was stated as complete and up-to-date.
 Particularly the phone offers for 1€ were liked by the participants.
 "Great, that there are mobile phones for one Euro, which look o.k. and
have many features."
 If the preferred phone model was not offered on the website, the range of
products was often criticized as too small.
 In this case the choice was also described as out of date.
 Participants always expect that the latest models are available on the website.
 "I searched for the Samsung E 840, thinking that XXX as market leader
always offers the most up-to-date phone models, even before other
providers offer them."
 " 'Top-Angebote' (top-offers) are on offer - latest mobile phones are
missing."

55. Area: Shop – 'Handys & Shop'
Mobile phones detail-page
? (radio)
 Understanding the meaning of the several
"function-icons" was a problem.
 "Icons completely inoperative!!!
They don't work for a comparison and
are not understandable without
explanations. Very, very bad."
 The icons did not suggest that further
explanations were available and
participants could therefore not find
them.
 The offered alt-tags were often not clearly understandable (e.g. Voice, Orga).
 The explanations were sometimes very long and it was necessary to read
carefully through them in order to understand which function was described.
 Most users only wanted to know which function was included, not its
purpose.

56. Time for Questions
WebWerte 56

57. www.sirvaluse.de
SirValUse Consulting GmbH | Schlossstraße 8g | 22041 Hamburg
Tel. +49 (40) 68 28 27-0 | Fax +49 (40) 68 28 27-20
WebWerte