Slides from session at the Outfox Your Competition event in Stockholm... About why we need a balance between performance, ux & content. And of course, how to measure performance…

2. @jeroentjepkema
Founder & CEO

3. Content
vs.
Performance

4. How we perceive performance....

5. How we perceive performance....
15%-20% worse than in reality

6. How we perceive performance....
15%-20% worse than in reality
Slow downs have more impact

7. How we perceive performance....
15%-20% worse than in reality
Slow downs have more impact
Task completion has positive impact

8. How we perceive performance....
15%-20% worse than in reality
Slow downs have more impact
Task completion has positive impact
Always compared to past experiences

9. A simple online business model:
Marketing
Conversion Optimization
New
visitors
Search
Tweets
Mentions
ADs seen
Conversion
rate
Growth
Number of
visits
Loss
Bounce
rate
Revenue
Pages per
visit
Time on
site
X
Order
value

10. Dutch consumers visit 18-23 travel websites
before they finally convert
Source: TNS/Nipo

11. Performance influence cycle
Search/Orientation phase
1
Perception of performance
http://twinkle100.measureworks.nl

12. Performance influence cycle
Stimulate content/
conversion
4
Search/Orientation phase
5
1
Perception of performance
http://twinkle100.measureworks.nl
Delivered
performance
3
2
Your website

13. Need
for
Speed
@jeroentjepkema, MeasureWorks
@jeroentjepkema, MeasureWorks

14. Abandonment"Rate"Across"200+"Sites"/"177+"Million"Page"Views"
Over"2"weeks"/"All#Browsers#
30"
Abandonment"Rate"(%)"
25"
20"
Abandonment*Rate*,*
All*Browsers*
15"
10"
5"
0"
0"
1"
2"
3"
Source: Gomez real user monitoring
4"
5"
6"
7"
8"
9"
Page"Load"Time"Band"(sec.)"
10"
11"
12"
13"
14"
15"
It only
takes a
second...

15. Abandonment"Rate"Across"200+"Sites"/"177+"Million"Page"Views"
Abandonment"Rate"Across"200+"Sites"/"177+"Million"Page"Views"
Over"2"weeks"/"All#Browsers#vs.#iPhone#Safari#
Over"2"weeks"/"All#Browsers#
25"
Abandonment"Rate"(%)"
30"
25"
Abandonment"Rate"(%)"
30"
20"
20"
15"
15"
10"
10"
5"
5"
0"
Abandonment*Rate*,*
All*Browsers*
Abandonment*Rate*,*
All*Browsers*
Abandonment*Rate*,*
iPhone*Safari*
0"
0"
1"
1"
2"
2"
3"
4"
5"
6"
7"
8"
9" 10" 11" 12" 13" 14" 15"
0"
3"
4"
5"
6"
7"
8"
9" 10" 11" 12" 13" 14" 15"
Page"Load"Time"Band"(sec.)"
Source: Gomez real user monitoring
Page"Load"Time"Band"(sec.)"
Source: Gomez real user monitoring
..To click
away

16. 0
Source: Jakob Nielsen

17. 1
0,3
0
Interaction: Let’s conversate...
Instantaneous: I like it!
Source: Jakob Nielsen

18. 3
1
0,3
0
Mmm, shall I click away?
Interaction: Let’s conversate...
Instantaneous: I like it!
Source: Jakob Nielsen

19. 10
Only if the task/content is relevant
3
1
0,3
0
Mmm, shall I click away?
Interaction: Let’s conversate...
Instantaneous: I like it!
Source: Jakob Nielsen

22. 0:00 sec.
Conversation
starts

23. 0:00 sec.
Conversation
starts
2.5 sec.
First reply

24. 4.5 sec.
Finish
conversation
0:00 sec.
Conversation
starts
2.5 sec.
First reply

25. 4.5 sec.
Finish
conversation
0:00 sec.
Conversation
starts
Comfort zone
Tolerated
2.5 sec.
First reply
Frustrated

26. The holy grail in experience is
“flow state”

27. “”
Flow is an
optimal experience
that is
“intrinsically enjoyable”

28. Provide relevant
content to support
task completion

29. Deliver it fast,
focus on perception

30. http://www.nytimes.com/2012/08/19/opinion/sunday/why-waiting-in-line-is-torture.html?pagewanted=all&_r=0

31. http://answers.yahoo.com/question/index?qid=1005081200005

32. http://amzn.to/1biSi6C

33. That’s not my
website?

34. Over 60% of people that experienced a slow website ranked
design 2 points lower compared to a fast experience
Slow
Average
Fast
% of respondents that rated the actual website speed
100
75
50
25
0
1
2
3
Design score (1=bad - 5=beautiful)
4
5

35. Experience
the
difference
Tested with webpagetest.org (3G traffic shaping, 2000Kbps Down, 1000 Kbps Up, 150ms latency)

37. Perception
vs.
Technology

38. Desktop shopping experience setup....
Buy a summer
holiday:
100 users
‣ From 5 selected websites
‣ Rank from fastest to
slowest after task execution
No measurements tools allowed ;-)
Desktop only,
either via LAN or WiFi

39. Perceived
Technology
(User Ranking)
(Full page download)
1
Sunweb
2
D-reizen
3
Neckermann
4
Arke
5
Globe
Technology vs. Perceived
Study performed by MeasureWorks for Emerce eTravel, 2013

40. Perceived
Technology
(User Ranking)
(Full page download)
1
Sunweb
Sunweb
2
D-reizen
Globe
3
Neckermann
Neckermann
4
Arke
D-reizen
Globe
Arke
5
Technology vs. Perceived
Study performed by MeasureWorks for Emerce eTravel, 2013

41. Perceived
Technology
(User Ranking)
(Full page download)
1
Sunweb
Sunweb
2
D-reizen
Globe
3
Neckermann
Neckermann
4
Arke
D-reizen
Globe
Arke
5
Technology vs. Perceived
Study performed by MeasureWorks for Emerce eTravel, 2013

42. Globe
Neckermann
Sundio
D-reizen
Arke

43. M-commerce?

44. Mobile shopping experience setup....
100 mobile users
Perform 2 tasks:
‣Buy a book
‣Buy a T-shirt
Mobile browsers only
via 3G, no WiFi

45. #1:
Speed

47. Functional issues
reported with
Zalando
50
37,5
Round 1
25
45
42,1
12,5
15,4
0
Zalando
HM
20
7,7
V&D
24
3,9
Tom Tailor
4
Design
1. Buy a T-shirt
Speed
Mobile Readiness
Other
2. Feedback
40
30
Round 2
20
10
12
0
54
34
Zalando
Mobile Only research:
- Task completion: Only use smartphone to buy a book
- N = 100, users range from 20-65
15,4
HM
8
V&D
Tom Tailor
21
Design
18
Speed
6
Mobile Readiness
Other

48. Nexus - Android - 3G
Tested with webpagetest.org (3G traffic shaping, 2000Kbps Down, 1000 Kbps Up, 150ms latency)

49. Nexus - Android - 3G
Tested with webpagetest.org (3G traffic shaping, 2000Kbps Down, 1000 Kbps Up, 150ms latency)

50. #2:
Usability

51. Bol.com
Selexyz
De Boekerij
ECI
Bruna
Cosmox
Mobile
Readiness

52. 100
75
50
46
44
26
24
25
16
0
Bol.com Selexyz
Bruna
Other
1. Buy a book
Mobile Only research:
- Task completion: Only use smartphone to buy a book
- N = 100, users range from 20-65
Design
Speed
16
Mobile Readiness
2. Feedback bol.com
16
Other
Design
12
Speed
Mobile Readiness
Other
3. Feedback from different stores

53. Mobile
Readiness

54. -
One touch?
+

55. Login or
not?

56. Mobile
Search

57. Mobile
Banking

58. The mobile user experience…
50 users of ING, ABN
and Rabobank
Rate the user
experience
Select home insurance

59. There’s an app
for that?!

60. Mobile satisfaction compared to desktop
80%
74%
77%
Usability
Speed
23%
Design
Content
Satisfaction
Mobile Banking research:
- Focus on task completion
- N = 50, users range from 20-65

61. Mobile satisfaction compared to desktop
80%
74%
77%
Preference per type of task
77%
72%
38%
34%
23%
Design
Content
Usability
Satisfaction
Mobile Banking research:
- Focus on task completion
- N = 50, users range from 20-65
Speed
Check Balance
Transfer
3d Party payments View history
Mobile task
preference

62. Mobile satisfaction compared to desktop
Mobile versus Desktop frequency
Preference per type of task
Mobile
80%
74%
77%
77%
72%
38%
34%
23%
Design
Content
Usability
Satisfaction
Mobile Banking research:
- Focus on task completion
- N = 50, users range from 20-65
Speed
Check Balance
Transfer
3d Party payments View history
Mobile task
preference
Daily
Weekly
2-weekly
Mobile Usage
Monthly

63. Mobile
websites?

64. Mobile
Desktop
71%
76%
82%
61%
34%
42%
37%
22%
Design
Content
Usability
Speed
Design
Task satisfaction Desktop vs. Mobile
Mobile Banking research:
- Focus on task completion
- N = 50, users range from 20-65
Content
Usability
Speed

66. Speed matters

67. Complete Web Monitoring

68. Web Analytics
Usability
Performance
(what did they do on
the site?)
(how did they
interact with it?)
(could they do what
they wanted to?)
Complete Web Monitoring
VoC
Social Media
Competition
(what were their
motivations?)
(what were they
saying?)
(what are they up
to?)

69. “Hard” data
Web Analytics
Usability
Performance
(what did they do on
the site?)
(how did they
interact with it?)
(could they do what
they wanted to?)
Complete Web Monitoring
VoC
Social Media
Competition
(what were their
motivations?)
(what were they
saying?)
(what are they up
to?)
“Soft” data

70. “Hard” data
Web Analytics
Usability
Performance
(what did they do on
the site?)
(how did they
interact with it?)
(could they do what
they wanted to?)
Complete Web Monitoring
VoC
Social Media
Competition
(what were their
motivations?)
(what were they
saying?)
(what are they up
to?)
“Soft” data

71. Data driven

72. #1:
Performance
SLA

73. Establish a baseline:

74. Establish a baseline:
A pre-defined set of metrics

75. Establish a baseline:
A pre-defined set of metrics
that describes normal behavior

76. Establish a baseline:
A pre-defined set of metrics
that describes normal behavior
in order to detect variancies

77. Establish a baseline:
A pre-defined set of metrics
that describes normal behavior
in order to detect variancies
and to be comparable within historic context

78. A practical (and easy DIY) example:

79. Purchasing a book,
must be completed (speed),
where every page loads under 3 sec.,
Customer journey
Metric: Speed
Target: Sec
using IE9 and higher,
User scenario
from any location in the Netherlands,
User locations
for 90% of all users,
every day between 6am and 12pm,
measured with Real User Monitoring.
Percentile
Window
Collection type
Read more: Metrics 101, Velocityconf 2010

80. Performance SLA

81. 100%
150%
125%
100%
75%
50%
50%
25%
25%
0%
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Sessions
Bounce rate
75%

82. LD50: 5.2 sec.
100%
150%
100%
125%
100%
75%
50%
50%
25%
25%
0%
0%
1
2
3
4
5
6
7
8
9
10
11
12
Baseline: 2,7 sec.
Poverty Line: 10,6 sec.
13
14
Sessions
Bounce rate
75%

83. #2:
Collecting
Data

84. (CLOUD) DATA CENTER
INTERNAL USERS
CUSTOMERS
INTERNET
Third-party/
Cloud Services
Storage
Major
ISP
DB Servers Web Servers
Network
Mainframe
Middleware
Servers
App
Servers
Load
Balancers
Local
ISP
Content
Mobile
Delivery
Networks Carriers

85. (CLOUD) DATA CENTER
INTERNAL USERS
CUSTOMERS
INTERNET
Third-party/
Cloud Services
Storage
Major
ISP
DB Servers Web Servers
Network
Mainframe
Middleware
Servers
App
Servers
Load
Balancers
This is what you control...
Local
ISP
Content
Mobile
Delivery
Networks Carriers

86. (CLOUD) DATA CENTER
INTERNAL USERS
CUSTOMERS
INTERNET
Third-party/
Cloud Services
Storage
Major
ISP
DB Servers Web Servers
Network
Mainframe
Middleware
Servers
App
Servers
Load
Balancers
This is what you control...
Local
ISP
Content
Mobile
Delivery
Networks Carriers
What you’re blamed for..

87. Measuring performance?

88. Measuring performance? Outside-in, from
the browser perspective...

89. Synthetic
Monitoring

90. Simulate business transactions

91. Lot’s of choice, select on ease of use

92. Ultimately, synthetic monitoring shows you if your
site’s working or not...
(best used for operations and service reports)

93. Real User
Monitoring

94. But, synthetic isn’t enough...

95. Synthetic heartbeat
Real Users
But, synthetic isn’t enough...

96. Relies on navigation timing API, custom variables can be added
Turn on
“Site Speed”
in your GA
account!
http://bit.ly/1ivGWTt

97. Or try
Soasta “Lite”
http://bit.ly/I5ngs8

98. Ultimately, Real User Monitoring shows you how many
users are affected by bad performance...
(best used for capacity management and conversion optimization)

99. #3:
Performance
analytics

100. Service levels
Service Level Report
KPI’s
082
084
Performance (avg)
Page 1
www.homepage.nl
Loadtime.
Loadtime
Availability (Document.
(.fully.loaded.)
complete)
Competition
You
5
97,0%
2,20
3,50
10%
85%
Page 2
ge.nl/autoverzekeren
99,0%
2,50
3,30
20%
90%
Page 3
overzekeren/afsluiten
95,0%
2,30
3,60
15%
70%
95,0%
2,43
3,57
20%
67%
1
Page 5
homepage.nl/zoeken
94,0%
2,48
3,62
23%
59%
Branche
2
Page 4
homepage.nl/contact
Competitors
0
Page 6
e.nl/productoverzicht
93,0%
2,53
3,67
25%
92,0%
2,58
3,72
28%
44%
3
33
34
35
ge.nl/ab8test/variant2
Page 8
91,0%
2,63
3,77
30%
37%
ge.nl/facebookplugins
Page 9
90,0%
2,68
3,82
33%
89,0%
2,73
3,87
35%
22%
ww.homepage.nl/faq
Page 11
88,0%
2,78
3,92
38%
14%
37
38
39
40
# 6
3,4 sec
Is your ranking
against
competitors
is the average
performance of your
competitors
40%
40% of your
compeitors are
faster than you
29%
page.nl/muziekpagina
Page 10
36
Week
52%
Page 7
ge.nl/ab8test/variant1
4
Source:
Availability: Gomez
Loadtime: Soasta
Competition: Webpagetest.org

101. Speed vs. Engagement per month (Non-Bounce)
90
78,75
56,25
Pageviews (#)
Non-Bounce rate (%)
67,5
45
33,75
22,5
11,25
0
0-0,5
0,5-1
1-1,5
1,5-2
2-2,5
2,5-3
3-3,5
3,5-4
4-5
5-6
Non-Bounce CategorienaamPageviews
#
6-7
7-8
8-9
9-10
>10

102. Online Satisfaction Report
100
Availability (%)
80
400
0
-400
-800
-1200
60
40
20
Page 1
Page 11
0
Mentions (#)
1200
1200
800
Page 2
Page 3
Page 4
Positive Messages
Source:
Availability: Gomez
Sentiment: Twitter, Facebook
Negative Messages
Data collected from ING.nl
Availability
Messages from Twitter & Facebook, collected with Coosto

103. Revenue risked
100.000 visits
(Performance/Time Period/Percentiles)
Optimal Flow (P_25)
Bouncerate (%)
Potential Conversion
2.1
100
49%
51.000
Page 2
4.0
100
51%
24.990
Page 3
2.0
100
60%
9996
Page 4
3.5
85
25%
6372
Actual Flow (P_85)
Speed (sec.)
Availability (%)
Bouncerate (%)
Actual Conversion
Page 1
4.1
100
59%
41.000
Page 2
6.3
100
54%
18.860
Page 3
Speed: Soasta
Availability: Webpagetest.org
Availability (%)
Page 1
Source:
Speed (sec.)
2.0
100
60%
7544
Page 4
4.7
85
53%
3013
0-2,7 sec.
2,7-3,6 sec.
>3,6 sec.

104. If anything else...

105. Deliver
fast(er)

106. Design for
Real Usage

107. Find out
where to
improve?

108. Focus on a fast
experience
…without it,
NO time for
content

109. Go check your site @
Webpagetest.org
(For settings: Akamai State of the Internet)

110. View slides: bit.ly/MW-OYCnfs13
Or join a meetup:
Online Optimizers & Analytics: bit.ly/1awKBwS

111. Thanks! More questions?
M: jtjepkema@measureworks.nl
T: @jeroentjepkema
W: www.measureworks.nl