My slides from StartupBootcamp Amsterdam workshop. Introducing the importance of performance from an end users point of view...

2. @jeroentjepkema
Performance :: Usability :: Experience

3. Are you ready for customers?
Why performance matters for your online strategy
Startup Bootcamp Amsterdam,
April 15, 2013

4. http://bit.ly/MW-DML13

5. Online Analytics

6. Every
website has
goals

8. Organic Search
Campaigns
Ad Network
Transactional site
Visitor
Offer
€"
Abondenment)
Upselling
Reach
Purchase step 1
€"
Purchase step 2
€"
Mailing,
alerts,
€" promotions
Conversion
€"
Disengagement) Enrolment
Impact)on)site)
€" Negative €" Positive

10. €"
Media site
Enrolment
Targeted
€"
embedded add
Add Network
Visitor
€"
€"
Advertiser site
Departure(
Impact(on(site(
€" Negative €" Positive

11. Web Analytics
measures the
movement towards
these goals

12. A simple analytics model:
A$en'on" Engagement( Conversion(
NEW
VISITORS
GROWTH CONVERSION
SEARCHES RATE
PAGES TIME
TWEETS
MENTIONS
NUMBER
OF VISITS
PER
VISIT
ON
SITE
x"
ADS SEEN ORDER
LOSS VALUE
BOUNCE
RATE

15. Why did customers drop off?
‣Price
‣Functional errors?
‣Performance issues?

16. Why did customers drop off?
‣Price
‣Functional errors?
‣Performance issues?
What’s the business impact?
‣Lost customers?
‣Revenue risked?
‣In Euros?

17. We need
context!

18. Complete Web Monitoring

19. Web Analytics
(what did they do on
the site?)
Complete Web Monitoring

20. Web Analytics Usability
(what did they do on (how did they
the site?) interact with it?)
Complete Web Monitoring

21. Web Analytics Usability Performance
(what did they do on (how did they (could they do what
the site?) interact with it?) they wanted to?)
Complete Web Monitoring

22. Web Analytics Usability Performance
(what did they do on (how did they (could they do what
the site?) interact with it?) they wanted to?)
Complete Web Monitoring
VoC
(what were their
motivations?)

23. Web Analytics Usability Performance
(what did they do on (how did they (could they do what
the site?) interact with it?) they wanted to?)
Complete Web Monitoring
VoC Social Media
(what were their (what were they
motivations?) saying?)

24. Web Analytics Usability Performance
(what did they do on (how did they (could they do what
the site?) interact with it?) they wanted to?)
Complete Web Monitoring
VoC Social Media Competition
(what were their (what were they (what are they up
motivations?) saying?) to?)

25. “Hard” data
Web Analytics Usability Performance
(what did they do on (how did they (could they do what
the site?) interact with it?) they wanted to?)
Complete Web Monitoring
VoC Social Media Competition
(what were their (what were they (what are they up
motivations?) saying?) to?)
“Soft” data

26. “Hard” data
Web Analytics Usability Performance
(what did they do on (how did they (could they do what
the site?) interact with it?) they wanted to?)
Complete Web Monitoring
VoC Social Media Competition
(what were their (what were they (what are they up
motivations?) saying?) to?)
Source: Complete Web Monitoring, Sean Power/Alistair Croll, 2009
“Soft” data

27. Performance?

28. Abandonment"Rate"Across"200+"Sites"/"177+"Million"Page"Views"
Over"2"weeks"/"All#Browsers#
30"
25"
Abandonment"Rate"(%)"
20"
Abandonment*Rate*,*
15" All*Browsers*
10"
5"
0"
0" 1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15"
Page"Load"Time"Band"(sec.)"
Source: Gomez real user monitoring

29. Abandonment"Rate"Across"200+"Sites"/"177+"Million"Page"Views"
Over"2"weeks"/"All#Browsers#
Over"2"weeks"/"All#Browsers#vs.#iPhone#Safari#
30"
25"
Abandonment"Rate"(%)"
20" Abandonment*Rate*,*
All*Browsers*
Abandonment*Rate*,*
15" All*Browsers*
Abandonment*Rate*,*
iPhone*Safari*
10"
5"
0"
0" 1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 11" 12" 13" 14" 15"
Page"Load"Time"Band"(sec.)"
Source: Gomez real user monitoring

30. I
(inter)action
@jeroentjepkema, MeasureWorks

31. 0

32. 0,3
Instantaneous: I like it!
0

33. 1
Interaction: Let’s conversate...
0,3
Instantaneous: I like it!
0

34. 3
Mmm, shall I click away?
1
Interaction: Let’s conversate...
0,3
Instantaneous: I like it!
0

35. 10
Only if the task/content is relevant
3
Mmm, shall I click away?
1
Interaction: Let’s conversate...
0,3
Instantaneous: I like it!
0

36. 10
Only if the task/content is relevant
3
Mmm, shall I click away?
1
Do I want to conversate?
This is where the magic happens
0,3
Do I like it?
0

37. How we perceive “interaction”?

38. How we perceive “interaction”?
Sensitive for specific tasks

39. How we perceive “interaction”?
Sensitive for specific tasks
Always compared to past experiences

40. How we perceive “interaction”?
Sensitive for specific tasks
Always compared to past experiences
Task completion has positive impact

41. How we perceive “interaction”?
Sensitive for specific tasks
Always compared to past experiences
Task completion has positive impact
Slowdowns hurts

42. Experience
the
difference
Tested with webpagetest.org (3G traffic shaping, 2000Kbps Down, 1000 Kbps Up, 150ms latency)

43. Ultimate goal = “flow state”

44. Relevant content,
to support
task completion

45. With fast delivery,
focus on interaction

46. “” Flow is an
“optimal experience”
that is
“intrinsically enjoyable”

47. Myth busted?

48. Experiment 1
Technology vs. Perceived

49. Which website would you recommend in terms of
performance?
Desktop only, fixed
Buy a summer holiday
100 users selection of shops
online
(Travel30)

50. Ranking Perceived Technology
1 Sunweb
2 Globe
3 Neckermann
4
D-reizen
5
Arke
Technology vs. Perceived Study performed by MeasureWorks for Emerce eTravel, 2013

51. Ranking Perceived Technology
1 Sunweb Sunweb
2 D-reizen Globe
3 Neckermann Neckermann
4
Arke D-reizen
5
Globe Arke
Technology vs. Perceived Study performed by MeasureWorks for Emerce eTravel, 2013

52. Ranking Perceived Technology
1 Sunweb Sunweb
2 D-reizen Globe
3 Neckermann Neckermann
4
Arke D-reizen
5
Globe Arke
Technology vs. Perceived Study performed by MeasureWorks for Emerce eTravel, 2013

53. Experiment 2

54. Mobile shopping experience setup....
Perform 2 tasks:
100 mobile users
‣Buy a book Mobile browsers only
‣Buy a T-shirt via 3G, no WiFi

55. Speed

57. 50
Many functional
37,5
issues reported
Round 1 with Zalando
25
42,1 45
12,5
20 24
15,4
7,7 3,9
0 4
Zalando HM V&D Tom Tailor
Design Speed Mobile Readiness Other
1. Buy a T-shirt 2. Review
40
30
Round 2
20
34 54
10
12 15,4 21 18
8 6
0
Zalando HM V&D Tom Tailor Design Speed Mobile Readiness Other
Mobile Only research:
- Task completion: Only use smartphone to buy a book
- N = 100, users range from 20-65

58. Nexus - Android - 3G
Tested with webpagetest.org (3G traffic shaping, 2000Kbps Down, 1000 Kbps Up, 150ms latency)

59. Nexus - Android - 3G
Tested with webpagetest.org (3G traffic shaping, 2000Kbps Down, 1000 Kbps Up, 150ms latency)

60. Mobile
readiness

61. 100
75
50
44 46
25 24 26
16 16 16
12
0
Bol.com Selexyz Bruna Other Design Speed Mobile Readiness Other Design Speed Mobile Readiness Other
1. Buy a book 2. Review 3. Buy at a different store
Mobile Only research:
- Task completion: Only use smartphone to buy a book
- N = 100, users range from 20-65

62. Bol.com Selexyz De Boekerij ECI Bruna Cosmox

63. Target your audience!
40%
Native Mobile Web
58% 55%
30%
20% 40% 40%
Mobile Web Native
42% 45%
25%
10%
18%
< 34 > 35
0%
Speed Readiness
< 34 > 35

64. Recipe for
performance

65. You
1.
Eco-System of
me

66. Deliver the same experience to all your (mobile) users

67. Regular website Native Application
Deliver the same experience to all your (mobile) users

68. Regular website Optimized Mobile Web Native Application
Deliver the same experience to all your (mobile) users

69. Mobile first

70. Prioritize your business goals
Mobile first

71. Deliver content accordingly
Prioritize your business goals
Mobile first

72. #2
Design for
(mobile)
use cases

73. Focus on
task
completion

74. With
mobile
conditions
in mind

75. Optimized for the right screen

77. Native app Mobile Web - Portrait
Mobile Web - Landscape

79. #fail

80. Why it matters? 76% of mobile
consumers won’t retry more than twice
Source: Compuware, 2012, Engaging the tablet user; What to expect from web sites?

81. Why it matters? 89% of (mobile)
consumers will leave to the competition
Source: Oracle, 2012, Customer Experience Survey

82. #3
Collecting
data

83. (CLOUD) DATA CENTER INTERNAL USERS INTERNET CUSTOMERS
Third-party/
Cloud Services
Storage DB Servers Web Servers Major Local
This is what you control... Network What you’re blamed for..
ISP
Content
ISP
Delivery Mobile
Load Networks Carriers
Middleware App Balancers
Mainframe Servers Servers

84. Measuring performance?

85. Measuring performance? Outside-in, from
the browser perspective...

86. Performance
Measurement
Toolkit

87. Synthethic
Real User Benchmark
Monitoring
Performance Measurement toolkit

88. Simulate business transactions

89. Via multiple devices & browsers

90. From multiple locations...

91. Used for error detection & Root Cause Analysis

92. Used for error detection & Root Cause Analysis

93. Used for error detection & Root Cause Analysis

94. Ultimately, synthetic monitoring shows you if your
site’s working or not...

95. But, synthetic isn’t enough...

96. Synthetic heartbeat
Real Users
But, synthetic isn’t enough...

97. Real User
Real User Benchmark
Monitoring
Performance Measurement toolkit

98. Navigation timing
2 ways of measuring...
Browser RUM
Disclaimer: There’s also third category Datacenter RUM, this will not be
covered in this out of the presentation. Contact me if your want details

99. h"p://www.w3.org/TR/naviga2on32ming/5
Some background info:
Navigation timing
http://66.7percentangel.com/2011/12/breaking-down-onload-event-performance-bookmarklet/
http://www.html5rocks.com/en/tutorials/webperformance/basics/
http://www.w3.org/TR/2011/CR-navigation-timing-20110315/#nt-dom-content-event-start

100. http://caniuse.com/nav-timing

101. 1 2 3 4
As pages execute, After onload tag send
Insert tag (.js file) into Pages are requested
tag collects detailed report for
(mobile) web pages from browser/device
performance metrics further analysis
tag.js
tag.js
tag.js
tag.js
Browser RUM

102. Gomez

103. Gomez

104. Soasta

105. Torbit

106. Relies on navigation timing API, custom variables can be added
Google Analytics

107. New Relic

108. Ultimately, Real User Monitoring shows you how many
users are affected by bad performance...

109. Real User
benchmarking
Performance Measurement toolkit

110. First view
11.349s
Real usage
Repeat view
4.357s

111. Average page load time per bandwith (seconds) for Dutch eTravel30
8,8 1,5 mbps
10 mbps
20 mbps
3,9
3,4
56%
13%
Bandwith

112. Average page load time per Browser (seconds) Dutch eTravel30
IE7
6,5
IE8
3,9 IE9
3,6
40%
8%
Devices used

113. Webpagetest.org

114. Object level
!
! Webpagetest.org
Optimization tips
!

115. Webpagetest.org

116. Ultimately, Real User benchmarking gives you periodic
insight in real usage scenario’s...

117. Performance
surveys
Performance Measurement toolkit

118. Usability is
everywhere

119. Homepage Homepage Last Minutes
Heatmaps

120. Expected Experience vs. Delivered Experience
Surveys

121. 7
Average experience rating
6
5
4
3
2
1
1 2 3 4 5 6 7
Average expectation rating
Find a website
Search for a book
Order a book

122. 7
Average experience rating
6
5
4
3
2
1
1 2 3 4 5 6 7
Average expectation rating
Find a website
Search for a book
Order a book

123. 7
Average experience rating
6
5
4
3 Fix it fast
2
1
1 2 3 4 5 6 7
Average expectation rating
Find a website
Search for a book
Order a book

124. Optimized vs
Non-optimized

125. Selecting items
- +

126. To login or not?

127. Ultimately, performance surveys give you periodic insight
in task completion issues...

128. Synthethic monitoring Real User Monitoring Real User benchmarking Performance surveys
Performance Measurement toolkit

129. Synthethic monitoring Real User Monitoring Real User benchmarking Performance surveys
Benefits Benefits Benefits Benefits
Heartbeat, runs without traffic Real usage information from Testing of user scenario’s Soft data feedback
Test specific customer journeys all users!! with real devices/bandwith Abandonment optimization
Object level detail Trending/Optimization Optimization details Test before go live
Collect detailed alerts, including Business impact Competitive scan
root cause analysis
Desktop/Mobile Site Desktop/Mobile Site Desktop/Mobile Site Real User
Desktop/Mobile Site
Websitepulse LogNormal Webpagetest Wufoo
Watchmouse Torbit Browserstack Usabilla
Alertsite New Relic Loop11
Google Analytics Crazyegg
Mobile Apps Mobile Apps Mobile Apps Mobile Apps
Gomez Gomez Perfecto Mobile Loop11
Keynote New Relic Device Anywhere
Localytics
Google Analytics
Performance Measurement toolkit

130. #4
Service Levels

131. Defining service levels & thresholds...

132. First thing is to establishing a baseline:

133. First thing is to establishing a baseline:
A pre-defined set of metrics

134. First thing is to establishing a baseline:
A pre-defined set of metrics
that describes normal behavior

135. First thing is to establishing a baseline:
A pre-defined set of metrics
that describes normal behavior
in order to detect variancies

136. First thing is to establishing a baseline:
A pre-defined set of metrics
that describes normal behavior
in order to detect variancies
and to be comparable within historic context

137. Then define service levels & thresholds...

138. Purchasing a book, Customer journey
must be completed (speed), Metric: Speed
where every page loads under 4 sec., Target: Sec
using IE8 and higher, User scenario
from any location in the Netherlands, User locations
for 95% of all users, Percentile
every day between 6am and 12pm, Window
measured with Real User Monitoring. Collection type
Source: Metrics 101, Velocityconf 2010

139. #5
Data Driven

140. Page
speed

141. Page Load Time: Homepage
Time to First Byte Start Render Document complete Fully loaded
Sunweb 0,20s 1,45s 3,08s 3,41
Arke 0,30s 3,10s 3,39s 5,51
Top30 Average 0,39s 2,97s 5,32s 7,04
Neckermann 0,44s 2,03s 5,40s 6,20
D-reizen 0,35s 4,30s 7,24s 7,97
Globe 0,32s 2,80s 7,79s 8,91

142. Time to first byte
Speed: 9,6s
TTFB: 0,71s
Worldticketcenter.nl
Speed: 13,6s
TTFB: 3,19s
NShispeed.nl

143. Page
size

144. Elements per page avg. Page Size
(Kb)
Sunweb 0% 15% 2% 80% 3% 233
D-reizen 22% 4% 48% 24% 2% 673
Globe 0% 33% 5% 50% 11% 713
Average 24% 6% 57% 10% 4% 1142
Arke 24% 6% 50% 20% 1% 1142
Neckermann 0% 27% 3% 68% 1% 1528
Javascript CSS Images Flash HTML

145. 3d Party
(content)

146. #Requests 3d Party objects % 3d Party
Sunweb 58 2 3%
Globe 84 5 6%
Neckermann 99 16 16%
D-reizen 111 19 17%
Travel30 average 90 17 19%
Arke 50 17 34%

148. Worldticketcenter.nl
30
25
22,0
20 19,0
15
10
5 3,2 3,6 3,1
2,9 2,5 2,6 2,5 2,5 2,7 2,8 2,5 2,6 2,7 3,0 2,7 2,5 2,8 2,8 2,6 2,5 2,5 2,8 2,6
2,4
0
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 1 2 3 4 5 6 7 8 9 10
31 May 2012 1 June 2012

149. Worldticketcenter.nl
30
25
20
15
10
5
0
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 1 2 3 4 5 6 7 8 9 10
31 May 2012 1 June 2012

150. First
visuals

151. Sunweb D-reizen Arke Globe Neckermann
5,25
5,05
4,9 4,9 4,9
4,8
4,6
4,3 4,2
3,1
2,8 2,8 2,9
2,4
2,05
2,3
1,9
2,1 Start 1,9
1,45 render
Home page Landingpage Sun Detailed Page Booking Page

152. Globe
Neckermann
Sunweb
D-reizen
Arke
First visuals Homepage

153. First visuals details page
Globe
Neckermann
Sunweb
D-reizen
Arke

154. “Variation”
Speed
variation

155. 5
4
3
2
1
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Daily Average Min Max

156. Revenue risked
5
4
3
above 15-20% variance is detected
2
1
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Daily Average Min Max

157. 9,26
8,98
7,79
7,79
7,23
7,24
7,24
7,29
6,67
5,4
5,40
4,97
4,80
4,61
3,38
3,39
3,08
3,08
2,97
1,89
Sunweb D-reizen Arke Globe Neckermann

158. Find the
gaps...

159. Loadtime
Loadtime
Functionaliteit Availability (Document
0-­‐2 0-­‐4
(
fully
l oaded
)
complete)
www.homepage.nl 97,0% 2,20 3,50 10% 85%
www.homepage.nl/autoverzekeren 99,0% 2,50 3,30 20% 90%
www.homepage.nl/autoverzekeren/afsluiten 95,0% 2,30 3,60 15% 70%
www.homepage.nl/contact 95,0% 2,43 3,57 20% 67%
www.homepage.nl/zoeken 94,0% 2,48 3,62 23% 59%
www.homepage.nl/productoverzicht 93,0% 2,53 3,67 25% 52%
www.homepage.nl/ab-­‐test/variant1 92,0% 2,58 3,72 28% 44%
www.homepage.nl/ab-­‐test/variant2 91,0% 2,63 3,77 30% 37%
www.homepage.nl/facebookplugins 90,0% 2,68 3,82 33% 29%
www.homepage.nl/muziekpagina 89,0% 2,73 3,87 35% 22%
www.homepage.nl/faq 88,0% 2,78 3,92 38% 14%
Service Level Report

160. 30 3000
Pagesize (kb)
20 2000 URL's 3rd
URL’s
URL's
10 1000
Page Size (KB)
0 0
33 34 35 36 37 38 39 40
Your website vs. Competition

161. Discovering patterns

162. Revenue risked 10.000 visitors
Optimale flow
Speed (s)
Availability (%)
Bouncerate (%)
Conversie
Page 1
5,5
100
70
3000
Page 2
13,0
100
75
750
Page 3
2,0
100
60
300
Page 4
3,5
88,5
5
252
10.000 visitors
Actual flow
Speed (s)
Availability (%)
Bouncerate (%)
Conversie
Page 1
8,0
100
75
2500
Page 2
15,5
100
80
500
Page 3
2,0
100
60
200
Page 4
8,0
88,5
5
168

163. Performance vs. Sentiment

164. Web%traffic%%(Pageviews)!
1.000.000!
1.200.000!
200.000!
400.000!
600.000!
800.000!
0!
Mei;09!
jun;09!
jul;09!
aug;09!
sep;09!
okt;09!
nov;09!
dec;09!
jan;10!
feb;10!
mrt;10!
apr110!
mei10!
jun;10!
jul;10!
aug;10!
sep;10!
okt;10!
nov;10!
dec;10!
Jan;11!
feb;11!
Max.%number%of%pageviews%per%hour%!
mrt;11!
apr;11!
Christmas/EOY%
mei;11!
MAX!capacity!
Safety!capacity!
Traffic!Realized!
Traffic!Forecast!
jun;11!
jul;11!
aug;11!
sep;11!
okt;11!
nov;11!
dec;11!
Capacity Management

165. #6
Continuous
performance
diet

166. Low hanging fruit
‣ Reduce page size
‣ Enable Gzip
‣ Reduce the number of
roundtrips
‣ Structure the page to improve
rendering & download
‣ Enable caching
‣ Clean up & remove duplicates
‣ Minify javascript

167. Low hanging fruit Advanced optimization
‣ Reduce page size ‣ Non-blocking scripts
‣ Enable Gzip ‣ Optimizing images
‣ Splitting payloads
‣ Reduce the number of ‣ Asynchronous loading
roundtrips
‣ Structure the page to improve ‣ Pre-loading
rendering & download ‣ Etc.
‣ Enable caching
‣ Clean up & remove duplicates
‣ Minify javascript

169. Page size:
1660 Kb
# requests:
89
Time to render:
1,29 sec.
Document complete:
3,68 sec.
Not Optimized

170. Page size: Page size:
1660 Kb 498 Kb
# requests: # requests:
89 48
Time to render: Time to render:
1,29 sec. 1,10 sec.
Document complete: Document complete:
3,68 sec. 1,54 sec.
Not Optimized Optimized

171. Page size: Page size:
1660 Kb 498 Kb
# requests: # requests:
89 48
Time to render: Time to render:
1,29 sec. Timeto first visual: 1,10 sec.
1,10 sec. vs 1,90 sec.
Document complete: Document complete:
3,68 sec. 1,54 sec.
Not Optimized Optimized

172. To summarize...

173. Performance is
a requirement

174. Create great
service
Net Performance Score

175. Real users,
real devices,
real data

176. Slice &
Dice

177. Detecting
Trends

178. Be ready
to recover

179. Create an
experience
that Don’t let them
click away
performs

180. Questions? Or, else... ;-)

181. Page Load Time: Homepage
10,5
9,7
9,4
8,2
6,8 6,9
5,8
5,1
4,6
4,2
3,9
3,7 3,7
3,4
3,1
2,5 2,5
1,9
7write Alive Shoes Easyschool Fuel iLost Kinems Presentain Swogo Twoodo
Synthetic Webpagetest

182. 25,2
23,5 23,7
17,8
16,6
15,3 15,7
14,7
13,6 13,5
10,5 10,2
9,4 9,7 9,4
8,7 8,9
8,2
7,6 7,7
6,9
4,2 4,6
3,9 3,7
3,4
2,3
7write Alive Shoes Easyschool Fuel iLost Kinems Presentain Swogo Twoodo
NL US Asia
Global coverage

183. iPhone 4s IE8
16
14,5
13
11 10,7
10,5
9,4 9,7
9,1 9,3
8,2
7,6
5,8
4,6
4,2 3,9
3,4 3,7
7write Alive Shoes Easyschool Fuel iLost Kinems Presentain Swogo Twoodo
Mobile

185. > 1Mb uncomprized text, can be reduced to 450Kb
> 640Kb Javascript, of which 1 is blocking (460Kb)

186. Cloudflare?

187. >400Kb images

189. Regular homepage
Optimized
homepage

193. Thanks! More questions?
M: jtjepkema@measureworks.nl
T: @jeroentjepkema
W: www.measureworks.nl