Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Metrics, metrics everywhere
(but where the heck do you start?)
@tameverts @cliffcrocker
#velocityconf
Who cares about performance today?
How do I measure performance?
How fast am I?
How fast should I be?
How do I get there?
The myth of a single metric
Start render DNS TCP TTFB
DOM loading DOM ready Page load Fully loaded
User timing Resource timing Requests Bytes in
Speed...
Who cares about performance?
“47% of consumers expect a web
page to load in 2 seconds or less.”
Akamai, 2009
1s = $27M DNS
144ms
Start render
1.59s
Hero image render
2.01s
How do I measure performance?
Androiddevicefragmentation
OpenSignal,August2015
How do we measure UX for mobile?
https://gist.github.com/larahogan/b681da601e3c94fdd3a6
RUM versus plus synthetic
Real User Monitoring 101
Technology for collecting performance metrics
directly from the end user’s browser
Involves instrumenting your site via Ja...
What makes RUM great
 Always on
 Every user, every browser, everywhere
 Able to capture human behavior/events
 Only ge...
Questions your RUM data can answer
What are
my users’
environments?
How do visitors move
through my site?
How are my
third-party scripts
performing in
real time?
What impact does
performance have
on my business?
Synthetic Monitoring 101
Uses automated agents (bots) to measure your
website from different physical locations
A set “path” or URL is defined
Test...
What makes synthetic monitoring great
 Rich data collected (waterfall, video,
filmstrip, HTTP headers)
 Consistent “clea...
Questions your synthetic data can answer
How do I compare to the competition?
How does the
design of my
pages affect
performance?
How does the newest version of my site
compare to previous versions?
How well am I sticking to my performance budget?
What if my third parties fail?
Original: 3.5s
SPOF: 22.7s
38© 2014 SOASTA CONFIDENTIAL - All rights reserved.
Why are these numbers so different?
I don’t trust your data. Your numb...
“But it loads so much faster for me!?!”
2015 Macbook Pro
Warm browser cache
FIOS
X86 – Windows 7 VM
Completely cold cache/...
boomerang.js
Episodes
W3C Performance
Working Group
How fast am I?
http://soasta.io/perftimings
NavigationTiming API
Browser support for NavigationTiming
http://caniuse.com/#feat=nav-timing
48© 2014 SOASTA CONFIDENTIAL - All rights reserved.
Network operations
Front-end developer
Marketing and site operations
D...
Use case: Measure
network performance
I need visibility into…
 issues with authoritative DNS servers
 impact of denial of service attacks
on end users
 effic...
Start render DNS TCP TTFB
DOM loading DOM ready Page load Fully loaded
User timing Resource timing Requests Bytes in
Speed...
Measuring DNS and TCP
function getPerf() {
var timing = window.performance.timing;
return {
dns: timing.domainLookupEnd -
...
What’s with all those zeros!
 Connection reuse
 DNS caching
 Prefetching
Focus on higher percentiles
85th percentile
Median (50th)
Use case: Measure
front-end browser events
How do I…
 understand the impact of back-end
versus front-end on page speed?
 investigate how DOM complexity affects
per...
Start render DNS TCP TTFB
DOM load event DOM ready Page load Fully loaded
User timing Resource timing Requests Bytes in
Sp...
Isolate front-end vs. back-end
Isolate front-end vs. back-end
function getPerf() {
var timing = window.performance.timing;
return {
ttfb: timing.response...
Front-end
Back-end
Measuring the critical rendering path
Investigate DOM events
function getPerf() {
var timing = window.performance.timing;
return {
DLoading: timing.domLoading –...
Understanding critical rendering path
 DOM Loading – browser begins parsing initial
bytes of the document
 DOM Interacti...
2618 DOM nodes
86 DOM nodes
Visualizing DOM complexity
Use case: Measure
object-level performance
I need to understand…
 how third-party content affects my
performance
 how long a group of assets takes to
download
 ho...
Start render DNS TCP TTFB
DOM loading DOM ready Page load Fully loaded
User timing Resource timing Requests Bytes in
Speed...
ResourceTiming interface
Browser support for Resource Timing
http://caniuse.com/#feat=resource-timing
Cross-Origin Resource Sharing (CORS)
Start/End time only unless Timing-Allow-Origin
HTTP Response Header defined
Timing-Al...
ResourceTiming
var rUrl = ‘http://www.akamai.com/images/img/cliff-crocker-
dualtone-150x150.png’;
var me = performance.get...
Other uses for ResourceTiming
 Slowest resources
 Time to first image (download)
 Response time by domain
 Time a grou...
Using Resource Groups
PLT impact not due
resource groups
PLT impact
correlates with
improvement
from image domains
Use case: Measure
the user experience
I just need to understand…
 when users perceive the page to
be ready
 how long until my page begins
to render
 when con...
Start render DNS TCP TTFB
DOM loading DOM ready Page load Fully loaded
User timing Resource timing Requests Bytes in
Speed...
The fallacy of “First Paint” in the wild
 Support for First Paint is not standardized between browsers
 Metric can be mi...
First Paint is not equal to Start Render!
Chrome – “First Paint” True Start Render
Start Render and filmstrips
UserTiming Interface
 Allows developers to measure performance of
their applications through high-precision
timestamps
 ...
Browser support for UserTiming
http://caniuse.com/#feat=user-timing
Measure duration between two marks
performance.mark(“startTask”);
//Some stuff you want to measure happens here
performanc...
How long does it
take to display
the main product
image on my site?
Record when an img loads
<img src=“image1.gif” onload=“performance.mark(‘image1’)”>
For more interesting examples, see:
Me...
Adoption of
UserTiming
for top 25
shopping sites
(Global)
Nope
User
Timing
Sales
http://soasta.io/perftimings
How do I measure performance
when the onload event loses
relevance?
Use case: Measure
performance of SPAs
http://www.ryan-williams.net/hacker-news-hiring-trends/2015/october.html
What is a SPA?
• Single Page Application
• HTML loads once
• Fluid user experience
• Can be great for performance
• Hard t...
onload event
visible resources
Measuring SPAs
• Accept the fact that onload no longer matters
• Tie into routing events of SPA framework
• Measure downlo...
How fast should I be?
Use case: Measure
business impact
I need to understand…
 how performance affects business KPIs
 how our site compares to our
competitors
Start render DNS TCP TTFB
DOM loading DOM ready Page load Fully loaded
User timing Resource timing Requests Bytes in
Speed...
http://www.soasta.com/blog/mobile-web-performance-monitoring-conversion-rate/
2% increase in conversions
for every 1 second of improvement
Cut load time by 80%
83% traffic increase
32% increase in time on
site
108% increase in
interaction rate with ads
So what?
You must look at your own data.
Not all pages are created equal
For a typical
ecommerce site,
conversion rate
drops by up to 50%
when “browse”
pages incre...
Not all pages are created equal
However, there is
much less impact
to conversion
when “checkout”
pages degrade
What is the Conversion Impact Score?
The Conversion Impact Score (CIS) is a relative score that ranks page groups
by their...
TL;DR
The Conversion Impact Score answers this question:
How much impact does the performance
of this page have on convers...
Conversion Impact Score
http://www.soasta.com/blog/website-monitoring-conversion-impact-score/
How do I get there?
Create a performance budget
Setting a Performance Budget
http://timkadlec.com/2013/01/setting-a-performance-budget/
Perfor...
Set meaningful, page-specific SLAs
Chapter 8:
Changing Culture
at Your Organization
Free download (until November 4)
http://soasta.io/time
ismoneybook
performancebacon.com
performancebeacon.com
Thanks!
Meet us at booth #51
Metrics, metrics everywhere (but where the heck do you start?)
Metrics, metrics everywhere (but where the heck do you start?)
Metrics, metrics everywhere (but where the heck do you start?)
Metrics, metrics everywhere (but where the heck do you start?)
Metrics, metrics everywhere (but where the heck do you start?)
Metrics, metrics everywhere (but where the heck do you start?)
Metrics, metrics everywhere (but where the heck do you start?)
Metrics, metrics everywhere (but where the heck do you start?)
Metrics, metrics everywhere (but where the heck do you start?)
Metrics, metrics everywhere (but where the heck do you start?)
Metrics, metrics everywhere (but where the heck do you start?)
Metrics, metrics everywhere (but where the heck do you start?)
Metrics, metrics everywhere (but where the heck do you start?)
Metrics, metrics everywhere (but where the heck do you start?)
Metrics, metrics everywhere (but where the heck do you start?)
Metrics, metrics everywhere (but where the heck do you start?)
Metrics, metrics everywhere (but where the heck do you start?)
Nächste SlideShare
Wird geladen in …5
×

Metrics, metrics everywhere (but where the heck do you start?)

3.238 Aufrufe

Veröffentlicht am

There’s no one-size-fits-all approach to metrics. In this session, Cliff Crocker and I walk through various metrics that answer performance questions from multiple perspectives — from designer and DevOps to CRO and CEO. You’ll walk away with a better understanding of your options, as well as a clear understanding of how to choose the right metric for the right audience.

Veröffentlicht in: Internet
  • Als Erste(r) kommentieren

Metrics, metrics everywhere (but where the heck do you start?)

  1. 1. Metrics, metrics everywhere (but where the heck do you start?)
  2. 2. @tameverts @cliffcrocker #velocityconf
  3. 3. Who cares about performance today? How do I measure performance? How fast am I? How fast should I be? How do I get there?
  4. 4. The myth of a single metric
  5. 5. Start render DNS TCP TTFB DOM loading DOM ready Page load Fully loaded User timing Resource timing Requests Bytes in Speed Index Pagespeed score 1s = $$ DOM elements DOM size Visually complete Redirect SSL negotiation
  6. 6. Who cares about performance?
  7. 7. “47% of consumers expect a web page to load in 2 seconds or less.” Akamai, 2009
  8. 8. 1s = $27M DNS 144ms Start render 1.59s Hero image render 2.01s
  9. 9. How do I measure performance?
  10. 10. Androiddevicefragmentation OpenSignal,August2015
  11. 11. How do we measure UX for mobile?
  12. 12. https://gist.github.com/larahogan/b681da601e3c94fdd3a6
  13. 13. RUM versus plus synthetic
  14. 14. Real User Monitoring 101
  15. 15. Technology for collecting performance metrics directly from the end user’s browser Involves instrumenting your site via JavaScript Measurements are fired across the network to a collection point through a small request object (beacon)
  16. 16. What makes RUM great  Always on  Every user, every browser, everywhere  Able to capture human behavior/events  Only getting better
  17. 17. Questions your RUM data can answer
  18. 18. What are my users’ environments?
  19. 19. How do visitors move through my site?
  20. 20. How are my third-party scripts performing in real time?
  21. 21. What impact does performance have on my business?
  22. 22. Synthetic Monitoring 101
  23. 23. Uses automated agents (bots) to measure your website from different physical locations A set “path” or URL is defined Tests are run either ad hoc or scheduled, and data is collected
  24. 24. What makes synthetic monitoring great  Rich data collected (waterfall, video, filmstrip, HTTP headers)  Consistent “clean room” baseline  Nothing to install  Doesn’t require users / ability to measure pre-production and competition
  25. 25. Questions your synthetic data can answer
  26. 26. How do I compare to the competition?
  27. 27. How does the design of my pages affect performance?
  28. 28. How does the newest version of my site compare to previous versions?
  29. 29. How well am I sticking to my performance budget?
  30. 30. What if my third parties fail? Original: 3.5s SPOF: 22.7s
  31. 31. 38© 2014 SOASTA CONFIDENTIAL - All rights reserved. Why are these numbers so different? I don’t trust your data. Your numbers are wrong. How are you calculating page load time? I can’t share two sets of numbers with the business.
  32. 32. “But it loads so much faster for me!?!” 2015 Macbook Pro Warm browser cache FIOS X86 – Windows 7 VM Completely cold cache/DNS Throttled bandwidth
  33. 33. boomerang.js Episodes
  34. 34. W3C Performance Working Group
  35. 35. How fast am I?
  36. 36. http://soasta.io/perftimings
  37. 37. NavigationTiming API
  38. 38. Browser support for NavigationTiming http://caniuse.com/#feat=nav-timing
  39. 39. 48© 2014 SOASTA CONFIDENTIAL - All rights reserved. Network operations Front-end developer Marketing and site operations Designer C-level
  40. 40. Use case: Measure network performance
  41. 41. I need visibility into…  issues with authoritative DNS servers  impact of denial of service attacks on end users  efficiency of connection re-use  tier 1 connectivity issues (load balancer, web server)
  42. 42. Start render DNS TCP TTFB DOM loading DOM ready Page load Fully loaded User timing Resource timing Requests Bytes in Speed Index Pagespeed score 1s = $$ DOM elements DOM size Visually complete Redirect SSL negotiation
  43. 43. Measuring DNS and TCP function getPerf() { var timing = window.performance.timing; return { dns: timing.domainLookupEnd - timing.domainLookupStart}; connect: timing.connectEnd - timing.connectStart}; }
  44. 44. What’s with all those zeros!  Connection reuse  DNS caching  Prefetching
  45. 45. Focus on higher percentiles 85th percentile Median (50th)
  46. 46. Use case: Measure front-end browser events
  47. 47. How do I…  understand the impact of back-end versus front-end on page speed?  investigate how DOM complexity affects performance?  measure a “fully loaded” page?
  48. 48. Start render DNS TCP TTFB DOM load event DOM ready Page load Fully loaded User timing Resource timing Requests Bytes in Speed Index Pagespeed score 1s = $$ DOM elements DOM size Visually complete Redirect SSL negotiation
  49. 49. Isolate front-end vs. back-end
  50. 50. Isolate front-end vs. back-end function getPerf() { var timing = window.performance.timing; return { ttfb: timing.responseStart - timing.connectEnd}; basePage: timing.responseEnd - timing.responseStart}; frontEnd: timing.loadEventStart - timing.responseEnd}; }
  51. 51. Front-end Back-end
  52. 52. Measuring the critical rendering path
  53. 53. Investigate DOM events function getPerf() { var timing = window.performance.timing; return { DLoading: timing.domLoading – timing.navigationStart}; DInt: timing.domInteractive – timing.navigationStart}; DContLoaded: timing.domContentLoadedEventEnd – timing.navigationStart; DContLoadTime: timing.domContentLoadedEventEnd – timing.domContentLoadedEventStart}; DComplete: timing.domComplete - timing.navigationStart}; PLoad: timing.loadEventStart -
  54. 54. Understanding critical rendering path  DOM Loading – browser begins parsing initial bytes of the document  DOM Interactive – doc parsed, DOM has been constructed  DOM Content Loaded – No blocking style sheets  DOM Complete – All processing complete, all assets downloaded https://developers.google.com/web/fundamentals/performance/critical-rendering-path
  55. 55. 2618 DOM nodes 86 DOM nodes Visualizing DOM complexity
  56. 56. Use case: Measure object-level performance
  57. 57. I need to understand…  how third-party content affects my performance  how long a group of assets takes to download  how assets served by a CDN are performing
  58. 58. Start render DNS TCP TTFB DOM loading DOM ready Page load Fully loaded User timing Resource timing Requests Bytes in Speed Index Pagespeed score 1s = $$ DOM elements DOM size Visually complete Redirect SSL negotiation
  59. 59. ResourceTiming interface
  60. 60. Browser support for Resource Timing http://caniuse.com/#feat=resource-timing
  61. 61. Cross-Origin Resource Sharing (CORS) Start/End time only unless Timing-Allow-Origin HTTP Response Header defined Timing-Allow-Origin = "Timing-Allow-Origin" ":" origin-list- or-null | "*"
  62. 62. ResourceTiming var rUrl = ‘http://www.akamai.com/images/img/cliff-crocker- dualtone-150x150.png’; var me = performance.getEntriesByName(rUrl)[0]; var timings = { loadTime: me.duration, dns: me.domainLookupEnd - me.domainLookupStart, tcp: me.connectEnd - me.connectStart, waiting: me.responseStart - me.requestStart, fetch: me.responseEnd - me.responseStart } Measuring a single resource
  63. 63. Other uses for ResourceTiming  Slowest resources  Time to first image (download)  Response time by domain  Time a group of assets  Response time by initiator type (element type)  Cache-hit ratio for resources For examples see: https://github.com/lognormal/beyond-page-metrics
  64. 64. Using Resource Groups PLT impact not due resource groups PLT impact correlates with improvement from image domains
  65. 65. Use case: Measure the user experience
  66. 66. I just need to understand…  when users perceive the page to be ready  how long until my page begins to render  when content above the fold is visible
  67. 67. Start render DNS TCP TTFB DOM loading DOM ready Page load Fully loaded User timing Resource timing Requests Bytes in Speed Index Pagespeed score 1s = $$ DOM elements DOM size Visually complete Redirect SSL negotiation
  68. 68. The fallacy of “First Paint” in the wild  Support for First Paint is not standardized between browsers  Metric can be misleading (i.e. painting a white screen)
  69. 69. First Paint is not equal to Start Render! Chrome – “First Paint” True Start Render
  70. 70. Start Render and filmstrips
  71. 71. UserTiming Interface  Allows developers to measure performance of their applications through high-precision timestamps  Consists of “marks” and “measures”  PerformanceMark: Timestamp  PerformanceMeasure: Duration between two given marks
  72. 72. Browser support for UserTiming http://caniuse.com/#feat=user-timing
  73. 73. Measure duration between two marks performance.mark(“startTask”); //Some stuff you want to measure happens here performance.mark(“stopTask”); //Measure the duration between the two marks performance.measure(“taskDuration”,“startTask”,“stopTask”);
  74. 74. How long does it take to display the main product image on my site?
  75. 75. Record when an img loads <img src=“image1.gif” onload=“performance.mark(‘image1’)”> For more interesting examples, see: Measure hero image delay http://www.stevesouders.com/blog/2015/05/12/hero-image-custom-metrics/ Measure aggregate times to get an “above fold time” http://blog.patrickmeenan.com/2013/07/measuring-performance-of-user- experience.html
  76. 76. Adoption of UserTiming for top 25 shopping sites (Global) Nope User Timing Sales
  77. 77. http://soasta.io/perftimings
  78. 78. How do I measure performance when the onload event loses relevance? Use case: Measure performance of SPAs
  79. 79. http://www.ryan-williams.net/hacker-news-hiring-trends/2015/october.html
  80. 80. What is a SPA? • Single Page Application • HTML loads once • Fluid user experience • Can be great for performance • Hard to measure
  81. 81. onload event visible resources
  82. 82. Measuring SPAs • Accept the fact that onload no longer matters • Tie into routing events of SPA framework • Measure downloads during soft refreshes • (support in boomerang.js for Angular and other SPA frameworks)
  83. 83. How fast should I be?
  84. 84. Use case: Measure business impact
  85. 85. I need to understand…  how performance affects business KPIs  how our site compares to our competitors
  86. 86. Start render DNS TCP TTFB DOM loading DOM ready Page load Fully loaded User timing Resource timing Requests Bytes in Speed Index Pagespeed score 1s = $$ DOM elements DOM size Visually complete Redirect SSL negotiation
  87. 87. http://www.soasta.com/blog/mobile-web-performance-monitoring-conversion-rate/
  88. 88. 2% increase in conversions for every 1 second of improvement
  89. 89. Cut load time by 80% 83% traffic increase 32% increase in time on site 108% increase in interaction rate with ads
  90. 90. So what? You must look at your own data.
  91. 91. Not all pages are created equal For a typical ecommerce site, conversion rate drops by up to 50% when “browse” pages increase from 1 to 6 seconds
  92. 92. Not all pages are created equal However, there is much less impact to conversion when “checkout” pages degrade
  93. 93. What is the Conversion Impact Score? The Conversion Impact Score (CIS) is a relative score that ranks page groups by their propensity to negatively impact conversions due to high load times. For each page group, the Conversion Impact Score is calculated using the proportion of overall requests that are associated with that group, along with the Spearman Ranked Correlation between its load times and number of conversions. The Conversion Impact Score will always be a number between -1 and 1, though scores much greater than zero should be very rare. The more negative the score, the more detrimental to conversions that high load times for that page group are, relative to the other page groups.
  94. 94. TL;DR The Conversion Impact Score answers this question: How much impact does the performance of this page have on conversions?
  95. 95. Conversion Impact Score
  96. 96. http://www.soasta.com/blog/website-monitoring-conversion-impact-score/
  97. 97. How do I get there?
  98. 98. Create a performance budget Setting a Performance Budget http://timkadlec.com/2013/01/setting-a-performance-budget/ Performance Budget Metrics http://timkadlec.com/2014/11/performance-budget-metrics/
  99. 99. Set meaningful, page-specific SLAs
  100. 100. Chapter 8: Changing Culture at Your Organization
  101. 101. Free download (until November 4) http://soasta.io/time ismoneybook
  102. 102. performancebacon.com performancebeacon.com
  103. 103. Thanks!
  104. 104. Meet us at booth #51

×