Measuring web performance. Velocity EU 2011

MEASURING WEB PERFORMANCE
Steve Thair
Seriti Consulting
@TheOpsMgr

Every measurement of web performance
you will ever make will be
wrong

(C) SERITI CONSULTING, 2011 08/11/2011 2

“The human perception of duration is
both subjective and variable”

http://en.wikipedia.org/wiki/Time_perception

“PERCEPTION IS VARIABLE…”

Go read Stoyan’s talk! http://velocityconf.com/velocity2010/public/schedule/detail/13019


Web
Performance

Subjective Objective


Case Studies
Subjective Focus Groups
“Qualitative
Interviews
techniques”
Video Analysis

Surveys

Javascript

Navigation timing
Objective Browser
Extensions
“Quantitative Custom Browsers
techniques”
Proxy timings

Web Server mods

Network sniffing


“I keep six honest serving-men
(They taught me all I knew);
Their names are What and Why and When
And How and Where and Who.”
Rudyard Kipling, The Elephant’s Tale


WHAT LEVEL DO YOU MEASURE?

Journey

Page
Object

CHOOSE YOUR METRIC!

https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html

4 Key “Raw” Metrics
• Time to First Byte (TTFB)
• Render Start Time
• DOMContentLoaded
• Page (onLoad) Load Time (PLT)


What about “Above the Fold” time?
• How long to “render of the static stuff in
the viewable area of the page”?
Limitations of AFT
– Only applicable to lab setting
– Does not reflect user perceived latency based on
functionality

http://assets.en.oreilly.com/1/event/62/Above%20the%20Fold%20Time_%20Measuring%20Web%20Page%20Performance
%20Visually%20Presentation.pdf

WHAT OTHER METRICS?

Apdex
Statistical
Metrics
Counts/Histograms

Raw Metrics

Apdex (t) =
(Satisfied Count + Tolerated Count / 2)
/ Total Samples
• A number between 0 and 1 that represents “user satisfaction”
• For technical reasons the “Tolerated” threshold is set to four
times the “Satisfied” Threshold so if your “Satisfied” threshold
(t) was 4 seconds then:
• 0 to 4 seconds = Satisfied
4 to 16 seconds = Tolerated
over 16 seconds = Frustrated.

http://apdex.org/

PERFORMANCE IS MULTI-DIMENSIONAL

Multiple Metrics
For Multiple URLS
From Different Locations
Using Different Tools
Across the Lifecycle
Over Time


The importance of
CONTEXT


Location Bandwidth

Wired, WiFi, 3G Latency

Operating
Cached objects
System

Addons &
Antivirus
Extensions

Browser Device

Time of Day Context Resolution


Who? When?
User Experience Design
(UX)

Developers
Prod
Develop
Testers Ops
SDLC
WebOps
“The Boss”
Build
QA
(CI)


WHERE – DEPENDS ON THE HOW & WHY…

Web Browser
Proxy Server
Internet
Synthetic versus Real-User

“Real User”

Firewall /
Synthetic Agent
Load-Balancer Web Server
(Reverse) Proxy Server

SPAN port or
Network tap
WiFi or 3G
Smartphone

Signal/Noise Ratio increases….
Network “Sniffer”

User/Browser metrics Server-based metrics

The Synthetic
Versus
Real-User
Debate


“…it's a question of when,
“Because you’re skipping the “last mile” not if active monitoring of websites
between the server and the user’s for availability and performance will
browser, you’re not seeing how your be obsolete.”
site actually performs in the real world” - Pat Meenan
- Josh Bixby

“You can have my active
monitoring when you pry it
from my cold, dead
hands…”
- Steve Thair

http://blog.patrickmeenan.com/2011/05/demise-of-active-website-monitoring.html
http://www.webperformancetoday.com/2011/07/05/web-performance-measurement-island-is-sinking/
http://www.seriticonsulting.com/blog/2011/5/21/you-can-have-my-active-monitoring-when-you-pry-it-from-my-co.html

Observational Study
Versus
Experiment

Experiment versus Observational Study
• Both typically have the goal of detecting a relationship between the
explanatory and response variables.

Experiment
• create differences in the explanatory variable and examine any resulting
changes in the response variable (cause-and-effect conclusion)

Observational Study
• observe differences in the explanatory variable and notice any related
differences in the response variable (association between variables)
http://www.math.utah.edu/~joseph/Chapter_09.pdf


Observational Study = Real-User

• “Watching” what happens in a
given population sample
• We can only observe… and try to
infer what is actually happening
• Many “confounding variables”
• High signal to noise
• Correlation


Location Bandwidth
Wired,
Latency
WiFi, 3G

Cached Operating
objects System

Addons &
Antivirus
Extensions

Browser Device

Time of
Day Context Resolution


Observational Study = Real-User Experiment = Synthetic

• “Watching” what happens in a • We “design” our experiment
given population sample
• We chose when, where, what,
• We can only observe… and try to how etc
infer what is actually happening
• We control the variables (as
• Many “confounding variables” much as possible)
• High signal to noise • Lower signal to noise
• Correlation • Causation*
* OK, real “root cause” analysis will probably take a lot more investigation,
I admit… but you get closer!


So which one is better?
Neither.
Complementary not Competing
“…Ultimately I'd love to see a hybrid model where
synthetic tests are triggered based on something
detected in the data (slowdown, drop in volume, etc) to
validate the issue or collect more data.
- Pat Meenan


API Call to Synthetic

Real-User Monitoring Controlled Test and
Use RUM as “Reality Check”
detect a change in a compare to baseline.
page’s performance

From Observation… By controlling the variables To Experiment…


Javascript
Back to the “How”…
Navigation timing
Objective Browser
Extensions
“Quantitative Custom Browsers
techniques”
Proxy timings

Web Server mods

Network sniffing


7 WAYS OF MEASURING WEBPERF
1. JavaScript timing e.g. Souder’s Episodes or Yahoo! Boomerang*
2. Navigation-Timing e.g GA SiteSpeed
3. Browser Extension e.g. HTTPwatch
4. Custom browser e.g. 3pmobile.com or (headless) PhantomJS.org
5. Proxy timing e.g. Charles proxy
6. Web Server Mod e.g. APM solutions
7. Network sniffing e.g. Atomic Labs Pion


COMPARING METHODS…
Measurement Method
Navigation- Browser Custom Proxy Web Server Network
Metric JavaScript
Timing API Extension Browser Debugger Mod sniffing
Charles APM
Example Product WebTuna SiteSpeed HTTPWatch 3PMobile Pion
Proxy Modules
"Blocked/Wait" No No Yes Yes Yes No No
DNS No Yes Yes Yes Yes No No
Connect No Yes Yes Yes Yes No Yes
Time to First Byte Partially Yes Yes Yes Yes Yes Yes
"Render Start" No No Yes Yes No No No
DOMReady Partially Yes Yes Yes No No No
"Page/HTTP
Partially Yes Yes Yes Yes No Partially
Complete"
OnLoad Event Yes Yes Yes Yes No No No
JS Execution Time Partially No Yes Yes No No No
Page-Level Yes Yes Yes Yes Partially Partially Partially
Object Level No No Yes Yes Yes Yes Yes
Good for RUM? Yes Yes Partially No No Partially Yes
Good for Mobile? Partially Partially Partially Partially Partially Partially Partially
Affects Measurement Yes No Yes Yes Yes Yes No


JAVASCRIPT TIMING – HOW IT WORKS
unLoad Event
var start = new Stick it in a Cookie Load the next page
Date().getTime()

PLT = onLoad Event
Send a beacon
var end = new
beacon.gif?time=plt end - start Date().getTime()



PROS & CONS OF JAVASCRIPT TIMING
Metric JavaScript • Pro’s
Example Product WebTuna
• Simple
"Blocked/Wait" No • Episodes/Boomerang provide custom timing for
DNS No developer instrumentation
Connect No
Time to First Byte Partially • Cons
"Render Start" No
DOMReady Partially • Relies on Javascript and Cookies
"Page/HTTP
Partially
Complete" • Only accurate for 2 nd page in journey
OnLoad Event Yes
JS Execution Time Partially • Can only really get a “page load metric” and a
Page-Level
Object Level
Yes
No
partial TTFB metric
Good for RUM?
Good for Mobile?
Yes
Partially
• “Observer effect” (and Javascript can break!)
Affects Measurement Yes


NAVIGATION-TIMING – HOW IT WORKS

onLoad Event var plt = now -
Send a beacon
var end = new performance.timing.
beacon.gif?time=plt
Date().getTime() navigationStart;


NAVIGATION TIMING METRICS


PROS & CONS OF NAVIGATION-TIMING
Metric
Navigation- • Pro’s
Timing API

Example Product SiteSpeed
• Even simpler!
"Blocked/Wait" No • Lots more metrics
DNS Yes
Connect Yes • More accurate
Time to First Byte Yes
"Render Start" No • Cons
DOMReady Yes
"Page/HTTP • Need browser support for API
Yes
Complete"
OnLoad Event Yes • IE9+ / Chrome 6+ / Firefox 7+
JS Execution Time No
Page-Level Yes • Relies on Javascript (for querying API & beacon)
Object Level No
Good for RUM? Yes
• “Observer effect”
Good for Mobile? Partially
Affects Measurement No
• Page-level only

A BIT MORE ABOUT GA SITESPEED…
• Just add one line for basic, free, real-user monitoring!
_gaq.push(['_setAccount', 'UA-12345-1']);
_gaq.push(['_trackPageview']);
_gaq.push(['_trackPageLoadTime']);
• Sampling appears to vary (a lot!)
• 10% of page visits by design but reported 2% to 100%
• Falls back to Google Toolbar if available (but NOT javascript timing)
• Will probably make you think perf is better than it really is…


BROWSER EXTENSION – HOW IT WORKS
That subscribes to
Write a browser Get your users to
a whole lot of API
extension… install it…
event listeners…

Send the timing
back to collector
E.g. showslow.com

https://developer.mozilla.org/en/XPCOM_Interface_Reference

PROS & CONS OF BROWSER EXTENSIONS
Metric
Browser • Pros
Extension
• Very complete metrics
Example Product HTTPWatch

"Blocked/Wait" Yes
• Object and Page level
DNS
Connect
Yes
Yes
• No javascript (in the page at least)!!!
Time to First Byte Yes • Great for continuous integration perf testing
"Render Start" Yes
DOMReady Yes • Cons
"Page/HTTP
Complete"
Yes
• Getting users to install it…
OnLoad Event Yes
JS Execution Time Yes • Not natively cross-browser
Page-Level Yes
Object Level Yes • Some browsers don’t support extensions
Good for RUM?
Good for Mobile?
Partially
Partially
• Especially mobile browsers!
Affects Measurement Yes • “Observer effect”

CUSTOM BROWSER – HOW IT WORKS
Add custom
Take some open Like WebKit or the instrumentation for
source browser code Android Browser performance
measurement

Send the timing back
Get users to to collector
install it…
E.g. 3pmobile.com


PROS & CONS OF CUSTOM BROWSER
Metric
Custom
• Pros
Browser

Example Product 3PMobile • Great when you can’t use extensions / javascript / cookies
"Blocked/Wait" Yes
ie. For mobile performance e.g. 3Pmobile.com
DNS Yes
Connect Yes
• Great for automation e.g. http://www.PhantomJS.org/
"Render Start" Yes
• Good metrics (depending on OS API availability)
DOMReady Yes
"Page/HTTP • Cons
Yes
Complete"
OnLoad Event Yes • Requires installation
JS Execution Time Yes
Page-Level Yes • Maintaining fidelity to “real browser” measurements
Object Level Yes
Good for RUM? No • “Observer Effect” (due to instrumentation code)


PROXY DEBUGGER – HOW IT WORKS
Change browser to
use debugging Proxy Debugging proxy
Export data to log
e.g. Charles or records each request
Fiddler


PROS & CONS OF PROXY DEBUGGER
Metric
Proxy • Pros
Debugger

Example Product
Fiddler • One simple change to browser config
Proxy
"Blocked/Wait" Yes • No Javascript / Cookies
DNS Yes
Connect Yes • Can offer bandwidth throttling
"Render Start" No
• Cons
DOMReady
"Page/HTTP
No
• Proxies significantly impact HTTP traffic
Yes
Complete"
• http://insidehttp.blogspot.com/2005/06/using-fiddler-for-
OnLoad Event No
JS Execution Time No
performance.html
Page-Level Partially
Object Level Yes
• No access to browser events
Good for RUM? No
• Concept of a “page” be problematic…


6 Keep-Alive connections per SERVER
Versus
8 Keep-Alive connections TOTAL per PROXY
(Firefox 7.0.1)

WEB SERVER MOD – HOW IT WORKS

Write a webserver Start a timer on Stop Timer on
Mod or ISAPI filter Request Response

Send the timing
back to collector
E.g. AppDynamics

http://www.apachetutor.org/dev/request

PROS & CONS OF WEB SERVER MOD
Metric
Web Server • Pros
Mod
APM • Great for Application Performance Management (APM)
Example Product
Modules
• Can be used in a “hybrid mode” with Javascript timing
"Blocked/Wait" No
DNS No • Measuring your “back-end” performance
Connect No
Time to First Byte Yes • Can be easy to deploy*
"Render Start" No
DOMReady No • Cons
"Page/HTTP
Complete"
No • Limited metrics, ignores network RTT and only sees origin
OnLoad Event No
requests
JS Execution Time No • “Observer Effect” (~5% server perf hit with APM?)
Object Level Yes • Concept of a “page” be problematic…
Good for RUM? Partially
Good for Mobile? Partially • Can be a pain to deploy*


NETWORK SNIFFING – HOW IT WORKS
Create a SPAN Promiscuous Assemble TCP/IP
port or network mode packet packets into
tap sniffing HTTP Requests

Record the timing Assemble HTTP
data in a Requests into
database “pages”


PROS & CONS OF NETWORK SNIFFING
Metric
Network • Pros
sniffing
• No “observer effect” (totally “passive”)
Example Product Pion

"Blocked/Wait" No • Very common “appliance-based” RUM solution
DNS No
Connect Yes
• Can be used in a “hybrid mode” with Javascript timing
Time to First Byte
"Render Start"
Yes
No
• Can be easy to deploy*
DOMReady No • Cons
"Page/HTTP
Partially
Complete" • Limited metrics and only sees origin requests
OnLoad Event No
JS Execution Time No • Not “cloud friendly” at present
Object Level Yes • Concept of a “page” be problematic…
Good for RUM? Yes
Good for Mobile? Partially • Can be a pain to deploy*
Affects Measurement No


SUMMARY
• Performance is subjective (but we try to make it objective)
• Performance is Multi-dimensional
• Context is critical
• “Observational Studies AND Experiments”
• Real User Monitoring AND Synthetic Monitoring
• 7 different measurement techniques each with Pros & Cons


@LDNWEBPERF USER GROUP!
• Join our London Web Performance Meetup
• http://www.meetup.com/London-Web-Performance-Group/
• Next Wednesday 16 th Nov - 7pm – London (Bank)
• WPO case study from www.thetimes.co.uk!
• Follow us on Twitter @LDNWebPerf
• #LDNWebPerf & #WebPerf


QUESTIONS?

http://mobro.co/TheOpsMgr


Measuring web performance. Velocity EU 2011

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Andere mochten auch

Andere mochten auch (6)

Ähnlich wie Measuring web performance. Velocity EU 2011

Ähnlich wie Measuring web performance. Velocity EU 2011 (20)

Mehr von Stephen Thair

Mehr von Stephen Thair (15)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Measuring web performance. Velocity EU 2011

Hinweis der Redaktion