1. Parvez Ahammad
Principal Engineer / Head of DS&ML Team
Instart Logic Inc.
July 2016
Perceptual Speed Index (PSI)
for Measuring above-fold visual
Quality of Experience of webpages
3. Why we (@ InstartLogic) care about metrics for end-user QoE
● Towards a data-driven ‘Control System’ for improving web performance
● Accurate measurement of end-user experience is key!
5. The space of end-to-end perf. metrics
• PA’s taxonomy of web performance measurements
A. Milestone timings (of various kinds)
B. Byte size counts (relative to various milestones)
C. Request counts
D. Metrics on above-fold visual QoE
E. ...
6. Quick Refresher on Speed Index
● Popular public metric on above-fold visual QoE
● Patrick Meenan (WPT / Google Inc.)
● Aggregate function on quickness of visual completion above-fold
● Goes beyond milestone timer based approaches
7. Measurement of visual progress in Speed Index
● Frame-by-frame VC progress is computed from pixel-histogram comparisons
● Pixel-wise similarity (mean histogram difference / MHD) doesn’t capture visual
perception !
● Perception of Shape / Color / Object similarity
12. pSI (perceptual Speed Index)
● Frame-by-frame VC progress computation using SSIM
perceptual Via SSIM
13. Experiment #1
● Key question: Does the MHD SSIM change capture “real” website
behavior?
● Experiments with IR-100 websites (April 2015)
● Presented to Pat Meenan in May 2015
● Let’s look at a few illustrative examples
14. Tracking Perceptual Change: SI vs. pSI – RalphLauren (2015)
Green: % visual completion via SSIM (PSI)
Red: % visual completion via MHD (SI)
15. Tracking Perceptual Change: SI vs. pSI – ChicosFas (2015)
Green: % visual completion via SSIM (PSI)
Red: % visual completion via MHD (SI)
16. Tracking Perceptual Change: SI vs. pSI – Etsy (2015)
Green: % visual completion via SSIM (PSI)
Red: % visual completion via MHD (SI)
17. Tracking Perceptual Change: SI vs. pSI – Express (2015)
Green: % visual completion via SSIM (PSI)
Red: % visual completion via MHD (SI)
18. Recap: Experiment #1
● Does the “MHD SSIM change” capture “real” website behavior?
● Experiments with IR-100 websites (April 2015)
● Presented to Pat Meenan in May 2015
● Preliminary conclusions
● SSIM based visual progress measurements match human
perception more closely than MHD
● Key observation: PSI appears higher when visual jitter exists
● Pop-up ads / large lay-out changes / etc.
● SSIM / MHD swap doesn’t affect websites without visual jitter
● ChicosFas / Etsy examples
19. Experiment #2
● How do SI/PSI compare on a larger sample of Alexa top-K websites?
● Experiments with Alexa top mobile websites (March/April 2016)
● Dataset of 500+ mobile above-fold videos
● Dataset provided by Pat Meenan
● All analysis complete (May 2016)
● Results sent to Pat
● What did we find out?
20. #1: SI and PSI are highly (linearly) correlated (ρ = 0.91)
21. #2: Median PSI (5020) is slightly higher than median SI (4857)
22. Experiment #2
● How do SI/PSI compare on a larger sample of Alexa top-K websites?
● Experiments with Alexa top mobile websites (March/April 2016)
● Dataset of 500+ mobile above-fold videos
● Dataset provided by Pat Meenan
● All analysis complete (May 2016)
● Results sent to Pat
● What did we find out?
● SI and PSI are linearly correlated
● Key observation: PSI is much higher than SI for webpages with
visual jitter / layout thrashing behavior
23. Summary (1)
● Metrics on above-fold visual QoE of web application delivery must
take human end-user perception into account.
● Speed Index
● Primarily focused on progress of above-fold loading
● Does not account for layout stability
● Perceptual Speed Index
● A perceptually oriented metric to measure above-fold visual QoE
● Designed to account for visual jitter (layout stability)
● Complementary to SI
● PSI implementation complete
● Open sourced GitHub: WPO-Foundation/visualmetrics
24. Summary (2)
● For more details and explanations:
● http://www.parvez-ahammad.org/blog/perceptual-speed-index-
psi-for-measuring-above-fold-visual-performance-of-webpages
