Improvements to user experience translate directly to real business metrics and the bottom line. To guide the business to making wise choices on user experience, you need an accurate picture of site performance for real users. In this talk, Steve Lerner will describe how eBay’s performance monitoring strategy has evolved, how the insights gained from real user monitoring have impacted eBay’s business, and some of the considerations that have shaped their in house implementation of Real User Monitoring to serve eBay’s massive global scale. See Steve Lerner's Edge Presentation: http://www.akamai.com/html/custconf/edgetv-commerce.html#real-user-monitoring The Akamai Edge Conference is a gathering of the industry revolutionaries who are committed to creating leading edge experiences, realizing the full potential of what is possible in a Faster Forward World. From customer innovation stories, industry panels, technical labs, partner and government forums to Web security and developers' tracks, there’s something for everyone at Edge 2013. Learn more at http://www.akamai.com/edge

1. Real User Monitoring
Getting Real Data from Real
Users in the Real World
Steve Lerner, Senior Member of Technical Staff, Network Engineering, Global Network Services
Rajasekhar Bhogi, Manager, Software Development, Platform Frameworks, US
Sheldon Shao, Member of Technical Staff, Software Engineer, Platform Frameworks, China

2. PLATFORM

3. Thought Experiment
Measuring Performance: Not As Obvious As It Seems
Which Scenario Has
Better Performance?
2meter 5watt radio
Reaches 3 miles
Perfect clarity
How Can It Be Improved?
10meter 100watt radio
Reaches Europe from NYC
Slightly distorted
3

4. Pontificated Points
Measurement metrics must
contain clear path for action
Otherwise suffer
analysis paralysis
Scale, geography, network, servers,
users, etc. all influence performance
• Performance measurement is
multifaceted
• No single method is enough
There is no silver bullet
Because werewolves
don’t exist
Image Source: Creative Commons via Wikipedia
4

5. Performance Measurement
Synthetic testing (ThousandEyes, Catchpoint, Keynote, etc)
•
•
•
•
Use small numbers of probes/agents to artificially measure your pages
Can be useful for network measurements and triaging issues
Small sample size and test agent structure makes unreliable for critical needs
CDNs game synthetic tests with caches near agents so measurements are always
perfect
Real User Metrics Are 100% Accurate But
• Cannot yet capture details about each object on page
• Generate massive data- need massive infrastructure and experienced team to
manage
• Upstream problems can create confounding- i.e. problems on a web server may
show overall slowness of RUM but not point to problem’s root cause
• Standard browser hooks for timing not supported by Safari- need to use proprietary
ones
• Ultimate goal would be to analyze every log line of every object but not yet feasible
Best Practice: Use Everything
• TRIANGULATE AND INTERPRET: Results need to be viewed in context, double
checked, and cross referenced
5

6. Goal: Continuous Improvement
eBay’s Top 5 Pages For Analysis
• Measured “from” a country’s users “to” a country site i.e. from Brazil
users to US site:
• Home page
• Search results
• View Item
• My eBay
• Login
Clear Path For Improvement Is Laid Out By Proper Timing
Measurement Tools For Example
•
•
•
•
•
•
DNS resolution
Connect
First byte
Navigation
Object download
Object start
6

7. eBay Synthetic Testing: ThousandEyes
Standard
Waterfall Used
To Troubleshoot
Object Issues
Pubmatic Ad
Server =
Ouch
7

8. eBay Synthetic Testing: ThousandEyes
Network Path
View Used For
Troubleshooting
Internet Issues
8

9. eBay Synthetic Testing: ThousandEyes
BGP View Used For
Troubleshooting
Routing Issues
9

10. eBay Real User Metrics aka Site Speed
eBay Real User Metrics (RUM)
• Program is called Site Speed and reporting is called Site Speed Gauge
• Designed and built by global team
Based On Standards
• W3C Web Performance Working Group http://www.w3.org/2010/webperf/
• “The mission of the Web Performance Working Group, part of the Rich
Web Client Activity, is to provide methods to measure aspects of
application performance of user agent features and APIs”
Anyone Can Build This!
• Open source libraries exist
• Boomerang: http://lognormal.github.io/boomerang/doc/
Anyone Can Buy This!
• Most synthetic test vendors already or will soon offer a RUM solution
10

11. Order of Operations for Site Speed Measurement
Before Navigation Timing API Support
Receive
Page
Request and
Choose
Sample
RateUsually 10%
Take time
stamps
•
•
•
•
Start
Head of Page
Page load
Others
Calculate
commonly
used metrics
Send precalculated
metrics back
to server via
beacon
Generate
reports,
dashboards
and alerts
With Navigation Timing API Support
Receive Page
Request and
Choose Sample
Rate- Usually
10%
Calculate
commonly used
metrics
Send precalculated
metrics back to
server via
beacon
Generate reports,
dashboards and
alerts
11

12. Visualization of Web Page Timing Elements
Proprietary code catches network
layer timing events to pass to
JavaScript beacon (Before
NavigationTimer API support)
Event Graph Source: https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html
12

13. Site Speed: Sample Sizes for UK Search Results
Page
10% Sampling = 350M views of Search
Results per day during this time period,
of UK page from UK users – to verify
sample size is acceptable
13

14. Site Speed Example: Akamai DSA in Italy
• eBay enabled Akamai Dynamic Site Acceleration (DSA) for www.ebay.it as a “better
bandwidth” service- no page caching, prefetching, or other page changes enabled
Clear Path
– Improve connection time and reduce latency to pages
• Static objects already on Akamai EdgeSuite for acceleration
• Speed Gauge example report:
– Connection Time (Connections End – Connection start): Median
– Content Download Time (Response End– Response Start): Median
14

15. Site Speed Results: Akamai DSA in Italy
DSA Enabled
Content
Download
58% Reduction
Connect Time
78% Reduction
15

16. TRIANGULATE! Synthetic Test View of DSA Activation
90% Reduction In Latency
From ThousandEyes
Agents To www.ebay.it
When DSA Enabled
16

17. Site Speed: Switching CDNs for View Item Photos
•View Item pages contain images and objects delivered on
their own domain i.e. i.ebayimg.com
Clear Path
–Switch image domain from one CDN to another with
better global coverage
•Measure: Median response of start to on-load event
(loadEventStart – responseStart)
17

18. Site Speed Results: Switching CDNs for View Item Photos
Improvements:
Philippines: 38%
Malaysia: 30%
China: 25%
Brazil: 19%
18

19. Site Speed Results: Enabling Global DNS Service
• eBay is enabling global DNS service – a global anycast deployment of DNS servers
Clear Path
– Enable global DNS service to reduce DNS resolution time
• First test of service is in India on www.ebay.in
57% Improvement
For DNS Lookup
Time In India When
India DNS Servers
Added
USA DNS Shown
For Scale
19

20. DNS: Triangulate Results
Outsource DNS
Service, Globally,
Is 63% Faster
Than Origin DNS
20

21. Site Speed: Global Heat Map
Green = faster,
Red = slower
21

22. Site Speed: USA Heat Map
Green = Faster,
Red = Slower
22

23. Site Speed: Browser View
View Of Home
Page, USA Users
(Some Areas
Redacted)
23

24. Site Speed: ISP View
View Of Home
Page, USA Users
(Some Areas
Redacted)
24

25. Site Speed Histogram: Speed at Percentile (BETA)
Which Page
Download Speed Is At
Which Percentile
Goal: Move Pages Left To Get Faster (Better) Speed
25

26. Site Speed Histogram: Count At Each Percentile (BETA)
Quantity Of
Datapoints At Each
Percentile Of
Speed
Goal: Move Users Left To Get Faster (Better) Speed
26

27. Purchases Per Visitor / Week
How Does Page Load Speed Impact Business?
But there is a
bigger
economics
story
10% Faster Page
= 1% Increase in
Purchases Per
Visitor / Week
Source: Published eBay study http://www.ebaytechblog.com/2013/03/29/measuring-real-user-experience-with-site-speed-gauge/
27

28. Summary, Conclusions, and Wisdom
Measurement Metrics Must Contain A Clear Path For Action
• Don’t Fall For (Or Supply) A Single Summary Metric- It Doesn’t Exist
Performance Measurement Requires A Multifaceted Approach
• Synthetic Tests
• Real User Metrics
• Ideally Log File Analysis
Triangulate Results To Understand Cause And Effect
Identify Economic Outcome
Keep Investigating, Evolving, And Breaking Things
Never Assume What Worked Or Had Success In The Past Will Continue To Do So
28

29. Steve Lerner
Senior Member of Technical Staff, Network Engineering
Global Network Services
slerner@ebay.com
Rajasekhar Bhogi
Manager, Software Development
Platform Frameworks, US
rbhogi@ebay.com
29