Measuring CDN performance and why you're doing it wrong

1. Measuring CDN Performance Hooman Beheshti VP Technology

2. Why this matters •  Performance is one of the main reasons we use a CDN •  Measurement often used during evaluation phase to compare CDNs –  Most of what we’ll talk about is in this context •  Seems easy, but isn’t •  Heavily vendor-‐inﬂuenced –  “Ok Google: deﬁne irony!”

3. Goals •  What does the measurement landscape look like •  Share measurement experiences •  Help guide towards good testing plan if/when you decide to do this

4. Background

5. Delivery: static/cached objects Client CDN Node Origin

6. Delivery: dynamic/uncached objects

7. What we’ll be focusing on •  Only on delivery and not all the other features CDNs provide •  How we measure •  Metrics to measure •  What to measure •  Some gotchas, misconceptions, and common mistakes

8. Measurement Techniques (how we measure)

9. Measurement techniques •  Pretend Users –  Synthetic tests –  Not actual users •  Real Users –  In the browser –  Actual users

10. Synthetic testing

11. Synthetic testing •  Usually a large network of test nodes all over the globe •  Highly scalable, can do lots of tests at once •  Many vendors that have this model – Examples: Catchpoint, Dynatrace(Gomez), Keynote, Pingdom, etc

12. Synthetic testing •  Built to do full performance and availability testing –  Lots of “monitors” – emulating what real users do –  DNS, Traceroute, Ping, Streaming, Mobile –  HTTP •  Object •  Browser •  Transactions/Flows •  Tests set up with some frequency to repeatedly test things –  Aggregates reported

13. Backbone nodes •  Test machines sitting in datacenters all around the globe •  Really good at: –  Availability and reachability –  Scale –  Backend problems –  Global reach •  Terrible indicators of raw performance –  No latency –  Inﬁnite bandwidth

14. Backbone nodes •  Test machines sitting in datacenters all around the globe •  Really good at: –  Availability and reachability –  Scale –  Backend problems –  Global reach •  Often terrible indicators of raw performance –  No latency –  Inﬁnite bandwidth

15. https://www.ﬂickr.com/photos/stars6/4381851322/

16. Last mile nodes •  Test machines sitting behind a real home-‐like internet connection •  Much better at reporting what you can expect from users, but sometimes unreliable •  Also not as dense in deployment

17. backbone last mile

18. Real users (RUM)

19. RUM •  Use javascript to collect timing metrics •  Can collect lots of things through browser APIs – Page metrics, asset metrics, user-‐deﬁned metrics

20. Use test assets •  Use this model to initiate tests in the browser •  Some vendors: – Cedexis, TurboBytes, CloudHarmony, more… – Usually, this isn’t their business, but the data drives their main business objectives •  You can build this yourself too

21. Use real assets in the page •  Collect timings from actual objects –  Resource timing •  Vendors –  SOASTA, New Relic, most synthetic vendors –  Boomerang (open source) –  Google Analytics User Timings

22. DATA, DATA, DATA •  For either RUM technique, we need A LOT of data •  Too much variance – Most vendors don’t use averages – Medians, percentiles, and histograms

23. Measurement Metrics

24. Client Server

25. Client Server 1 x RTT

26. Client Server DNS DNS

27. TCP Client Server DNS DNS

28. TCP Client Server DNS DNS (TLS)

29. TCP Client Server DNS DNS (TLS) HTTP (TTFB)

30. TCP Client Server DNS DNS (TLS) HTTP (TTFB) HTTP (Download)

31. DNS TCP (TLS) TTFB Download (TTLB-‐TTFB) Time

32. DNS TCP (TLS) TTFB Download (TTLB-‐TTFB) Time DNS RTT to DNS server, DNS iterations, DNS caching and TTLs

33. DNS TCP (TLS) TTFB Download (TTLB-‐TTFB) Time DNS TCP RTT to DNS server, DNS iterations, DNS caching and TTLs RTT to cache server (CDN footprint & routing algorithms)

34. DNS TCP (TLS) TTFB Download (TTLB-‐TTFB) Time DNS TCP (TLS) RTT to DNS server, DNS iterations, DNS caching and TTLs RTT to cache server (CDN footprint & routing algorithms) RTT to cache server (or RTTs depending on TLS False Start), eﬃciency of TLS engine

35. DNS TCP (TLS) TTFB Download (TTLB-‐TTFB) Time DNS TCP (TLS) TTFB RTT to DNS server, DNS iterations, DNS caching and TTLs RTT to cache server (CDN footprint & routing algorithms) RTT to cache server (or RTTs depending on TLS False Start), efficiency of TLS engine RTT to where the object is stored + storage efficiency (different for requests to origin); lower bound = network RTT

36. DNS TCP (TLS) TTFB Download (TTLB-‐TTFB) Time DNS TCP (TLS) TTFB TTLB-‐TTFB RTT to DNS server, DNS iterations, DNS caching and TTLs RTT to cache server (CDN footprint & routing algorithms) RTT to cache server (or RTTs depending on TLS False Start), efficiency of TLS engine RTT to where the object is stored + storage efficiency (different for requests to origin); lower bound = network RTT Bandwidth, congestion avoidance algorithms (and RTT!)

37. Core object metrics •  Not every request experiences every metric: – DNS: once per domain – TCP/TLS setup once per connection – TTFB/Download for every object (not already in browser cache)

38. Resource timing http://www.w3.org/TR/resource-‐timing/

39. Resource timing window.performance.getEntries()

40. Mistakes we make (when evaluating)

41. CDN X vs CDN Y

42. “I’ll pick an image from my home page, use backbone synthetic tests from all over the world and pick the CDN that has the fastest average time” “let’s test an asset via RUM on a million page views a day and pick the fastest CDN” “let’s run webpagetest on both CDNs and go with whichever has a faster page load time” ~$time curl –v http://…

43. we measure the wrong thing

44. Web application: objects •  Your application should determine what you test: –  Objects served from the edge –  Objects served from origin (through CDN) •  If HTML is from origin (through CDN), we must measure it –  Essential to critical page metrics

45. Web application: object sizes

46. •  On any page –  DNS queries only happen a small number of times –  6 TCP connections per domain –  1 TLS setup per connection –  Many many many HTTP fetches •  Core metrics –  TTFB –  Download (TTLB-‐TTFB) if important large objects –  Should have a good idea of DNS/ TCP/TLS, but less critical

47. Web application •  If CDN only for static/cacheable objects: –  One or two representative assets –  TTFB and maybe download most important Client CDN Node

48. X-Cache: HIT

49. Web application •  If CDN also for whole site (HTML going through CDN) –  Sample of key HTML pages, delivered from origin –  TTFB will show eﬃciency of routing (and connection management) to origin –  TTLB will show eﬃciency of delivery Web Server Client CDN Node

50. Web application •  If CDN also for whole site (HTML going through CDN) –  Sample of key HTML pages, delivered from origin –  TTFB will show eﬃciency of routing (and connection management) to origin –  TTLB will show eﬃciency of delivery Web Server Client CDN Node CDN Node

51. we measure the wrong way

52. Backbone Nodes (For true performance measurements)

53. % of tests msec TCP Connect Time Histogram (BB nodes)

54. object metrics or page metrics

55. Download: 15Mbps Upload: 5Mbps Latency: 10 ms, 25 ms

56. 10 msec 25 msec

57. 10 msec 25 msec

58. onload Speed Index Start Render 10 msec 25 msec

59. What the…??? •  We always assume “all things equal” •  Too many factors aﬀect page load time –  3rd parties (sometimes varying), content form origin, layout, JS execution, etc •  Too much variance Source: httparchive.org

60. To be clear… •  Always use webpagetest (or something like it) to understand your application’s performance proﬁle •  Continue to monitor application performance, and always spot check •  Be extremely careful when using it to compare CDN performance, it can mislead you –  If using RUM to measure page metrics, with lots of data, things become a little more meaningful (data volume handles variance)

61. we overgeneralize and draw the wrong conclusions

62. Cache hit ratios

63. Cache hit ratio: traditional calculation 1 -‐ Requests to Origin Total Requests

64. Origin

65. Origin Cache

66. TCP Origin Cache

67. HTTP Origin Cache

68. Origin Cache HTTP

72. Origin Cache HOT COLD

73. Origin Cache cache “hit”

74. Cache hit ratio: traditional calculation 1 -‐ Requests to Origin Total Requests

75. Isn’t this better? Hits Total Requests @edge

76. Isn’t this better? Hits Hits + Misses @edge

77. Cache hit ratio vs. 1 -‐ Requests to Origin Total Requests Hits Hits + Misses @edge

78. Cache hit ratio vs. 1 -‐ Requests to Origin Total Requests Hits Hits + Misses @edge Oﬄoad

79. Cache hit ratio vs. 1 -‐ Requests to Origin Total Requests Hits Hits + Misses @edge Oﬄoad Performance

80. Eﬀect on long tail content

81. Eﬀect on long tail content (long tail: Cacheable but seldom fetched)

82. Popular Medium Tail (1hr) Long tail (6hr)

83. Popular Medium Tail (1hr) Long tail (6hr) Connect (median) Popular 14msec 1hr Tail 15msec 6hr Tail 16msec

84. Popular Medium Tail (1hr) Long tail (6hr) Connect (median) Popular 14msec 1hr Tail 15msec 6hr Tail 16msec 6,400+ measurements 77,000+ measurements 38,000+ measurements

85. Popular Medium Tail (1hr) Long tail (6hr) Connect (median) Wait (median) Popular 14msec 19msec 1hr Tail 15msec 26msec 6hr Tail 16msec 32msec 6,400+ measurements 77,000+ measurements 38,000+ measurements

86. Popular Medium Tail (1hr) Long tail (6hr) Isn’t this better?

89. After all that….

90. How much of this really matter? (when trying to choose between multiple CDNs)

91. The bigger picture •  It’s really easy to lock in on a metric •  Performance absolutely matters •  True performance isn’t always as easy to measure

92. We must ask questions …

93. What’s the storage model and how does it aﬀect long tail content?

94. What should I expect with cache hit ratios for oﬄoad and performance?

95. Footprint? (is what I’m testing the same as what I’m buying?)

96. HTTP vs TLS footprint?

97. Can I serve stale content if necessary? (stale-while-revalidate & stale-if-error)

98. What if I can cache something I didn’t think I could?

99. Key takeaways •  Everything is application-‐dependent –  Evaluate how your application works and what impacts performance the most •  Don’t get locked into a single number/metric •  Always know your application performance and bottlenecks •  Be mindful of the bigger picture •  Don’t stop measuring!

100. Thank you! hooman@fastly.com oﬃce hours Friday @lunch

Measuring CDN performance and why you're doing it wrong

Fastly

Measuring CDN performance and why you're doing it wrong