This document discusses how web performance impacts business metrics and provides recommendations for optimization. It begins by showing how small improvements in page load time (e.g. from 6 seconds to 1.2 seconds) led to large increases in revenue and traffic for Shopzilla. Similarly, Amazon saw a 1% drop in sales for every 100ms increase in latency. The document then examines factors that impact web performance like DNS lookups, connections, wait times, and loading content. It provides tips for optimization like reducing payload sizes and leveraging caching. Throughout, it emphasizes the importance of monitoring and troubleshooting performance issues.

1. 5/19/2011 CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2010 . ALL RIGHTS RESERVED Michael Mikikian - 1 Peeling the Web Performance Onion Drit Suljoti, Co-founder Catchpoint Systems drit@catchpoint.com

2. Speed Matters Shopzilla Performance 6sec to 1.2 sec Revenue +12% Pageviews+25% Yahoo 400ms improvement Traffic +9% Length of an eye blink is ~400ms

3. Speed Matters Amazon Latency +100ms Sales drop 1% / 250mil Google 400ms latency Searches drop 0.59%

4. What Impacts Speed 5/19/2011 CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2010 . ALL RIGHTS RESERVED 4

5. What Impacts Speed 5/19/2011 CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2010 . ALL RIGHTS RESERVED 5 Webpage Response (ms): 2,254 # Requests: 102 # Hosts: 11 # Connections: 16

6. Web Performance Optimization Reduce Pack Tight Merge Leverage Cache Partner Up

7. WPO Works 5/19/2011 CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2010 . ALL RIGHTS RESERVED For Ethan Selzer - 7

8. 80 / 20 Rule 5/19/2011 CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2011 . ALL RIGHTS RESERVED 8 Steve Souders: “80% of the end-user response time is spent on the front-end. Start there.” When things fail, a single request takes 80% of the time!

9. Great Performance is More than SPEED Availability can users access it? Reliability is it working consistently and properly?

10. Monitoring

11. Web Performance Monitoring Synthetic: simulates End User activity Measures the end to end transaction Continuous, 24x7 Detect anomalies Track performance metrics over time Measures every component

12. The Foundations of the Web IP

13. Simple HTTP Transaction DNS Lookup: The client tries to resolve the domain name for the request. Client sends DNS Query to local ISP DNS server. DNS server responds with the IP address for hostname.com Connect: Client establishes TCP connection with the IP address of hostname.com Client sends SYN packet. Web server sends SYN-ACK packet. Client answers with ACK packet, concluding the three-way TCP connection establishment. Send: Client sends the HTTP request to the web server. Wait: Client waits for the server to respond to the request. Web server processes the request, finds the resource, and sends the response to the Client. Client receives the first byte of the first packet from the web server, which contains the HTTP Response headers and content. Load: Client loads the content of the response. Web server sends second TCP segment with the PSH flag set. Client sends ACK. (Client sends ACK every two segments it receives. from the host) Web server sends third TCP segment with HTTP_Continue. Close: Client sends a a FIN packet to close the TCP connection.

14. DNS

16. Avoid multiple CNAMEs.

17. Rely on distributed NS server or utilize companies like 3Crowd, Cotendo, Dyn, UltraDNS,...

18. Ensure DNS servers capacity.

19. Avoid using Registrars, usually unreliable and slow.

21. Use Case: DNS DNS Provider Switch

22. Connect DNS Lookup: The client tries to resolve the domain name for the request. Client sends DNS Query to local ISP DNS server. DNS server responds with the IP address for hostname.com Connect: Client establishes TCP connection with the IP address of hostname.com Client sends SYN packet. Web server sends SYN-ACK packet. Client answers with ACK packet, concluding the three-way TCP connection establishment. Send: Client sends the HTTP request to the web server. Wait: Client waits for the server to respond to the request. Web server processes the request, finds the resource, and sends the response to the Client. Client receives the first byte of the first packet from the web server, which contains the HTTP Response headers and content. Load: Client loads the content of the response. Web server sends second TCP segment with the PSH flag set. Client sends ACK. (Client sends ACK every two segments it receives. from the host) Web server sends third TCP segment with HTTP_Continue. Close: Client sends a a FIN packet to close the TCP connection.

23. Connect What impacts Connect times? Packet loss / network stability Latency Routers Load Balancers Network Cards Server Load Common issues: Bad configuration of Load Balancers, Routers, Switches and the NIC cards! Latency/Packet Loss/Link speed TCP offloading Capacity

24. Use Case: Connect

25. Use Case: Connect Capacity Problems at Datacenter

26. Wait DNS Lookup: The client tries to resolve the domain name for the request. Client sends DNS Query to local ISP DNS server. DNS server responds with the IP address for hostname.com Connect: Client establishes TCP connection with the IP address of hostname.com Client sends SYN packet. Web server sends SYN-ACK packet. Client answers with ACK packet, concluding the three-way TCP connection establishment. Send: Client sends the HTTP request to the web server. Wait: Client waits for the server to respond to the request. Web server processes the request, finds the resource, and sends the response to the Client. Client receives the first byte of the first packet from the web server, which contains the HTTP Response headers and content. Load: Client loads the content of the response. Web server sends second TCP segment with the PSH flag set. Client sends ACK. (Client sends ACK every two segments it receives. from the host) Web server sends third TCP segment with HTTP_Continue. Close: Client sends a a FIN packet to close the TCP connection.

27. Wait What impacts Wait times? Network factors The Application code The Application Load / Capacity Any other dependencies (database lookups, other servers, apis…) Common issues: Application Load - Profile your capacity / perform load testing Database utilization, locks…, Memory Leaks, Bugs…

28. Use Case: Wait

29. Use Case: Wait

30. Use Case: Wait - Database Impact DB Problems

31. Load/Receive DNS Lookup: The client tries to resolve the domain name for the request. Client sends DNS Query to local ISP DNS server. DNS server responds with the IP address for hostname.com Connect: Client establishes TCP connection with the IP address of hostname.com Client sends SYN packet. Web server sends SYN-ACK packet. Client answers with ACK packet, concluding the three-way TCP connection establishment. Send: Client sends the HTTP request to the web server. Wait: Client waits for the server to respond to the request. Web server processes the request, finds the resource, and sends the response to the Client. Client receives the first byte of the first packet from the web server, which contains the HTTP Response headers and content. Load: Client loads the content of the response. Web server sends second TCP segment with the PSH flag set. Client sends ACK. (Client sends ACK every two segments it receives. from the host) Web server sends third TCP segment with HTTP_Continue. Close: Client sends a a FIN packet to close the TCP connection.

32. Load/Receive What impacts Wait times? Network factors The Application code The Application Load / Capacity Any other dependencies (database lookups, other servers, apis…) Common issues: Application Load - Profile your capacity / Load testing Database utilization, locks…, Memory Leaks, Bugs… Bad chunked encoding!

33. Factors You Control Applications/System Architecture, Code, DB, SQL Queries, … Configurations Partners: DNS, CDN, ISP Links, Datacenter, Optimization, Ads… ISP Peering Hardware Server, CPU, RAM, I/O, Routers,… 5/19/2011 CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2010 . ALL RIGHTS RESERVED For Ethan Selzer - 29

34. Factors You CANNOT Control Internet Fiber cut in DC, Latency East Coast to West Coast Last Mile The internet link to the end user! End User Local Network Misconfigured router, computers downloading/uploading torrents! OS/Browser of the end user Configurations, Add-ons, installed malware… 5/19/2011 CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2010 . ALL RIGHTS RESERVED For Ethan Selzer - 30

37. HTML Content Each request relies on HTTP - same rules apply! Pay attention to bottlenecks Primary URL Inline JavaScript requests Front end code impacts browser processing!

38. Oops … bad httpd.conf file 5/19/2011 CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2011 . ALL RIGHTS RESERVED 34

39. Oops … bad httpd.conf file 5/19/2011 CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2011 . ALL RIGHTS RESERVED 35 Apache configuration! No More Keep Alives

41. Setup monitoring for every key provider / hostname / requests

42. DNS

43. Origin servers

44. if Multiple datacenters, monitor them individuallyBaseline very often, track improvement or degradation Keep an eye on performance during critical business hours (Volume) Compare data pre-release and post release Review Daily / Weekly /Monthly Alerts on % delta

46. Firebug

47. IE 9 – Developer tool

48. HTTPWatch

49. Wireshark

50. Fiddler

51. Yslow

52. Dynatrace

53. Dig for DNS

55. Google Page speed (http://pagespeed.googlelabs.com/)

56. Closing the Loop Monitor Action Plan  Optimize Monitor… do it again It isn’t over till it’s over. And it’s never over. It is not a sprint, it is a never ending marathon!

57. CATCHPOINT SYSTEMS . CONFIDENTIAL . © 2011 . ALL RIGHTS RESERVED 39 THANK YOU. http://blog.catchpoint.com @catchpoint

59. Where is the problem? is it specific to a country? a city? a customer? a datacenter?

60. Who is affected? All customers? Some customers? Just the monitoring tools?

61. When did it start? Does it always happen? is it sporadic?

62. Why?Did something change? Did we roll out something? Did the environment change? Did the customer behavior change?

63. How did this happen? Change management? Router upgrade? Did someone upload a crazy ad?