eBook can be downloaded here: http://www.yottaa.com/ebook-eight-wpo-apm-best-practices-for-the-holiday-season

1. Ensuring Success &
Avoiding Site Disruptions:
Tips for WebOps, WebDev and
Marketing
Eight WPO/APM Best Practices
for the Holiday Season
Yottaa, Inc.
101 Summer Street
Boston MA 02110
http://www.yottaa.com
© 2012 Yottaa, Inc. All Rights Reserved.

2. Contents
• Introduction
• Tip #1: Monitor performance & set up alerts
• Tip #2: Use real browsers to monitor user experience
• Tip #3: Monitor key pages
• Tip #4: Trend user experience
• Tip #5: See what your users see
• Tip #6: Establish performance & SLA goals
• Tip #7: Capture & diagnose site & app issues
• Tip #8: Carry out statistically meaningful sampling
• Appendix: Key site performance and user experience
metrics
2

3. Introduction
• Holiday shopping is expected to rise nearly
17% this season, and sites large and small
want the competitive edge to capture as much
revenue as possible.
• Is your site ready for the holiday rush? How do
the best sites monitor and manage their site
performance and availability?
• In this eBook, we identify the best practices to
help you be proactive and prevent site
problems during the holiday season.
3

4. 24/7 Monitoring & Alerting
• Set up 24/7 monitoring
• Set up Email/text/SMS alerts
Get started for free here

5. Best Practices
• Configure issue definitions • Customize Alert Escalations
to different people
Critical Error: Recommend Senior Management:
network connectivity and page Consider text messaging
status code>400 as critical for critical errors not
errors (this is the default resolved after a certain
setting) duration
Error: Recommend Development Team:
HTML content check Consider email alerting for
as an error critical errors, errors and
maybe warnings
Warning: Recommend treat
page asset issues, user Operations Team: Consider
experience issues and text messaging for critical
network performance issues errors and errors
as warnings

6. Real Browser Monitoring
• Only real browser monitoring provides visibility
into your users’ experience
• With Firefox, IE and Chrome browsers from locations
across the planet, you can capture your users’ experience
and see exactly what your users see

7. Best Practices
• Select monitor agent • Monitoring • Select the
Agent: frequency: locations your
• Use HTTP agent for APIs
HTTP and HTTP services – For availability users are from
• Use HTTP agent if you monitoring, try
are only interested in 5 min or less • To monitor
service availability and – For Operations logged in pages,
network performance
Team, try 5 min
• Use browser agents
or less
use Yottaa’s
if you are interested
in front end as well as – For Development “Advanced” tab
backend performance
and availability, including
Team, try 10 min (more later)
to 30 min
monitoring third party
assets/widgets
Browser Agent
Choice:
• Pick the most popular
browser for your site
• IE9 is still the most popular
browser in general, but
your site may differ

8. Monitor Pages Key for Your Business
Monitor:
• App pages
• Pages behind a log-in
• Customer sites
(specific examples on next page)
Using cookies and other methods,
monitor pages behind log-in,
user sessions such as shopping cart, etc.

9. Best Practices: Which Pages to Monitor
Web Sites & User
Mobile Apps SaaS
Registration
Web Sites & SaaS Apps:
Mobile Apps: • User dashboard page
• Home page • Key pages behind
• Key landing pages user login
(e.g., User registration
page)
• Pages using different e-Commerce
technologies (e.g., User
forum, knowledge base)
• Dynamically-generated e-Commerce Sites:
pages • Product catalog page
• Pages from different • Product detail page
origin servers
• Customer shopping cart page
• Customer checkout page

10. Best Practice: Performance Dashboard

11. Track User Experience Trending
Pages behind login can be
really slow, and perform
inconsistently

12. Best Practices: What Trends to Look For?
• Look at the Summary Section for Average Values
– If the average values are way out of the range, consider
you have problems
• Look at the Trending Curves:
– Trending curves should be as flat as possible (consistent)
– If there are sudden spikes, these spikes are typically problems
• Click into them to look at the data samples to see what happened
– If the trending curve goes up and down too much, you may have
a consistency issue
• A consistent experience means
confidence and reliability
• Trending Curves to look into:
– Time to Interact
– Time to Start Render
– Time to Last Byte
(see appendix for definitions)

13. See What Your Users See
• Capture user
experience as video
and screenshot
filmstrips
• Correlate with user
experience & site
performance metrics

14. Best Practices
• See What Your • Analyze the • Stats to Think
Users See Page Load About:
– Look at the film strip, Waterfall – How many requests
would you be happy are there? Too many
as a user? – Are there any requests can mean
particularly slow a slow experience
– Play the film strip loading assets?
as a video to get – How many third party
a realistic replay – Are there any HTTP assets? Too many
of the actual user status code errors? of them can mean
experience – Are their obvious an unreliable user
gaps in the waterfall? experience
• Which typically means
heavy client side
JavaScript execution
blocking the entire
browser
HTTP

15. Ensure Performance & SLA Objectives
Get started for free here

16. Capture & Diagnose Site & App Issues
• Define and monitor the issues that matter
to your business
• Gain 360 visibility into all factors impacting your site:
– Server issues
– Page errors
– Network problems
– 3rd-party asset issues

17. Capture & Diagnose Site & App Issues

18. Best Practices: Dealing with Issues
• Configure issue definitions • If you are only interested
in monitoring
certain domains,
Critical Error: Recommend
network connectivity and page you can do so
status code>400 as critical (simply configure it)
errors (this is the default
setting) • Check the Issue
Listing Page to
Error: Recommend
HTML content check see how many
as an error events each
Warning: Recommend treat issue has
page asset issues, user generated
experience issues and
– The more events, the higher
network performance issues
as warnings priority to tackle the issue

19. Statistically Meaningful Testing
• Eliminate random noise
• Schedule tests across geographies,
browsers, last mile connectivities
Get started for free here

20. Best Practices
• Monitoring & Testing • Operations Team:
– Use WebsiteTest.com
Site Monitor:
to diagnose
Is good for continuous
monitoring and issue issues
alerting
WebSite Test: Is great
for on-demand testing • Development Team:
and diagnosis – Use WebsiteTest.com
• Statistically to analyze & improve site
Meaningful Testing:
Collect enough
number of samples
to improve signal to
noise ratio

21. Improve Web & Mobile Experience
• Explore every asset’s impact on performance
• Identify bottlenecks using waterfall chart
(In this example – the Facebook widget is slowing down the page load!)

22. Best Practices
• Analyze • Check Data • Analyze Individual
Trending First Samples Related Data Samples
– Use “Edit Filter” to “Odd” Trending – Does the page loading
to compare key filmstrip look ok?
– To determine the root
metrics such as Time – What does the page
cause for “odd”
to Last Byte, Time to load waterfall show?
Start Render and – Any error or warning?
Any particular slow
Time to Interact assets, HTTP errors?
– Identify “Odd”
data points from
trending lines
– Identify if any issue is
a consistent issue or
only temporary

23. Appendix: A Primer on
User Experience &
Site Performance Metrics
23

24. Characterizing User Experience
Get started for free here

25. Time to Title
• Time at which the browser is able to display
the title of the site in the title bar.
• The first visual indicator to the user that the
loading process is working.
• Typically, ~1 sec.
Get started for free here

26. Time to Start Render
• (Also know as “Time to First Paint”)
• The time at which browser draws the first
visual element of a web page.
• Up to that point, a blank screen  high risk
of abandonment if takes too long.
• Typically, ~2 sec.
Get started for free here

27. Time to Display
• time at which browser has finished parsing the
HTML web page, constructed the Document
Object Model (DOM) and displayed the HTML
document.
• (However, the assets in the document such as
images may not have been loaded yet.)
• Typically, ~5 sec.
Get started for free here

28. Time to Interact
• The length of time it took before a user could
interact with the page (type into a field, click on a
link, etc.).
• Typically, ~6 sec.

29. Yottaa Site Monitor
• 24/7 Site Monitoring is always on alert & ready
to help
• Count on Yottaa to monitor user experience &
identify problems
• Multivariate testing & web performance
troubleshooting with real-browsers
• See what your users see from any device, any
location, & any browser
Get started for free here
29