Load Testing Essentials for Peak Season Performance

©2016 SOASTA, All rights reserved.
Tom Chavez @tomchavez
Mike Ostenberg @mikeostenberg
Load testing for peak season:
What to test and how to test

©2016 SOASTA, All rights reserved.
• How fast is your site now?
• Measure everything
• Too much data? Science!
• Prioritize what to fix
• Optimize your site
• What to test?
• Test early and often – shift
left and right
• How to test?

©2016 SOASTA, All rights reserved. 3
Measure, Optimize, Test – Repeat

How fast is your site now?
• Google says 1000ms!
• No single right answer
• Not all pages are equal
o Some pages should be faster than others
o The slowest pages might not be the ones to fix first

How is your site now?
• mCommerce is global
o US is not the benchmark
• 39% of mobile users
are unhappy with their online experiences
o Top reasons: slow pages and site freezes/crashes
• 57% have experienced problems when using mobile
• 3 out of 4 consider mobile slower than desktop
• 46% will not return to a site that performs poorly
• 27% of smartphone users exceed data cap every month

When is your Peak Season?
• Peak is year-round
o Super Bowl Sunday – 110 million viewers
o Bi-weekly or monthly payroll processing
o Valentine’s Day, President’s Day, Mother’s Day, Father’s Day
o Quarterly processing and filing
o US Tax filing
o Spring, Summer, Fall & Winter season sales
o July 4th, Christmas in July
o Back to School  We are here
o Olympics
o Halloween
o Election Day
o Singles Day – world’s biggest shopping day, $9B in 12 hours
o Black Friday, Cyber Monday and Holidays

Measure Real, Synthetic, and Virtual Users
• Real Users (RUM) provide deep data about your visitors
o No other way to know everything about your users on your site
• Synthetic Users tell you how parts of your site are doing
o When no real users are visiting, synthetic users are still coming
• Virtual Users tell you how your site will do at scale
o Can’t get a million friends to visit your site at the same time?
• Collect all the data and save it in the cloud!
o Storage is cheap in the cloud, plus easier to analyze there
• Add App Performance Management (APM) for a complete picture
The Three Musketeers of Performance

Other Things to Measure
• Session paths
o How are your users going through your site
• Mobile App Performance
o How are your mobile apps performing?
o What are users doing in your mobile apps?
• Revenue
o How much money are you making from each user and where?
• How fast are your competitors’ pages?
o You can’t insert code on their sites but you can measure
Your company may be measuring these already

Analyze
• Benchmark and baseline your site
o How fast is your site now? How about the competition?
• Correlate performance and outcome
o Is poor performance affecting the business?
• Find root causes for performance issues
o Use RUM and APM data
• Fix pages that impact most
o Make a prioritized punch list of the pages to fix
Use the data you’ve collected to gain insights

Baseline Your Performance
How is your site doing? How does it compare to the competition?

Find Root Causes for Poor Performance
Not all problems are from your app or back end systems

Page Visits vs. Performance

Page Visits vs. Performance by Device

Prioritize What to Fix – Greatest Impact
The poorest performing page might not be the most important page

Page Bloat
The average web page is more than 2MB in size

Page Bloat
Where are all those kilobytes going

Doesn’t Bandwidth Solve the Problem?

Comparing Network Performance

25x Faster Bandwidth != 25x Faster Load Time

Prepare for Peak Events
Optimize content for peak load events
• Simplify website content
o Keep only the most important content on the page
• Reduce infrastructure requests
o Block unnecessary requests, out-of-region visitors
• Leverage historical comparisons
o Review past traffic levels to predict future levels

Creating a Test Plan – Session Analysis
Which paths are the most important to users?

Creating a Test Plan – Peak Load
What was the busiest Day, Hour, Minute, Second?

Creating a Test Plan – Page Flows
What were most popular page flows with think times?

Creating a Test Plan -
Which paths should I test, what think times, at what load, from where?

Network Emulation
• Testing to match realistic situations
o Users are not all from one location on a fast network
o Packets get delayed, lost, re-ordered
• Apps perform differently as network conditions worsen
o Reduced throughput
o Timeouts occur, connections are lost
o Transactions take longer, longer sessions
Adding real network modeling to load and performance testing

Start Testing Yesterday
• How long will it take?
o 300 test cases x 2 browsers = 600 cases
o 600 cases x 30-minutes/case = 300 hours
o 300 hours / 6 hours/day = 50 days
o 50 days / 5 people = 10 days
• And for mobile:
o 300 text cases x 26 devices = 7800 cases
o 7800 cases x 30 minutes/case = 3900 hours
o 3900 hours / 6 hours/day = 650 days
o 650- days/5 people = 130 days of test – 26 weeks!
And even earlier for Mobile

Don’t Wait Til the Last Minute to Test!

Test Right and Test Left, even into Production
• Performance Testing in development
o Set performance SLAs for new code and test in development,
same as unit and functional tests which code must pass
• Performance Testing in staging
o Higher loads in near-production systems
• Performance Testing in Production
o Full scale load tests to expected load and beyond
o Find weak and stress points and prioritize fixes into project
Performance Testing is Continuous

Test Continuously
Test from Development to Integration to Staging to Production

Test in Production
Some issues will only be found at scale load on production systems
Don’t forget your third party providers: CDNs, external scripts

Measure Mobile App Performance Under Load
All mobile apps perform well when there is no load

Automate Your Testing
Use Continuous Integration tools to automate testing

Now That the Test is Over
• Analyze results for future areas to address
o Weak or stress areas that aren’t likely this time but …
• Identify third parties that were affected by load
o Coordinate with suppliers to address issues
o Align schedules or at least be aware of their changes
Even if the test passed, you’re not done

Issues Will Happen
• “I am 100% confident that everything we tested will work
just fine.” – Nordstrom Performance Leader
o When loads are different than what you modeled, may have
problems
o E.g. At Nordstrom, during a recent anniversary sale, mobile sign-
ins were 9x greater than anticipated
o Used RUM to get to the root cause of the issue
Realize that Issues Will Happen Where You Can’t Predict Them

Questions, Resources & Next Steps
• Holiday Readiness
o soasta.com/holiday-readiness
• The Performance Beacon
o soasta.com/blog
o soasta.com/blog/2015-popular-posts-web-performance
• Upcoming Webinars
o soasta.com/webinars: RUM-Based Testing
o Recordings of past webinars
• CloudTest on-Demand
o soasta.com/cloudtest: SOASTA will build & run tests for you
Resources

“The leader in performance analytics.”
www.soasta.com
SOASTA

CloudTest
• Load testing from development to production
o Continuous testing at any scale, on-prem or in the cloud
• Real-time analytics with customizable dashboards
o Safely test in production with full visibility
• Run tests live or automatically
o Full hands-on control or hands-off automation
• Full end-to-end view
o Front-end performance to back-end systems
Fast, scalable, affordable load testing for performance optimization

Load Testing Essentials for Peak Season Performance

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (16)

Ähnlich wie Load Testing Essentials for Peak Season Performance

Ähnlich wie Load Testing Essentials for Peak Season Performance (20)

Mehr von SOASTA

Mehr von SOASTA (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Load Testing Essentials for Peak Season Performance

Hinweis der Redaktion