Testing in production for online applications has evolved into a critical component of successful performance testing strategies. Dan Bartow explains the fundamentals of cloud computing, its application to full-scale performance validation, and the practices and techniques needed to design and execute a successful testing-in-production strategy. Drawing on his experiences, Dan describes the methodology he has used for testing numerous online applications in a production environment with minimal disruption. He explains how to create a performance testing strategy to give your team critical data about how your online application performs and scales. Learn how to create a robust lab-to-production ecosystem that delivers the answers about what will happen when peak traffic hits your site. Take back practical approaches to mitigate the three most common problems—security, test data, and potential live customer impact—that arise when embarking on testing in production.

1. TK
PM Tutorial
10/1/2013 1:00:00 PM
"Production Performance
Testing in the Cloud"
Presented by:
Dan Bartow
SOASTA
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Dan Bartow
SOASTA, Inc.
At SOASTA, Dan Bartow is Vice President of Product Management for the industry leading and
award-winning product CloudTest. Prior to joining SOASTA, Dan was Senior Manager of
Engineering at Intuit where his team was responsible for the speed and stability of TurboTax
Online. During the past decade he has been responsible for the performance of websites for
dozens of leading consumer brands.

3. 8/27/2013
Create a One-Page Capacity
Model for High-Traffic Web
Applications
Dan Bartow
SOASTA
About the Speaker
SOASTA
VP Product Management
CloudTest Evangelist
Intuit
Sr. Manger, Engineering
TurboTax Online and E-com
ATG
Sr. Deployment Engineer
Work
American Airlines, Best Buy, Target, Turbotax Online, QuickenOnline, MySpace,
Dennys, Dominos, Mattel, Hallmark, FAA, US Army, AT&T Wireless, Alcatel,
Newsweek, Oprah, NeimanMarcus, SBC, Plantronics, Kodak, JCrew, Cingular,
Newell Rubbermaid
1

4. 8/27/2013
Poor performance.
The Problems
1.
No knowledge of what the critical metrics are at each tier
of an applications architecture
1.
No record of what the critical metric values were at the
peak last year
1.
No idea of what those metrics should be at the next peak
traffic day
1.
No indication of what the capacity is today and how that
relates to the upcoming peak
2

5. 8/27/2013
The Solution
(or at least a huge step in the right direction)
The one page capacity model
What one looks like
3

6. 8/27/2013
How to build it
Step 1.
Create a simple architecture diagram, containing mostly
infrastructure, that lists key technologies in play for each tier
You don’t need this
4

7. 8/27/2013
Getting closer
What to include in the diagram
1.
2.
3.
Key infrastructure components
Critical services and their infrastructure where possible
Third party components (if applicable)
Hint: If you can’t test it, you probably don’t need to include it. #controversial
5

8. 8/27/2013
A reasonable diagram
How to build it
Step 2.
Identify the metrics that should be tracked for each tier
6

9. 8/27/2013
Three layers of monitoring
Layer 1 - Customer
Concurrent sessions
Pages (requests) / sec
Layer 2 - Server
CPU %
Memory
Network IO
Disk IO
Layer 3 - Container
JVM Heap usage
Threads / second
Login servlet invocations / sec
Database connections
To make this all happen (and be
really useful)
You need comprehensive monitoring
You need to be able able to test to scale
7

10. 8/27/2013
Metrics examples for common
technologies
Metric examples for common
technologies
CPU
Memory
TX/sec (especially SSL TX)
Bandwidth/sec
CPU
Memory
Threads / sec
Container
Application
CPU
Memory
Threads or processes / sec
Key GET/POST actions
Key method calls
External connections
Container
Application
CPU
Memory
JVM heap usage
Garbage collection interval
Threads / sec
Key servlet invocation count (login, order
placement)
Database connection count
Key servlet execution time
Container
CPU
Memory
Connections
Queries / sec
(SELECT/INSERT/UPDATE/
DELETE)
Table or row locks
8

11. 8/27/2013
How to build it
Step 3.
Fill in the values for your peak last year (the peak second)
How to build it
Step 4.
Project and do simple math for the upcoming year. Base
this math on top level percentages and honor relationships
between metrics where needed.
9

12. 8/27/2013
Contact Information
Dan Bartow
VP Product Management
SOASTA Inc.
Email:
Twitter:
LinkedIn:
Blog:
dan@soasta.com
www.twitter.com/PerfDan
www.linkedin.com/in/danbartow/
www.soasta.com
10