Presentation to Large Scale Production Engineering meetup at Yahoo campus on June 27, 2012. Dan Bartow, VP Product Management at SOASTA discussed critical performance metrics.

1. 1© 2012 SOASTA. All rights reserved.#lspe Meetup

2. 2© 2012 SOASTA. All rights reserved. June 27th
, 2012
SOASTA : 12/2009 - Present
VP, Product Management
Intuit : 4/2007 – 12/2009
Sr. Manger, Engineering
TurboTax Online and E-Com
ATG : 10/2000 – 4/2007
Sr. Deployment Engineer
American Airlines, Best Buy, Target, Turbotax Online, Quicken Online, MySpace, Dennys, Dominos,
Mattel, Hallmark, FAA, US Army, AT&T Wireless, Alcatel, Newsweek, Oprah, Neiman Marcus,
Plantronics, Kodak, Jcrew, Newell Rubbermaid, Walmart, Target, Paychex, Fidelity Investments

3. 3© 2012 SOASTA. All rights reserved. June 27th
, 2012
Poor
Performance
Can be painful.

4. 4© 2012 SOASTA. All rights reserved. June 27th
, 2012

5. 5© 2012 SOASTA. All rights reserved. June 27th
, 2012
Full Scale – To and Above Expected Peak (2x/3x)Full Scale – To and Above Expected Peak (2x/3x)Full Scale – To and Above Expected Peak (2x/3x)Full Scale – To and Above Expected Peak (2x/3x)
In ProductionIn ProductionIn ProductionIn Production
Usually Live and During Off-HoursUsually Live and During Off-HoursUsually Live and During Off-HoursUsually Live and During Off-Hours
Millions of Concurrent UsersMillions of Concurrent Users
Tens of Millions of Page Views per HourTens of Millions of Page Views per Hour
Many Thousands of Orders per MinuteMany Thousands of Orders per Minute
Millions of Concurrent UsersMillions of Concurrent Users
Tens of Millions of Page Views per HourTens of Millions of Page Views per Hour
Many Thousands of Orders per MinuteMany Thousands of Orders per Minute

6. 6© 2012 SOASTA. All rights reserved. June 27th
, 2012

7. 7© 2012 SOASTA. All rights reserved. June 27th
, 2012
ss
Memory
leaks
Scale of Test
Stage/Team
Dev
& Test
Release
& Deploy
Network
& Ops
CDN file placement
Load Balancer configuration
Network bandwidth
Network configuration
DNS routing
Inadequate server resources
Default configuration settings
Unbalanced web serversAuto-scaling failures
Latency between systems
Slow third-party plug-ins
Garbage collection
Database thread counts
Inefficient database queriesSlow pages
Conflict with other apps
Test Lab Staging Production (100% +++)
Search technology limits
Method-level tuning
Max sockets exceeded
Firewall max capacity
Global latency variance
Security bottlenecks
Tuning for full-scale…and well beyond

8. 8© 2012 SOASTA. All rights reserved. June 27th
, 2012
Response Time
Revenue
$$$




9. 9© 2012 SOASTA. All rights reserved. June 27th
, 2012

10. 10© 2012 SOASTA. All rights reserved. June 27th
, 2012
Real-Time (within a few seconds)Real-Time (within a few seconds)Real-Time (within a few seconds)Real-Time (within a few seconds)
Combine and CorrelateCombine and CorrelateCombine and CorrelateCombine and Correlate

11. 11© 2012 SOASTA. All rights reserved. June 27th
, 2012
ApplicationApplicationApplicationApplication
Level 3
InfrastructureInfrastructureInfrastructureInfrastructure
Level 2Level 1
ExternalExternalExternalExternal

12. 12© 2012 SOASTA. All rights reserved. June 27th
, 2012
ApplicationApplicationApplicationApplication
Level 3
InfrastructureInfrastructureInfrastructureInfrastructure
Level 2Level 1
ExternalExternalExternalExternal
1.1. Response TimeResponse Time
2.2. Three CriticalThree Critical
TransactionsTransactions
(counts and(counts and
durations)durations)
3.3. ErrorsErrors
4.4. BandwidthBandwidth
1.1. Response TimeResponse Time
2.2. Three CriticalThree Critical
TransactionsTransactions
(counts and(counts and
durations)durations)
3.3. ErrorsErrors
4.4. BandwidthBandwidth
1.1. CPUCPU
2.2. MemoryMemory
3.3. Network and DiskNetwork and Disk
I/OI/O
4.4. Load BalancerLoad Balancer
1.1. CPUCPU
2.2. SSL TX/SecSSL TX/Sec
1.1. CPUCPU
2.2. MemoryMemory
3.3. Network and DiskNetwork and Disk
I/OI/O
4.4. Load BalancerLoad Balancer
1.1. CPUCPU
2.2. SSL TX/SecSSL TX/Sec
1.1. Container threadContainer thread
usageusage
2.2. ContainerContainer
memory usagememory usage
3.3. Method responseMethod response
timetime
4.4. ConcurrencyConcurrency
1.1. Container threadContainer thread
usageusage
2.2. ContainerContainer
memory usagememory usage
3.3. Method responseMethod response
timetime
4.4. ConcurrencyConcurrency
A few at each tier - This can be a ton of data just by itselfA few at each tier - This can be a ton of data just by itselfA few at each tier - This can be a ton of data just by itselfA few at each tier - This can be a ton of data just by itself

13. 13© 2012 SOASTA. All rights reserved. June 27th
, 2012
Work in LayersWork in Layers
• Pick three at eachPick three at each
levellevel
• Average responseAverage response
time is kingtime is king
Keep it Real (Time)Keep it Real (Time)
Actionable MetricsActionable Metrics
• Start from the outside andStart from the outside and
work inwardswork inwards
• Must be able to easilyMust be able to easily
combine and correlatecombine and correlate
metrics – it’s aboutmetrics – it’s about
reltionshipsreltionships
• Data within a fewData within a few
seconds isseconds is
important to beimportant to be
actionableactionable
Start SimpleStart Simple

14. 14© 2012 SOASTA. All rights reserved. June 27th
, 2012
Email: dan@soasta.com
Twitter: www.twitter.com/PerfDan
LinkedIn: www.linkedin.com/in/danbartow