This document summarizes a presentation on performance testing fundamentals for managers. It discusses what performance testing is, the components and lifecycle of performance testing. Performance testing determines if a system can meet business goals like user expectations, load capacity, data integrity and scalability. It identifies when a system will degrade or fail under load. The lifecycle involves initial planning, design, analysis, building tests, executing tests and analyzing results in iterative cycles.

1. Performance Testing
Fundamentals for Managers
Presented by:
Jeff Roth
VP of Consulting Services
BenchmarkQA
Date:
March 22, 2012

2. Agenda
Performance Testing 101
Components of Performance Testing
Performance Testing Lifecycle
© 2012 BenchmarkQA Slide 2

3. What is Performance Testing?
Batch Jobs Stress
Load
Volume
Scalability
System Benchmark Failure Point
Baseline
Durability
Failover
System Optimization
Infrastructure Validation
System Tuning
Desktop Benchmark
© 2012 BenchmarkQA Slide 3

4. Business Goals
Will my system meet my goals?
User expectations (response time, accuracy)
Load capacity (user volume, cost of configuration)
Data integrity (at normal loads, peak loads, and failure)
Stand up over time (durability)
Failover capability (continuity through failure)
System growth (scalability)
When will my system degrade and when will it fail?
What do I need (infrastructure) for effective
performance?
© 2012 BenchmarkQA Slide 4

5. Please excuse me while I shout .
PERFORMANCE TESTING IS
DEFINED BY WHAT YOU
WANT TO ACCOMPLISH
NOT WHAT TOOLS YOU OWN
© 2012 BenchmarkQA Slide 5

6. What Can Performance Testing Do?
Before we answer this question, let s start with a little
anatomy of IT systems
© 2012 BenchmarkQA Slide 6

7. User Interfaces
Input Keyboard, Touch Screen, Voice,
Camera, Hand Scanner, etc.
Output Screen, Printer, Sound, etc.
Processing CPU, GPU
Storage RAM, Disk, SDD, etc.
Connectivity LAN, WAN, WiFi, 3G/4G
© 2012 BenchmarkQA Slide 7

8. Performance Testing Devices
Business Goals
Configuration Optimization
Best hardware for the job
OS/Software configuration
Baseline Device Performance
Embedded devices software dependent upon hardware
With and without connectivity
© 2012 BenchmarkQA Slide 8

9. Performance Testing Systems
Include devices or not
Real vs. simulated integration points
Isolate risk areas
Iterative process
Load testing tools
© 2012 BenchmarkQA Slide 9

10. COMPONENTS OF A
PERFORMANCE TEST
© 2012 BenchmarkQA

11. What Makes up a Performance Test
Load Generator Target of Test
Web Farm Mainframe
Controller
Databases
© 2012 BenchmarkQA Slide 11

12. What Makes up a Performance Test
Virtual User Script
Home Page Search Item Select Item
Activity/Business process
© 2012 BenchmarkQA Slide 12

13. What Makes up a Performance Test
Technical Details of a Virtual User Script
Protocol - HTTP, AJAX, SOAP, SAP, CITRIX, etc.
Uniqueness
Session variables
User IDs
Test Data
Files
Correlate within the script
Made up
Think time
© 2012 BenchmarkQA Slide 13

14. Performance Test Scenario
One or more test scripts
Quantity of VUsers per script
Velocity of activity per user
Designate ramp up & test schedule based on goal
© 2012 BenchmarkQA Slide 14

15. Ramp Up And Failure
Number of
Percent of VUsers Axis
Errors Axis
Errors begin to
occur @ 400
VUsers
© 2012 BenchmarkQA Slide 15

16. PERFORMANCE TEST
LIFECYCLE
© 2012 BenchmarkQA

17. PMBOK - PROJECT MANAGEMENT PHASES
Software Development Lifecycle
SELECTION INITIATION PLANNING EXECUTION & CONTROL CLOSURE
Project Definition Requirements Analysis
& Design
Technical Analysis &
(SDLC)
Design
Build/Integrate & Test
User Acceptance
Test
Deployment &
Wrap-Up
Performance Testing
Initial Planning Initial
Design
Planning
Lifecycle
Analyze Build
Execute
© 2012 BenchmarkQA Slide 17

18. Performance Testing Lifecycle
Initial
Design
Planning
The Performance Test
lifecycle requires some initial
Analyze Build
upfront planning work, and
becomes a cyclical effort to
achieve your performance
goals. Execute
Typical engagements are 7 to 9 weeks
including initial planning.
© 2012 BenchmarkQA Slide 18

19. 4 Test Effort Scope
4 Software Components
4 Target System Architecture
4 High Volume Business Processes and Resource
Intensive System Activities
4 Workload Profiles
4 Performance Test Goals
Participants 4 Resources
4 Target Test System Hardware
Ø Project Stakeholders
Ø Project Manager 4 Project Personnel and Test Effort Roles
Ø System Architects
4 Test Data Management Plan
Ø Key SMEs
Ø Performance Architect 4 Test Tools (load injection hardware and software)
4 Test Schedule
Duration 4 Project Schedule
4 Project Dependencies
1 to 4 Weeks 4 Performance Test Work Breakdown Structure
© 2012 BenchmarkQA Slide 19

20. 4 Load Test Tools
4 Test Script IDE
4 Load Generation Farm
4 Protocol Licensing (Web, Winsock, Citrix, etc.)
4 Business Process
4 Screen by Screen Walk Through
4 Usage Frequency and Pacing
4 Test Data & Test User IDs
Participants
4 System Intensive Processes
Ø Business SMEs
Ø System SMEs 4 Batch Jobs
Ø Performance Architect 4 Reporting Systems
Ø Performance
Engineers
4 Interfaces/Feeds
4 Test Monitoring Strategy
4 Servers/Databases/Network
Duration 4 Integrate with Load Test Tool (Security
Permissions)
1 to 4 Weeks
4 Data Center/ SME System Monitoring
© 2012 BenchmarkQA Slide 20

21. 4 Test Scripts
4 Record Business Processes
4 Correlate Data
4 Parameterized Data
4 Pacing
4 End User Response Time Measurements
4 Typically 4 to 8 Scripts
4 Test Scenarios
Participants 4 User Ramp Up Scenario
Ø Business SMEs 4 Frequency of Business Process Activities
Ø System SMEs
4 User Managed Activities
Ø Performance
Engineers 4 Results Storage
4 Verify Test Readiness
Duration 4 User Concurrency
2 to 12 Weeks 4 Multi-Script Concurrency
4 Load Generators
© 2012 BenchmarkQA Slide 21

22. 4 Schedule Execution
4 No Users in Target System
4 Anti-Virus Scans, Automated Backups
4 Off Hours Scheduling
4 Monitor Execution
4 Load Test Progress
4 Load Test System
Participants 4 Real Time Response Time Metrics
Ø Project Stakeholders 4 Target System Utilization Under Load
Ø Project Manager 4 General Execution Approach
Ø System Architect
Ø Key SMEs 4 Small Load First To Confirm System Is Stable
Ø Performance
Engineers 4 Reasonable User Load Ramp Up
4 More Than One Test Execution To Confirm
Goals
Duration
1 to 8 Weeks
© 2012 BenchmarkQA Slide 22

23. Typical User Response Graph
© 2012 BenchmarkQA Slide 23

24. 4 Test Goal Assessment
4 Response Time Performance
4 System Utilization
4 Bottleneck Identification
4 Correlate Response Metrics with System
Utilization Metrics
4 Isolate Causal System/User Activity
Participants 4 Remediation Strategy
Ø Project Stakeholders 4 Configuration Changes
Ø Project Manager
Ø System Architect
4 System Changes
Ø Business SMEs 4 Hardware Changes
Ø System SMEs
Ø Performance Architect
4 Test Script Changes
& Engineers 4 Load Scenario Change
4 Monitoring Changes
Duration
1 to 4 Weeks Repeat Cycle Until Goals Achieved
© 2012 BenchmarkQA Slide 24

25. Things to Remember
Set clear objectives
To ensure all team members agree on the questions performance
testing will answer
non-functional requirements
solution requirements specification
To understand the expectations of the business owners
To know when testing is complete
Strive for realism
Subject the system under test to conditions that match production as
closely as possible
If possible use the actual production hardware for the performance test
© 2012 BenchmarkQA Slide 25

26. Things to Remember
Understand what happens
Every component of a system must be monitored and data collected
Subject matter experts analyze collected data
Provide meaningful recommendations
Must improve the end-user experience
Must be based on a cost-benefit analysis
© 2012 BenchmarkQA Slide 26

27. Thank You!
For additional information on Consulting Services with
BenchmarkQA, please contact:
Jeff Roth
952.392.2384
jeff.roth@benchmarkqa.com
© 2012 BenchmarkQA