An Introduction to Software Performance Engineering

Enterprise Software
Performance Engineering
Building systems that scream so your users don’t:
A key component of risk management
Presented By: Walter Kuketz
Hosted By: Frank Days

© 2012 Collaborative Consulting

Housekeeping

Presentation will last 45 Submit questions via the Slides will be made
minutes chat window available

2
2012 Collaborative Consulting

Purpose of today’s webinar

Review the current challenges of being a performance engineer

Overview of the five knowledge areas of Software performance engineering body of knowldge

Where to inject PE tasks and activities into the SDLC to build more performant applications

Overview of application performance management (end-to-end user experience)

Roles and career path overview

Case study – 10X event

3

Performance engineering challenges today

Highly • Complex and tangled system of systems
distributed • Distributed Infrastructure
apps • Distributed production operations

• Performance engineering and testing
Distributed
• Architecture and development
teams
• Business

Evolving & • Mobile devices
changing • Third party services
application • SaaS products
architectures • Cloud

Changing • Driven by mobile anywhere
usage
• Flash traffic events
patterns

4

Performance engineering challenges

User
Enterprise Budget Team skills
experience
• Release • User populations • Constant cost • Performance
schedules, mont are exploding and budget organization;
hly, quarterly pressure skillsets, multi
• Web site disciplined,
• Project based performance • Maintaining a communication,
directly impacts meaningful split team
revenue performance lab
(production-like) • Training and
career paths

5

No clear standards in the market for PE

Test automation technician
Capacity planner

?
Performance tester
?
? Performance Engineer
Performance analyst

?
Performance Architect
?
Benchmarks
?
Troubleshooting
Performance Data engineer
?

6

Goal of the SPE Body of Knowledge

Define the profession of performance
Define profession engineering

Establish Body of Establish a body of knowledge for
Knowldge performance engineering

Establish Establish a common language for roles,
common responsibilities, and practice areas using
language the form of Knowledge Areas

Define Define the competencies shared across
competencies the knowledge areas

Capture common Capture common techniques for sharing
techniques across the industry

7

The SPE Body of Knowledge

Enterprise Software Performance Engineering

Planning, coordination, information sharing, and control
Knowledge areas

Performance
validation and testing Problem
Application
SDLC & (PVT) detection
performance
architecture and
management
(SA) Capacity planning (APM)
resolution
(PDR)
(CP)

Underlying competencies and roles for delivering

Techniques supporting the knowledge areas

8

The Enterprise PE team: There must be a practice leader

A shared service and clearing house for all performance, scalability and
capacity planning information. Transforms performance information to
business knowledge and advantage.

Communicates across the teams to share information

For instance, the production workload and the
Operations PT performance test workload – are they in sync?

Architecture and Make sure the architecture team knows how the
production new release, or newest build behave in production.

The business risks are clearly identified and
Business direction communicated.

Can provide performance guidelines to the
Development teams developers, must influence the unit testing process.

9

Information sharing across the knowledge areas

Enterprise PE
Capacity
planning and planning Triage
monitoring

SPE impacts Architects,
Results,
Developers, testing, and
Forecast Workload operations
Workload scalability

NFR’s Performance User Exp
Volumetrics, validation, Workload
Utilization
workload testing

SDLC, Results
Architecture Observations APM
, NFR

Growth plans
Workload
NFR’s

10

PE: Knowledge areas

SA PVT CP APM PDR
Software development Performance testing and Capacity planning Application performance Problem detection and
lifecycle and architecture validation management resolution

11

PE: Software development lifecycle and architecture

SA PVT CP APM PDR

12

PE: SA (software development lifecycle)
Waterfall methodologies

Functional Performance
Define Design Develop Deploy
test test

Need to plan for Build code to performance
performance testing requirements

Define

Deploy Agile Design
methods

Test Build

Can a system be designed to support 10 TPS and 1 second response time work @ 100 TPS and 1 second?

13

SDLC: Define and design
Define Design
1. Project risk scorecard 1. Technology Architecture validated
- Key business transactions to the performance Usage
identified Scenarios. How will it achieve the
- Workflow constraints key performance requirements?
Sub-second
- SLA’s and penalties response time. Define the scalability approach.
Flash marketing
2. Workload model and usage event. 2. Scalability to support flash events
scenarios
3. Environments: include any
3. Business volumes with peak
required changes to develop and
factors
test the performance usage
4. Business performance scenarios.
requirements

Communication to your downstream processes and people
Prepare to deal with flash events

Design to avoid flash events
(they are expensive. E.g. Work with business why does the outbound email
have to send email to all xM customers at the same time

14

SDLC: Develop and functional testing
Functions are tested. Performance needs to be analyzed.

Develop Functional test
• The functions are mapped to software • Verify business functions and key
components. The performance req’s are performance functions.
assigned to the components.
• Verify the key performance workflows
• Unit testing for early validation of the and functions.
performance req’s for the software
components. Response time • Define and execute micro-performance
measurements. testing in the QA process.

• Integration testing include micro- • Produce report on for each key
performance testing of key functions. performance Usage Scenario with
Code review of the key performance response time and throughput.
functions.

• Enhance unit and integration test for
early validation of key performance
functions

Greatest impact on performance and scalability, but you must know the goals.
Communication to your downstream processes and people

15

Performance budgets
Do your developers know how much time they have for their piece of the transaction?

Code runs
everywhere.
SERVER

WEB APP SVR

PROXY
MQ/ESB

LDAP CORBA

DCOM

Respond to the user
in 2 seconds. Web
Services
Datacenter

16

PE: Performance testing and validation

SA PVT CP APM PDR

17

PE: PTV (performance testing and validation)

What is changing? Where does it need to be validated? What (all) is being measured?

The project: Why are we testing and what are the success criteria?
Prerequisites:

Business goals and non-functional requirements are defined

Workload and scenarios defined

Environment defined (where are we testing)

Application in a stable state

Configuration management and release management ready

Core performance team identified

Supporting team committed

18

Organization: Performance team value
You still need to explain to the rest of the organization what you do

Business and Early involvement
application area with the applications Test execution Environment
knowledge team

• Deep understanding of • Performance engineers • Well-understood • Testing environment is
the application being involved in design workload model and a close approximation
tested sessions characterization of production, including
database size
• Deep understanding of • Understand the nature • All components in the
the technical of the data required for test are monitored and • Differences between
architecture of the testing reported on test and production are
application clearly known
• Able to understand • A defined triage process
• Understanding of the technical gaps to meet for troubleshooting • Release and
business area requirements not configuration
• A process to manage
supported by the supported by the tool management in place
the inevitable
applications unplanned tests and and procedures
enforced
test archive exists

Performance team value

19

Performance team reference model

Primary test, design, and execution team Performance engineering support skills

Planning and
Application
project
Performance architect
management
engineer

Technology
SME

Test
Results
automation
analyst Data
technician
engineering

Must determine the core team members vs. on-demand team

20

Enterprise performance team

PT problem Strategy and
PT technology
resolution planning
PMO
board (architecture)**

Infrastructure**

Performance
Performance test engineering and
execution team optimization

Application
Vendor Application
support
management deployment

Vendor IBM Oracle EMC Informatica Vendor Vendor

Vendors

21

Model test case execution
In the end, the test has to be run.

Prepare test
Prepare test Confirm
Select test harness online Initiate test
0 case or batch
monitoring
components
environment
readiness
case 1
messages

No
Problem
analysis and Evaluate
triage team impact of
Select test Normal test
1 case completion?
change

Yes
Yes Gather test
Analysis for Create test
results
specific report; publish
Txn RT, CPU,
DB
change? to repository 2
No

Are Categorize Can you
Implement
changes change – enter retest with Rerun the test
2 required for
Yes
change to CM
configuration
change
the
Yes
case
retest? tracking tool change?

No No

Move to next Move to next
test case 0 test scenario

22

PE: Capacity planning

SA PVT CP APM PDR

23

PE: CP (capacity planning)
The right system at the right price!

Plan for • Forecast workloads
future • Plan for usage

Analyze
• Track utilization of
existing systems current
capacity
• Track workload
Determine • Define system to
capacity support workloads
requirements • Test
• Define service
levels

Workloads drive system transactions, which drive resource utilization

24

Capacity: Use of resources
The overall system has a capacity; each component has a capacity.

SERVER

WEB APP SVR

PROXY
MQ/ESB

LDAP CORBA

DCOM

Web
Services
Datacenter

25

Workload model validation

Actual workload Synthetic workload

Model calibration
System System
Measured response times, Measured response times,
throughputs, utilizations, etc. throughputs, utilizations, etc.

Acceptable No

Yes

26

PE: Application performance management

SA PVT CP APM PDR

27

Five elements of APM (Gartner)

Monitoring the performance of complex distributed applications

End user experience measurement – browser/mobile
1 device

2 Create a model of the run-time environment (discovery)

Profile the performance and behavior of user-defined
3 transactions

Performance metrics from each of the
4 applications/systems technical components (Webserver,
App server, Database, etc.)

5 Application performance management database

28

End user experience monitoring

29

Track key user experience metrics

Mobile Total end-user response time
Browser rendering time
Network latency

Mobile Real User Monitoring
Rendering

30

And see every hop of the transaction

Mobile

Proxy
server
Web server
App server
Data warehouse gateway
Mainframe
Database

Dynamic Transaction
Path Detection
Rendering

31

PE: Problem detection and resolution

SA PVT CP APM PDR

32

Performance issues and root-cause analysis

This leverages several knowledge areas

A solution must be found quickly (under pressure)

Performance issues are found in:
• Preproduction environment
• Production and operations environments

Key business transactions are not meeting their service level objectives:
• End user experience is degrading (acute or chronic)
• Overnight processing is taking longer
• Stability and performance issues

33

When problems occur

Flash web site events (all at once)

New code released causes problems

Infrastructure upgrade

Consolidation of applications and servers

New workload model (batch and online)

34

Underlying competencies

Analytical
Project Communication
thinking
management skills
problem solver

Business Monitoring and
Statistics
Knowledge tuning

Workload
Behavioral Application and
model
characteristics tool knowledge
Underlying development
competencies

Test data Test execution
Forecasting
management process

35

Technical competencies

Code Deep dive Resource
Database profiling tools
utilization
monitoring tools (HP Diagnostics,
tools
CA Wily, etc, system
(Static analysis, Jprobe, etc)
diagnostics) level

End to end Comprehensive
Statistics and
performance results collection
Distributions
tools and reporting

Technical
competencies Test design Queuing
and and Test data
Little’s law
execution modeling creation
tools tools

36

PE Techniques

Brainstorming Data modeling

Creating a workload model Performance testing

Triage approach Static code analysis

Logging and
System metrics collection
instrumentation

37

PE roles: The team – many possible roles

Managing and developing the core team members

1 Technical Project manager

2 Performance Architect

3 Sr. Performance Engineer

4 Performance Engineer

5 Sr. Test automation technician

6 Test automation technician

7 Performance data engineer

8 Sr. Capacity planner

9 Capacity planner

10 Modeler

38

Career path: Performance engineer foundation

Performance
Engineer

Grow your Performance
testing
leadership and Lead
communication technical Workload
role models

Technical Production
architecture triage
Core Additional
technical tech
competency Communication
competency

Grow your breadth of capabilities
39

Case study: Preparing for a flash marketing event

Planning a large event to introduce a new product

• Send out email to all registered customers (500,000 currently), there is a link in the email.
• Run multiple TV ad’s encouraging people to register and then come to the event, results in
100,000 new registrations
• No new functionality required to support the event
• Potential 600,000 users of the web site

The business is forecasting at least a 10X increase during the
event, in terms of product sales

• How does this translate into system workload and utilization?
• Do we need more servers?
• The current system was not really designed for such a flash event

40

PE: Process to prepare for the event (10X)

CP CP PVT SA

Forecast the
workload Tune application
based on results
APM PDR

Measure the current Plan and execute
workload and PT for forecast
resource utilization workload

41

CP: Workload model

CP
The CP team uses this information to forecast the
new workload 10X.

Step Transaction – by product Percentage
1 Login 100
2 Search product 100
3 View product detail 100
4 Add to shopping cart 50
5 Confirm purchase method 50
6 Shipping information 50
7 Send confirmation email 50

42

CP: Analyze current capacity

CP Review current Production workload, and the under lying system
utilization.
• Average workload
• Current peak workload

Response
time

Service level

100%
System utilization

43

PVT: planning, executing, and analysis

PVT
• Use the workload model from CP
• Enable APM during testing
• Work with the Application Architect

Methodology is iterative and tactical

Define business
Review infrastructure
activity profiles & Design & build tests
& architecture
service levels

• Identify risk areas • Types & numbers of • Test data generation
• Review configuration users • Create test scripts
settings, topology, & sizing • Business activities & • User & transaction profiles
• Define points of frequencies • Infrastructure configuration
measurement
Iterate testing & tuning

44

Questions?

Thank you!

Get your free copy
SharePath RUM today!

More information:
www.collaborative.com
info@correlsense.com
www.correlsense.com/demo
www.real-user-monitoring.com

45

An Introduction to Software Performance Engineering

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie An Introduction to Software Performance Engineering

Ähnlich wie An Introduction to Software Performance Engineering (20)

Mehr von Correlsense

Mehr von Correlsense (18)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

An Introduction to Software Performance Engineering

Hinweis der Redaktion