2. Performance modelling background
• My background is analysis of distributed systems, middleware, GRID,
architecture, performance, benchmarking (e.g. SPECjAppServer),
sensor web performance, etc
• Since 2007 project in NICTA to develop tools to assist mostly
government systems of systems to perform better in advance
• Service Oriented Performance Modelling tool
• Model driven (SOA performance meta model)
• GUI
• Simulation for metric prediction
• Enables modelling at level of workloads, composite and simple services,
servers.
• Used during early, middle, later lifecycle for lots of real systems
16/03/2016 Performance Assurance Pty Ltd 2
3. Performance modelling background
• BUT Manual model building (structure, parameterisation, calibration) is
• Time consuming
• Expensive
• Error prone
• Limited to model complexity that can be built manually
• Not easily repeatable or maintainable
• Not accurate enough for some problems (need high quality and quantity of
performance data)
• Not fast enough for agile development
• Last 3 years we have been a start up company, have to make $$$$$$
• Most customers have APM products
• All customers want to increase speed and number of releases, reduce time and
costs of testing
• Solution is to use automatic model building from APM data
• Cheaper and faster and more accurate
• Solves new problems, e.g. DevOps
16/03/2016 Performance Assurance Pty Ltd 3
4. Automatic performance modelling from APM
data
• Only use available APM data
• Use automatable (or potentially automatable) ways of getting the
data from the APM into our Service Oriented Performance Modelling
(SOPM) modelling/simulation tool (SaaS)
• Automatically build and parameterise the performance data from the
APM data
• Multiple model types with various trade-offs, accuracy for
capacity/response times, and model complexity/ability to change
model aspects
• Currently different model types are produced as part of the APM ->
modelling tool transformation phase
16/03/2016 Performance Assurance Pty Ltd 4
10. Experiences with three projects
• Project 1
• P2V migration
• Project 2
• C2V test -> prod
• Project 3
• DevOps
• Focus of this talk, come to main ICPE talk for others
16/03/2016 Performance Assurance Pty Ltd 10
11. Project 3
• Devops
• Focus on response time SLAs
• Deployment/resources
• Faster cycle time
• More releases
• Less and cheaper testing
• Challenge
• Proprietary in-house APM tool
• “Profile point” times only
• Required pre-processing (using Hive)
16/03/2016 Performance Assurance Pty Ltd 11
12. Focus
• Risk service
• Heavily used
• Multiple services
• New services added all the time
• Services had different time and memory profiles
• Would a new service break the SLA?
• Baseline model accurate to 10% response time
16/03/2016 Performance Assurance Pty Ltd 12
13. Alternatives modelled
• Changing transaction mix
• Changing arrival rates
• Making some services asynchronous, concurrent
• Adding new risk assessment services
• More complex
• Optimising deployment of services to multiple servers taking into account
memory and CPU usage, and response time
• A type of box/bin packing problem
• 4 services out of 30 used 50% of CPU
16/03/2016 Performance Assurance Pty Ltd 13
14. Challenges
• Pre-processing APM data “profile points”
• Low load for APM data sample c.f. target load
• Used calibration from load tests on pre-production to improve accuracy
• No CPU time breakdown from APM data
• But GC had a profile point (and was significant)
• Transaction types not in APM data
• Had to infer them, either too few or too many
16/03/2016 Performance Assurance Pty Ltd 14
18. DevOps
• Goal is to shift left and shift right
• Shift right
• Build and continuously maintain performance model of production to accurately model
response times, scalability, capacity and resource requirements under target production
loads
• Shift left
• Calibrate production performance model for development
• Enable developers to make code changes, explore impact with unit tests and
development APM to incrementally rebuild performance models
• To understand likely performance and scalability impact
• Speed up development cycle as no longer have to wait (weeks) for performance testing
16/03/2016 Performance Assurance Pty Ltd 18
19. Existing Dev, Test, Prod lifecycle: Delays in
feedback: Takes weeks per iteration, test env is a
bottleneck, environments are different
Dev Test Prod
Late Feedback Late Feedback
Deploy to test Deploy to prod
20. DevOps + APM: earlier but not completely
accurate performance feedback
i.e. environments are different so APM data is
different across lifecycle
Dev Test Prod
Late Feedback
Deploy to test Deploy to prod
APM APM APM
Earlier Feedback
21. DevOps + APM + Modelling: Earlier more accurate
performance predictions -> decreased cycle time
16/03/2016 Performance Assurance Pty Ltd 21
Dev Test Prod
Deploy to test Deploy to prod
APM APM APM
Early Feedback
Base
Model
Dev
Model
Incremental updates to
Base model with dev changes
Baseline model buildDev Model Update
Calibrate prod model for dev
22. Benefits
• Changes in code in Dev
• Unit test
• APM performance data
• Incrementally update calibrated performance model
• Predict performance and scalability impact for Prod env
• Cheaper and faster than waiting for testing and deployment to Prod
• Sensitivity analysis could determine areas of greater sensitivity to
changes and thresholds
• These would be subject to more rigorous modelling and testing
16/03/2016 Performance Assurance Pty Ltd 22
23. DevOps + APM + Modelling: In reality lots of
dev, different environments
16/03/2016 Performance Assurance Pty Ltd 23
Dev Test
Prod
Deploy to prod
APM APM APM
Base
Model
Dev
Model
Baseline model build
Dev
APM
Dev
Model
Dev
APM
Dev
Model
Dev
APM
Dev
Model
24. Challenges
• Calibration of performance models for use in Dev from Test and Prod
• Once predictions are made how do we test if they are supported by the APM
data or not? i.e. if null hypothesis is “changes in dev will have no impact on
prod”, how do we determine if this is supported by evidence or not?
• Is it scalable?
• Lots of developers and changes to subsets of code
• Concurrent and compounding changes would need centralised model with all changes incorporated
• What about changes to infrastructure code that could impact everything?
• How to support this in Dev APM and modelling tools
• ROI
• Depending on cost of testing, cost of initial setting up APM and modelling tools and
incremental costs, number of tests and modelling predictions per cycle, and value of reduced
cycle times and earlier performance predictions, ROI may occur earlier or later or never…
• Example
• Assumes model calibrated once per release from performance APM data
• Assumes one actual load test per release
• What’s tradeoff between multiple tests per release vs 1 test and multiple modelling predictions?
16/03/2016 Performance Assurance Pty Ltd 24
25. Costs: Modelling cheaper after 3 changes
16/03/2016 Performance Assurance Pty Ltd 25
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
0 2 4 6 8 10 12
Cost($)
Number of changes tested/modelled
Costs of LoadTest only and HybridApproach
LoadTestOnly Modelling
26. Speed: Average hours to test/model a number of
code changes (per model calibration)
16/03/2016 Performance Assurance Pty Ltd 26
0
10
20
30
40
50
60
70
80
90
0 2 4 6 8 10 12
Averagetime(hours)
Number of changes per calibrated model
Average hours to test/model changes
LoadTestAvgHourPerChange ModellingAvgHoursPerChange
27. Send us your data
• Free trial of simple Dynatrace capacity models
• http://www.performance-assurance.com.au/send-us-your-data/
• http://www.performance-assurance.com.au/introduction-to-
automatic-model-building/
• Send us a sample Dyntrace session file and we’ll send you a link to a
demo capacity model
• Particularly interested in trending technologies and use cases, e.g.
Micro-services, Containers, Big Data, IoT, etc
• Free Personal Dynatrace license from: http://bit.ly/dtpersonal
16/03/2016 Performance Assurance Pty Ltd 29
