Systematic decision support for architectural design decisions is a major concern for software architects of evolving service-oriented systems. In practice, architects often analyse the expected performance and reliability of design alternatives based on prototypes or former experience. Model-driven prediction methods claim to uncover the tradeoffs between different alternatives quantitatively while being more cost-effective and less error-prone. However, they often suffer from weak tool support and focus on single quality attributes. Furthermore, there is limited evidence on their effectiveness based on documented industrial case studies. Thus, we have applied a novel, model-driven prediction method called Q-ImPrESS on a large-scale process control system consisting of several million lines of code from the automation domain to evaluate its evolution scenarios. This presentation reports our experiences with the method and lessons learned. Benefits of Q-ImPrESS are the good architectural decision support and comprehensive tool framework, while one drawback is the time-consuming data collection.

1. © ABB Group
January 29, 2015 | Slide 1
Q-ImPrESS
Quality Impact Prediction
for Evolving Service-oriented Software
Heiko Koziolek, DECRC/I1 Industrial Software Technologies, 2010-11-26

2. © ABB Group
January 29, 2015 | Slide 2
Agenda
Context &
Motivation
ABB
Demonstrator
Applying
Q-ImPrESS
Lessons
Learned

3. EU Project Q-ImPrESS
Context
© ABB Group
January 29, 2015 | Slide 3
First
Discussions
01/2008
Demonstrators
finished
12/201004/2009
Tools available
for testing
12/2008
Requirements
finished
12/2009
First Iteration
Demonstrators
Consortium members
Q-ImPrESS tools
Ericsson demonstrator ABB demonstrator

4. Industrial automation: Process Control Systems
© ABB Group
January 29, 2015 | Slide 4

5. Motivation
Software development at ABB
 Continuous evolution of ABB software systems
 New requirements, technologies, failure reports
Software maintenance and evolution
are a large cost factor for ABB software development
 Current practice
 Design decisions for evolutions based on experience
 Prototyping for new technologies, performance impacts
 Limited systematic tradeoff analyses
Apply model-based prediction methods
for systematic tradeoff analyses
to save costs and achieve higher quality?
© ABB Group
January 29, 2015 | Slide 5

6. Q-ImPrESS Method
Overview
Eclipse
Modeling
Framework
Q-ImPrESS Workbench
Backbone
Layered Queueing Networks
or Palladio/SimuCOM
Discrete Time
Markov Chains
KAMP cost
calculator
QVT Model
Transformations

7. ABB Demonstrator: Process control system
Topology
© ABB Group
January 29, 2015 | Slide 7
Plant / Office Network
Network
Isolation
Device
Remote
Workplaces
Firewall
Internet
Remote
Workplaces
Redundant Network
Workplaces
Controllers
Servers
Fieldbus
Remote I/O and
Field devices
Focus on the
server-side
software

8. Reverse Engineering in Q-ImPrESS
Process/Method
© ABB Group
January 29, 2015 | Slide 8
SoMoX3
Q-ImPrESS
Model
C++
Source
1
Structural Investigation of Software Systems
3
Software Model eXtractor
2
Generalized Abstract Syntax Tree
SISSy1
GAST2

9. Reverse Engineering in Q-ImPrESS
Sample Result for 50KLOC subsystem
© ABB Group
January 29, 2015 | Slide 9

10.  SISSy parser not robust against Microsoft C++ code
 Configuring of the metrics intransparent
 Reference decomposition essential
 Limited scalability, unclear if faster than manual modelling
 Static analysis of limited use for QoS prediction
Reverse Engineering
Results from ABB code
Lines
of
Code
Time for
parsing,
clustering
Percentage
of reconstructed
higher level
components
Qualitative
evaluation
(--, -, o, +, ++)
ABB Component 1 10 0.5h / 1h 100% +
ABB Component 2 250 4h / 16h 0% --
ABB Component 3 3300 >60h / ? n/a --
© ABB Group
January 29, 2015 | Slide 10

11. Manual Modeling
Process/Method
Analyzed
existing
architectural
documentation
Created test-
bed (2 quad-
cores, emulated
controllers)
Installed and
configured the
system (>100h)
Identified 4
critical usage
scenarios
Configured load
drivers with
typical
workloads
Decided for
abstr. level
(processes =
components)
Recorded
component
transitions
Wrote script to
derive transition
probabilities
Discussed
validity with
architects and
developers
© ABB Group
January 29, 2015 | Slide 11

12. Manual Modeling
Q-ImPrESS model of the ABB process control system
© ABB Group
January 29, 2015 | Slide 12

13. Performance Prediction
Process
Applied Service
Demand Law
(S = X/U)
Measured X, U
for each
process
(PerfMon)
Determined 20
CPU and HD
resource
demands
Parametrized
resource
demands
(linear regr.)
Abstracted
memory,
network,
virtualization
Integrated
demands into
Q-ImPrESS
model
Performed
predictions
(SimuCom,
LQN)
© ABB Group
January 29, 2015 | Slide 13

14. Performance Prediction
Sample predictions for different allocation scenarios
© ABB Group
January 29, 2015 | Slide 14
Heiko Koziolek. Performance Prediction for an Industrial Control System.
Submitted to ACM/SPEC International Conference on Performance Engineering (ICPE‘11)

15.  Achieved prediction error below 30 percent
 Data collection consumed more time than expected
 Support for service-oriented systems limited
 Easy to analyse different evolution scenarios
 Many bottlenecks below the architectural level
Performance Prediction
Results
Workload PerfMon
Measured
SimuCom
Prediction
Error (%) LQNS
Prediction
Error (%)
30 17.146 12.467 27.288 12.464 27.305
60 26.681 22.366 16.174 22.343 16.260
90 31.902 32.347 1.395 32.322 1.317
120 39.016 42.432 8.754 42.329 8.490
150 51.929 51.943 0.027 51.760 0.326
© ABB Group
January 29, 2015 | Slide 15

16. Reliability Prediction
Data Collection
Acquired
access to bug
tracking system
Selected
growth model
from IEEE
1633-2008
Performed
curve fitting on
bug report
dates (CASRE)
Determined
failure rates per
subsystem
Validated failure
rates against
code metrics
Annotated
components in
Q-ImPrESS
model
Performed
reliability
predictions
(PRISM)
© ABB Group
January 29, 2015 | Slide 16

17. Reliability Prediction
Sample sensitivity analysis
© ABB Group
January 29, 2015 | Slide 17
Heiko Koziolek, Bastian Schlich, and Carlos Bilich.
A Large-Scale Industrial Case Study on Architecture-based Software Reliability Analysis.
Proc. 21st IEEE International Symposium on Software Reliability Engineering (ISSRE'10)

18. Reliability Prediction
Results
© ABB Group
January 29, 2015 | Slide 18
 Possible use of results for future test budget distribution
 Abstraction level too high: no hardware, no concurrency
 Data collection for lower abstraction levels difficult
 Many statistical assumptions (e.g., Markovian control flow)
 Validation inherently difficult: would take years to get data
 Difficult to establish trust into the results
 Data collection method not applicable for new systems
 Limited step-by-step guidance and best practice known
 Limited amount of case studies in literature

19. Tradeoff Analysis
Design space exploration with PerOpteryx
© ABB Group
January 29, 2015 | Slide 19
 Manually specified degrees of freedom in the model
(processor speeds, component allocations, COTS)
 PerOpteryx applied an evolutionary algorithm (NSGA II)
to search the space spanned by the degrees of freedom
 Determines optimal performance/reliability/costs tradeoffs
 Could be applied for future sizing of the system
Anne Martens, Heiko Koziolek, Steffen Becker, and Ralf Reussner.
Automatically improve software architecture models for performance, reliability, and cost using evolutionary algorithms.
Proc. ACM/SPEC International Conference on Performance Engineering (ICPE'10)

20.  Analytical Hierachy Process (from decision support)
 Requires manual pairwise comparison of candidates
 Applied on 4 alternatives, specified 24 weights
 Alternative 3 best because of the preference for reliability
 Method not scalable: e.g., 180 comparisons for 10 alt.
Tradeoff Analysis
Weighing Alternatives with AHP Wizard
© ABB Group
January 29, 2015 | Slide 20
Leo Hatvani, Anton Jansen, Cristina Seceleanu and Paul Pettersson.
Integrated Tool for Trade-off Analysis of Quality-of-Service Attributes
Proc. 2nd Int. Workshop on the Quality of Service-oriented Software Systems (QUAOSS’10)

21. Lessons Learned
Experience with Q-ImPrESS method and tools
 Integrated approach with comprehensive tool support
 Limited integration with other tools and methods
 Cost-effective only for large systems
 Limited expressiveness of the Q-ImPrESS meta model
 Missing data collection support
 Complexity of evolution scenarios determines accuracy
 Iterative modelling and strong goal-orientation required
© ABB Group
January 29, 2015 | Slide 21

22. Lessons Learned
Cost estimations in person hours
 Cost estimation (in person hours) based on our experience
 Best case: input data or test-bed readily available
 Worst case: unknown system, new data collection method
 We actually needed more effort than the worst case!
© ABB Group
January 29, 2015 | Slide 22

23. Conclusions
 Q-ImPrESS reverse engineering
 Parser not robust for MS C++,
clustering intransparent
 Q-ImPrESS modeling tools
 Well-integrated, graphical editors
 Q-ImPrESS performance prediction
 Accurate results possible
 Q-ImPrESS reliability prediciton
 Many assumptions, still primitive models
 More research for efficient data
collection needed
 Internal ABB follow-up planned
© ABB Group
January 29, 2015 | Slide 23

24. © ABB Group
January 29, 2015 | Slide 24