Software Reliability Engineering

Software Reliability Engineering
DeFacto Model

Hans Sassenburg (SE-CURE AG)
Lucian Voinea (SolidSource BV)

SRE - DeFacto Model © SolidSource BV, SE-CURE AG, 2010. 1

Contents
1. Introduction
2. Basic Model
3. Measures and Metrics
4. Extended Model – DeFacto

Appendix: Tool Examples


1. Introduction
• Software reliability is the probability of failure-free
operation with respect to execution time and environment.
• Software reliability engineering (SRE) is the quantitative
study of the operational behavior of software-based
systems with respect to user requirements concerning
reliability.
• SRE is increasingly adopted by many companies as it
becomes an important economic consideration
(competitive advantage, dependencies, safety regulations),
others still regard reliability a cost rather than a value.
• Creditable software reliability techniques are still in urgent
need.
• Hardware failures are repeatable and more predictable
(wear-out) while software failures are largely unpredictable.

System-of-System Characteristics
• Continuous evolution and deployment
– There is an increasing need to integrate new capabilities into a system
while it is operating.
– New and different capabilities are deployed, and unused capabilities
will be dropped; the system evolves not in phases, but continuously.
• Heterogeneous, inconsistent, and changing elements
– A system is not constructed from uniform parts: there are always some
misfits, especially as the system is extended and repaired.
• Erosion of the people/system boundary
– People are not just users of a system; they are elements of the system,
affecting its overall emergent behavior.
• Normal failures
– Software and hardware failures are the norm rather than the
exception.


Errors, Defects and Failures
• Error
– Mistake made by a human action or tool during
programming.

• Defect (Fault)
– Manifestation of an error during execution.

• Failure
– Incorrect software behaviour in operational context due to
a defect.


Errors, Defects and Failures
Errors (actual number unknown)
Manifested defects
Manifested failures Defect and failure
rates drop after
releasing and
continue to grow
Testing
depending on
contexts of use
(operational profile),
and number of users
Unit/integration
Size test coverage
Complexity

System
test coverage Time

Pre-release Post-release


Post-release situation
Released errors

Theoretical Errors (actual number unknown)
maximum
Manifested defects (actual number unknown)
Manifested failures

Released defects
Feature
coverage
during
operation

Released manifested failures

Post-release


SRE Components

1. Error 2. Defect 3. Failure
prevention removal forecasting

• to avoid, by • to detect, by • to estimate, by
construction, verification statistical
error and validation, modeling, the
occurrences the existence presence of
of errors and defects and
remove them occurrence of
failures

feedback loops


2. Basic Model
• For software reliability prediction, you need a model of
the relationship between the predicted variable and
other measurable variables.
• Assumptions
– We can accurately measure some property of software or
process.
– A relationship exists between what we can measure and
what we want to know.
– This relationship is understood, has been validated, and
can be expressed in terms of a formula or model.
• Few metrics have been demonstrated to be predictable
or related to software reliability.


Problems with Existing Models
• Too many models, often overly complex
• Wrong assumptions
–Operation environment same as testing environment;
–Once a failure occurs, the fault causing it is removed;
–Fault removal will not introduce new faults;
–Pre-release defects can be used as predictor for post-
release failures;
–…


Simplified Approach
• Distinguish in the reliability model three phases
from the product lifecycle
Error introduction
• Pre-release (especially during design and implementation)
Defect identification
• Pre-release (especially during testing)
Failure manifestation
• Pre and post-release
• Determine for each phase the main factors that
influence reliability


Basic Model with Reliability Factors

Drivers Defect Operational profile
• Problem complexity identification • System Testing
• Design complexity • Contexts of use
• Module and
• Code complexity integration testing • Agility of demand
• Domain knowledge • Model verification • Number of users
• Project constraints

Error Failure
introduction Detection methods manifestation


3. Measures and Metrics
• Error introduction
Factor: Design and code complexity unnecessarily high.
Consequence: overlook achievable execution states during
programming = more errors.

• Defect identification
Factor: Code limitedly verified
Consequence: limited number of execution states investigated.

• Failure manifestation
Factor: Contexts of use not always considered/known.
Consequence: clients use the product in ways that reveal
unconsidered execution states.


Error Introduction
Factor: Design and code complexity unnecessarily high

Measures Metrics
Design complexity Fan-in
Fan-out

Code complexity Size
McCabe cyclomatic complexity
Code duplication
Change propagation
Halstead Volume


Defect Identification
Factor: Code limitedly verified

Measures Metrics
Module testing Code coverage during testing
completion

Integration testing Code and interface coverage during testing
completion


Failure Manifestation
Factor: Contexts of use not always considered/known

Measures Metrics
System test Number of verified requirements with
completion Boundary and Pairwise testing.

Usage context Number of verified requirements (with
Boundary and Pairwise testing) from the
requirements that are needed to
implement the functionality used 80% of
the time.


Feedback Loops
• It can be argued that software parts with many errors
being found are likely to have more errors remaining,
since the same factors that produced the error affect
the remainder of the code as well.
• This means that failure occurrences are not
independent, and in fact, will exhibit correlations that
depend on the operational profile as well as on the
system itself.
• It is therefore crucial discovering these correlations
across multi-dimensional data to improve the error
prevention and defect removal processes.


4. Extended Model - DeFacto
• Basic Model
– Distinguishes three phases with reliability factors.

• Extended Model - DeFacto
– Assign measures to those reliability factors that
are relevant and derive appropriate metrics;
– Customize model to specific contexts by
adding/removing factors and calibrating factor
weights via correlations discovered using historical
data.


DeFacto – Example Metrics

Defect
• Design complexity identification • System Testing
Fan-in, Fan-out Requirement
• Module + coverage with
Integration testing Boundary and
• Code complexity
Code coverage Pairwise testing
McCabe, Duplication

Error Failure
introduction manifestation


Appendix: SolidFX = Fact eXtractor for C/C++

• The Fact eXtractor (SolidFX) is a Error
standalone, non-intrusive solution for
static analysis of industry-size projects
introduction
written in the C and C++ programming measurement
languages.
• SolidFX uses proprietary technology to
analyze even the most complex C/C++
code bases efficiently and robustly.
• SourceFX offers predefined analysis
scenarios and metrics to measure
C/C++ code quality, maintainability,
modularity, detect potential bugs and Automatically
extract design from source code – all
for coding faster, cleaner, better.
compute
McCabe complexity,
Fan-in, Fan-out,…
www.solidsourceit.com


Appendix: SolidSDD = Software Duplication Detector
• The Source Code Duplication Detector
(SolidSDD) is a standalone application Error
for detecting and managing source introduction
code duplication (i.e., code clones) in
software. measurement
• It can be used to analyze large projects
and detect code that has been cloned
(e.g., via cut-n-paste operations) during
development.
• The currently supported programming
languages are C, C++, C# and Java. In
addition to identifying the code clone
fragments, SolidSDD offers an intuitive
graphical interface for assessing the
code duplication characteristics and the
location of the duplicated fragments in
the code stack. Automatically
• SolidSDD is able to cope with variations
in the duplicated code; not only exact compute
clones are detected but also duplicated
fragments that have been modified by code duplication
renaming variables or by
inserting/removing code.

Appendix: SolidSX = Software eXplorer
• The Software Explorer (SolidSX) is a
standalone Windows application that Model
gives insight in large software systems.
SolidSX creates high-quality calibration
visualizations that simultaneously show
the structure, dependencies, metrics
on all types of source code elements
(files, classes, methods, fields, etc.).
• By using hardware-accelerated
graphics, SolidSX is able to display large
amounts of information in a clear and
concise manner and provides fast and
easy exploration through large source
codes.
• SolidSX uses plug-ins to import
dependencies and metrics from Discover correlations
different sources.
across
multidimensional data


Appendix: SolidSTA = Software Trend Analyzer
• The Software Trend Analyzer (SolidSTA)
is a standalone, non-intrusive solution Error
for monitoring and investigating introduction
software trends.
measurement
• SolidSTA uses a number of proprietary
and standard metric analyses to assess Model calibration
the evolution of software quality
indicators for industry-size code
versioning repositories.
• The set of metrics can be extended
with custom analyses via a plug-in
system with an open API.
SolidSTA presents the analyses results
in an intuitive way to enable users to
discover trend correlations and make Compute change propagation
fact-based informed decisions.
• Overviews of team activity or system and
metrics can be produced in minutes.
• No repository management expertise is
analyze software
required. history/trends.


Software Benchmarking Organization

www.sw-benchmarking.org
• The Software Benchmarking Organization (SBO) was
founded in 2009 by SE-CURE AG and SolidSource BV.
• The objective is to provide benchmarking services to
the software industry.
• The portfolio includes benchmarking studies,
workshops, and supporting products.
• The portfolio is brought to the market through an
international network with accredited partners.

Contact Information
• Dr. Lucian Voinea
T +31 (40) 203 4290
M +31 (6) 1436 3842
E voinea@sw-benchmarking.org

• Dr. Hans Sassenburg:
T +41 (33) 733 4682
M +41 (79) 231 6600
E sassenburg@sw-benchmarking.org


Software Reliability Engineering

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (17)

Ähnlich wie Software Reliability Engineering

Ähnlich wie Software Reliability Engineering (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Software Reliability Engineering