METRICON 2010, Germany

1. Master thesis presentation
Impact of design complexity
on software quality
Student: Nguyen Duc Anh
First supervisor: Marcus Ciolkowski, Fraunhofer IESE
Second supervisor: Sebastian Barney, BTH
General supervisor: Prof. Dr. Dr. h.c. Dieter Rombach
Feb 21, 2013 1
© Fraunhofer IESE

2. Agenda
 Motivation
 Problem statement
 Research methodology
 Research result
 Threat of validity
 Conclusion
 Future work
Feb 21, 2013 2
© Fraunhofer IESE

3. Motivation
High complexity leads to high cost and low quality
Feb 21, 2013 3
© Fraunhofer IESE

4. Problem statement
 SQ1: Which cost & quality attributes are predicted
using design complexity metrics?
W hat is the  SQ2: What (kind of ) design complexity metrics are
impact most frequently used in literature?
of
 SQ3: Which complexity metrics are potential
design
predictors of quality attribute?
complexity
on  SQ4: Is there an overall influence of these metrics
software on quality attributes? If yes, what are the impacts of
cost &quality those metrics on those attributes?
?  SQ5: If no, what explains the inconsistency between
studies? Is this explanation consistent across
different metrics?
Feb 21, 2013 4
© Fraunhofer IESE

5. Research methodology
W hat is the Search for relevant publications
impact Extract information about design complexity
of metrics & quality attributes
design
complexity Extract numerical representation of impact
on relationship & context factors
software
Synthesize data & interpret results
cost &quality
?
Feb 21, 2013 5
© Fraunhofer IESE

6. Study selection result
Search range: 1960 to 2010
Scope: Object oriented metrics
Feb 21, 2013 6
© Fraunhofer IESE

7. Research result
SQ1 - Which quality attributes are predicted using software design metrics?
Probability of a
module to be faulty
Effort to maintain a
software module
Number of fault per
LOC
Probability of a
module to be
changed
Cost (effort) is excluded due to lack of sufficient number of investigated studies
Feb 21, 2013 7
© Fraunhofer IESE

8. Research result
SQ2- What kind of complexity metrics is most frequently used in literature?
Design complexity dimension
No of studies
Feb 21, 2013 8
© Fraunhofer IESE

9. Research result
SQ2- What complexity metrics is most frequently used in literature?
Design complexity metric: Chidamber & Kemerer (CK) metric set (*)
Fault proneness Maintainability
No of No of
Metric Type Metric Type
studies studies
NOC (Number Of Children) inheritance 28 WMC scale 9
DIT (Depth of Inheritance Tree) inheritance 27 RFC coupling 8
CBO (Coupling Between Object) coupling 22 DIT coupling 7
LCOM (Lack of Cohesion cohesion 22 NOC inheritance 6
between Method) CBO coupling 4
WMC (Weighted Method Count) scale 22 LCOM cohesion 3
RFC (Response For a Class) coupling 21 … … 3
… … 12
Feb 21, 2013 9
( )S . C id m e a dC . K m rr“AM t s u efr be t
* .R h a b r n .F e ee, er S it o O jc
ic
© Fraunhofer IESE O ie tdD s n IEEE Trans. Softw. Eng., v l2 ,
r ne eig ,” o. 0
1 9 , p . 4 64 3
94 p 7- 9

10. Research result
SQ3 - Which complexity metrics are potential predictors of fault proneness?
 Potential prediction – Statistical correlation analysis
 Correlation coefficient
 Spearman
 Odds ratios (estimated from univariate logistic regression model)
 Significant correlation
 Vote counting
 Count the number of reported significant impacts over total
number of studies
Feb 21, 2013 10
© Fraunhofer IESE

11. Research result
SQ3 - Which complexity metrics are potential predictors of fault proneness?
(Ex a m p le : Vo te c o unting fo r Sp e a rm a n c o rre la tio n c o e ffic ie nt in Fa ult p ro ne ne s s s tud ie s )
Out comes ≤ 50% 
≥ 50% no
No of Proportiona Positive
Metric No of non l ratio of + impact !
positive impact
studies No of + No of - impact?
significant
NOC 19 6 1 1 2 3%
2 No
DIT 14 2 0 1 2 1%
4 No
CBO 17 10 0 7 5%
9 Ys
e
LCOM 14 6 0 8 4%
3 No Except NOC,
W MC 26 18 0 8 6%
9 Ys
e DIT, LCOM listed
RFC 15 9 0 6 6%
0 Ys
e
metrics are
W MC McCabe 16 11 0 5 6%
9 Ys
e
SDMC 6 6 0 0 10
0% Ys
e potential
AMC 6 6 0 0 10
0% Ys
e predictor of fault
NIM 6 6 0 0 10
0% Ys
e proneness !
NCM 6 6 0 0 10
0% Ys
e
NTM 6 6 0 0 10
0% Ys
e
Feb 21, 2013 11
© Fraunhofer IESE

12. Research result
SQ3 - Which complexity metrics are potential predictors of fault proneness?
 Strength of correlation (*)
Trivial Small Medium Large
 Meta analysis
 Synthesize reported correlation coefficients
 Assess the agreement among studies about aggregated result
Feb 21, 2013 12
(*) J. Cohen, Statistical Power Analysis for the Behavioral Science,
© Fraunhofer IESE Lawrence Erlbaum Hillsdale, New Jersey, 1988.

13. Research result
SQ4 - Is there an overall influence of these metrics on fault proneness?
95% confidence interval
of aggregated correlation
coefficient between the
metric and fault
proneness
Trivial Small Medium Large
 Scale, coupling metrics are stronger correlated than cohesion,
inheritance metric
 LOC is strongest correlated to fault proneness
Feb 21, 2013 13
© Fraunhofer IESE

14. Research result
SQ4 - Is there an overall influence of these metrics on fault proneness?
(Ex a m p le : M ta a na ly s is fo r Sp e a rm a n c o e ffic ie nt o f m e tric RFC in Fa ult p ro ne ne s s s tud ie s )
e
Forest plot of RFC
Aggregated results
Global Spearman 0.31
coefficient
95% Confidence [0.22;0.40]
Interval
P-value 0.000
Feb 21, 2013 14
© Fraunhofer IESE

15. Research result
SQ4 - Is there an overall influence of these metrics on fault proneness?
(Ex a m p le : M ta a na ly s is fo r Sp e a rm a n c o e ffic ie nt o f m e tric RFC in Fa ult p ro ne ne s s s tud ie s )
e
Is this result consistent across studies?
Metric I2
I2 test for heterogeneity !
CO
B 9%
5
DT
I 8%
3
N C
O 7%
5
LO
CM 7%
4
RC
F 7%
8
RFC: I2=78% W C
M 9%
3
LC
O 8%
4
Feb 21, 2013 15
© Fraunhofer IESE

16. Research result
SQ4* - How many cases is enough to draw the statistically significant conclusion?
(Ex a m p le : Po we r a na ly s is fo r Sp e a rm a n c o e ffic ie nt o f m e tric RFC in Fa ult p ro ne ne s s s tud ie s )
α value 0.1
Tails 2
Expected effect size 0.31
Expected power 80%
Number of cases needed: 60 cases !
Feb 21, 2013 16
© Fraunhofer IESE

17. Research result
SQ5: What explains the inconsistency between studies? Is this
explanation consistent across different metrics?
 Moderator variable
 Programming Language: C++ & Java
 Project type: Open source, Closed source academic & Closed source
industry
 Defect collection phase: Pre release defects & Post release defects
 Business domain: Embedded system & Information system
 Dataset size: Small, Medium & Large
 Are the correlations different across each moderator variable?
Feb 21, 2013 17
© Fraunhofer IESE

18. Research result
SQ5: What explains the inconsistency between studies? Is this
explanation consistent across different metrics?
Metric
Programming Project type Defect col. Business Dataset size Variance
Language Phase Domain explanation
CO
B 6% 4% 83% 4 % 8% in percent
DT
I 3% 0% 2%
0 0 % 1%
N C
O 3%
4 2%4 1%
5 2%2 1%
4
LO
CM 1% 0% 60% 0 % 6%
RC
F 5% 3% 78% 3 % 2%
W C
M 3%
2 4% 60% 4 % 3%
LC
O 7% 2% 51% 1%5 0%
 Remaining inconsistency is still excessive
 No consistent explanation for heterogeneity across metrics !
Feb 21, 2013 18
© Fraunhofer IESE

19. Comparison of results with perception in literature
Vote counting & meta analysis common claims in literature
Common claims in literature In Lit. Ours
The more classes a given class is
coupled, the more likely that class is Yes Yes
faulty
The more methods that can potentially
be executed in response to a message Yes Yes
received by an object of a given class,
the more likely inheritance tree for a
The deeper the that class is faulty
given class is, the more likely that class Yes No
is faulty
The more immediate sub-classes a given
class has, the more likely that class is No No
faulty
The less similar methods within a given
class, the more likely that class is Yes No
faulty
The more local methods a given class Yes Yes
has, the more likely that class is faulty
The 2013
Feb 21, larger size a given class has, the Yes Yes
19
more likely that class is faulty
Do the effects of CK metrics differ Yes No
© Fraunhofer IESE
across different programming languages
Do the effects of CK metrics differ

20. Limitation
 Internal validity
 Selection of publications
 Quality of selected studies.
 External validity
 Limitation to models with single complexity metric
 Limitation to object oriented systems
 Conclusion validity
 Lack of comparable studies
 Lack of reported context information
Feb 21, 2013 20
© Fraunhofer IESE

21. Conclusion
 SQ1: Most common predicted attributes:
 Fault proneness & Maintainability
 SQ2: Most common design complexity dimension & metric:
 Coupling: CBO, RFC
 Scale: WMC
 Inheritance: DIT, NOC
 Cohesion: LCOM
 SQ3,4: Overall impact of design complexity on software quality:
 Moderate impact of WMC, CBO, RFC on fault proneness
 LOC shows strongest impact on fault proneness !
 SQ5: What explains the inconsistency between studies?
 Not able to explain for the inconsistency
 Defect collection phase explains part of the inconsistency
21
© Fraunhofer IESE

22. Interpretation
 Look for quality predictor in source code: LOC
 Look for quality predictor in design: CBO, RFC and WMC
 Build different prediction models for pre release and post
release defect
 Need context information to increase predictive performance
 Adapt the design metrics for any software systems
Feb 21, 2013 22
© Fraunhofer IESE

23. Future work
Construction of a generic
Quality benchmarking
model prediction
System A System B
CBO XXX CBO XXX
RFC XXX ? RFC XXX
WMC XXX WMC XXX
LCOM XXX LCOM XXX
DIT XXX DIT XXX
Feb 21, 2013 23
© Fraunhofer IESE

24. Q&A
Feb 21, 2013 24
© Fraunhofer IESE