1. Source Code Quality Evaluation:
The SQALE method
SQALE: Software Quality Assessment based on Lifecycle Expectations
December 2011
Author: Jean-Louis Letouzey

2. Agenda
The needs for a Source Code evaluation method
Issues with current aggregation rules
The SQALE method structure
The SQALE Quality Model
The SQALE Analysis Model
The SQALE indices and indicators
SQALE in practice
2

3. The SQALE Method: Summary
Software Quality Assessment based on Lifecycle Expectations
Has been developed by experts, independent of any tool vendor
Focus on the diagnostic objectivity (precision, no false-positive)
Easy to understand, to implement and to deploy
Avoids practices that damage measurement results
Aggregation with averages that generates compensation effects
Notation on a delimited interval that generates threshold effect
Promotes simple principles
Source code quality is a non functional requirement that should be
specified, then verified
Evaluating quality is measuring the remaining needed workload to fix
all non-conformities
Is a robust method for identifying and managing the Technical
Debt

4. inspearit and source code evaluation
inspearit is not a tool vendor, inspearit is an independant
company
inspearit thinks that the method come first, then the tools
to support it
inspearit assist large accounts to implement source code
analysis with SQALE
Our customers reported us issues with current methods and
tools
Difficulties to understand the meaning and usage of indicators
Too much false positive
No support for remediation decision (what are the priorities?)
5

5. The needs for a source code
evaluation method
Ability to objectively evaluate and monitor software
development products in order to anticipate issues
Aligned with best measurement practices
Ability to compare
Source code versions
Different products with different usage/history
Development teams or subcontractors performance
Capacity to provide useful inputs to an improvement plan
Capacity to support decisions: Ex. Two teams working on two
similar projects
The first one, delivered 3 weeks in advance but with 100
coding practice issues
The second, delivered 1 week in advance but with 15 coding
practice issues
Which project is the most efficient and effective?
6

6. High level requirements for an
evaluation method
Quantified, Objective, Precise, Sensitive
Implementable by automated static
analysis tools
Reproducible by the implementation of
one tool to another (produce the same
findings based on non ambigous
definitions, rules…)
Provide guidance for tailoring this
standard model to:
Any language
Different severity levels (business critical,
life critical etc.)
The SQALE method has been developped as a solution to all these requirements
7

7. Agenda
The needs for a Source Code evaluation method
Issues with current aggregation rules
The SQALE method structure
The SQALE Quality Model
The SQALE Analysis Model
The SQALE indices and indicators
SQALE in practice
8

8. Summary of the challenge for
an evaluation method
Two hierarchies:
Challenge / Need:
AH QH Provide a quality related
“measure” or “score” for each
A Hierarchy A Hierarchy of Quality couple {A,Q} of the 2 hierarchies
of Artifacts char. and sub-char.
= Quality Model
Portfolio Quality
Q
Domain Maintainability Characteristic
A
AppliA
Sub-characteristic
Component Measure/rule
File
Get a measure, a score which characterizes, represents the
evaluated concept
9

9. Measurement basics:
The representation condition
“The condition that, if one software entity is less than another
entity in terms of a selected attribute, then any software measure
for that attribute must associate a smaller number to the first
entity than it does to the second entity” [1]
Real World Mathematical World
Measurement function
Attribute Mes(Aa) Impact on :
- Measure/rule choice
Aa - Normalization functions
Mes(Ab)
Artifacts
- Aggregation rules
Ab
Mes(Ac)
A representative
Ac measure keep the
relationship
established in the Mes(Aa) > Mes(Ab)
Aa >> Ab >> Ac real world Mes(Ab) > Mes(Ac)
[1] N.E. Fenton and S. L. Pfleeger, Software Metrics: A rigourous & Practical Approach,
10
second edition, ISBN 053495425-1,PWS Publishing Company, Boston, 1997

10. The representation condition
applied to aggregation
?
Aggregation The aggregation should represent
rule the basic findings
Issues should be reported up to the
highest level of the hierarchy
Issues:
AgScore_a AgScore_b The aggregate score should report
the improvement
Version b >> Version a
AgScore_b > AgScore_a
Version a Improvement Version b
We have identified some effects that violate this condition
11

11. The masking effect
The masking effect appears when the aggregate value is not
sensitive to the variation of one of the base values
Example:
n
ia
1
2
3
4
5
ed
ax
le
le
le
le
le
in
Fi
Fi
Fi
Fi
Fi
M
M
M
MyAppli Va A A C E E A E C
MyAppli Vb A A C D E A E C
An improvement from Va to Vb on file 4 does not impact the
aggregate score
Aggregations by Min, Max and Median violate the representation condition
12

12. The compensation effect
The compensation effect appears on aggregation functions
such as: mean, weighted mean, median
Example:
Comment ratio (target for the project : >30% per file)
ge
10
ra
1
2
3
4
5
6
7
8
9
ve
le
le
le
le
le
le
le
le
le
le
Fi
Fi
Fi
Fi
Fi
Fi
Fi
Fi
Fi
Fi
A
MyAppli Va 5% 34% 48% 47% 31% 37% 33% 35% 4% 39% 31%
While 2 files do have “maintainability” issue, the average is OK
In real life, lack of comment in files 1 & 9 won’t be compensated
by abundance of comments in file 3 & 4
Aggregations by average (weighted or not) violate the representation condition
That ‘s one reason why most Quality Dashboards are not precise
13

13. The type of scale and
allowed aggregations
The measurement theory is precise about allowed aggregation
[1]
Scale Valid Transformation Main Valid Agregation
Nominal 1 to 1 mapping None
Ordinal Monotonic increasing Min, Max, Median
function
More Choice
Interval M' = aM + B (a>0) Min, Max, Median,
Average
Ratio M' = aM (a>0) Min, Max, Median, Mean,
Average, Sum, Distance
(Euclidian or other)
Absolute M' = M All
Due to the representation condition, some combinations (scale type,
aggregation) should be rejected
[1] N.E. Fenton and S. L. Pfleeger, Software Metrics: A rigourous & Practical Approach,
14
second edition, ISBN 053495425-1,PWS Publishing Company, Boston, 1997

14. Synthesis of our analysis
Synthesis of allowed operations and aggregation issues depending
on scale type
Average, Weighted
Scale Min, Max, median Sum, Distance
average
Not allowed Not allowed Not allowed
Nominal
Potentially not Not allowed Not allowed
Ordinal
representative
Potentially not Potentially not Not allowed
Interval
representative representative
Potentially not Potentially not Representative
Ratio
representative representative
Potentially not Potentially not Representative
Absolute
representative representative
Within SQALE, we choose to normalize all measures on a ratio scale and
to aggregate the normalized values by summation
15

15. Agenda
The needs for a Source Code evaluation method
Issues with current aggregation rules
The SQALE method structure
The SQALE Quality Model
The SQALE Analysis Model
The SQALE indices and indicators
SQALE in practice
16

16. The SQALE method: Structure
Implementation/Tools
Tailoring
4 concepts
8 Fundamental Principles
Measurement theory and representativity
17

17. The SQALE 8 Fundamental Principles
1. The quality of the source code is a non-functional requirement
2. The requirements in relation to the quality of the source code have to be
formalised according to the same quality criteria such as any other functional
requirement
3. Assessing the quality of a source code is in essence assessing the distance
between its state and its expected quality objective
4. The SQALE Method assesses the distance to the conformity with the
requirements by considering the necessary remediation cost of bringing the
source code to conformity
5. The SQALE Method respects the representation condition
6. The SQALE Method uses addition for aggregating the remediation costs and for
calculating its quality indicators
7. The SQALE Method’s Quality Model is orthogonal
8. The SQALE Method’s Quality Model takes the software’s lifecycle into account
18

18. The 4 main concepts of
the SQALE method
Static analysis tools
Source
Code
1 Quality Model 2 Analysis Model 3 Indices 4 Indicators
Testabilité Fiabilité Evolutivité Efficacité Maintenabilité
Maintenabilité 589
Efficacité 248 248
SQI
Remediation
Evolutivité 1 480 1 480 1 480
List of source
functions
Fiabilité 548 548 548 548
code related Findings Remediation Testabilité 6 535 6 535 6 535 6 535 6 535
Aggregation STI
costs table
6 535 7 083 8 563 8 811 9 400
requirements Table rules SRI
…
SQID
…
19

19. Agenda
The needs for a Source Code evaluation method
Issues with current aggregation rules
The SQALE method structure
The SQALE Quality Model
The SQALE Analysis Model
The SQALE indices and indicators
SQALE in practice
20

20. Back to the fundamentals of Quality
You buy a new car
• How will you feel, if the delivered car has only 5 cylinders and 290 hp?
21

21. Back to the fundamentals of Quality
In 1979, Philip Crosby in his famous book “Quality is free”
established the 4 principles of Quality:
the definition of quality is conformance to requirements
the system of quality is prevention
the performance standard is zero defects
the measurement of quality is the price of nonconformance
Since that time, the vision and definition of quality has been
extended to a much wider scope including customer satisfaction
But anyway, quality is still at least “Conformance to requirements”
22

22. The SQALE Quality Model: source code
requirements
An organized set of expectations (requirements) Requirement
based on lifecycle needs Sub-characteristic
Understandability
Reuse Readability
Reusability
Ram related efficiency
Portability
Rom related efficiency
Maintain
Maintainability
Characteristic CPU related efficiency
Archi. related changeability
Deliver Reusability
Security …
Logic related changeability
Efficiency
…
Portability
Data related changeability
no public data
Evolve
Changeability Maintainability Fault tolerance Number of derived class <=10
Reliability …
Security Architecture related reliability
Test
Logic related reliability
Testability Efficiency
Instruction related reliability
Code Changeability
Data related reliability
Reliability
Integration Testing testability
Testability Unit Testing testability
Requirements, appear only once within the Quality Model,
when they are first needed. They are checked with relevant static analysis tools
23

23. Requirements: type of issues
It is important to use a SQALE Quality Model that covers all
the types of code issues
Useless code,
un-optimized code
Presentation, structurness
Excessive coupling,
Maintainability Hard coded data
Efficiency
Changeability Potential logic errors,
exception management,
Reliability
test coverage
Testability
Copy and Paste, internal
structure of methods
When deployed, the SQALE Quality Model contains
from 30 to 100+ requirements tailored to the organization context
24

24. Agenda
The needs for a Source Code evaluation method
Issues with current aggregation rules
The SQALE method structure
The SQALE Quality Model
The SQALE Analysis Model
The SQALE indices and indicators
SQALE in practice
25

25. The SQALE Analysis Model
Quality Indexes represent the remediation effort needed to
refactor artifacts in order to comply with the Quality Model
Part level indexes are aggregated by adding all file indexes Part level indexes
Σ
Sub
characteristic
indexes
Remediation
4.1
Analysis
function
Tool
Non conformity table Remediation cost table Σ
“Understandability index” for the selected file 4.1
“Characteristic indexes” are aggregated by adding “Subcharacteristic indexes”
26

26. The SQALE Analysis Model
For a given couple {A,Q}, SQALE provides a simple rule to
Quality
calculate the associated score
Q
Maintainability Characteristic
Sub-characteristic
Portfolio Measure/rule
Domain Req_1 Req_2 Req_3 Req_4 Req_5 Req_6 … … … … Req_99
File_1
File_2
A AppliA
File_3
File_4
… Σ
…
Component …
…
…
File
File_99999
Remediation costs table
The positions into the 2 hierarchies define the perimeter of remediation costs to be added
27

27. The SQALE Analysis Model:
remediation factors
How findings are transformed into costs?
• SQALE use « Remediation Functions » that are associated to types of
Non Conformity. The standard SQALE Analysis Model contains 5 types
which correspond to different « remediation lifecycles »
• These Types and values are proposed by default. I it is recommended
to extend /taylor them at Organization/Project/Application level
Estimated cost for fixing one Non Conformity of Type4 is: 5 Work Units
28

28. Agenda
The needs for a Source Code evaluation method
Issues with current aggregation rules
The SQALE method structure
The SQALE Quality Model
The SQALE Analysis Model
The SQALE indices and indicators
SQALE in practice
29

29. 3° concept: SQALE indices
The SQALE characteristic indices:
Artifact remediation workload
- SQALE Testability Index: STI for all requirements
- SQALE Reliability Index: SRI associated to testability
- SQALE Changeability Index: SCI
- SQALE Security Index: SSI
- SQALE Efficiency Index: SEI
- SQALE Maintainability Index: SMI
- SQALE Portability Index: SPI
- SQALE Reusability Index: SRuI
The global SQALE Quality Index: SQI
- This is the Technical Debt
SQALE index densities: SQID, STID etc
Example: Density by KLOC
(1,000 lines of code)
30

30. 4° concept: The 3 SQALE indicators
SQALE Indices
SQI
STI
SQALE Kiviat
SRI
…
SQID Rating Grid
…
Based on the ratio (in %) Testabilité Fiabilité Evolutivité Efficacité Maintenabilité
Remediation cost / Development cost
Maintenabilité 589
Efficacité 248 248
Example of Testability rating Evolutivité 1 480 1 480 1 480
Remediation cost (STI): 4.36 hours Fiabilité 548 548 548 548
Development cost: 250 hours Testabilité 6 535 6 535 6 535 6 535 6 535
Ratio: 1.7 % Rating: “C” 6 535 7 083 8 563 8 811 9 400
31

31. The SQALE Pyramid:
A two points of view indicator
Interpreting the SQALE indices An external view that represents
the percieved quality evaluated by
consolidation of the hierarchy of
characteristics
An analytic view provided by
orthogonal characteristics
One understands impact of Maintainability
each Non Conformity and Σ
Efficiency
improvement on quality
characteristic and life cycle Changeability
Σ
issues.
Reliability
Σ
Σ
Testability
© Copyright inspearit 2007-20011
32

32. The perceived benefits
As SQALE requires to specify the quality of the code, the quality
measure is objective
The SQALE quality index represents a workload, a cost. It is the
concrete “Technical debt” of the project
It is easy to monitor simultaneously:
the remaining workload associated to functionalities
the debt associated to code quality
and update accordingly the project‘s planning
Technical debt may be aggregated at any portfolio granularity
Technical debt density allows to compare versions, applications,
subcontractors…
As SQALE does not violate the representation condition, SQALE thus
provides usable meanings to source code measurements
33

33. Agenda
The needs for a Source Code evaluation method
Issues with current aggregation rules
The SQALE method structure
The SQALE Quality Model
The SQALE Analysis Model
The SQALE indices and indicators
SQALE in practice
34

34. Using SQALE: Tools
PRIVATE,
METRIXWARE:
SQALE index
distribution analysis
35

35. Using SQALE: Tools
SQUORE: Dashboard
36

36. Using SQALE: Tools
SONAR: Monitoring the source code quality
(Technical Debt)
37

37. Using SQALE: Tools
SONAR: Tailored Dashboard
38

38. Using SQALE: Tools
SONAR: Portfolio Management
39

39. Using SQALE: Deployment
0 Initialization 1 Tailoring 2 Implementation 3 Deployment
Method training
Planification Choose and Tool Deployment,
Development of Tailored
Stake holders Implementation of training coaching
SQALE models
Perimeter … the solution Monitor and improve
Specify the tool solution
Identify the On day training for Choose the solution to Perform awareness
sponsor and stake the team: The SQALE be implemented within session
holders, define the Method the organization
roadmap Coach and support the
Develop a tailored Implementation of the users
Define the most SQALE Quality Model tailored models within
usefull use cases the selected solution Monitor the solution and
of source code Develop a tailored define an improvement
analysis: SQALE Analysis Model Implementation of the plan (identification and
selected indicators and implementation of new
Build the « Source Validation of both reports within the requirements)
code analysis » models trough a pilot selected solution
Project team
project
Update process,
Specify the tool associated deliverables
solution including and training to prepare
recommanded deployment
indicators
Support Training, Support Seminars
Workshop preparation Training
and animation Coaching
Support
40

40. The SQALE Discovery Kit
Discover the fundamentals principles and benefits of the SQALE method with
the“SQALE Discovery Kit”. This package contains:
A one day training session on the SQALE Method
The identification (through dedicated interviews) of your main “use cases” of
source code analysis within your organisation’s context
The development (through dedicated Workshops) of your own quality and
analysis models
These models will be tailored to your environment and will be the basis for defining
and evaluating the quality of your source code (for one of the following language:
Java, C, C++, Cobol)
The concrete assessment of one of your application using the SQALE method
and your tailored quality and analysis models including a detailed assessment
report
Workshop on how to interpret and use the results
At the end you will get:
Your tailored models for one of your development language
An evaluation report
Direction for actions
41
Total duration: about 20 days

41. Thanks
To know more about SQALE: http: /www.sqale.org
To do more with SQALE, contact inspearit
just sqale it
© inspearit - Author - Place
42 Saturday, December 17, 2011