Introduced Waseda Software Quality Benchmark (WSQB) based on SQuaRE as a result of IPA/RISE Research at ISO SC7/WG6 Plenary Meeting Kuantan

1. IPA RISE 2015-16
Waseda Software Quality Benchmark
Professor, Waseda University
Visiting Professor, National Institute of Informatics
Outside Director, SYSTEM INFORMATION
ISO/IEC/JTC1 SC7/WG20 Convenor
washizaki@waseda.jp
Hironori Washizaki
In corporation with Prof. Azuma and Mr. Komiyama from
ISO/IEC/JTC1/SC7/WG6, Mr. Tsuda and Dr. Honda from
Waseda U., and Computer Software Association Japan (CSAJ)

2. Executive Summary
2
• 21 package software and SaaS
• Waseda Software Quality
Benchmark (WSQB2017)
http://www.washi.cs.waseda.ac.jp/?page_id=3479
• ISO/IEC 25000 SQuaRE-
based concrete measurement
• Relationships among different
quality characteristics
0
20
40
60
80
100
機能適合性
性能効率性
互換性
使用性
信頼性
セキュリティ
保守性
移植性
0
5
10
1 2 3 4 5 6
12 vendors
21 products
IV&V org.
Delegate
Result
ISO/IEC JTC1/SC7/WG6
SQuaRE editorsCooperation
Eval. result

3. Quality Evaluation Framework
3
Q1. Any path going through internal
servers only?
Q2. Any path going through outside
servers?
Q3. Any P2P communications?
…
G. The events or actions cannot
be repudiated later through
communication channels (paths).
M. Signed communication path ratio
= #Signed_paths / #Total_paths
E.g. Non-repudiation
Waseda U. Team Vendor
1 Concretize SQuaRE
measurements by
GQM
2 Prepare measurement
methods: data forms,
static analysis,
questionnaire, user-
testing
3 Conduct code
static analysis,
user-testing
Fill data
forms,
questionnaire
4 Measure and
evaluate quality
Scores by using percentile
E.g., Top 30% = 0.7
HighLow
Histogram
Measured value

4. Internal/External Quality
• 66 measurements (i.e., metrics)
• Measurement coverage: 34% of 66×21 products
4
#Measurements defined
4
10
2
9
13
6 8
5
0
1
0
0
3
0
4
1
機能適合性 性能効率性 互換性 使用性 信頼性 セキュリティ 保守性 移植性
測定値あり 測定値なしMeasured
NOT
measured
Functional
suitability
Performance
efficiency
Compatibility Usability Reliability Security Maintainability Portability

5. Quality in Use
User questionnaire
• Standard questionnaire
– E.g., Are you satisfied … ?
• 3 products
User testing
• 10 products
5
Waseda U.
Team
Vendor
1 Declare
functions
2 Extract core
functions
3 Define normal
test cases
4 Define
exceptional
test cases
5 Conduct test
6 Measure quality
Coverage 24%
Measured
NOT
measured

6. Result: Quality Characteristics
6
Functional
suitability
Performance
efficiency
Compatibility Usability
Reliability Security Maintainability Portability
Effectiveness Efficiency Satisfaction Freedom from
risk
Context
coverage

7. Relationships among characteristics
7
Internal/External Quality Quality in Use
p-value < 0.1
性能効率性 互換性 使用性 信頼性 セキュリティ 保守性 移植性 有効性 効率性 満足性 リスク回避性 利用状況網羅性
機能適合性 0. 31 0. 19 -0. 72 0. 37 -0. 05 0. 50 0. 31 -0. 14 0. 52 1. 00 1. 00 1. 00
性能効率性 0. 44 0. 24 0. 36 -0. 17 0. 37 0. 32 0. 32 -0. 10 -0. 50 -0. 50 -0. 50
互換性 0. 04 0. 17 -0. 06 0. 36 -0. 04 -0. 14 0. 05 -0. 50 -0. 50 -0. 50
使用性 0. 17 -0. 21 0. 11 0. 44 -0. 09 -0. 20 -1. 00 -1. 00 -1. 00
信頼性 0. 30 0. 41 0. 45 -0. 08 0. 11 1. 00 1. 00 1. 00
セキュリティ -0. 06 0. 19 0. 64 -0. 34 0. 50 0. 50 0. 50
保守性 0. 26 -0. 29 0. 01 1. 00 1. 00 1. 00
移植性 -0. 21 0. 67 0. 50 0. 50 0. 50
有効性 0. 03 -1. 00 -1. 00 -1. 00
効率性 1. 00 1. 00 1. 00
満足性 1. 00 1. 00
リスク回避性 1. 00
Func.
Perf.
Comp.
Usa.
Relia.
Sec.
Main.
Port.
Effe.
Effic.
Sati.
Free.
Perf. Comp. Usa. Relia. Sec. Main. Port. Effe. Effic. Sati. Free. Cont.

8. Analysis by Contexts
8
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
機能
適…
性能
効…
互換
性
使用
性
信頼
性
セキュ
リティ
保守
性
移植
性
有効
性
効率
性
満足
性
リスク
回…
利用
状…
パッケージ
(n=17)
クラウド(n=4)
区別なし
(n=21)
Package
Cloud
ALL
Functional
suitability
Performance
efficiency
Compatibility
Usability
Reliability
Security
MaintainabilityPortability
Effectiveness
Efficiency
Satisfaction
Freedom from
risk
Context
coverage

9. Recommendations
9
Waseda Software Quality Benchmark
http://www.washi.cs.waseda.ac.jp/?page_id=3479
Industry
ISO/IEC
JTC1/SC7/WG6
Continuous measurement
and PSQ-Certification
Improve
SQuaRE
4. Incorporate concrete measurements into SQuaRE
5. Address properties specific to Agile and Cloud
1. Low security and compatibility in some products.
Necessary to address these in IoT era.
2. Negative correlation between usability and functionality.
Need to adopt user-centered development.
3. Limited data and goals. Necessary to measure/benchmark
Improve quality
management

10. Publications
 K. Honda, H. Washizaki and Y. Fukazawa, “Generalized Software Reliability Model
Considering Uncertainty and Dynamics: Model and Applications,” International
Journal of Software Engineering and Knowledge Engineering (IJSEKE), pp.1-29,
2017.
 K. Honda, H. Washizaki, Y. Fukazawa, et al., “Detection of Unexpected Situations
by Applying Software Reliability Growth Models to Test Phases,” 26th IEEE
International Symposium on Software Reliability Engineering (ISSRE 2015),
Industry Track
 H. Nakai, N. Tsuda, K. Honda, H. Washizaki, and Y. Fukazawa, “Initial Framework
for a Software Quality Evaluation based on ISO/IEC 25022 and ISO/IEC
25023,” Poster, The 2016 IEEE International Conference on Software Quality,
Reliability & Security (QRS 2016)
 K. Honda, N. Nakamura, H. Washizaki and Y. Fukazawa, “An Industrial Case Study
of Project Management Using Cross Project Software Reliability Growth Model,”
Poster, The 2016 IEEE International Conference on Software Quality, Reliability &
Security (QRS 2016)
 K. Honda, N. Nakamura, H. Washizaki and Y. Fukazawa, “Case Study: Project
Management Using Cross Project Software Reliability Growth Model,” IEEE
International Workshop on Trustworthy Computing, 2016
 K. Honda, N. Nakamura, H. Washizaki and Y. Fukazawa, “Case Study: Project
Management Using Cross Project Software Reliability Growth Model Considering
System Scale,” 27th IEEE International Symposium on Software Reliability
Engineering (ISSRE 2016), Industry Track,
 H. Nakai, N. Tsuda, K. Honda, H. Washizaki, Y. Fukazawa, “Evaluating Software
Product Quality based on SQuaRE Series,” IEEE TENCON 2016
10