Software code review, i.e., the practice of having third-party team members critique changes to a software system, is a well-established best practice in both open source and proprietary software domains. Prior work has shown that the formal code inspections of the past tend to improve the quality of software delivered by students and small teams. However, the formal code inspection process mandates strict review criteria (e.g., in-person meetings and reviewer checklists) to ensure a base level of review quality, while the modern, lightweight code reviewing process does not. Although recent work explores the modern code review process qualitatively, little research quantitatively explores the relationship between properties of the modern code review process and software quality. Hence, in this paper, we study the relationship between software quality and: (1) code review coverage, i.e., the proportion of changes that have been code reviewed, and (2) code review participation, i.e., the degree of reviewer involvement in the code review process. Through a case study of the Qt, VTK, and ITK projects, we find that both code review coverage and participation share a significant link with software quality. Low code review coverage and participation are estimated to produce components with up to two and five additional post-release defects respectively. Our results empirically confirm the intuition that poorly reviewed code has a negative impact on software quality in large systems using modern reviewing tools.

1. The Impact of Code Review
Coverage and Participation
on Software Quality
Shane
McIntosh
Ahmed E.
Hassan
Bram
Adams
Yasutaka
Kamei
@shane_mcintosh
shanemcintosh@acm.org

2. Patch
+
+
-
-
Patch
+
+
-
-
2
The modern, tool-supported code review process
Gerrit
Shane
Bram
Yasu

3. Patch
+
+
-
-
Patch
+
+
-
-
2
The modern, tool-supported code review process
Gerrit
Shane
Bram
Yasu

4. Patch
+
+
-
-
Patch
+
+
-
-
2
The modern, tool-supported code review process
Gerrit
Request
review
Shane
Bram
Yasu

5. Patch
+
+
-
-
Patch
+
+
-
-
2
The modern, tool-supported code review process
Gerrit
Watch about NULL
values at line 3!
Shane
Bram
Yasu

6. Patch
+
+
-
-
Patch
+
+
-
-
2
The modern, tool-supported code review process
Gerrit
Your code sucks!
Watch about NULL
values at line 3!
Shane
Bram
Yasu

7. Patch
+
+
-
-
Patch
+
+
-
-
2
The modern, tool-supported code review process
Gerrit
Your code sucks!
Watch about NULL
values at line 3!
Shane
Bram
Yasu

8. The modern, tool-supported code review process
3
Gerrit
Shane
Bram
Yasu

9. The modern, tool-supported code review process
3
Gerrit
Shane
Bram
Yasu

10. The modern, tool-supported code review process
3
Upstream
Gerrit
Shane
Bram
Yasu

11. The modern, tool-supported code review process
3
Upstream
Gerrit
Shane
Bram
Yasu
Changes are automatically
integrated after review
criteria are satisfied

13. Traditional!
Code Inspections
Mandated reviewer
checklists and in-person
meetings

14. Traditional!
Code Inspections
Mandated reviewer
checklists and in-person
meetings
Measurably!
improves!
software!
quality

15. Traditional!
Code Inspections
Mandated reviewer
checklists and in-person
meetings
Modern!
Code Review
Lightweight, tool-supported
Measurably!
improves!
software!
quality

16. Traditional!
Code Inspections
Mandated reviewer
checklists and in-person
meetings
Modern!
Code Review
Lightweight, tool-supported
Measurably!
improves!
software!
quality
???

17. Do lax reviewing practices have an
impact on software quality?
Expert code!
reviewer

18. 6
Upstream
Development
Feature development
Defect repairing
Commit types
Reviews can be linked to commits in
version control systems

19. 6
Upstream
Development
Feature development
Defect repairing
Commit types
Reviews can be linked to commits in
version control systems

20. 7
commit b5b46f398c1a04ab443bf22ec08c5efbf72c161e
Author: Alan Alpert <alan.alpert@nokia.com>
Date: Thu Mar 1 20:09:52 2012 +1000
!
Add a hidden XandYAxis enum value
For Qt 5 XandYAxis is being renamed to XAndYAxis to more consistently
follow capitalization rules. Add an undocumented XandYAxis variable to
ease porting.
Change-Id: Id9e41dd5578373f5f557937da889a9326ff12e53
Reviewed-by: Martin Jones <martin.jones@jollamobile.com>
Gerrit
Upstream
Reviews can be linked to commits in
version control systems

21. 7
commit b5b46f398c1a04ab443bf22ec08c5efbf72c161e
Author: Alan Alpert <alan.alpert@nokia.com>
Date: Thu Mar 1 20:09:52 2012 +1000
!
Add a hidden XandYAxis enum value
For Qt 5 XandYAxis is being renamed to XAndYAxis to more consistently
follow capitalization rules. Add an undocumented XandYAxis variable to
ease porting.
Change-Id: Id9e41dd5578373f5f557937da889a9326ff12e53
Reviewed-by: Martin Jones <martin.jones@jollamobile.com>
Gerrit
Upstream
Reviews can be linked to commits in
version control systems

22. 7
commit b5b46f398c1a04ab443bf22ec08c5efbf72c161e
Author: Alan Alpert <alan.alpert@nokia.com>
Date: Thu Mar 1 20:09:52 2012 +1000
!
Add a hidden XandYAxis enum value
For Qt 5 XandYAxis is being renamed to XAndYAxis to more consistently
follow capitalization rules. Add an undocumented XandYAxis variable to
ease porting.
Change-Id: Id9e41dd5578373f5f557937da889a9326ff12e53
Reviewed-by: Martin Jones <martin.jones@jollamobile.com>
Gerrit
Upstream
Reviews can be linked to commits in
version control systems

23. 7
commit b5b46f398c1a04ab443bf22ec08c5efbf72c161e
Author: Alan Alpert <alan.alpert@nokia.com>
Date: Thu Mar 1 20:09:52 2012 +1000
!
Add a hidden XandYAxis enum value
For Qt 5 XandYAxis is being renamed to XAndYAxis to more consistently
follow capitalization rules. Add an undocumented XandYAxis variable to
ease porting.
Change-Id: Id9e41dd5578373f5f557937da889a9326ff12e53
Reviewed-by: Martin Jones <martin.jones@jollamobile.com>
Gerrit
Upstream
Reviews can be linked to commits in
version control systems

24. 8
The Baseline

25. 8
Product Metrics
The Baseline

26. 8
Product Metrics
e.g., SLOC, complexity
The Baseline

27. 8
Product Metrics
Process Metrics
e.g., SLOC, complexity
The Baseline

28. 8
Product Metrics
Process Metrics
e.g., SLOC, complexity
e.g., Prior defects, churn
The Baseline

29. 8
Product Metrics
Process Metrics
Human Factor Metrics
e.g., SLOC, complexity
e.g., Prior defects, churn
The Baseline

30. 8
Product Metrics
Process Metrics
Human Factor Metrics
e.g., SLOC, complexity
e.g., Prior defects, churn
e.g., # Minor contributors
The Baseline

31. 9
The Context

32. The Context
1,339 components
254 defect-prone
v5.0.0
218 components
24 defect-prone
v4.3.0
170 components
15 defect-prone
v5.10.0
1,337 components
187 defect-prone
v5.1.0
10

33. ParticipationCoverage
Do lax reviewing practices have an
impact on software quality?

34. ParticipationCoverage
Do lax reviewing practices have an
impact on software quality?

35. 13
Measuring review coverage using the
proportion of reviewed changes
13
Upstream
Development

36. 13
Measuring review coverage using the
proportion of reviewed changes
13
Upstream
Development
Gerrit

37. 13
Measuring review coverage using the
proportion of reviewed changes
13
Upstream
Development
Gerrit
7 out of 10 changes are!
linked to code reviews

38. 14
Concrete Predicted Defect Count
Defect
Model
Create artificial component setting explanatory
variables to median values
Artificial
Comp.
500SLOC
Explanatory!
Variable Value
Reviewed
changes
100%
Churn 150

39. 15
Defect
Model
Calculate number of predicted defects
Artificial
Comp.
500SLOC
Explanatory!
Variable Value
Reviewed
changes
Churn 150
Reviewed
changes
PredictedDefectCount
Concrete Predicted Defect Count
100%

40. 15
Defect
Model
Calculate number of predicted defects
Artificial
Comp.
500SLOC
Explanatory!
Variable Value
Reviewed
changes
Churn 150
Reviewed
changes
0
PredictedDefectCount
Concrete Predicted Defect Count
100%

41. 15
Defect
Model
Calculate number of predicted defects
Artificial
Comp.
500SLOC
Explanatory!
Variable Value
Reviewed
changes
Churn 150
Reviewed
changes
0
PredictedDefectCount
Concrete Predicted Defect Count
100%

42. 16
Defect
Model
Artificial
Comp.
500SLOC
Explanatory!
Variable Value
Reviewed
changes
90%
Churn 150
Modify variable under test while keeping other
explanatory variables constant
Reviewed
changes
PredictedDefectCount
Concrete Predicted Defect Count

43. 16
Defect
Model
Artificial
Comp.
500SLOC
Explanatory!
Variable Value
Reviewed
changes
90%
Churn 150
Modify variable under test while keeping other
explanatory variables constant
Reviewed
changes
PredictedDefectCount
Concrete Predicted Defect Count

44. 16
Defect
Model
Artificial
Comp.
500SLOC
Explanatory!
Variable Value
Reviewed
changes
90%
Churn 150
1
Modify variable under test while keeping other
explanatory variables constant
Reviewed
changes
PredictedDefectCount
Concrete Predicted Defect Count

45. 16
Defect
Model
Artificial
Comp.
500SLOC
Explanatory!
Variable Value
Reviewed
changes
90%
Churn 150
1
Modify variable under test while keeping other
explanatory variables constant
Reviewed
changes
PredictedDefectCount
Concrete Predicted Defect Count

46. ● ●
● ● ● ● ● ● ● ●
● ● ● ● ● ● ●0
1
2
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Proportion of Reviewed Changes
ConcretePredictedDefectCount
● Qt 5.0.0
VTK 5.10.0
17
Review coverage doesn’t tell the whole story

47. ● ●
● ● ● ● ● ● ● ●
● ● ● ● ● ● ●0
1
2
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Proportion of Reviewed Changes
ConcretePredictedDefectCount
● Qt 5.0.0
VTK 5.10.0
17
Review coverage doesn’t tell the whole story
Not a significant factor
for Qt 5.1.0 or ITK

48. ParticipationCoverage
Do lax reviewing practices have an
impact on software quality?

49. ParticipationCoverage
Do lax reviewing practices have an
impact on software quality?
!
Coverage is linked to
quality in two of the four
studied systems, yet it
doesn’t tell the whole
story

50. ParticipationCoverage
Do lax reviewing practices have an
impact on software quality?
!
Coverage is linked to
quality in two of the four
studied systems, yet it
doesn’t tell the whole
story

51. 20
Measuring review participation using
proportions of lax reviews
I❤️
Me
Self-approved
changes

52. 20
Measuring review participation using
proportions of lax reviews
I❤️
Me
Self-approved
changes
Hastily-reviewed
changes

53. 20
Measuring review participation using
proportions of lax reviews
I❤️
Me
Self-approved
changes
Hastily-reviewed
changes
Changes without
discussion

54. 21
Measuring review participation using
proportions of lax reviews
I❤️
Me
Self-approved
changes
Hastily-reviewed
changes
Changes without
discussion

55. ● ● ●
● ●
● ● ●
● ● ●
● ● ●
● ● ●
0
1
2
3
4
5
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Proportion of Self−Approved Changes
ConcretePredictedDefectCount
22
Lax participation leads to many
post-release defects in our models

56. ● ● ●
● ●
● ● ●
● ● ●
● ● ●
● ● ●
0
1
2
3
4
5
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Proportion of Self−Approved Changes
ConcretePredictedDefectCount
22
5 additional
post-release defects
Lax participation leads to many
post-release defects in our models

57. ● ● ●
● ●
● ● ●
● ● ●
● ● ●
● ● ●
0
1
2
3
4
5
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Proportion of Self−Approved Changes
ConcretePredictedDefectCount
22
5 additional
post-release defects
Small increase
leads to defects
Lax participation leads to many
post-release defects in our models

58. ParticipationCoverage
Do lax reviewing practices have an
impact on software quality?
!
Coverage is linked to
quality in two of the four
studied systems, yet it
doesn’t tell the whole
story

59. ParticipationCoverage
Do lax reviewing practices have an
impact on software quality?
!
Coverage is linked to
quality in two of the four
studied systems, yet it
doesn’t tell the whole
story
!
Components w/ lax
review participation are
estimated to have several
additional post-release
defects