The use of automatic static analysis has been a software engineering best practice for decades. However, we still do not know a lot about its use in real-world software projects: How prevalent is the use of Automated Static Analysis Tools (ASATs) such as FindBugs and JSHint? How do developers use these tools, and how does their use evolve over time? We research these questions in two studies on nine different ASATs for Java, JavaScript, Ruby, and Python with a population of 122 and 168,214 open-source projects. To compare warnings across the ASATs, we introduce the General Defect Classification (GDC) and provide a grounded-theory-derived mapping of 1,825 ASAT-specific warnings to 16 top-level GDC classes. Our results show that ASAT use is widespread, but not ubiquitous, and that projects typically do not enforce a strict policy on ASAT use. Most ASAT configurations deviate slightly from the default, but hardly any introduce new custom analyses. Only a very small set of default ASAT analyses is widely changed. Finally, most ASAT configurations, once introduced, never change. If they do, the changes are small and have a tendency to occur within one day of the configuration’s initial introduction.

1. Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software

2. Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software
Moritz Beller
@Inventitech

3. Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software
Moritz Beller
@Inventitech
Radjino Bholanath, Andy Zaidman

4. Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software
Moritz Beller
@Inventitech
Radjino Bholanath, Andy Zaidman
Shane McIntosh

5. Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png

6. Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png

7. Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png

8. Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png

9. Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png

10. Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png

11. RQ1: How Prevalent Are ASATs?
Image: http://www.valueinvestasia.com/wp-content/uploads/2015/03/odd-one-out.jpg

12. 122 popular OSS projects

13. RQ1: How Prevalent Are ASATs?

14. RQ1: How Prevalent Are ASATs?
122

15. RQ1: How Prevalent Are ASATs?
122

16. RQ1: How Prevalent Are ASATs?
122

17. RQ1: How Prevalent Are ASATs?
122
122

18. RQ1: How Prevalent Are ASATs?
122
122

19. RQ1: How Prevalent Are ASATs?
122
36
122

20. RQ1: How Prevalent Are ASATs?
122
36
122

21. RQ1: How Prevalent Are ASATs?
122
36
122

22. RQ1: How Prevalent Are ASATs?

23. Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
RQ1: How Prevalent Are ASATs?

24. Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
RQ1: How Prevalent Are ASATs?

25. Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
RQ1: How Prevalent Are ASATs?

26. Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
RQ1: How Prevalent Are ASATs?

27. Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
RQ1: How Prevalent Are ASATs?

28. Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
1) (Automated) investigation of repository information is an
approximation of real ASAT use
RQ1: How Prevalent Are ASATs?

29. Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
1) (Automated) investigation of repository information is an
approximation of real ASAT use
2) We cannot infer how a project uses ASATs from a repository
analysis alone
RQ1: How Prevalent Are ASATs?

30. RQ2: How are ASATs configured?

31. RQ2: How are ASATs configured?

32. RQ2: How are ASATs configured?
checkstyle.xml

33. RQ2: How are ASATs configured?
filename
checkstyle.xml

34. RQ2: How are ASATs configured?
filename
checkstyle.xml

35. RQ2: How are ASATs configured?
filename
checkstyle.xml
parse rules

36. RQ2: How are ASATs configured?
filename
checkstyle.xml
enable
parse rules

37. RQ2: How are ASATs configured?
filename
checkstyle.xml
enable
re-configure
parse rules

38. RQ2: How are ASATs configured?
filename
checkstyle.xml
enable
re-configure
parse rules
custom

39. General Defect Classification (GDC)

40. General Defect Classification (GDC)

41. General Defect Classification (GDC)
RQ1:
9

42. General Defect Classification (GDC)
RQ1:
9
Checkstyle, FindBugs, PMD, ESLint, JSCS, JSHint, JSL, PYLint, RuboCop

43. General Defect Classification (GDC)
1,825
RQ1:
9
Checkstyle, FindBugs, PMD, ESLint, JSCS, JSHint, JSL, PYLint, RuboCop

44. RQ2: How are ASATs configured?

45. RQ2: How are ASATs configured?

46. RQ2: How are ASATs configured?

47. 168,425
RQ2: How are ASATs configured?

48. 168,425
RQ2: How are ASATs configured?

49. 168,425
RQ2: How are ASATs configured?

50. 168,425
RQ2: How are ASATs configured?

51. RQ2.1: How Popular Are Certain ASATs?
Tool Language Configuration Files
Checkstyle Java 18,785
FindBugs Java 2,090
PMD Java 7,458
ESLint JavaScript 4,435
JSCS JavaScript 11,677
JSHint JavaScript 108,770
JSL JavaScript 862
Pylint Python 4,071
RuboCop Ruby 10,066
Total - 168,405

52. RQ2.1: How Popular Are Certain ASATs?
Tool Language Configuration Files
Checkstyle Java 18,785
FindBugs Java 2,090
PMD Java 7,458
ESLint JavaScript 4,435
JSCS JavaScript 11,677
JSHint JavaScript 108,770
JSL JavaScript 862
Pylint Python 4,071
RuboCop Ruby 10,066
Total - 168,405

53. RQ 2.2: Which Rules Do Developers Enable?

54. RQ 2.2: Which Rules Do Developers Enable?

55. 65%
RQ 2.2: Which Rules Do Developers Enable?

56. 35% 65%
RQ 2.2: Which Rules Do Developers Enable?

57. 35% 65%
ASATs perform poorly at finding
functional defects. Wagner et al.
RQ 2.2: Which Rules Do Developers Enable?

58. RQ 2.2: Which Rules Do Developers Enable?
0%
00%
00%
00%
00%
00%
00%
00%
00%
00%
00%
Checkstyle ESLint FindBugs JSCS JSHint JSL PMD Pylint RuboCop
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%

59. 0%
00%
00%
00%
00%
00%
00%
00%
00%
00%
00%
Checkstyle ESLint FindBugs JSCS JSHint JSL PMD Pylint RuboCop
RQ 2.2: Which Rules Do Developers Enable?
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%

60. RQ 2.2: Which Rules Do Developers Enable?
We: This is great!
Image: Phil Skipper, The Potentialist,http://i2.wp.com/thepotentiality.com/assets/media/Kurt-Enthusiasm.jpg

61. RQ 2.2: Which Rules Do Developers Enable?
We: This is great!
Image: Phil Skipper, The Potentialist,http://i2.wp.com/thepotentiality.com/assets/media/Kurt-Enthusiasm.jpg

62. RQ 2.2: Which Rules Do Developers Enable?
ASAT Developers*: Don't care.
Image: NiBiS, http://bidab.nibis.de/PICT/traurig.jpg

63. RQ 2.2: Which Rules Do Developers Enable?
ASAT Developers*: Don't care.
Image: NiBiS, http://bidab.nibis.de/PICT/traurig.jpg

64. RQ 2.3: How Good Is The Default?

65. RQ 2.3: How Good Is The Default?
Most ASAT configurations
deviate from the default.

66. RQ 2.3: How Good Is The Default?
Most ASAT configurations
deviate from the default.

67. RQ 2.3: How Good Is The Default?
Most ASAT configurations
deviate from the default.

68. RQ 2.3: How Good Is The Default?
But, typically only have one
change from the default …
Most ASAT configurations
deviate from the default.

69. RQ 2.3: How Good Is The Default?
But, typically only have one change from the
default …
●
- Addition
Most ASAT configurations
deviate from the default.

70. RQ 2.3: How Good Is The Default?
But, typically only have one change from the default …
●
- Addition
●
- Deletion
Most ASAT configurations
deviate from the default.

71. RQ 2.3: How Good Is The Default?
But, typically only have one change from the default …
●
- Addition
●
- Deletion
●
- Re-configuration/Custom analysis
Most ASAT configurations
deviate from the default.

72. RQ2: Open Questions

73. ●
Why do projects favor certain GDC rule categories from
ASATs?
RQ2: Open Questions

74. ●
Why do projects favor certain GDC rule categories from
ASATs?
●
Can ASAT developers better fit their default configurations to
their users' needs?
RQ2: Open Questions

75. ●
Why do projects favor certain GDC rule categories from ASATs?
●
Can ASAT developers better fit their default configurations to
their users' needs?
●
Do 'dynamic' languages require more ASAT use?
RQ2: Open Questions

76. ●
Why do projects favor certain GDC rule categories from ASATs?
●
Can ASAT developers better fit their default configurations to
their users' needs?
●
Do 'dynamic' languages require more ASAT use?
RQ2: Open Questions

77. RQ3: How Do ASAT Configurations Evolve?
Image: Daimler AG, http://5komma6.mercedes-benz-passion.com/wp-content/uploads/2013/06/s-class-lineup.jpg

78. RQ 3.1: How Often Do Changes Occur?

79. RQ 3.1: How Often Do Changes Occur?

80. RQ 3.1: How Often Do Changes Occur?
>80%
“never”

81. RQ 3.2: When Do Changes Occur?

82. RQ 3.2: When Do Changes Occur?

83. RQ 3.2: When Do Changes Occur?
<20%
of files

84. RQ 3.3: How Big Are The Changes?

85. RQ 3: Open Questions

86. ●
Why do ASAT configurations not typically evolve?
RQ 3: Open Questions

87. ●
Why do ASAT configurations not typically evolve?
●
How are ASATs used in a CI-environment?
RQ 3: Open Questions

88. Moritz Beller
@Inventitech
Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software

89. Moritz Beller
@Inventitech
Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software

90. Moritz Beller
@Inventitech
Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software

91. Moritz Beller
@Inventitech
Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software