Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software

Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software
Moritz Beller
@Inventitech

Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software
Moritz Beller
@Inventitech
Radjino Bholanath, Andy Zaidman

Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software
Moritz Beller
@Inventitech
Radjino Bholanath, Andy Zaidman
Shane McIntosh

Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png

Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png

Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png

Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png

Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png

Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png

RQ1: How Prevalent Are ASATs?
Image: http://www.valueinvestasia.com/wp-content/uploads/2015/03/odd-one-out.jpg

122 popular OSS projects

RQ1: How Prevalent Are ASATs?

RQ1: How Prevalent Are ASATs?
122

RQ1: How Prevalent Are ASATs?
122

RQ1: How Prevalent Are ASATs?
122

RQ1: How Prevalent Are ASATs?
122
122

RQ1: How Prevalent Are ASATs?
122
122

RQ1: How Prevalent Are ASATs?
122
36
122

RQ1: How Prevalent Are ASATs?
122
36
122

RQ1: How Prevalent Are ASATs?
122
36
122

RQ1: How Prevalent Are ASATs?

Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
RQ1: How Prevalent Are ASATs?

Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
RQ1: How Prevalent Are ASATs?

Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
RQ1: How Prevalent Are ASATs?

Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
RQ1: How Prevalent Are ASATs?

Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
RQ1: How Prevalent Are ASATs?

Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
1) (Automated) investigation of repository information is an
approximation of real ASAT use
RQ1: How Prevalent Are ASATs?

Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
1) (Automated) investigation of repository information is an
approximation of real ASAT use
2) We cannot infer how a project uses ASATs from a repository
analysis alone
RQ1: How Prevalent Are ASATs?

RQ2: How are ASATs configured?

RQ2: How are ASATs configured?

RQ2: How are ASATs configured?
checkstyle.xml

RQ2: How are ASATs configured?
filename
checkstyle.xml

RQ2: How are ASATs configured?
filename
checkstyle.xml

RQ2: How are ASATs configured?
filename
checkstyle.xml
parse rules

RQ2: How are ASATs configured?
filename
checkstyle.xml
enable
parse rules

RQ2: How are ASATs configured?
filename
checkstyle.xml
enable
re-configure
parse rules

RQ2: How are ASATs configured?
filename
checkstyle.xml
enable
re-configure
parse rules
custom

General Defect Classification (GDC)

General Defect Classification (GDC)

General Defect Classification (GDC)
RQ1:
9

General Defect Classification (GDC)
RQ1:
9
Checkstyle, FindBugs, PMD, ESLint, JSCS, JSHint, JSL, PYLint, RuboCop

General Defect Classification (GDC)
1,825
RQ1:
9
Checkstyle, FindBugs, PMD, ESLint, JSCS, JSHint, JSL, PYLint, RuboCop

RQ2: How are ASATs configured?

RQ2: How are ASATs configured?

RQ2: How are ASATs configured?

168,425
RQ2: How are ASATs configured?

168,425
RQ2: How are ASATs configured?

168,425
RQ2: How are ASATs configured?

168,425
RQ2: How are ASATs configured?

RQ2.1: How Popular Are Certain ASATs?
Tool Language Configuration Files
Checkstyle Java 18,785
FindBugs Java 2,090
PMD Java 7,458
ESLint JavaScript 4,435
JSCS JavaScript 11,677
JSHint JavaScript 108,770
JSL JavaScript 862
Pylint Python 4,071
RuboCop Ruby 10,066
Total - 168,405

RQ2.1: How Popular Are Certain ASATs?
Tool Language Configuration Files
Checkstyle Java 18,785
FindBugs Java 2,090
PMD Java 7,458
ESLint JavaScript 4,435
JSCS JavaScript 11,677
JSHint JavaScript 108,770
JSL JavaScript 862
Pylint Python 4,071
RuboCop Ruby 10,066
Total - 168,405

RQ 2.2: Which Rules Do Developers Enable?

RQ 2.2: Which Rules Do Developers Enable?

65%
RQ 2.2: Which Rules Do Developers Enable?

35% 65%
RQ 2.2: Which Rules Do Developers Enable?

35% 65%
ASATs perform poorly at finding
functional defects. Wagner et al.
RQ 2.2: Which Rules Do Developers Enable?

RQ 2.2: Which Rules Do Developers Enable?
0%
00%
00%
00%
00%
00%
00%
00%
00%
00%
00%
Checkstyle ESLint FindBugs JSCS JSHint JSL PMD Pylint RuboCop
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%

0%
00%
00%
00%
00%
00%
00%
00%
00%
00%
00%
Checkstyle ESLint FindBugs JSCS JSHint JSL PMD Pylint RuboCop
RQ 2.2: Which Rules Do Developers Enable?
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%

RQ 2.2: Which Rules Do Developers Enable?
We: This is great!
Image: Phil Skipper, The Potentialist,http://i2.wp.com/thepotentiality.com/assets/media/Kurt-Enthusiasm.jpg

RQ 2.2: Which Rules Do Developers Enable?
We: This is great!
Image: Phil Skipper, The Potentialist,http://i2.wp.com/thepotentiality.com/assets/media/Kurt-Enthusiasm.jpg

RQ 2.2: Which Rules Do Developers Enable?
ASAT Developers*: Don't care.
Image: NiBiS, http://bidab.nibis.de/PICT/traurig.jpg

RQ 2.2: Which Rules Do Developers Enable?
ASAT Developers*: Don't care.
Image: NiBiS, http://bidab.nibis.de/PICT/traurig.jpg

RQ 2.3: How Good Is The Default?

RQ 2.3: How Good Is The Default?
Most ASAT configurations
deviate from the default.

RQ 2.3: How Good Is The Default?
Most ASAT configurations
deviate from the default.

RQ 2.3: How Good Is The Default?
Most ASAT configurations
deviate from the default.

RQ 2.3: How Good Is The Default?
But, typically only have one
change from the default …
Most ASAT configurations
deviate from the default.

RQ 2.3: How Good Is The Default?
But, typically only have one change from the
default …
●
- Addition
Most ASAT configurations
deviate from the default.

RQ 2.3: How Good Is The Default?
But, typically only have one change from the default …
●
- Addition
●
- Deletion
Most ASAT configurations
deviate from the default.

RQ 2.3: How Good Is The Default?
But, typically only have one change from the default …
●
- Addition
●
- Deletion
●
- Re-configuration/Custom analysis
Most ASAT configurations
deviate from the default.

RQ2: Open Questions

●
Why do projects favor certain GDC rule categories from
ASATs?
RQ2: Open Questions

●
Why do projects favor certain GDC rule categories from
ASATs?
●
Can ASAT developers better fit their default configurations to
their users' needs?
RQ2: Open Questions

●
Why do projects favor certain GDC rule categories from ASATs?
●
Can ASAT developers better fit their default configurations to
their users' needs?
●
Do 'dynamic' languages require more ASAT use?
RQ2: Open Questions

●
Why do projects favor certain GDC rule categories from ASATs?
●
Can ASAT developers better fit their default configurations to
their users' needs?
●
Do 'dynamic' languages require more ASAT use?
RQ2: Open Questions

RQ3: How Do ASAT Configurations Evolve?
Image: Daimler AG, http://5komma6.mercedes-benz-passion.com/wp-content/uploads/2013/06/s-class-lineup.jpg

RQ 3.1: How Often Do Changes Occur?

RQ 3.1: How Often Do Changes Occur?

RQ 3.1: How Often Do Changes Occur?
>80%
“never”

RQ 3.2: When Do Changes Occur?

RQ 3.2: When Do Changes Occur?

RQ 3.2: When Do Changes Occur?
<20%
of files

RQ 3.3: How Big Are The Changes?

RQ 3: Open Questions

●
Why do ASAT configurations not typically evolve?
RQ 3: Open Questions

●
Why do ASAT configurations not typically evolve?
●
How are ASATs used in a CI-environment?
RQ 3: Open Questions

Moritz Beller
@Inventitech
Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software

Moritz Beller
@Inventitech
Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software

Moritz Beller
@Inventitech
Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software

Moritz Beller
@Inventitech
Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software