Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software
Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software
Moritz Beller
@Inventitech
Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software
Moritz Beller
@Inventitech
Radjin...
Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software
Moritz Beller
@Inventitech
Radjin...
Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_7...
Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_7...
Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_7...
Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_7...
Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_7...
Automatic Static Analysis Tools (ASATs)
Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_7...
RQ1: How Prevalent Are ASATs?
Image: http://www.valueinvestasia.com/wp-content/uploads/2015/03/odd-one-out.jpg
122 popular OSS projects
RQ1: How Prevalent Are ASATs?
RQ1: How Prevalent Are ASATs?
122
RQ1: How Prevalent Are ASATs?
122
RQ1: How Prevalent Are ASATs?
122
RQ1: How Prevalent Are ASATs?
122
122
RQ1: How Prevalent Are ASATs?
122
122
RQ1: How Prevalent Are ASATs?
122
36
122
RQ1: How Prevalent Are ASATs?
122
36
122
RQ1: How Prevalent Are ASATs?
122
36
122
RQ1: How Prevalent Are ASATs?
Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
RQ1: How Prevalent Are ASA...
Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
RQ1: How Prevalent Are ASA...
Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
RQ1: How Prevalent Are ASA...
Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
RQ1: How Prevalent Are ASA...
Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
RQ1: How Prevalent Are ASA...
Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
1) (Automated) investigati...
Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT
122 59% 23% -
36 77% 36% 36%
1) (Automated) investigati...
RQ2: How are ASATs configured?
RQ2: How are ASATs configured?
RQ2: How are ASATs configured?
checkstyle.xml
RQ2: How are ASATs configured?
filename
checkstyle.xml
RQ2: How are ASATs configured?
filename
checkstyle.xml
RQ2: How are ASATs configured?
filename
checkstyle.xml
parse rules
RQ2: How are ASATs configured?
filename
checkstyle.xml
enable
parse rules
RQ2: How are ASATs configured?
filename
checkstyle.xml
enable
re-configure
parse rules
RQ2: How are ASATs configured?
filename
checkstyle.xml
enable
re-configure
parse rules
custom
General Defect Classification (GDC)
General Defect Classification (GDC)
General Defect Classification (GDC)
RQ1:
9
General Defect Classification (GDC)
RQ1:
9
Checkstyle, FindBugs, PMD, ESLint, JSCS, JSHint, JSL, PYLint, RuboCop
General Defect Classification (GDC)
1,825
RQ1:
9
Checkstyle, FindBugs, PMD, ESLint, JSCS, JSHint, JSL, PYLint, RuboCop
RQ2: How are ASATs configured?
RQ2: How are ASATs configured?
RQ2: How are ASATs configured?
168,425
RQ2: How are ASATs configured?
168,425
RQ2: How are ASATs configured?
168,425
RQ2: How are ASATs configured?
168,425
RQ2: How are ASATs configured?
RQ2.1: How Popular Are Certain ASATs?
Tool Language Configuration Files
Checkstyle Java 18,785
FindBugs Java 2,090
PMD Jav...
RQ2.1: How Popular Are Certain ASATs?
Tool Language Configuration Files
Checkstyle Java 18,785
FindBugs Java 2,090
PMD Jav...
RQ 2.2: Which Rules Do Developers Enable?
RQ 2.2: Which Rules Do Developers Enable?
65%
RQ 2.2: Which Rules Do Developers Enable?
35% 65%
RQ 2.2: Which Rules Do Developers Enable?
35% 65%
ASATs perform poorly at finding
functional defects. Wagner et al.
RQ 2.2: Which Rules Do Developers Enable?
RQ 2.2: Which Rules Do Developers Enable?
0%
00%
00%
00%
00%
00%
00%
00%
00%
00%
00%
Checkstyle ESLint FindBugs JSCS JSHin...
0%
00%
00%
00%
00%
00%
00%
00%
00%
00%
00%
Checkstyle ESLint FindBugs JSCS JSHint JSL PMD Pylint RuboCop
RQ 2.2: Which Rul...
RQ 2.2: Which Rules Do Developers Enable?
We: This is great!
Image: Phil Skipper, The Potentialist,http://i2.wp.com/thepot...
RQ 2.2: Which Rules Do Developers Enable?
We: This is great!
Image: Phil Skipper, The Potentialist,http://i2.wp.com/thepot...
RQ 2.2: Which Rules Do Developers Enable?
ASAT Developers*: Don't care.
Image: NiBiS, http://bidab.nibis.de/PICT/traurig.j...
RQ 2.2: Which Rules Do Developers Enable?
ASAT Developers*: Don't care.
Image: NiBiS, http://bidab.nibis.de/PICT/traurig.j...
RQ 2.3: How Good Is The Default?
RQ 2.3: How Good Is The Default?
Most ASAT configurations
deviate from the default.
RQ 2.3: How Good Is The Default?
Most ASAT configurations
deviate from the default.
RQ 2.3: How Good Is The Default?
Most ASAT configurations
deviate from the default.
RQ 2.3: How Good Is The Default?
But, typically only have one
change from the default …
Most ASAT configurations
deviate f...
RQ 2.3: How Good Is The Default?
But, typically only have one change from the
default …
●
- Addition
Most ASAT configurati...
RQ 2.3: How Good Is The Default?
But, typically only have one change from the default …
●
- Addition
●
- Deletion
Most ASA...
RQ 2.3: How Good Is The Default?
But, typically only have one change from the default …
●
- Addition
●
- Deletion
●
- Re-c...
RQ2: Open Questions
●
Why do projects favor certain GDC rule categories from
ASATs?
RQ2: Open Questions
●
Why do projects favor certain GDC rule categories from
ASATs?
●
Can ASAT developers better fit their default configurati...
●
Why do projects favor certain GDC rule categories from ASATs?
●
Can ASAT developers better fit their default configurati...
●
Why do projects favor certain GDC rule categories from ASATs?
●
Can ASAT developers better fit their default configurati...
RQ3: How Do ASAT Configurations Evolve?
Image: Daimler AG, http://5komma6.mercedes-benz-passion.com/wp-content/uploads/201...
RQ 3.1: How Often Do Changes Occur?
RQ 3.1: How Often Do Changes Occur?
RQ 3.1: How Often Do Changes Occur?
>80%
“never”
RQ 3.2: When Do Changes Occur?
RQ 3.2: When Do Changes Occur?
RQ 3.2: When Do Changes Occur?
<20%
of files
RQ 3.3: How Big Are The Changes?
RQ 3: Open Questions
●
Why do ASAT configurations not typically evolve?
RQ 3: Open Questions
●
Why do ASAT configurations not typically evolve?
●
How are ASATs used in a CI-environment?
RQ 3: Open Questions
Moritz Beller
@Inventitech
Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software
Moritz Beller
@Inventitech
Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software
Moritz Beller
@Inventitech
Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software
Moritz Beller
@Inventitech
Analyzing the State of Static Analysis:
A Large-Scale Evaluation
in Open Source Software
Nächste SlideShare
Wird geladen in …5
×

Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software

763 Aufrufe

Veröffentlicht am

The use of automatic static analysis has been a software engineering best practice for decades. However, we still do not know a lot about its use in real-world software projects: How prevalent is the use of Automated Static Analysis Tools (ASATs) such as FindBugs and JSHint? How do developers use these tools, and how does their use evolve over time? We research these questions in two studies on nine different ASATs for Java, JavaScript, Ruby, and Python with a population of 122 and 168,214 open-source projects. To compare warnings across the ASATs, we introduce the General Defect Classification (GDC) and provide a grounded-theory-derived mapping of 1,825 ASAT-specific warnings to 16 top-level GDC classes. Our results show that ASAT use is widespread, but not ubiquitous, and that projects typically do not enforce a strict policy on ASAT use. Most ASAT configurations deviate slightly from the default, but hardly any introduce new custom analyses. Only a very small set of default ASAT analyses is widely changed. Finally, most ASAT configurations, once introduced, never change. If they do, the changes are small and have a tendency to occur within one day of the configuration’s initial introduction.

Veröffentlicht in: Software
0 Kommentare
0 Gefällt mir
Statistik
Notizen
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Keine Downloads
Aufrufe
Aufrufe insgesamt
763
Auf SlideShare
0
Aus Einbettungen
0
Anzahl an Einbettungen
16
Aktionen
Geteilt
0
Downloads
3
Kommentare
0
Gefällt mir
0
Einbettungen 0
Keine Einbettungen

Keine Notizen für die Folie

Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software

  1. 1. Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software
  2. 2. Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software Moritz Beller @Inventitech
  3. 3. Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software Moritz Beller @Inventitech Radjino Bholanath, Andy Zaidman
  4. 4. Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software Moritz Beller @Inventitech Radjino Bholanath, Andy Zaidman Shane McIntosh
  5. 5. Automatic Static Analysis Tools (ASATs) Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png
  6. 6. Automatic Static Analysis Tools (ASATs) Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png
  7. 7. Automatic Static Analysis Tools (ASATs) Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png
  8. 8. Automatic Static Analysis Tools (ASATs) Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png
  9. 9. Automatic Static Analysis Tools (ASATs) Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png
  10. 10. Automatic Static Analysis Tools (ASATs) Image: https://pixabay.com/static/uploads/photo/2015/06/12/18/31/cute-807306_960_720.png
  11. 11. RQ1: How Prevalent Are ASATs? Image: http://www.valueinvestasia.com/wp-content/uploads/2015/03/odd-one-out.jpg
  12. 12. 122 popular OSS projects
  13. 13. RQ1: How Prevalent Are ASATs?
  14. 14. RQ1: How Prevalent Are ASATs? 122
  15. 15. RQ1: How Prevalent Are ASATs? 122
  16. 16. RQ1: How Prevalent Are ASATs? 122
  17. 17. RQ1: How Prevalent Are ASATs? 122 122
  18. 18. RQ1: How Prevalent Are ASATs? 122 122
  19. 19. RQ1: How Prevalent Are ASATs? 122 36 122
  20. 20. RQ1: How Prevalent Are ASATs? 122 36 122
  21. 21. RQ1: How Prevalent Are ASATs? 122 36 122
  22. 22. RQ1: How Prevalent Are ASATs?
  23. 23. Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT 122 59% 23% - 36 77% 36% 36% RQ1: How Prevalent Are ASATs?
  24. 24. Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT 122 59% 23% - 36 77% 36% 36% RQ1: How Prevalent Are ASATs?
  25. 25. Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT 122 59% 23% - 36 77% 36% 36% RQ1: How Prevalent Are ASATs?
  26. 26. Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT 122 59% 23% - 36 77% 36% 36% RQ1: How Prevalent Are ASATs?
  27. 27. Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT 122 59% 23% - 36 77% 36% 36% RQ1: How Prevalent Are ASATs?
  28. 28. Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT 122 59% 23% - 36 77% 36% 36% 1) (Automated) investigation of repository information is an approximation of real ASAT use RQ1: How Prevalent Are ASATs?
  29. 29. Source Amount of Projects Using ASATs Using >1 ASAT Enforcing ASAT 122 59% 23% - 36 77% 36% 36% 1) (Automated) investigation of repository information is an approximation of real ASAT use 2) We cannot infer how a project uses ASATs from a repository analysis alone RQ1: How Prevalent Are ASATs?
  30. 30. RQ2: How are ASATs configured?
  31. 31. RQ2: How are ASATs configured?
  32. 32. RQ2: How are ASATs configured? checkstyle.xml
  33. 33. RQ2: How are ASATs configured? filename checkstyle.xml
  34. 34. RQ2: How are ASATs configured? filename checkstyle.xml
  35. 35. RQ2: How are ASATs configured? filename checkstyle.xml parse rules
  36. 36. RQ2: How are ASATs configured? filename checkstyle.xml enable parse rules
  37. 37. RQ2: How are ASATs configured? filename checkstyle.xml enable re-configure parse rules
  38. 38. RQ2: How are ASATs configured? filename checkstyle.xml enable re-configure parse rules custom
  39. 39. General Defect Classification (GDC)
  40. 40. General Defect Classification (GDC)
  41. 41. General Defect Classification (GDC) RQ1: 9
  42. 42. General Defect Classification (GDC) RQ1: 9 Checkstyle, FindBugs, PMD, ESLint, JSCS, JSHint, JSL, PYLint, RuboCop
  43. 43. General Defect Classification (GDC) 1,825 RQ1: 9 Checkstyle, FindBugs, PMD, ESLint, JSCS, JSHint, JSL, PYLint, RuboCop
  44. 44. RQ2: How are ASATs configured?
  45. 45. RQ2: How are ASATs configured?
  46. 46. RQ2: How are ASATs configured?
  47. 47. 168,425 RQ2: How are ASATs configured?
  48. 48. 168,425 RQ2: How are ASATs configured?
  49. 49. 168,425 RQ2: How are ASATs configured?
  50. 50. 168,425 RQ2: How are ASATs configured?
  51. 51. RQ2.1: How Popular Are Certain ASATs? Tool Language Configuration Files Checkstyle Java 18,785 FindBugs Java 2,090 PMD Java 7,458 ESLint JavaScript 4,435 JSCS JavaScript 11,677 JSHint JavaScript 108,770 JSL JavaScript 862 Pylint Python 4,071 RuboCop Ruby 10,066 Total - 168,405
  52. 52. RQ2.1: How Popular Are Certain ASATs? Tool Language Configuration Files Checkstyle Java 18,785 FindBugs Java 2,090 PMD Java 7,458 ESLint JavaScript 4,435 JSCS JavaScript 11,677 JSHint JavaScript 108,770 JSL JavaScript 862 Pylint Python 4,071 RuboCop Ruby 10,066 Total - 168,405
  53. 53. RQ 2.2: Which Rules Do Developers Enable?
  54. 54. RQ 2.2: Which Rules Do Developers Enable?
  55. 55. 65% RQ 2.2: Which Rules Do Developers Enable?
  56. 56. 35% 65% RQ 2.2: Which Rules Do Developers Enable?
  57. 57. 35% 65% ASATs perform poorly at finding functional defects. Wagner et al. RQ 2.2: Which Rules Do Developers Enable?
  58. 58. RQ 2.2: Which Rules Do Developers Enable? 0% 00% 00% 00% 00% 00% 00% 00% 00% 00% 00% Checkstyle ESLint FindBugs JSCS JSHint JSL PMD Pylint RuboCop 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%
  59. 59. 0% 00% 00% 00% 00% 00% 00% 00% 00% 00% 00% Checkstyle ESLint FindBugs JSCS JSHint JSL PMD Pylint RuboCop RQ 2.2: Which Rules Do Developers Enable? 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%
  60. 60. RQ 2.2: Which Rules Do Developers Enable? We: This is great! Image: Phil Skipper, The Potentialist,http://i2.wp.com/thepotentiality.com/assets/media/Kurt-Enthusiasm.jpg
  61. 61. RQ 2.2: Which Rules Do Developers Enable? We: This is great! Image: Phil Skipper, The Potentialist,http://i2.wp.com/thepotentiality.com/assets/media/Kurt-Enthusiasm.jpg
  62. 62. RQ 2.2: Which Rules Do Developers Enable? ASAT Developers*: Don't care. Image: NiBiS, http://bidab.nibis.de/PICT/traurig.jpg
  63. 63. RQ 2.2: Which Rules Do Developers Enable? ASAT Developers*: Don't care. Image: NiBiS, http://bidab.nibis.de/PICT/traurig.jpg
  64. 64. RQ 2.3: How Good Is The Default?
  65. 65. RQ 2.3: How Good Is The Default? Most ASAT configurations deviate from the default.
  66. 66. RQ 2.3: How Good Is The Default? Most ASAT configurations deviate from the default.
  67. 67. RQ 2.3: How Good Is The Default? Most ASAT configurations deviate from the default.
  68. 68. RQ 2.3: How Good Is The Default? But, typically only have one change from the default … Most ASAT configurations deviate from the default.
  69. 69. RQ 2.3: How Good Is The Default? But, typically only have one change from the default … ● - Addition Most ASAT configurations deviate from the default.
  70. 70. RQ 2.3: How Good Is The Default? But, typically only have one change from the default … ● - Addition ● - Deletion Most ASAT configurations deviate from the default.
  71. 71. RQ 2.3: How Good Is The Default? But, typically only have one change from the default … ● - Addition ● - Deletion ● - Re-configuration/Custom analysis Most ASAT configurations deviate from the default.
  72. 72. RQ2: Open Questions
  73. 73. ● Why do projects favor certain GDC rule categories from ASATs? RQ2: Open Questions
  74. 74. ● Why do projects favor certain GDC rule categories from ASATs? ● Can ASAT developers better fit their default configurations to their users' needs? RQ2: Open Questions
  75. 75. ● Why do projects favor certain GDC rule categories from ASATs? ● Can ASAT developers better fit their default configurations to their users' needs? ● Do 'dynamic' languages require more ASAT use? RQ2: Open Questions
  76. 76. ● Why do projects favor certain GDC rule categories from ASATs? ● Can ASAT developers better fit their default configurations to their users' needs? ● Do 'dynamic' languages require more ASAT use? RQ2: Open Questions
  77. 77. RQ3: How Do ASAT Configurations Evolve? Image: Daimler AG, http://5komma6.mercedes-benz-passion.com/wp-content/uploads/2013/06/s-class-lineup.jpg
  78. 78. RQ 3.1: How Often Do Changes Occur?
  79. 79. RQ 3.1: How Often Do Changes Occur?
  80. 80. RQ 3.1: How Often Do Changes Occur? >80% “never”
  81. 81. RQ 3.2: When Do Changes Occur?
  82. 82. RQ 3.2: When Do Changes Occur?
  83. 83. RQ 3.2: When Do Changes Occur? <20% of files
  84. 84. RQ 3.3: How Big Are The Changes?
  85. 85. RQ 3: Open Questions
  86. 86. ● Why do ASAT configurations not typically evolve? RQ 3: Open Questions
  87. 87. ● Why do ASAT configurations not typically evolve? ● How are ASATs used in a CI-environment? RQ 3: Open Questions
  88. 88. Moritz Beller @Inventitech Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software
  89. 89. Moritz Beller @Inventitech Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software
  90. 90. Moritz Beller @Inventitech Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software
  91. 91. Moritz Beller @Inventitech Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software

×