Job application talk.

1. Mining Software Archives
to Support Software Development
Tom Zimmermann
Saarland University

2. Software Development
Hello
Build Calgary!

3. Software Development
Build

4. Collaboration

5. Collaboration

6. Collaboration
Comm.
Archive

7. Collaboration
Version
Comm.
Archive
Archive

8. Collaboration
Version
Comm. Bug
Archive
Archive Database

9. Collaboration
Version
Comm. Bug
Archive
Archive Database
Mining Software Archives

10. Mining Software Archives

11. Mining Software Archives
eROSE BugCache Vulture

12. eROSE
Related Changes
(ICSE 2004, TSE 2005)
Tom Zimmermann • Saarland University
Peter Weißgerber • University of Trier
Stephan Diehl • University of Trier
Andreas Zeller • Saarland University

16. Developers who changed this function
also changed...

17. eROSE: Guiding Developers
Customers who
bought this item also
bought...
Purchase
History

18. eROSE: Guiding Developers
Developers who
Customers who
changed this function
bought this item also
also changed...
bought...
Version
Purchase
Archive
History

22. eROSE suggests further locations.

24. eROSE prevents incomplete changes.

25. Processing CVS data

26. Processing CVS data

27. Processing CVS data
1. Comparing files
2. Building transactions

28. Comparing Files

29. Comparing Files
A()
B()
C()
D()
E()

30. Comparing Files
A() A()
B() F()
C() B()
D() D()
E() E()

31. Comparing Files
A() A()
B() F()
C() B()
D() D()
E() E()

32. Building Transactions
CVS
150,000

33. Building Transactions
2003-02-19 (aweinand): fixed #13332
CVS
createGeneralPage()
createTextComparePage()
150,000
fKeys[]
initDefaults()
buildnotes_compare.html
PatchMessages.properties
plugin.properties

34. Building Transactions
same author + message + time
2003-02-19 (aweinand): fixed #13332
CVS
createGeneralPage()
createTextComparePage()
150,000
fKeys[]
initDefaults()
buildnotes_compare.html
PatchMessages.properties
plugin.properties

35. Mining Associations
User changes fKeys[] and initDefaults()

36. Mining Associations

37. Mining Associations
EROSE
finds past
transactions

38. Mining Associations
#756 #6721 #21078
EROSE fKeys[] fKeys[] fKeys[]
initDefaults() initDefaults() initDefaults()
finds past ... ... ...
transactions plugin.properties plugin.properties plugin.properties
#42432 #51345 #59998 #71003
fKeys[] fKeys[] fKeys[] fKeys[]
initDefaults() initDefaults() initDefaults() initDefaults()
... ... ... ...
plugin.properties plugin.properties plugin.properties plugin.properties
#87264 #91220 #101823 #104223
fKeys[] fKeys[] fKeys[] fKeys[]
initDefaults() initDefaults() initDefaults() initDefaults()
... ... ... ...
plugin.properties plugin.properties plugin.properties

39. Mining Associations
#756 #6721 #21078
EROSE fKeys[] fKeys[] fKeys[]
initDefaults() initDefaults() initDefaults()
finds past ... ... ...
transactions plugin.properties plugin.properties plugin.properties
#42432 #51345 #59998 #71003
{fKeys[], initDefaults()} {plugin.properties}
fKeys[] fKeys[] fKeys[] fKeys[]
initDefaults() initDefaults() initDefaults() initDefaults()
Support 10, Confidence 10/11 = 0.909
... ... ... ...
plugin.properties plugin.properties plugin.properties plugin.properties
#87264 #91220 #101823 #104223
fKeys[] fKeys[] fKeys[] fKeys[]
initDefaults() initDefaults() initDefaults() initDefaults()
... ... ... ...
plugin.properties plugin.properties plugin.properties

40. Evaluation
GIMP
PostgreSQL
KOffice
jEdit

41. Evaluation
EROSE predicts 33% of all changed entities.
GIMP
(files: 44%)
PostgreSQL
KOffice
jEdit

42. Evaluation
EROSE predicts 33% of all changed entities.
GIMP
(files: 44%)
In 70% of all transactions, EROSE’s topmost
three suggestions contain a changed entity.
PostgreSQL
(files: 72%)
KOffice
jEdit

43. Evaluation
EROSE predicts 33% of all changed entities.
GIMP
(files: 44%)
In 70% of all transactions, EROSE’s topmost
three suggestions contain a changed entity.
PostgreSQL
(files: 72%)
EROSE learns quickly (within 30 days).
KOffice
jEdit

44. eROSE
Related Changes
(ICSE 2004, TSE 2005)
guides developers
non-program elements
(documentation)
learns quickly

45. BugCache
Predicting Defects
(ASE 2006, ICSE 2007)
`
Sung Kim • MIT
Tom Zimmermann • Saarland University
Jim Whitehead • Univ. of California SC
Andreas Zeller • Saarland University

46. The Problem
How should we
allocate our resources
for quality assurance?

47. One Solution
List with elements that
(will) have defects
List is adaptive, i.e.,
it changes over time

48. One Solution
List with elements that
(will) have defects
Cache
List is adaptive, i.e.,
it changes over time

49. The BugCache Model
What is loaded in the
cache?
Cache size: 2
Hypothesis: Temporal locality between defects

50. The BugCache Model
What is loaded in the
cache?
Cache size: 2
Hypothesis: Temporal locality between defects

51. The BugCache Model
What is loaded in the
cache?
Cache size: 2
Hypothesis: Temporal locality between defects

52. The BugCache Model
What is loaded in the
cache?
Cache size: 2
Hypothesis: Temporal locality between defects

53. The BugCache Model
What is loaded in the
cache?
Cache size: 2
Hypothesis: Temporal locality between defects

54. The BugCache Model
What is loaded in the
cache?
Cache size: 2
Miss
Hypothesis: Temporal locality between defects

55. The BugCache Model
What is loaded in the
cache?
Cache size: 2
Miss
Hypothesis: Temporal locality between defects

56. The BugCache Model
Cache size: 2
Miss

57. The BugCache Model
Cache size: 2
Miss

58. The BugCache Model
Cache size: 2
Miss Hit

59. The BugCache Model
Cache size: 2
Miss Hit

60. The BugCache Model
Cache size: 2
Miss Hit Miss

61. The BugCache Model
Cache size: 2
Miss Hit Miss

62. The BugCache Model
Cache size: 2
Miss Hit Miss
Hit rate = #Hits / #Defects = 33.3%

63. The BugCache Model
Cache size: 2
Miss Hit Miss

64. The BugCache Model
Cache size: 2
Miss Hit Miss

65. The BugCache Model
Cache size: 2
Miss Hit Miss Miss

66. The BugCache Model
Cache size: 2
Miss Hit Miss Miss

67. The BugCache Model
Cache size: 2
Miss Hit Miss Miss

68. Loading Elements
Temporal locality – as shown before
Spatial locality – load “nearby” elements
(i.e., co-changed before)
Changed-entity locality – load changed elements
New-entity locality – load new elements
Initial pre-fetch – start with a loaded cache

69. Evaluation
Mozilla
jEdit
PostgreSQL Columba

70. Hit Rates
Methods Files
Project BugCache FixCache BugCache FixCache
Apache 1.3 59.6% 61.5% 83.9% 81.5%
Columba 58.9% 67.6% 83.5% 83.0%
Eclipse 64.5% 71.6% 95.1% 95.0%
JEdit 50.5% 48.9% 85.7% 85.4%
Mozilla 49.3% 55.0% 93.3% 88.0%
PostgreSQL 61.9% 59.2% 73.9% 71.0%
Subversion 68.3% 43.8% 82.0% 81.3%
Cache size = 10%

71. Hit Rates
Methods Files
Project BugCache FixCache BugCache FixCache
Apache 1.3 59.6% 61.5% 83.9% 81.5%
Columba 58.9% 67.6% 83.5% 83.0%
Eclipse 64.5% 71.6% 95.1% 95.0%
JEdit 50.5% 48.9% 85.7% 85.4%
Mozilla 49.3% 55.0% 93.3% 88.0%
PostgreSQL 61.9% 59.2% 73.9% 71.0%
Subversion 68.3% 43.8% 82.0% 81.3%
Cache size = 10%

72. Reasons for Hits
Initial pre-fetch
Spatial locality 18%
18%
Initial pre-fetch
Temporal locality
Temporal locality Spatial locality
Changed-entity locality
60% New-entity locality

73. Warning Developers
“Safe” Location
(not in FixCache)
Risky Location
(red, in FixCache)

74. BugCache
Predicting Defects
(ASE 2006, ICSE 2007)
temporal locality
adaptive
hit rates of 71%~95%

75. Vulture
Predicting
Security Vulnerabilities
(Work in Progress)
Stephan Neuhaus • Saarland University
Tom Zimmermann • Saarland University
Andreas Zeller • Saarland University

76. Firefox/Mozilla
>700 developers 228,365 commits
14,368 C/C++ files
1,012,512 revisions
(10,452 components)

77. >700 developers 228,365 commits
14,368 C/C++ files
1,012,512 revisions
(10,452 components)

78. Vulnerabilities

79. Vulnerabilities

80. Vulnerabilities
0
Vulnerabilities

81. Vulnerabilities
Security Advisory 2005-12
Title: Livefeed bookmarks can steal cookies
Impact: High
Products: Firefox
Description: Earlier versions of Firefox allowed
javascript: and data: URLs as Livefeed bookmarks.
When they updated the URL would be run in the
context of the current page and could be used to
steal cookies or data displayed on the page. If the
user were on a page with elevated privileges (for
example, about:config) when the Livefeed was
updated, the feed URL could potentially run
arbitrary code on the user's machine.
0
Vulnerabilities

82. Vulnerabilities
0
Vulnerabilities

83. Vulnerabilities
Security Advisory 2005-13
Title: Window Injection Spoofing
Severity: Low
Products: Firefox, Mozilla Suite
Description: A website can inject content into a
popup opened by another site if the target name
of the popup window is known. An attacker who
knows you are going to visit that other site could
spoof the contents of the popup.
0
Vulnerabilities

84. Vulnerabilities
Security Advisory 2005-15
2005-41
2005-16
2006-76
2005-14
Title: Heap overflow possible security dialogs
Title: Spoofing escalation via DOM property
XSS quot;secure sitequot;window's Function
Privilege download and in UTF8 to object
SSL using outer indicator spoofing
Impact: Moderate
Unicode conversion
overrides High
with overlapping windows
Severity:
Products:Critical 2.0
Severity: High
Products: Firefox Mozilla Suite
Firefox,
Description:Various schemesdemonstrated
Products: Firefox, Thunderbird, Mozilla Suitethat
Description: moz_bug_r_a4 were reported
Mozilla Suite
Description: It thepossible forreportedstringin
the Function prototype regressionlock icon to with
that could causeMichael Kraxsitequot; UTF8 several
moz_bug_r_a4 a described
is quot;secure demonstrates that
the download dialog trigger details overflow be
bug 355161 couldto and security dialogs the
exploitsand show attacker the ability tothe wrong
invalid sequences certificate a heap bypass can of
appear giving an be exploited to for install
malicious could be data. by requiring would
spoofed byUnicode cross Exploitability only
convertedcode or steal data,phishers to an that
site. These against used site script (XSS)
protections partially covering them with make
injection, which could be used to particularly a
the user do commonplace users get click onin
overlapping window. Some actionsstealthe string
depend on the attackers abilityto may not notice
their spoofs look more legitimate, like credentials
or the buggyhide the and browser or perform
link or window from arbitrary sitescommon
thesensitive the context menu. Theshowing the
intoOS opendataborderaddress barweb content is
windows that converter. General statusbar
destructive actions on privileged rule out
cause in what appears to be of a logged-in and
bisectingeach case was behalf a single dialog,user.
converted elsewhere but we can'tUI code the be
true location.
(quot;chromequot;) being overly attack.
convinced by the spoofing text of the top-most
possibility of a successfultrusting of DOM nodes
from the content window.
window to click on the quot;Allowquot; or quot;Openquot; button
of the window below.
0
Vulnerabilities

85. Vulnerabilities
0
Vulnerabilities

86. Vulnerabilities
10,452 components
424 vulnerable
4.05%
0
Vulnerabilities

87. Vulnerabilities
What other
components are
vulnerable?
0
Vulnerabilities

88. Vulnerabilities
0
Vulnerabilities

89. Vulnerabilities
0
Vulnerabilities ?

90. Vulnerabilities
Is this new
component likely
to be vulnerable?
0
Vulnerabilities ?

91. Vulture
Code
Vulnerability Version Code
Code
Database Archive Code
Redo diagram

92. Vulture
Code
Vulnerability Version Code
Code
Database Archive Code
Redo diagram
Vulture

93. Vulture
Code
Vulnerability Version Code
Code
Database Archive Code
Redo diagram
Vulture
Component
Component Component

94. Vulture
Code
Vulnerability Version Code
Code
Database Archive Code
Redo diagram
Vulture Predictor
Component
Component Component

95. Vulture
Code
Vulnerability Version Code
Code Code
Database Archive Code
Redo diagram
Vulture Predictor
Component
Component Component

96. Correlations

97. Correlations
Programmer Code Complexity
Language

98. Correlations
Code Complexity
Language

99. Correlations
Language

100. Correlations
Language
Problem Domain

101. Imports

102. Imports
GUI Database Certificates OS

103. Imports
GUI Database Certificates OS

104. Imports
GUI Database Certificates OS

105. Example (1)
nsIContent.h
nsIContentUtils.h
nsIScriptSecurityManager.h

106. Example (1)
nsIContent.h
import
nsIContentUtils.h
nsIScriptSecurityManager.h

107. Example (1)
✘
✘ ✘
✘ ✘
✘ nsIContent.h
✘
✘ ✘
✘✘ ✘ import
✘
✘ ✘ nsIContentUtils.h
✘ ✘
95.5%
✘
✔ ✘
✘ ✘
nsIScriptSecurityManager.h

108. Example (2)
nsIPrivateDOMEvent.h
nsReadableUtils.h

109. Example (2)
import nsIPrivateDOMEvent.h
nsReadableUtils.h

110. Example (2)
✘
✘ ✘
✘ ✘
✘
✘ ✘
✘ ✘ import nsIPrivateDOMEvent.h
✘
✘ ✘
✘
100%
✘
✘
✘
✘ ✘ nsReadableUtils.h

111. Research Questions
• How well do imports predict vulnerabilities?
• Can imports be used for
− classification (vulnerable or not) and for
− regression (number of vulnerabilities)?

112. Input Data
nsCOMArray 0
nsIDocument.h 1
nspr_md.h 0
nsDOMClassInfo 10
EmbedGTKTools 0
MozillaControl.cpp 0
nsDOMClassInfo has had 10
vulnerability-related bug reports

113. Input Data
e. am t.h
h
e.
re Fr c
bt ack nne
e
or
St o
di h
s/fi h
m
ns PC
st le.
9, h
ut o.h
sy pl.
9
il.h
IX
Im
05
ns
ss
nsCOMArray 0 1 0 0 0 1 0 0
nsIDocument.h 1 0 0 1 0 0 1 0
nspr_md.h 0 0 1 1 0 0 1 0
nsDOMClassInfo 10 0 0 1 0 1 0 0
EmbedGTKTools 0 0 0 0 0 1 0 0
MozillaControl.cpp 0 0 1 0 1 0 0 0
nsDOMClassInfo has had 10 nsDOMClassInfo imports
vulnerability-related bug reports “nsIXPConnect.h”

114. Distribution
ibution of MFSAs Distribution of Bug Reports
300
Number of Components
20 50
5
12
5 7 9 11 13 13579 13 17 24
umber of MFSAs Number of Bug Reports

115. Experiments
• 40 randomtraining set, 3,484 rows in validation set
splits
6,968 rows in
• Classification recall and precision
Train SVM, compute
• Regression rank correlation on top 1%
Train SVM, compute
• SVM: linear kernel10GB ofdefault parameters
with
R implementation (up to main memory)

116. Results
(a) Precision and Recall (b) Rank Correlation
0.55
1.0
●
●
●
●
● ●
●
Cumulative Distribution
●
0.8
●
●
0.50
●
● ●
●
● ●
●● ●
●
●●● ●● ●
● ●
0.6
Precision
●
●
●
● ●
0.45
●
●● ●
●
● ● ● ●
● ●
0.4
●
● ● ●
●●● ● ● ●
● ● ●
●
●
0.40
●
● ●
●
● ●
0.2
●
●● ● ●
●
● ● ●
●
●
●
0.35
●
0.0
●
0.55 0.60 0.65 0.70 0.75 0.2 0.3 0.4 0.5 0.6 0.7
Recall Rank Correlation

117. Results
(a) Precision and Recall (b) Rank Correlation
0.55
1.0
●
●
●
●
● ●
●
Cumulative Distribution
●
0.8
●
●
0.50
●
● ●
●
● ●
●● ●
●
●●● ●● ●
● ●
0.6
Precision
●
●
●
● ●
0.45
●
●● ●
●
● ● ● ●
● ●
0.4
●
● ● ●
●●● ● ● ●
● ● ●
●
●
0.40
●
● ●
●
● ●
0.2
●
●● ● ●
●
● ● ●
●
●
●
0.35
●
0.0
●
0.55 0.60 0.65 0.70 0.75 0.2 0.3 0.4 0.5 0.6 0.7
Recall Rank Correlation
45% (about 1/2) of predictions correct

118. Results
(a) Precision and Recall (b) Rank Correlation
0.55
1.0
●
●
●
●
● ●
●
Cumulative Distribution
●
0.8
●
●
0.50
●
● ●
●
● ●
●● ●
●
●●● ●● ●
● ●
0.6
Precision
●
●
●
● ●
0.45
●
●● ●
●
● ● ● ●
● ●
0.4
●
● ● ●
●●● ● ● ●
● ● ●
●
●
0.40
●
● ●
●
● ●
0.2
●
●● ● ●
●
● ● ●
●
●
●
0.35
●
0.0
●
0.55 0.60 0.65 0.70 0.75 0.2 0.3 0.4 0.5 0.6 0.7
Recall Rank Correlation
2/3 of all vulnerable components detected
45% (about 1/2) of predictions correct

119. Results
(a) Precision and Recall (b) Rank Correlation
0.55
1.0
●
●
●
●
● ●
●
Cumulative Distribution
●
0.8
●
●
0.50
●
● ●
●
● ●
●● ●
●
●●● ●● ●
● ●
0.6
Precision
●
●
●
● ●
0.45
●
●● ●
●
● ● ● ●
● ●
0.4
●
● ● ●
●●● ● ● ●
● ● ●
●
●
0.40
●
● ●
●
● ●
0.2
●
●● ● ●
●
● ● ●
●
●
●
0.35
●
0.0
●
0.55 0.60 0.65 0.70 0.75 0.2 0.3 0.4 0.5 0.6 0.7
Recall Rank Correlation
2/3 of all vulnerable components detected
45% (about 1/2) of predictions correct

120. Results
moderately strong correlation (mostly significant at p < 0.01)
(a) Precision and Recall (b) Rank Correlation
0.55
1.0
●
●
●
●
● ●
●
Cumulative Distribution
●
0.8
●
●
0.50
●
● ●
●
● ●
●● ●
●
●●● ●● ●
● ●
0.6
Precision
●
●
●
● ●
0.45
●
●● ●
●
● ● ● ●
● ●
0.4
●
● ● ●
●●● ● ● ●
● ● ●
●
●
0.40
●
● ●
●
● ●
0.2
●
●● ● ●
●
● ● ●
●
●
●
0.35
●
0.0
●
0.55 0.60 0.65 0.70 0.75 0.2 0.3 0.4 0.5 0.6 0.7
Recall Rank Correlation
2/3 of all vulnerable components detected
45% (about 1/2) of predictions correct

121. Ranking

122. Ranking
Rank Component Actual Rank
1 nsDOMClassInfo 3
2 SGridRowLayout 95
3 xpcprivate 6
4 jsxml 2
5 nsGenericHTMLElement 8
6 jsgc 3
7 nsISEnvironment 12
8 jsfun 1
9 nsHTMLLabelElement 18
10 nsHttpTransaction 35
... (3,474 components)

123. Ranking
Rank Component Actual Rank
1 nsDOMClassInfo 3
2 SGridRowLayout 95
3 xpcprivate 6
4 jsxml 2
5 nsGenericHTMLElement 8
6 jsgc 3
7 nsISEnvironment 12
8 jsfun 1
9 nsHTMLLabelElement 18
10 nsHttpTransaction 35
... (3,474 components)

124. Ranking
Rank Component Actual Rank
1 nsDOMClassInfo 3
2 SGridRowLayout 95
3 xpcprivate 6
4 jsxml 2
5 nsGenericHTMLElement 8
6 jsgc 3
7 nsISEnvironment 12
8 jsfun 1
9 nsHTMLLabelElement 18
10 nsHttpTransaction 35
... (3,474 components)

125. Ranking
Rank Component Actual Rank
1 nsDOMClassInfo 3
2 SGridRowLayout 95
3 xpcprivate 6
4 jsxml 2
5 nsGenericHTMLElement 8
6 jsgc 3
7 nsISEnvironment 12
8 jsfun 1
9 nsHTMLLabelElement 18
10 nsHttpTransaction 35
... (3,474 components)

126. Similar Results for Bugs
Packages + Import relationships
(ISESE 2006)
Precision: 66.7% Recall: 69.4%
Binaries + Dependencies
(Internship @ Microsoft Research, 2006)
Precision: 64.4% Recall: 75.3%

127. Vulture
Predicting
Security Vulnerabilities
(Work in Progress)
locates past + predicts new
vulnerabilities
problem domain

128. Future
Work
?

129. #1: Mining across Projects
• Complement source
code search engines
with mining techniques.
• Large-scale mining
(144,000 SF projects)

130. #2: Developer Buddy
MOCKUP

131. eROSE BugCache Vulture

132. automatic
eROSE BugCache Vulture

133. automatic
large-scale
eROSE BugCache Vulture

134. automatic
large-scale
eROSE BugCache Vulture
tool-oriented

135. automatic
large-scale
Empirical Software
Engineering 2.0
tool-oriented

136. automatic
large-scale
Empirical Software
Engineering 2.0
tool-oriented Thanks! Questions?