The Mining Software Repositories (MSR) research community has grown significantly since the first MSR workshop was held in 2004. As the community continues to broaden its scope and deepens its expertise, it is worthwhile to reflect on the best practices that our community has developed over the past decade of research. We identify these best practices by surveying past MSR conferences and workshops. To that end, we review all 117 full papers published in the MSR proceedings between 2004 and 2012. We extract 268 comments from these papers, and categorize them using a grounded theory methodology. From this evaluation, four high-level themes were identified: data acquisition and preparation, synthesis, analysis, and sharing/replication. Within each theme we identify several common recommendations, and also examine how these recommendations have evolved over the past decade. In an effort to make this survey a living artifact, we also provide a public forum that contains the extracted recommendations in the hopes that the MSR community can engage in a continuing discussion on our evolving best practices.
1. The MSR Cookbook
Mining a Decade of Research
Hadi Hemmati, Sarah Nadi, Olga Baysal, Oleksii Kononenko,
Wei Wang, Reid Holmes, Michael W. Godfrey
David R. Cheriton School of Computer Science
University of Waterloo, Canada
MSR-2013, May 19, 2013
4. Methodology
Review Open coding
MSR 2004–2012 (9 yrs)
270 papers
117 full papers
Comments:
Generalizable observations
or suggestions supported
by evidence in the paper
268 comments
4 themes with
16 recommendations
Recommendation:
Comments supported
by evidence from at least
5 papers
4
6. Top 5 Recommendations
Validate your assumptions and
heuristics; repos are noisy
[22 c, 17 p]
Watch out for collinearities and
skewness when synthesizing
models from data
[22 c, 17 p]
[20 c,16 p]
Manually verify outputs
[12 c, 10 p]
Sometimes other measures
work better than precision/
recall
Sometimes practicality
trumps statistical rigour
[16 c, 15 p]
6
7. Theme/Recommendation Trends
• MSR research is maturing as a field
• Apparent shift from data extraction toward deeper
analysis of the results and their practical use
7
8. Online Forum of MSR Cookbook
!"#$%%&'()*+'(,-./00*1(%2030404-4%5&.%6
8
9. Take-away Message
9
• Newcomers to MSR have questions
• Recommendations of best practices for conducting
MSR research
• Online forum to learn, discuss and contribute
• Check the paper for the full list of guidelines!
