For the first time in the history of software engineering, we can both broadly and deeply analyze the behavior and dynamics of software development projects. This has become possible because of open source, which is publicly developed software. In this presentation, I will discuss our recent findings about open source software, its development process, and programmer behavior. I also discuss the challenges we encountered when quantitatively mining software repositories for such insights.

1. Six Easy Pieces (of Quantitatively Analyzing Open Source Software) ‏ Dirk Riehle SAP Research, SAP Labs LLC dirk@riehle.org, www.riehle.org, twitter.com/driehle

3. Talk Overview (Agenda) ‏ The Growth of Open Source Software Data Mining for Fun and Profit Efficiently Estimating Commit Sizes Developer Activity in Open Source Software Projects 1. 2. 3. 5. The Commit Size Distribution of Open Source 4. The Commenting Practice of Open Source 6. Team Size Evolution in Open Source Projects 7. Conclusions 8.

4. The Growth of Open Source Software Amit Deshpande, Dirk Riehle. “The Total Growth of Open Source.” In Proceedings of the Fourth Conference on Open Source Systems (OSS 2008). Springer Verlag, 2008. Page 197-209. http://www.riehle.org/2008/03/14/the-total-growth-of-open-source/

5. Source Code Growth in Open Source SLoC = source lines of code

6. Model of Source Code Growth where, y: Total open source lines of code x: Time from Jan 1995 to Dec 2006 in months 0.964 y = 2E+06*e 0.0464x Lower bound 0.961 y = 784098*e 0.0555x Upper bound R-square value Model Approach

7. Project Growth in Open Source

8. Model of Project Growth where, y: Total number of open source projects x: Time from Jan 1995 to Dec 2006 in months 0.956 y = 7.1511e 0.0499x R-square value Model

10. Data Mining for Fun and Profit Oliver Arafat, Amit Deshpande, Philipp Hofmann, Dirk Riehle. http://www.riehle.org/publications/

13. Open Source Analytics Tool Chain

14. Efficiently Estimating Commit Sizes Philipp Hofmann, Dirk Riehle. “Estimating Commit Sizes Efficiently.” In Proceedings of the 5th International Conference on Open Source Systems (OSS 2009). Springer Verlag, 2009. Forthcoming. http://www.riehle.org/2009/02/11/estimating-commit-sizes-efficiently/

16. What Diff Does 4,5c4,6 < d < f --- > e > e > e 7a9 > j 9d10 < n a b c e e e g h j m a b c d f g h m n 01: 02: 03: 04: 05: 06: 07: 08: 09: 10: 11: diff a.txt b.txt b.txt a.txt

19. Garden Variety of Heuristics 5.44 0 Linear Estimation 7 40.35 -5.95 Ldiff 6 30.87 -3.06 GNU diff –d 5 19.55 -1.96 GNU diff 4 7.68 -0.27 Bounds Mean 3 6.39 -4.41 Upper Bound 2 16.64 3.86 Lower Bound 1 Error Standard Deviation Error Mean Approach

20. Visual Comparison of Heuristics

22. The Commit Size Distribution of Open Source Oliver Arafat, Dirk Riehle. “The Commit Size Distribution of Open Source Software.” In Proceedings of the 42nd Hawaiian International Conference on System Science (HICSS-42). IEEE Press: 2009. Page 1-8. http://www.riehle.org/2008/09/23/ the-commit-size-distribution-of-open-source-software/

23. The Overall Commit Size Distribution

24. The Dominance of Small Commits

26. Developer Activity in Open Source Software Projects Dirk Riehle, Oliver Arafat, Amit Deshpande. “Developer Activity in Open Source Software Projects.” In preparation. Amit Deshpande, Dirk Riehle. “Continuous Integration in Open Source Software Development.” In Proceedings of the Fourth Conference on Open Source Systems (OSS 2008). Springer Verlag, 2008. Page 273-280. http://www.riehle.org/2008/03/08/ continuous-integration-in-open-source-software-development/

27. Average Commit Size

28. Average Commit Frequency

30. The Commenting Practice of Open Source Oliver Arafat, Dirk Riehle. “The Comment Density of Open Source Software Code.” In Companion to Proceedings of the 31st International Conference on Software Engineering (ICSE 2009). IEEE Press, 2009: Forthcoming. http://www.riehle.org/2009/02/04/ the-comment-density-of-open-source-software-code/

32. Comment Density by Programming Language 273 7% 10% Perl 6. 534 8% 11% Python 5. 276 9% 16% Javascript 4. 1621 8% 18% C/C++ 3. 559 12% 22% php 2. 1085 11% 26% Java 1. Population Size Stddev [%] Average [%] Language #

33. Comment Density by Commit Size

34. Comment Density by Team Size

35. Comment Density by Project Age

37. Team Size Evolution in Open Source Projects Philipp Hofmann, Dirk Riehle. “Team Size Evolution in Open Source Software Projects.” In preparation.

38. Teams Size Evolution Figure

41. Thank you! dirk@riehle.org, www.riehle.org, twitter.com/driehle Comments are welcome! ‏