The research community in Software Engineering and Software Testing in particular builds many of its contributions on a set of mutually shared expectations. Despite the fact that they form the basis of many publications as well as open-source and commercial testing applications, these common expectations and beliefs are rarely ever questioned. For example, Frederic Brooks’ statement that testing takes half of the development time seems to have manifested itself within the community since he first made it in the “Mythical Man Month” in 1975. With this paper, we report on the surprising results of a large-scale field study with 416 software engineers whose development activity we closely monitored over the course of five months, resulting in over 13 years of recorded work time in their integrated development environments (IDEs). Our findings question several commonly shared assumptions and beliefs about testing and might be contributing factors to the observed bug proneness of software in practice: the majority of developers in our study does not test; developers rarely run their tests in the IDE; Test-Driven Development (TDD) is not widely practiced; and, last but not least, software developers only spend a quarter of their work time engineering tests, whereas they think they test half of their time.

1. When, How, and Why Developers
(Do Not) Test in Their IDEs

2. When, How, and Why Developers
(Do Not) Test in Their IDEs
Moritz Beller
@Inventitech

7. Image: https://en.wikipedia.org/wiki/Delft, CC-BY

8. Image: https://en.wikipedia.org/wiki/Delft, CC-BY

9. Image: https://en.wikipedia.org/wiki/Delft, CC-BY

10. Image: https://en.wikipedia.org/wiki/Province_of_Bergamo, CC-BY

11. Image: http://www.scarymommy.com/angry-baby/

12. Image: http://www.scarymommy.com/angry-baby/

13. I am one of
you!
Image: http://www.scarymommy.com/angry-baby/

14. Source: Moeez, https://commons.wikimedia.org/wiki/File:Thank_You!.jpg, CC-BY

15. Image: http://www.jccschools.com/news/what_s_new/district_testing_descriptions

16. Developer Writes Tests
Image: http://www.jccschools.com/news/what_s_new/district_testing_descriptions

17. Developer Writes Tests Mine Repos for Tests
Image: http://www.jccschools.com/news/what_s_new/district_testing_descriptions

18. Developer Writes Tests Mine Repos for Tests
Image: http://www.jccschools.com/news/what_s_new/district_testing_descriptions

19. ?
Developer Writes Tests Mine Repos for Tests
Image: http://www.jccschools.com/news/what_s_new/district_testing_descriptions

20. RQ1. How Do Developers Test?

21. RQ1. How Do Developers Test?
RQ2. How Do Developers Manage Failing Tests?

22. RQ1. How Do Developers Test?
RQ3. Do Developers Follow Test-Driven Development?
RQ2. How Do Developers Manage Failing Tests?

23. RQ1. How Do Developers Test?
RQ3. Do Developers Follow Test-Driven Development?
RQ4. How Much Do Developers Test?
RQ2. How Do Developers Manage Failing Tests?

24. IDE!

25. IDE!

26. IDE!

28. Image: https://www.ckras.com/EPS-Entry-GameServer-mieten

29. Image: https://www.ckras.com/EPS-Entry-GameServer-mieten

31. → Interval

34. → Edit Distance

39. 2013 2014 2015

40. Initial WatchDog
prototype with ~10
users
2013 2014 2015

41. Initial WatchDog
prototype with ~10
users
2013 2014 2015
1.0

42. Initial WatchDog
prototype with ~10
users
2013 2014 2015
1.0
40

43. Initial WatchDog
prototype with ~10
users
2013 2014 2015
1.0
ICSE NIER
Evaluation
with 40
students
40

44. Initial WatchDog
prototype with ~10
users
2013 2014 2015
1.0
ICSE NIER
Evaluation
with 40
students
40

45. Initial WatchDog
prototype with ~10
users
2013 2014 2015
1.0
ICSE NIER
Evaluation
with 40
students
40 416

46. Initial WatchDog
prototype with ~10
users
2013 2014 2015
1.0
ICSE NIER
Evaluation
with 40
students
ESEC/FSE
Studying
416 developers
40 416

47. Initial WatchDog
prototype with ~10
users
2013 2014 2015
1.0
ICSE NIER
Evaluation
with 40
students
ESEC/FSE
Studying
416 developers
1.5
40 416

48. Initial WatchDog
prototype with ~10
users
2013 2014 2015
1.0
ICSE NIER
Evaluation
with 40
students
ESEC/FSE
Studying
416 developers
?
1,936
1.5
40 416

49. 416
Source: https://en.wikipedia.org/wiki/Pair_programming, CC-BY

51. WatchDog World Domination Tour 2015

52. 416

53. 416
●
Projects from industry
●
Famous and not-so-famous OSS
projects, e.g. Apache
●
Private projects
●
Anonymous projects
●
5 months

54. 416
●
Projects from industry
●
Famous and not-so-famous OSS
projects, e.g. Apache
●
Private projects
●
Anonymous projects
●
5 months

55. 416
●
Projects from industry
●
Famous and not-so-famous OSS
projects, e.g. Apache
●
Private projects
●
Anonymous projects
●
5 months
●
68 countries
●
24,255 hours of work time
●
~13.7 developer years
●
460 unique projects
●
5,665 sessions

56. 416
●
Projects from industry
●
Famous and not-so-famous OSS
projects, e.g. Apache
●
Private projects
●
Anonymous projects
●
5 months
●
68 countries
●
24,255 hours of work time
●
~13.7 developer years
●
460 unique projects
●
5,665 sessions

57. RQ1. How Do Developers Test?

58. RQ1. How Do Developers Test?
How Common is Testing?

59. RQ1. How Do Developers Test?
How Common is Testing?
Projects

60. RQ1. How Do Developers Test?
How Common is Testing?
460
Projects

61. RQ1. How Do Developers Test?
How Common is Testing?
460
200 (43%)
Projects

62. RQ1. How Do Developers Test?
How Common is Testing?
460
200 (43%)
Projects

63. RQ1. How Do Developers Test?
How Common is Testing?
460
200 (43%) Actually worked on
tests
Projects

64. RQ1. How Do Developers Test?
How Common is Testing?
460
200 (43%) Actually worked on
tests
Projects
1) Most projects do not work with tests.

65. RQ1. How Do Developers Test?
What are Tests? Every class that ...

66. RQ1. How Do Developers Test?
What are Tests? Every class that ...
●
is a Junit test.
●
contains traces of a testing Framework (Mockito,
PowerMock, TestNG, …).
●
has “Test” in its name.

67. RQ1. How Do Developers Test?
What are Tests? Every class that ...
●
is a Junit test.
●
contains traces of a testing Framework (Mockito,
PowerMock, TestNG, …).
●
has “Test” in its name.

68. RQ1. How Do Developers Test?
What are Tests? Every class that ...
●
is a Junit test.
●
contains traces of a testing Framework (Mockito,
PowerMock, TestNG, …).
●
has “Test” in its name.

69. RQ1. How Do Developers Test?

70. RQ1. How Do Developers Test?
How Common is Executing Tests?

71. RQ1. How Do Developers Test?
How Common is Executing Tests?
sessions

72. RQ1. How Do Developers Test?
How Common is Executing Tests?
3,424
sessions

73. RQ1. How Do Developers Test?
How Common is Executing Tests?
3,424
527 (15%)
sessions
≥ 1 test run

74. RQ1. How Do Developers Test?
How Common is Executing Tests?
3,424
527 (15%)
sessions
≥ 1 test run
2) Very rare, even for projects with tests.

75. RQ1. How Do Developers Test?
Image: TylerFrotierPhoto, http://tylerfortierphoto.deviantart.com/art/Lonely-Sheep-244657922

76. RQ1. How Do Developers Test?
How Many Tests Do Developers Execute?
Image: TylerFrotierPhoto, http://tylerfortierphoto.deviantart.com/art/Lonely-Sheep-244657922

77. RQ1. How Do Developers Test?
How Many Tests Do Developers Execute?
Image: TylerFrotierPhoto, http://tylerfortierphoto.deviantart.com/art/Lonely-Sheep-244657922

78. RQ1. How Do Developers Test?
How Many Tests Do Developers Execute?
3) One (90%), or all of them (9%).
Image: TylerFrotierPhoto, http://tylerfortierphoto.deviantart.com/art/Lonely-Sheep-244657922

79. RQ1. How Do Developers Test?
Image: https://blog.shareaholic.com/retail-sites-page-load-time/stopwatch

80. RQ1. How Do Developers Test?
How Long is a Test Run?
Image: https://blog.shareaholic.com/retail-sites-page-load-time/stopwatch

81. RQ1. How Do Developers Test?
How Long is a Test Run?
50% < 0.5s
75% < 5s
95% < 2m
Image: https://blog.shareaholic.com/retail-sites-page-load-time/stopwatch

82. RQ1. How Do Developers Test?
How Long is a Test Run?
50% < 0.5s
75% < 5s
95% < 2m
Image: https://blog.shareaholic.com/retail-sites-page-load-time/stopwatch

83. RQ1. How Do Developers Test?
How Long is a Test Run?
50% < 0.5s
75% < 5s
95% < 2m
Image: https://blog.shareaholic.com/retail-sites-page-load-time/stopwatch

84. RQ1. How Do Developers Test?
How Long is a Test Run?
50% < 0.5s
75% < 5s
95% < 2m
4) Very short.
Image: https://blog.shareaholic.com/retail-sites-page-load-time/stopwatch

85. RQ1. How Do Developers Test?

86. RQ1. How Do Developers Test?
1) Most projects do not actively work with tests.
2) Even projects with tests execute them very
rarely.
3) Most test executions contain 1 test.
4) Most test executions are very short.

87. RQ1. How Do Developers Test?
1) Most projects do not actively work with tests.
2) Even projects with tests execute them very
rarely.
3) Most test executions contain 1 test.
4) Most test executions are very short.

88. RQ1. How Do Developers Test?
1) Most projects do not actively work with tests.
2) Even projects with tests execute them very
rarely.
3) Most test executions contain 1 test.
4) Most test executions are very short.

89. RQ1. How Do Developers Test?
1) Most projects do not actively work with tests.
2) Even projects with tests execute them very
rarely.
3) Most test executions contain 1 test.
4) Most test executions are very short.

90. RQ2. How Do Developers Manage Failing Tests?
Image: https://www.linkedin.com/pulse/admit-failure-early-james-ient

91. RQ2. How Do Developers Manage Failing Tests?
How Many Test Executions Fail?
Image: https://www.linkedin.com/pulse/admit-failure-early-james-ient

92. RQ2. How Do Developers Manage Failing Tests?
The majority, 65%.
How Many Test Executions Fail?
Image: https://www.linkedin.com/pulse/admit-failure-early-james-ient

93. RQ2. How Do Developers Manage Failing Tests?

94. RQ2. How Do Developers Manage Failing Tests?

95. RQ2. How Do Developers Manage Failing Tests?

96. RQ2. How Do Developers Manage Failing Tests?

97. RQ2. How Do Developers Manage Failing Tests?

98. RQ2. How Do Developers Manage Failing Tests?
1) Dive into production code
2) Switch away from Eclipse
3) Almost no debug perspective
4) No frustration-closing of IDE

99. RQ2. How Do Developers Manage Failing Tests?
1) Dive into production code
2) Switch away from Eclipse
3) Almost no debug perspective
4) No frustration-closing of IDE

100. RQ2. How Do Developers Manage Failing Tests?
1) Dive into production code
2) Switch away from Eclipse
3) Almost no debug perspective
4) No frustration-closing of IDE

101. RQ2. How Do Developers Manage Failing Tests?
1) Dive into production code
2) Switch away from Eclipse
3) Almost no debug perspective
4) No frustration-closing of IDE

102. RQ2. How Do Developers Manage Failing Tests?

103. minutes
RQ2. How Do Developers Manage Failing Tests?

104. minutes
RQ2. How Do Developers Manage Failing Tests?

105. 50% of test failures are fixed within 10 minutes.
minutes
RQ2. How Do Developers Manage Failing Tests?

106. RQ2. How Do Developers Manage Failing Tests?

107. RQ2. How Do Developers Manage Failing Tests?
1) Most test executions fail.
2) Developers have certain patterns of managing
failed test executions.
3) Most test repairs are fast.

108. RQ2. How Do Developers Manage Failing Tests?
1) Most test executions fail.
2) Developers have certain patterns of managing
failed test executions.
3) Most test repairs are fast.

109. RQ2. How Do Developers Manage Failing Tests?
1) Most test executions fail.
2) Developers have certain patterns of managing
failed test executions.
3) Most test repairs are fast.

110. RQ3. Do Developers Follow TDD?
What is TDD?
Image: http://www.agilenutshell.com/test_driven_development

111. RQ3. Do Developers Follow TDD?
What is TDD?
Image: http://www.agilenutshell.com/test_driven_development

112. RQ3. Do Developers Follow TDD?
What is TDD?
Image: http://www.agilenutshell.com/test_driven_development

113. RQ3. Do Developers Follow TDD?
What is TDD?
Image: http://www.agilenutshell.com/test_driven_development

114. RQ3. Do Developers Follow TDD?
What is TDD?
Image: http://www.agilenutshell.com/test_driven_development

115. RQ3. Do Developers Follow TDD?
Method?

116. RQ3. Do Developers Follow TDD?
Method?

117. RQ3. Do Developers Follow TDD?
Method?
Test phase

118. RQ3. Do Developers Follow TDD?
Method?
Test phase

119. RQ3. Do Developers Follow TDD?
Method?
Test phase Refactoring

120. RQ3. Do Developers Follow TDD?
Method?
Test phase Refactoring
2% of developers

121. RQ3. Do Developers Follow TDD?
Method?
Test phase Refactoring
2% of developers
28%

122. RQ3. Do Developers Follow TDD?
Method?
Test phase Refactoring
2% of developers
72%28%

123. RQ3. Do Developers Follow TDD?
Image: http://www.agilenutshell.com/test_driven_development

124. RQ3. Do Developers Follow TDD?
Image: http://www.agilenutshell.com/test_driven_development

125. RQ3. Do Developers Follow TDD?
(Almost) nobody follows TDD.
Image: http://www.agilenutshell.com/test_driven_development

126. RQ3. Do Developers Follow TDD?

127. “You followed TDD 38.6% of your
development changes (so, in
words, quite often).”
Sven Amann, letsdeveloper.com
RQ3. Do Developers Follow TDD?

128. RQ4. How Much Do Developers Test?
Software Testing takes ...

129. RQ4. How Much Do Developers Test?
Brooks, 1975
Software Testing takes ...

130. RQ4. How Much Do Developers Test?
50%
… of Project EffortBrooks, 1975
Software Testing takes ...

131. How (Much)
Do You
Test?
Image: http://www.agilenutshell.com/test_driven_development

132. RQ4. How Much Do Developers Test?
Projects
Delta(Production% Est - Real)

133. RQ4. How Much Do Developers Test?
Projects
Delta(Production% Est - Real)

134. RQ4. How Much Do Developers Test?
Projects
Delta(Production% Est - Real)

135. RQ4. How Much Do Developers Test?
%testingwork

136. RQ4. How Much Do Developers Test?
Estimated 48%
%testingwork

137. RQ4. How Much Do Developers Test?
Estimated 48% Reality 25%
%testingwork

138. RQ4. How Much Do Developers Test?
Estimated 48% Reality 25%
%testingwork
not test execution time!

139. Conclusion: Testing Paradox

140. Conclusion: Testing Paradox
1) Test executions are rare

141. Conclusion: Testing Paradox
1) Test executions are rare
2) Yet, 25% of work time on testing

142. Conclusion: Testing Paradox
1) Test executions are rare
“Running Eclipse UI Platform tests
locally blocks my machine … I
offload running them to the CI.”
Famous Eclipse Dude
2) Yet, 25% of work time on testing

143. @Inventitech
Moritz Beller, TU Delft
When, How, and Why Developers
(Do Not) Test in Their IDEs

144. @Inventitech
Moritz Beller, TU Delft
When, How, and Why Developers
(Do Not) Test in Their IDEs

145. @Inventitech
Moritz Beller, TU Delft
When, How, and Why Developers
(Do Not) Test in Their IDEs

146. @Inventitech
Moritz Beller, TU Delft
When, How, and Why Developers
(Do Not) Test in Their IDEs