Continuous Integration (CI) has become a best practice of modern software development. Yet, at present, we have a shortfall of insight into the testing practices that are common in CI-based software development. In particular, we seek quantifiable evidence on how central testing is to the CI process, how strongly the project language influences testing, whether different integration environments are valuable and if testing on the CI can serve as a surrogate to local testing in the IDE. In an analysis of 2,640,825 Java and Ruby builds on Travis CI, we find that testing is the single most important reason why builds fail. Moreover, the programming language has a strong influence on both the number of executed tests, their run time, and proneness to fail. The use of multiple integration environments leads to 10% more failures being caught at build time. However, testing on Travis CI does not seem an adequate surrogate for running tests locally in the IDE. To further research on Travis CI with GitHub, we introduce TravisTorrent.

1. Oops, My Tests Broke the Build:
An Explorative Analysis of Travis CI
with GitHub

2. Oops, My Tests Broke the Build:
An Explorative Analysis of Travis CI
with GitHub
Moritz Beller
@Inventitech

3. Oops, My Tests Broke the Build:
An Explorative Analysis of Travis CI
with GitHub
Moritz Beller
@Inventitech
Georgios Gousios

4. Oops, My Tests Broke the Build:
An Explorative Analysis of Travis CI
with GitHub
Moritz Beller
@Inventitech
Georgios Gousios
Andy Zaidman

5. How This Paper Came To Be

6. How This Paper Came To Be

7. How This Paper Came To Be

8. Travis What?

9. Travis What?

10. Travis What?

11. P1. Using CI is a widespread best practice

12. P1. Using CI is a widespread best practice
CI has become an integral quality
assurance practice.

13. P. M. Duvall, S. Matyas, and A. Glover, “Continuous
integration: improving software quality and
reducing risk”
P1. Using CI is a widespread best practice
CI has become an integral quality
assurance practice.

14. P1. Using CI is a widespread best practice

15. P1. Using CI is a widespread best practice

16. P1. Using CI is a widespread best practice

17. P1. Using CI is a widespread best practice

18. P1. Using CI is a widespread best practice

19. P1. Using CI is a widespread best practice

20. Around 30% of GITHUB
projects that could
potentially use the free
Travis CI, also make use it.
P1. Using CI is a widespread best practice

21. Around 30% of GITHUB
projects that could
potentially use the free
Travis CI, also make use it.
Caveat: In 2016!
P1. Using CI is a widespread best practice

22. TODO: Add background with
Sun
P1. Using CI is a widespread best practice

23. TODO: Add background with
Sun
P1. Using CI is a widespread best practice
Filter Criteria

24. TODO: Add background with
Sun
P1. Using CI is a widespread best practice
Filter Criteria
●
Non-toy, non-fork projects
(we want the original!)

25. TODO: Add background with
Sun
P1. Using CI is a widespread best practice
Filter Criteria
●
Non-toy, non-fork projects
(we want the original!)
●
> 50 stars

26. TODO: Add background with
Sun
P1. Using CI is a widespread best practice
Filter Criteria
●
Non-toy, non-fork projects
(we want the original!)
●
> 50 stars
●
A longer (> 50) Travis build
history

27. TODO: Add background with
Sun
P1. Using CI is a widespread best practice
Filter Criteria
●
Non-toy, non-fork projects
(we want the original!)
●
> 50 stars
●
A longer (> 50) Travis build
history
E. Kalliamvakou et al.:
“An in-depth study of the
promises and perils of mining
GitHub”

28. TODO: Add background with
Sun
P1. Using CI is a widespread best practice
Filter Criteria
●
Non-toy, non-fork projects
(we want the original!)
●
> 50 stars
●
A longer (> 50) Travis build
history
E. Kalliamvakou et al.:
“An in-depth study of the
promises and perils of mining
GitHub”

29. TODO: Add background with
Sun
P1. Using CI is a widespread best practice
Filter Criteria
●
Non-toy, non-fork projects
(we want the original!)
●
> 50 stars
●
A longer (> 50) Travis build
history
E. Kalliamvakou et al.:
“An in-depth study of the
promises and perils of mining
GitHub”

30. TODO: Add background with
Sun
P1. Using CI is a widespread best practice
Filter Criteria
●
Non-toy, non-fork projects
(we want the original!)
●
> 50 stars
●
A longer (> 50) Travis build
history
E. Kalliamvakou et al.:
“An in-depth study of the
promises and perils of mining
GitHub”
Java
●
423 projects

31. TODO: Add background with
Sun
P1. Using CI is a widespread best practice
Filter Criteria
●
Non-toy, non-fork projects
(we want the original!)
●
> 50 stars
●
A longer (> 50) Travis build
history
E. Kalliamvakou et al.:
“An in-depth study of the
promises and perils of mining
GitHub”
Ruby
●
936 projects
Java
●
423 projects

32. P1. Using CI is a widespread best practice
~

33. TODO: Add background with
Sun
P2. Testing is central to CI

34. TODO: Add background with
Sun
P2. Testing is central to CI
CI is the best practice in which
developers not only integrate their
work into a shared mainline
frequently, but also verify the quality
continuously through testing.

35. TODO: Add background with
Sun
P2. Testing is central to CI
CI is the best practice in which
developers not only integrate their
work into a shared mainline
frequently, but also verify the quality
continuously through testing.
It is known.

37. <TL;DR>...</TL;DR>

38. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

39. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>

40. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>

41. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>

42. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>

43. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>
●
Parsing Log files for Java: Maven, Gradle, Ant

44. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>
●
Parsing Log files for Java: Maven, Gradle, Ant

45. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>
●
Parsing Log files for Java: Maven, Gradle, Ant

46. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>
●
Parsing Log files for Java: Maven, Gradle, Ant
●
Parsing Log files for Ruby: bundler and rake

47. <TL;DR>...</TL;DR>
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running nl.tudelft.watchdog.ClientVersionCheckerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.04 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
<TL;DR>...</TL;DR>
●
Parsing Log files for Java: Maven, Gradle, Ant
●
Parsing Log files for Ruby: bundler and rake

48. P2. Testing is central to CI

49. ●
69% of Java and 87.5% of the Ruby projects:
test runs
P2. Testing is central to CI

50. ●
69% of Java and 87.5% of the Ruby projects:
test runs
1:0 for Ruby!
P2. Testing is central to CI

51. ●
69% of Java and 87.5% of the Ruby projects:
test runs
1:0 for Ruby!
P2. Testing is central to CI
●
Overall, 81% of the projects:
test runs

52. ●
69% of Java and 87.5% of the Ruby projects:
test runs
1:0 for Ruby!
P2. Testing is central to CI
●
Overall, 81% of the projects:
test runs
●
96% of builds:
test runs

53. P2. Testing is central to CI

54. P2. Testing is central to CI

55. P2. Testing is central to CI

56. Testing is the single most important
reason for broken builds, more
prevalent than compile errors, missing
dependencies, build cancellations and
provisioning problems together.
P2. Testing is central to CI

57. Testing is the single most important
reason for broken builds, more
prevalent than compile errors, missing
dependencies, build cancellations and
provisioning problems together.
P2. Testing is central to CI

58. TODO: Add background with
Sun
P3. Testing is language(-type) dependent
Does a dynamically typed language
(like Ruby) need as much testing as a
statically typed language (like Java)?

59. TODO: Add background with
Sun
P3. Testing is language(-type) dependent
Does a dynamically typed language
(like Ruby) need as much testing as a
statically typed language (like Java)?
S. Kleinschmager, S. Hanenberg, R. Robbes, E.
Tanter, A. Stefik: Do static type systems improve the
maintainability of software systems? An empirical
study.

60. P3. Testing is language(-type) dependent

61. P3. Testing is language(-type) dependent

62. P3. Testing is language(-type) dependent

63. P3. Testing is language(-type) dependent
Tests Build Result→

64. P3. Testing is language(-type) dependent
Tests Build Result→

65. P3. Testing is language(-type) dependent
Tests Build Result→

66. P3. Testing is language(-type) dependent
Tests Build Result→

67. Ruby projects have 10x more tests
than Java projects, yet execute faster.
Ruby builds are 4x more likely to fail
than Java projects.
P3. Testing is language(-type) dependent

68. Ruby projects have 10x more tests
than Java projects, yet execute faster.
Ruby builds are 4x more likely to fail
than Java projects.
P3. Testing is language(-type) dependent

69. Building and testing in different
environments delivers additional
value.
P4. Are more build environments helpful?

70. Building and testing in different
environments delivers additional
value.
We don’t know.
P4. Are more build environments helpful?

71. P4. Are more build environments helpful?

72. P4. Are more build environments helpful?

73. P4. Are more build environments helpful?

74. Yes, but 8x more so in Ruby than in
Java.
P4. Are more build environments helpful?

75. Yes, but 8x more so in Ruby than in
Java.
P4. Are more build environments helpful?

76. Moritz Beller @Inventitech
Oops, My Tests Broke the Build:
An Explorative Analysis of Travis CI with GitHub

77. Moritz Beller @Inventitech
Oops, My Tests Broke the Build:
An Explorative Analysis of Travis CI with GitHub

78. Moritz Beller @Inventitech
Oops, My Tests Broke the Build:
An Explorative Analysis of Travis CI with GitHub

79. Moritz Beller @Inventitech
Oops, My Tests Broke the Build:
An Explorative Analysis of Travis CI with GitHub

80. Moritz Beller @Inventitech
Oops, My Tests Broke the Build:
An Explorative Analysis of Travis CI with GitHub