Presented at STPcon 2011 on leveraging your automated tests by putting them under CI. Covers value proposition of CI, challenges to implement, and ideas for scaling.

1. TURBOCHARGE YOUR
AUTOMATED TESTS
WITH CONTINUOUS
INTEGRATION
STPCON 2011
1

2. WHY AM I QUALIFIED TO BE
UP HERE?
• President of OpenSource
Connections
• Contributor to CruiseControl
and Continuum CI projects
• Member of Apache Software
Foundation
• Presenter at conferences
(OSCON, ApacheCON, jTDS,
ExpoQA, STPcon 2009!)

3. WRITER

4. AUTHOR

5. FATHER

6. AGILISTA

7. AGENDA
CI in a Flash
Value Proposition of CI
Challenges in using CI
Scaling CI for LARGE suites of tests
Wrap Up
7

8. CI IN A FLASH!
8

9. DEFINITION
a fully automated and reproducible build, including testing, that
runs many times a day - Martin Fowler
http://martinfowler.com/articles/continuousIntegration.html

10. MARTIN FOWLER
• Coinedthe term
Continuous Integration
• Chief
Scientist for
ThoughtWorks
• Speaksand Blogs on
software:
• http://martinfowler.com/

11. CONCEPTUAL

12. MAPPING
diagram credit to Brian Di Croce @ http://blog.briandicroce.com/2008/03/17/setting-up-a-continuous-integration-
environment-preface/

13. FEEDBACK MECHANISM
From: cruisecontrol@opensourceconnections.com
Subject: [CruiseControl] agenda_direcciones build 370 fixed
Date: October 7, 2008 2:22:50 PM EDT
To: epugh@opensourceconnections.com
The build has been fixed.
CHANGES
-------
New revision 370 detected
Revision 370 committed by epugh on 2008-10-07 18:22:03
mysql esta mejor que sqlite
M /branches/ERIC_EXPO_08_DEMO/agenda_direcciones/config/database.yml
See http://cruise.opensourceconnections.com:3333/builds/agenda_direcciones/370 for details.

14. VALUE PROPOSITION
OF CI
14

15. ACTIVITY
Turn to the person next to you, and see how many
non-obvious things you have in common.

16. LIFE WITH OUT CI
• Unstable code, integration is difficult
• Many build errors reported
• Only one person on the team can
build the entire project
• Getting “test ready” state is hard
• Feedback cycle is long
Every day is a struggle to be productive

17. LIFE WITH CI
• The build process is simple
and repeatable
• Elimination of Human Error
• Bugs stay fixed!
• Feedback cycle is very rapid
Every day you know you can produce better software!

18. photo credit to Jay Grieves @ www.bigswingingdeveloper.com

19. RISK REDUCTION
• What is in the build?
• What has changed between
builds?
• How do I verify
functionality?

20. REDUCE HUMAN ERROR

21. MY AH-HAH! MOMENT

22. REDUCE REPETITIVE
PROCESSES
• DRY: Don’t Repeat Yourself
• Script Everything!
• “ant deploy”

23. MAKE PAINFUL THINGS
PAINLESS!
photo credit to Paul Duvall @ http://www.integratebutton.com/

24. PRODUCE DEPLOYABLE
SOFTWARE
• It’s what we are here to do!
• Always in a “Demo Ready” state
• No last minute administrivia tasks to get the product
ready
• Installer is built
• Documentation is written
• Changelog is written

25. ENABLE PROJECT VISIBILITY

27. INCREASE PROJECT
CONFIDENCE

29. photo credit to ThoughtWorks

30. FACILITATE
COMMUNICATION

31. photo credit to Eric Pugh @ www.opensourceconnections.com

34. CHALLENGES IN USING
CI
34

35. WHAT PREVENTS TEAMS
FROM USING CI?

36. CULTURAL CHALLENGES

37. FEAR OF COMMITMENT
• Developers and Testers need to
commit changes frequently!
• Incremental small changes make
finding integration issues simpler
drawing credit to Tom Armitage @ http://infovore.org/archives/2007/03/14/the-continuous-integration-barometer/

38. THIS ISN’T 1984

39. FEAR OF BLAME

40. • CI needs a champion who acts as the ambassador
to the leaders of the organization.
• Thought leaders in the organization who can help
encourage team to accept the process changes that
CI brings
•A very successful test case
•A greenfield project if possible

41. CI ON A DOLLAR A DAY
"From now on, our code in revision control will always build
successfully and pass its tests.". - James Shore
• Build Computer
• Ridiculous Toy
• Annoying Bell
• Automated Build
• Group Agreement
http://jamesshore.com/Blog/Continuous-Integration-on-a-Dollar-a-Day.html

42. credit to Paul Julius @ http://www.anthillpro.com/blogs/anthillpro-blog/2009/09/11/
enterprise_ci_cultural_maturity.html

43. STRUCTURAL CHALLENGES

44. TESTABILITY
• Modularity? • Perform code complexity
analysis to see if some parts
• Coupling to underlying are more amenable to
platform? testing.
• Requiresexternal services
to be up and available for
testing?

45. TIGHT COUPLING
photo credit to Jay Grieves @ www.bigswingingdeveloper.com

46. 10 MINUTE RULE
photo credit to Jay Grieves @ www.bigswingingdeveloper.com

47. CUP OF COFFEE METRIC
What happens in the time it takes to get a cup of coffee can
tell you a lot about the quality of your continuous integration
system. - Paul Julius
• Builds shouldn’t take any longer!
• Gettinga cup of coffee or drink
of water breaks up your day
• Givesyou a chance to review
your work
• Requires a moduler fast build
http://pauljulius.com/blog/2009/09/14/cup-of-coffee-metric-for-continuous-integration/

48. PAUL JULIUS
• co-founder of CITCON
• Original
developer of
CruiseControl
• Speaksand Blogs on CI
extensively
• http://pauljulius.com/blog/

49. 789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
!"#$%#&'
!"#$%&'"()*+ ?-#4+!@)1-#)+ 0)/)3*)31A+ 95$1B'3C+ ,-'&*+D54E+
,-'&*+.#)/" ,-'&*+.#)/" :)/4"'#45A !"#$%&!'(#%)* >2+.3$/"(4#"
0)*'1$#)*+ ,-'&*+6&-"#)5 .)1-5)+
,-'&*+2$1('3) !"#$%&!'(#%)*
643#'3-4-"+
.4&'*+ ,-'&*"
.4-51)+643#54&

50. GETTING TOO FOCUSED ON
GREEN BAR
Glare of the Green Bar in CI can
obscure issues with your product
James Bach

51. TOOL CHALLENGES

52. INVEST IN CI SERVER
X
• The CI system must be as well maintained as
the Source Control system.
• The CI system performs builds very quickly.
✓
• Who has responsibility for CI? It’s very
important to have a “throat to choke”.
• No false alarms. If there are false alarms then
the team loses confidence in the CI system.
•

53. MONKEY CI
photo credit to Eric Pugh @ www.opensourceconnections.com

54. MAKING CI THE CORE OF TOO
MANY SYSTEMS

55. MAKING DECISIONS BASED
ON YOUR AVAILABLE
TOOLING

56. !"#$%&'()
789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
F)&/)5+.15'/#" D-&&A+.15'/#)* .)&GH.)5I'1)+ L-"(H,-##43+ ?-#4E$#'1+
9)"#+0)/&4A" L54*+0)/&4AE)3# L54*-1#'43+
0)/&4A"
?-#4+0)/&4A+#4+ M$#)*N+?-#4E$#'1+
;3)+!3IJ L54E4#'43"
24"#&A+ D-&&A+.#$3*$5*'K)*+
.#$3*$5*'K)* 0)/&4A"

57. !"#$!%&'(
789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
944&H,$")*+ .'&4)*+:)/45#'3C F'"#45'1$&+ 95)3*'3C L5)*'1#'I)
:)/45#" :)/45#" 24*)&'3C
L-%&'"(+ 654""H.'&4
>'"'%'&'#AO+:)/45#+ P$#)"#+:)/45#" 654""+D-31#'43+ ?3$&A"'"
:-33)5 :)/45#+?11)""
,$"'1+
:)/45#+?3$&A"'"

58. SCALING CI FOR LARGE
SUITES OF TESTS
58

59. MULTI STAGE CI
• Reduce the amount of
integration required
• Teamsintegrate as modules
are marked as ready.
• Maven2 was built with this in
mind!
Damon Poole @ http://damonpoole.blogspot.com/2008/01/advanced-multi-stage-continous.html

60. BREAK UP TESTS
• “preflight” or “smoke” tests
• longer integration tests
• periodic full site generation and report generation

61. THROW
HARDWARE AT THE PROBLEM

62. PARALLELISM
• Ifyour suite of
tests take 24
hours to run, can
you spin it up on
24 machines and
take an hour?
• Testing a web
application with
Selenium?
Selenium Grid!

63. RUN TO CLOUD!

64. VIRTUAL SERVERS!

65. PREDICTIVE TESTING
• Some commercial tools support predicting which tests will fail
based on previous behavior, and running those first!

66. DON’T DO IT ALL AT ONCE
• Justlike when embarking on a new exercise program, don’t
just go full tilt, work up to it!
• Start
with the tests that are easily put under CI first, then
move up the food chain.

67. SLIDING “WINDOW”
• For a test suite that takes a
week to run, can you take a
sliding chunk, plus some
random tests?
• Ideas
from DNA
sequencing!
• http://en.wikipedia.org/
wiki/DNA_sequencing

68. 789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
!"#!$%&
.4E)++ :)C5)""'43+ .4E)+.#$#'1+ !@#)3"'I)+ RSST+
9)"#+?-#4E$#'43 9)"#"+:-3+Q'#(+ ?3$&A"'"+<")* <3'#+9)"#" 9)"#+64I)5$C)
,-'&*
?-#4E$#)*+ !@#)3"'I)+
D-31#'43$&+9)"#" ?-#4E$#)*+
D-31#'43+9)"#"
:'"BH,$")*+
2$3-$&+9)"#"
.)1-5'#A+.1$3"

69. LOTS OF
ADVANCED
TECHNIQUES
!
credit to Christine Mitterbauer @ Microdoc

70. WRAP UP

71. !P!2!89.+;D+!89!:L:7.!+67
789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
!"#$%#&'
!"#$%&'"()*+ ?-#4+!@)1-#)+ 0)/)3*)31A+ 95$1B'3C+ ,-'&*+D54E+
,-'&*+.#)/" ,-'&*+.#)/" :)/4"'#45A !"#$%&!'(#%)* >2+.3$/"(4#"
0)*'1$#)*+ ,-'&*+6&-"#)5 .)1-5)+
,-'&*+2$1('3) !"#$%&!'(#%)*
643#'3-4-"+
.4&'*+ ,-'&*"
.4-51)+643#54&
!"#$%&'()
789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
F)&/)5+.15'/#" D-&&A+.15'/#)* .)&GH.)5I'1)+ L-"(H,-##43+ ?-#4E$#'1+
9)"#+0)/&4A" L54*+0)/&4AE)3# L54*-1#'43+
0)/&4A"
?-#4+0)/&4A+#4+ M$#)*N+?-#4E$#'1+
;3)+!3IJ L54E4#'43"
24"#&A+ D-&&A+.#$3*$5*'K)*+
.#$3*$5*'K)* 0)/&4A"
789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
!"#!$%&
.4E)++ :)C5)""'43+ .4E)+.#$#'1+ !@#)3"'I)+ RSST+
9)"#+?-#4E$#'43 9)"#"+:-3+Q'#(+ ?3$&A"'"+<")* <3'#+9)"#" 9)"#+64I)5$C)
,-'&*
?-#4E$#)*+ !@#)3"'I)+
D-31#'43$&+9)"#" ?-#4E$#)*+
D-31#'43+9)"#"
:'"BH,$")*+
2$3-$&+9)"#"
.)1-5'#A+.1$3"
!"#$!%&'(
789:;0<69;:= 8;>76! 789!:2!07?9! ?0>?86!0 78.?8!
944&H,$")*+ .'&4)*+:)/45#'3C F'"#45'1$&+ 95)3*'3C L5)*'1#'I)
:)/45#" :)/45#" 24*)&'3C
L-%&'"(+ 654""H.'&4
>'"'%'&'#AO+:)/45#+ P$#)"#+:)/45#" 654""+D-31#'43+ ?3$&A"'"
:-33)5 :)/45#+?11)""
,$"'1+
:)/45#+?3$&A"'"
'!()*+,"$-!).$/ !"#$%&"!'
!"#$%&'()*&)+(&,%-$(.(/0$&12&3"-+(/-4&3)5$6&78/-$.".$(
!""#$%%&&&'()"!*++#,-'.-/%!"/+%,01-2,.01%&!*"03#(#0,1

72. Don’t forget the people aspect!
credit to Paul Julius @ http://www.anthillpro.com/blogs/anthillpro-blog/2009/09/11/
enterprise_ci_cultural_maturity.html

73. SELECTING A CI SYSTEM

74. APPROACHES
• Internal Hosted System
• Cloud Based Solution
• Completely Hosted

75. CI MATRIX
Matrix of differences between CI tools: http://tinyurl.com/39hol7

76. HUDSON / JENKINS
• Hudson, recently
forked as Jenkins.

77. CI IN A BOX
• Amazon EC2 Hosted instance
• http://www.ciinabox.com/

78. MIKECI.COM

79. WHERE TO NEXT?

80. CITCON
• Open Spaces Style
• Twice a year, US and International
• Portland, Oregon, April 2010
• http://www.citconf.com/

82. CI YAHOO GROUP
http://tech.groups.yahoo.com/group/citcon/

83. THE BOOK
Continuous Integration:
Improving Software Quality

84. THE CI GUYS
http://ci-guys.com/
Free online consulting

85. THANK YOU!
• twitter: dep4b
• speakerrate: http://www.speakerrate.com/epugh/
• email: epugh@opensourceconnections.com
85