Kohsuke Kawaguchi's Jenkins talk at 2011 June Silicon Valley DevOps meetup.

1. Jenkins

2. What’s Jenkins? ©2010 CloudBees, Inc. All Rights Reserved

3. What’s Jenkins? Jenkins is a “continuous integration” server It looks for changes and builds your project It publishes the result It keeps track of how your project is used elsewhere It brings transparency to projects 3

4. What’s Jenkins? Community-driven open-source project http://jenkins-ci.org/ Used to be called Hudson Get going with “java -jar jenkins.war” Extensible architecture Lots of community developed plugins Public plugins as well as in-house plugins Very active 7 years old, ~400 releases to date 300+ plugins, 250+ committers 4

5. Used Everywhere 5 5

6. What does Jenkins do? Notice changes and checking out Polling, post-commit hooks ClearCase, Perforce, Git, Mercurial, … Runs builds and tests Lots of build/reporting tools integrations 6

7. What does Jenkins do? Whatever you can script can be run in Jenkins It needs to be non-interactive Very flexible and can work with any build/test tools/frameworks But it is not by itself a build/test framework 7

8. Non-Java Suport Jenkins is used to build non-Java projects Ruby, Python, PHP, C++, .NET, … Rapidly expanding as many of those languages do not have viable CI server Various tool/report integrations done by community 8

9. Demo ©2010 CloudBees, Inc. All Rights Reserved

10. Why do I care? Life before Jenkins RE nightly build breaks. It blocks SQE Dev rebuilds before commit to avoid this People waste time Life after Jenkins Dev just commits it right away Dev moves on to work on another thing Jenkins will tell you in 5 mins if a build broke 10

11. Why do I care? Life before Jenkins Dev makes a change Monday AM SQE nightly finds a bug Monday night Dev fixes it Tuesday Lot of wasted time Life after Jenkins Dev makes a change Monday AM Jenkins finds a regression 30 mins later Dev can fix it before lunch 11

12. Why do I care? Life before Jenkins SQE test runs every night, results sent out in e-mail After the enthusiasm of the 1st week is gone, nobody looks at them anymore Regressions go unnoticed until it’s too late Life after Jenkins Tests run Jenkins after every commit E-mail sent out only when tests start failing So it manages to keep people’s attention 12

13. Got the idea? Automation Reduce turn-around time Make things transparent Remove people from the loop Save people’s time Push jobs to servers, keep workstations idle for you 13

14. Advanced Integration Techniques ©2010 CloudBees, Inc. All Rights Reserved

15. Doing Distributed Builds 15 Photo by skreuzer

16. Why? You need to use multiple computers because… You need different environments You need isolation You have non-trivial work loads There’s only so much you can do with 1 computer

17. Lots of computers = tedious Pains grow as your build farm grows Software installation problem Node failure problem Remote maintenance problem … What does Jenkins do to help you with these?

18. Installing new slaves For first 20 or so slaves, we did it manually Insert CD, click, type, click, type, click, … But that doesn’t scale Then we automated Available as “Jenkins PXE Plugin”

20. ISO images of OSYour corporate IT guy & his DHCP server Slaves Power on, hit F12 PC boots from network (PXE)

22. ISO images of OSYour corporate IT guy & his DHCP server Slaves Power on, hit F12 PC boots from network (PXE) Choose OS from menu Installs non-interactively

23. Automated System Installations Supports OpenSolaris, Ubuntu, CentOS, Fedora Trivial with most Linux Cooperate with Windows, too Quite useful outside Jenkins, too No more broken CD drives No more CD-Rs

24. Master Serves HTTP requests Stores all important info Slaves 170KB single jar Assumed to be unreliable Scale to at least 100 Distributed builds with Jenkins

25. Building on slaves Slave agent is a little Java program Runs locally on the machine that it controls Access files, fork processes, etc., on behalf of master Communication with master Needs a bi-di byte stream But very flexible in how this gets set up No shared file system, no network topology constraints, etc

26. How master and slaves start talking For Unix slaves, via SSH Only need SSHD on slaves We even install Java for you We just need a host name

27. How master and slaves start talking For Windows, DCOM We just need admin user name and password No manual intervention Works even from Unix masters

28. How master and slaves start talking Via Java Web Start When master cannot see slaves A separate socket connection is made

29. Automating JNLP launch Once started, can be installed as Windows service

30. Automating JNLP launch Emulate the JNLP client headlessly $ java -jar slave.jar -jnlpUrl URL

31. Automated Tool Installations JDK from http://java.sun.com/ It automatically chooses the right bundle

32. Automated Tool Installations Ant and Maven from Apache General purpose implementations Grab an archive from some URL and extract Run arbitrary shell script Extensible

33. Labels Tags that group slaves together Name: alpha Label: windows 32bit Name: bravo Label: linux 32bit Name: bravo Label: linux 64bit

34. Labels Name: alpha Label: windows 32bit Name: bravo Label: linux 32bit This job needs to run on “linux” Name: bravo Label: linux 64bit

35. Labels Name: alpha Label: windows 32bit Name: bravo Label: linux 32bit This job needs to run on “linux && 32bit” Name: bravo Label: linux 64bit

36. More Label Benefits Labels can be anything Geographic locations Availability of proprietary software installed Better resource utilization Jenkins have maximum freedom Resilient to node addition/removal Treat servers like livestocks, not like pets

37. Making builds sticky We want jobs to be mostly on the same slave Faster check out Consistent results Minimizes disk consumption But do it softly Jenkins uses consistent hash to achieve this More schedule controls become possible: Use faster machines more frequently Slowly ramp up newly installed slaves * http://en.wikipedia.org/wiki/Consistent_hashing

38. Forecasting failures Jenkins monitors key health metrics of slaves Low disk space, insufficient swap Clock out of sync Extensible Slaves put offline automatically Catch problems before it breaks builds 36

39. Clean up mess after builds Kill run-away processes Daemons, background processes left by your build Works on Windows, Linux, Mac, and Solaris

40. Cloud support Slaves can come and go Current/future workload and queue are all monitored continuously We can let Jenkins provision slaves on cloud on-the-fly

41. Jenkins EC2 plugin Automatically provision slaves on EC2 on demand Pick the right AMI depending on demand Start slave agent Shut down unused instances * http://code.google.com/p/typica/

42. Integration with VMWare Similar mechanism exists with VMWare Create a pool of virtual machines upfront Jenkins powers them up/down based on load Optionally revert to snapshot at the end Achieves benefits similar to EC2 without going public Nectar Feature

43. Matrix Project Photo: http://www.flickr.com/photos/amarilloposters/4380080793/ 41

44. Motivation You often do the same thing with slight variations Compile C++ code for different platforms Test with different browsers Or more generally, think of it as for (x in [a,b,c]) { for (y in [d,e,f]) { … doSomethingWith(x,y,z,..); … } }

45. Model Define axes One axis ≈ one for loop Choose from pre-defined types of Axis Slave axis: pick slaves by their names or their labels e.g., linux, solaris, and windows Generic axis: arbitrary values exposed as environment variables

46. Matrix Project Gimmicks Filtering Otherwise combinations increase exponentially Not all combinations make sense Use boolean expression to trim down the size Or tell Jenkins to cut the workload to N% Jenkins will thin out the combinations by itself (label==“windows”).implies(browser==“iexplore”) &&(label==“mac”).implies(browser==“safari”)

47. Demo

48. Build Promotion

49. Progressive Exposure AKA Build pipeline Once built, bits go through progressively more expensive “certification” Think of tests, deployment to QA, staging, etc. Some builds fail in the middle, others go through

50. Then it gets more complex Each stage can be several parallel jobs Needs to be retryable You can’t afford to execute every pipeline until it fails

51. Hence “build promotion” Configuration “Foo gets promoted if FooTest passes” “When Foo gets promoted, do X, Y, and Z” Sequence of events Foo #N successfully completes FooTest #M executes on Foo #N FooTest #M successfully completes Foo #N gets promoted X, Y, and Z happens on Foo #N

52. More on build promotion Promotion condition can be more complex If FooTest, BarTest, and ZotTest all passes If PHB manually approves If upstream project gets promoted Extensible Promotion actions can be anything Copy files, execute scripts, trigger jobs, …

53. More on build promotion Multiple different promotion levels Foo #5 FooIntTest #9 FooIntTest #10 FooTest #10 BarTest #16 Deploy #19

54. Promotion benefits No rebuilding of binaries Asynchronous and naturally throttled Audit Trail

55. Demo