1. Doing More WithYour Hudson Kohsuke Kawaguchi kohsuke@infradna.com InfraDNA, Inc.

2. Hudson Project Open-source CI server Written in Java but used for all kinds of languages Emphasis on ease of installation and use “java -jar hudson.war” execution Or your OS-specific package Configure everything from browsers 2 https://hudson-labs.org/

3. Ecosystem The real reason why Hudson thrives The same reason why Eclipse thrives Extensibility 330+ community-developed public plugins Up from 230+ last year in JavaOne 3

4. Growing Hudson Adoption Eclipse Community Survey 9.1% in 2009 -> 21.8% in 2010 Polling by John Ferguson 4

5. Install Base is Growing

6. Growing Hudson Adoption 6 Source: indeed.com

7. InfraDNA http://infradna.com/ I left Oracle this April and started a company around Hudson Support Subscriptions On open-source Hudson + plugins Or on ICHCI: a commercial value-added distribution Consulting Troubleshooting, custom plugins, etc. Visit us at partner kiosk outside exhibit floor 7

8. First steps to CI Hudson notices that there is a change Check out the source code Do some beefy computation Builds Run tests Record and publish results Report to people Only when necessary 8

9. But You Can Do So Much More! Once you automate, you can do a lot Yet average Hudson users only use very small part of Hudson Hence this talk 9

10. Interconnecting Jobs aka Workflows 10

11. Inter-Project Interactions You can host many projects on 1 Hudson Lots of ways to have them interact with each other Increased parallelism is good 11

12. Building Block #1: triggering When a build is done, kick off something else “upstream”: the build that’s kicking off “downstream”: the build being kicked off Triggering is asynchronous A large graph forms a cascading behaviour Upstream doesn’t wait for downstream Downstream can tell who is the upstream 12 Foo #10 Bar #153

13. Test/Build Separation Decompose test run into suite runs Trigger them all from build time Build #10 Build #11 Build #12 … Test X Test Y Test Z 13

14. Test/Build Separation Technique Do not recompile the bits during tests Instead, copy the pre-built bits Use the copy artifact plugin Or wget/curl via permalink 14

15. Build/Deploy Separation 1 job to build software, another to deploy it Deployment can be rerun without rebuild Failed build, new server, etc. Helps promote separate responsibilities 15

16. Building Block #2: join plugin Fan out, and then join back 16 time Build #10 Build #11 Build #12 … Test X Test Y Report Generation

17. Join Configuration 17

18. Building Block #3: Fingerprints MD5 checksums recorded during builds x.jar in product Z #112 had a bug. Where did it come from? Did it pass all tests? 18

19. Fingerprints The same mechanism works for … Tracking 3rd party dependencies Tracking the version of the build tools used Fingerprinting is critical when you have inter-related jobs Used to show relationship in UI Used to aggregate test reports 19

20. Build Promotion Sometimes plain-vanilla triggering is not desirable Example: Build X Run smoke test suites X1 and X2 If they both pass, deploy X to QA site Run QA test If it pass, deploy X to staging site 20

21. Build Promotion Hudson’s solution: build promotion Model Define a promotion level “Cleared Smoke tests” Define a promotion criteria “Successful completion of X1test and X2test” Define what happens after promotion “Tag the source tree and trigger Xqa” 21

22. More Gimmicks in Build Promotion Define multiple promotions E.g., “cleared smoke test” & “cleared QA test” Convenient for exposing builds to progressively more expensive work Promotion criteria can be “manual approval” We often can’t automate away everything! Last promoted build gets a permalink 22

23. Demo

24. Getting More From Your Maven Builds 24 Photo: http://www.flickr.com/photos/seeminglee/4556156477/

25. Maven Integration Highlights POM comprehension Simplified configuration More gets inferred automatically Snapshot dependencies among builds 25

26. Dealing With Large Maven Build Parallel module build Most projects have short but wide dependency chain Let Hudson build on module-by-module basis Wherever possible, build in parallel 26

27. Dealing With Large Maven Build Incremental builds Build changed modules and their descendants SCM determines what has changed 27

28. Private Maven Repository Cause ~/.m2/repository is not concurrent safe Multiple builds of the same project Different branches, test environments, JDK, etc. Problem Artifacts overwrite each other Artifacts change under the carpet Solution Allocate ~/.m2/repository per job Consider a repository manager 28

29. After-the-fact Deployment Default “mvn deploy” behavior harmful Think about a failure in the middle Better to let the build complete, then deploy 29

30. Really After-the-fact Deployment Deploy much after the build has finished 30

31. Matrix Project 31 Photo: http://www.flickr.com/photos/amarilloposters/4380080793/

32. Motivation You often do the same thing with slight variations Compile C++ code for different platforms Test with different browsers Or more generally, think of it as 32 for (x in [a,b,c]) { for (y in [d,e,f]) { … doSomethingWith(x,y,z,..); … } }

33. Model Define axes One axis ≈ one for loop Choose from pre-defined types of Axis Slave axis: pick slaves by their names or their labels e.g., linux, solaris, and windows Generic axis: arbitrary values exposed as environment variables 33

34. Matrix Project Gimmicks Filtering Otherwise combinations increase exponentially Not all combinations make sense Use boolean expression to trim down the size Or tell Hudson to cut the workload to N% Hudson will thin out the combinations by itself 34 (label==“windows”).implies(browser==“iexplore”) &&(label==“mac”).implies(browser==“safari”)

35. Demo

36. 36 Doing Selenium Tests on Hudson

37. Selenium Grid Automatically installs Selenium grid binaries On all Hudson slaves Automatically start Selenium RCs and connect them Starts Hub on Hudson master port 4444 Hudson cluster now accessible as Selenium RC farm from anywhere 37

38. Selenium Grid 38 Use Hudson slaves asSelenium RC nodes Hudson slaves Hudson master (selenium hub)

39. Choosing Environment Use Hudson’s labels to choose the slave that runs the browser *firefoxRun Firefox somewhere (but no guarantee that the chosen slave has it) windows:*firefoxPick a slave that has the “windows” label, then run Firefox windows&sfbay:*firefoxPick a slave that has both “windows” and “sfbay” label, then run Firefox 39

40. Testing with multiple browsers Use “matrix project” type Run the same tests with different browsers Aggregate the results 40 Firefox tests IE tests Safari tests

41. Selenium Grid w/Hudson Benefits No sysadmin overhead for setting up Selenium RCs Especially nice on heterogeneous network Reuse your existing investment in Hudson cluster 41

42. Launching browsers inside Hudson Hudson typically runs without GUI aka “Headless” Browsers need GUI So you need to make GUI environment accessible to Hudson and its slaves 42

43. On Unix Use Hudson xvncplugin Implements X protocol w/o a real GUI Needs xvnc binaries to be installed everywhereapt-get install vnc4server Hudson will start/stop xvnc and control $DISPLAY automatically 43

44. On Windows Windows has the concept called “WindowStation” Equivalent of X server in Windows WinSta0is the only interactive desktop So even headless processes can launch browsers. In theory. 44

45. On Windows, in practice GUI programs can hang with a dialog For example, IE8 setup wizard Thus it’s often easier to just let them access WinSta0 45

46. On Windows, really in practice But only SYSTEM user can interact with desktop Different user account creates different environments that often trip other things Fallback Configure Windows for auto-login Use start up script to start JNLP slaves 46

47. On Mac User that runs Hudson slaves needs to be also logged in to the desktop There appears to be no way to run GUI processes headlessly 47

48. Conclusions If you aren’t using Hudson yet… What are you waiting for? If you are using Hudson… You probably can get more out of it 48

49. Q&A Kohsuke Kawaguchi kohsuke@infradna.com InfraDNA, Inc.