Improving Engineering Processes using Hudson - Spark IT 2010

1. Improving Engineering Process
Using Hudson
<Insert Picture Here>
Arun Gupta, Java EE & GlassFish Guy
Oracle Corp
http://blogs.sun.com/arungupta, @arungupta

2. Never send a human to do a machine's job.

3. Throughout human history, we have been
dependent on machines to survive.

4. Rise of Continuous Integration
Offload from people, push to computers
$
computers
people
Time

5. Spend more CPU power to help you
First on your laptops and workstations
●
● IDEs are at the forefront
And then to the servers
●
● a.k.a. “Continuous Integration”
● More frequent build/test executions
● Static code analysis tools
● And more to come

6. Hudson (http://hudson-ci.org)
● Open-source CI server
● Emphasis on ease of installation and use
● “java -jar hudson.war” execution
● Or your OS-specific package
● Configure everything from browser
● Extensibility
● 230+ community-developed public plugins
● By 230+ contributors
● Estimated 13,000 installations

7. "D
as
hb
oa
rd
[H
ud
so
n]
"

8. Basic Feature Set
● Hudson notices that there is a code change
● Checks out the source code
● CVS, Subversion, ClearCase, Mercurial, Accurev,
Perforce, Git, StarTeam, …
● Builds Ant, Maven, shell script, NAnt, …
● Record and publish results
● Tool Integration
● Findbugs, Corbertura, ...

9. Provides Feedback
● RSS feeds
● Email/IM notification
● Tray application to notify developers
● Linux/Windows (Toolbar), Mac (Growl)
● ...

10. Demo

11. Why do I care?
Life before Hudson
●
● Dev forgets to commit a file
● Nightly build breaks, Testing blocked
● Resources not utilized efficiently
Life after Hudson
●
● Hudson will tell you in 5 mins if a build broke
● No one else notices that you forgot to commit a
file ;-)

12. Why do I care?
● Life before Hudson
● Dev makes a change Monday AM
● Test nightly finds a bug Monday night
● Dev fixes it Tuesday
● Again, under utilization of resources
Life after Hudson
●
● Dev makes a change Monday AM
● Hudson finds a regression few mins later
● Dev can fix it before lunch

13. Why do I care?
Life before Hudson
●
● Test runs every night, results in e-mail
● Excitement fades after 1 week
● Regressions go unnoticed until it’s too late
Life after Hudson
●
● Tests run Hudson after every commit
● E-mail sent out only when tests start failing
● So it manages to keep people’s attention

14. Matrix projects
Run the same thing on
●
different environments
● JDKs
● Databases
● OSes
● App servers
● XML parsers
● ...
Results aggregated
●

15. FindBugs integration

16. Tracking changes
16

17. … and more
Browse workspaces Build time trend report

18. Inter-Project Support
Host many projects on 1 Hudson
●
● 1+1 > 2
You can…
●
● Execute tests separately and correlate them
with builds
● Mark builds as “promoted” based on test
outcome
● Track which build of libraries are used where
● . . .

19. Test/Build Separation
●Tests often dominate the build time
●Multiple tests are run in parallel
● Improves turn-around time time
JAX-WS #10 JAX-WS #11 JAX-WS #12 …
Unit test with GF
Unit test with Tomcat
SQE test
Test w/o
3OS x 2JDK = 6 tests
container

20. Build Promotion
CI produces a lot of “successful” builds
●
● Often overwhelming to downstream
consumers
●Run tests as fast as you can
●If a build passes tests, promote it
20

21. Got the idea?
 Automation
 Reduce turn-around time
 Make things transparent
 Remove people from the loop
 Save people’s time
 Push jobs to servers, keep workstations
idle for you

22. Doing Distributed Builds
22

23. Distributed builds with Hudson
Master
●
● Serves HTTP requests slave
● Stores all important info 1
slave slave
Slaves
●
8 2
● 170KB single JAR
● Assumed to be
slave slave
unreliable 7 Master 3
● Scale to at least 100
Link
●
slave slave
6 4
● Single bi-di byte stream slave
● No other requirements 5

24. Automated System Installations
● Hudson + PXE plugin
● ISO images of OS
Your corporate IT guy
& his DHCP server
● Slaves
● Power on, hit F12
● PC boots from network (PXE)
● Choose OS from menu
● Installs non-interactively

25. Automated Tool Installation - JDK
JDK from http://java.sun.com/
●
● Automatically chooses the right bundle
● Always up to date with new releases
25

26. Automated Tool Installation - Apache
 Ant and Maven from Apache
26

27. Heterogeneous Cluster Challenge
 Builds/tests need to run in specific environment
 Dependency on individual nodes hurts utilization
jobs slaves
Wombat Windows
Windows test #1
GlassFish Windows
Windows test #2
Hudson Solaris
Windows test #1
Hudson
Solaris test

28. Labels to rescue
 Label is a group of slaves
 Tie jobs to labels
jobs slaves
Wombat Windows
Windows test #1
Windows
GlassFish Windows
Windows test #2
Hudson Solaris
Windows test #1
Solaris
Hudson Windows
Solaris test #3

29. Forecasting failures
Hudson monitors key health metrics of
●
slaves
● Low disk space, insufficient swap
● Clock out of sync
● Extensible
Slaves go offline automatically
●

30. Load Statistics Monitoring

31. When it’s time to add more slaves
31

32. Hudson EC2 plugin
Automatically provisions slaves on EC2 on
●
demand
● Based on the current load
●Picks the right AMI depending on demand
●Starts slave agent
●Shuts down unused instances
Can run Hudson master in the cloud too
●

33. And a lot more . . .
 IDE plugins
 iPhone/Android apps
 REST API and CLI
 Deployment Automation
 Hadoop, Selenium
 …

34. HUMANS: ALL YOUR COMPUTER ARE BELONG
TO US

35. Conclusions
CI is here to stay
●
● Using lots of PCs is a key
Hudson is very easy to get started
●
● Once started, there is a lot you can do
hudson-ci.org
●