How we use system Groovy scripts to automate housekeeping and monitoring of our Jenkins instances.

1. Managing Jenkins with Jenkins
Tweet @garethbowles with feedback!
Monday, October 28, 13
Hi, I’m Gareth from Netflix. It’s great to be back for my third JUC !
I’m going to talk about how we use Jenkins’ System Groovy Scripts feature to automate the management of our Jenkins infrastructure.
I hope you’ll get inspired by some of these ideas to try them out yourselves. As this is a short talk there may not be much time for
questions, but find me afterwards or tweet me if you want to follow up.

2. @
•
•
•
•
•
•
3200 jobs
3TB build data
6 masters
~100 slaves / ~250 executors
15% build failure rate
All 700 engineers can set up / run jobs
Tweet @garethbowles with feedback!
Monday, October 28, 13
We have a pretty big Jenkins setup, it’s the centrepiece of the pipeline that builds our streaming service.
Our main master has 3200 builds, 3TB of build data data
We have a growing number of masters as we split jobs between different masters for speed and reliability
Housekeeping and monitoring has to be automated due to this scale. We need to make sure our slaves are online, disk space for builds is
kept under control, builds run on the right slaves, and be able to update the configuration of many jobs or slaves at a time.
And because all our engineers have job configuration access, we need to make sure that some basic conventions are followed and nothing
gets too out of control.

3. How can we automate all the Jenkins things ?
Tweet @garethbowles with feedback!
Monday, October 28, 13
Jenkins has quite a few options for automation.

4. How can we automate all the Jenkins things ?
•
Remote API - nice, but limited update functions
Tweet @garethbowles with feedback!
Monday, October 28, 13
Jenkins has quite a few options for automation.

5. How can we automate all the Jenkins things ?
•
•
Remote API - nice, but limited update functions
Jenkins CLI - ditto
Tweet @garethbowles with feedback!
Monday, October 28, 13
Jenkins has quite a few options for automation.

6. How can we automate all the Jenkins things ?
•
•
•
Remote API - nice, but limited update functions
Jenkins CLI - ditto
Custom plugins - powerful but heavyweight
Tweet @garethbowles with feedback!
Monday, October 28, 13
Jenkins has quite a few options for automation.

7. A Better Option
Tweet @garethbowles with feedback!
Monday, October 28, 13
Here’s a more lightweight but still very powerful way to automate Jenkins tasks.

8. System Groovy Scripts
Tweet @garethbowles with feedback!
Monday, October 28, 13
Using system Groovy scripts gives you full access to Jenkins’ JVM and object model just like a plugin, but the scripts are much easier to
write and maintain.
You get the benefit of Jenkins’ built in features for tracking script results, scheduling runs, notifying users of the results and visualizing
data (with the plot plugin for example).
Many of our needs aren’t generic enough to make it worth writing a plugin, anyway.
But it’s always good to script a task, even if you think it’s a one-off - that way if you’re wrong you get to re-run it, and you can track it
back easily if something goes wrong.

9. System Groovy Scripts
•
Full access to the Jenkins master’s JVM
Tweet @garethbowles with feedback!
Monday, October 28, 13
Using system Groovy scripts gives you full access to Jenkins’ JVM and object model just like a plugin, but the scripts are much easier to
write and maintain.
You get the benefit of Jenkins’ built in features for tracking script results, scheduling runs, notifying users of the results and visualizing
data (with the plot plugin for example).
Many of our needs aren’t generic enough to make it worth writing a plugin, anyway.
But it’s always good to script a task, even if you think it’s a one-off - that way if you’re wrong you get to re-run it, and you can track it
back easily if something goes wrong.

10. System Groovy Scripts
•
•
Full access to the Jenkins master’s JVM
Much quicker than writing a plugin
Tweet @garethbowles with feedback!
Monday, October 28, 13
Using system Groovy scripts gives you full access to Jenkins’ JVM and object model just like a plugin, but the scripts are much easier to
write and maintain.
You get the benefit of Jenkins’ built in features for tracking script results, scheduling runs, notifying users of the results and visualizing
data (with the plot plugin for example).
Many of our needs aren’t generic enough to make it worth writing a plugin, anyway.
But it’s always good to script a task, even if you think it’s a one-off - that way if you’re wrong you get to re-run it, and you can track it
back easily if something goes wrong.

11. System Groovy Scripts
•
•
•
Full access to the Jenkins master’s JVM
Much quicker than writing a plugin
Make use of Jenkins' job scheduling, logging,
notification and data visualization
Tweet @garethbowles with feedback!
Monday, October 28, 13
Using system Groovy scripts gives you full access to Jenkins’ JVM and object model just like a plugin, but the scripts are much easier to
write and maintain.
You get the benefit of Jenkins’ built in features for tracking script results, scheduling runs, notifying users of the results and visualizing
data (with the plot plugin for example).
Many of our needs aren’t generic enough to make it worth writing a plugin, anyway.
But it’s always good to script a task, even if you think it’s a one-off - that way if you’re wrong you get to re-run it, and you can track it
back easily if something goes wrong.

12. System Groovy Scripts
•
•
•
Full access to the Jenkins master’s JVM
•
Needs the Groovy plugin: https://wiki.jenkinsci.org/display/JENKINS/Groovy+plugin
Much quicker than writing a plugin
Make use of Jenkins' job scheduling, logging,
notification and data visualization
Tweet @garethbowles with feedback!
Monday, October 28, 13
Using system Groovy scripts gives you full access to Jenkins’ JVM and object model just like a plugin, but the scripts are much easier to
write and maintain.
You get the benefit of Jenkins’ built in features for tracking script results, scheduling runs, notifying users of the results and visualizing
data (with the plot plugin for example).
Many of our needs aren’t generic enough to make it worth writing a plugin, anyway.
But it’s always good to script a task, even if you think it’s a one-off - that way if you’re wrong you get to re-run it, and you can track it
back easily if something goes wrong.

13. Examples
Tweet @garethbowles with feedback!
Monday, October 28, 13
Here are a few examples of the work we do with system Groovy scripts.
If a job hasn’t built successfully in 90 days or more, we disable it to reduce the number of wasted executions of broken jobs.
We can relabel a set of slaves with a given group of labels - useful if we want to update the configuration of our EC2 slaves, we can
remove labels from the old slaves so that no more builds are sent to them, then wait for builds to finish before terminating them.
We monitor our standard slave pool for builds that failed with remoting errors that can affect other builds, and disconnect / reconnect the
slaves (a reconnect usually works, otherwise we leave the slave offline for analysis).
We monitor custom build slaves and send notifications to the owners if they get disconnected.
We scan all enabled jobs to make sure that the number of builds to keep is set (the default is to keep all builds, which can chew up disk
space fast).
We make sure that all jobs have the “restrict where this build can be run” field set, so that Linux builds don’t end up on a Windows slave
for example.

14. •
Examples
Disable old or broken jobs
Tweet @garethbowles with feedback!
Monday, October 28, 13
Here are a few examples of the work we do with system Groovy scripts.
If a job hasn’t built successfully in 90 days or more, we disable it to reduce the number of wasted executions of broken jobs.
We can relabel a set of slaves with a given group of labels - useful if we want to update the configuration of our EC2 slaves, we can
remove labels from the old slaves so that no more builds are sent to them, then wait for builds to finish before terminating them.
We monitor our standard slave pool for builds that failed with remoting errors that can affect other builds, and disconnect / reconnect the
slaves (a reconnect usually works, otherwise we leave the slave offline for analysis).
We monitor custom build slaves and send notifications to the owners if they get disconnected.
We scan all enabled jobs to make sure that the number of builds to keep is set (the default is to keep all builds, which can chew up disk
space fast).
We make sure that all jobs have the “restrict where this build can be run” field set, so that Linux builds don’t end up on a Windows slave
for example.

15. •
•
Examples
Disable old or broken jobs
Relabel a set of slaves
Tweet @garethbowles with feedback!
Monday, October 28, 13
Here are a few examples of the work we do with system Groovy scripts.
If a job hasn’t built successfully in 90 days or more, we disable it to reduce the number of wasted executions of broken jobs.
We can relabel a set of slaves with a given group of labels - useful if we want to update the configuration of our EC2 slaves, we can
remove labels from the old slaves so that no more builds are sent to them, then wait for builds to finish before terminating them.
We monitor our standard slave pool for builds that failed with remoting errors that can affect other builds, and disconnect / reconnect the
slaves (a reconnect usually works, otherwise we leave the slave offline for analysis).
We monitor custom build slaves and send notifications to the owners if they get disconnected.
We scan all enabled jobs to make sure that the number of builds to keep is set (the default is to keep all builds, which can chew up disk
space fast).
We make sure that all jobs have the “restrict where this build can be run” field set, so that Linux builds don’t end up on a Windows slave
for example.

16. •
•
•
Examples
Disable old or broken jobs
Relabel a set of slaves
Monitor slave uptime
Tweet @garethbowles with feedback!
Monday, October 28, 13
Here are a few examples of the work we do with system Groovy scripts.
If a job hasn’t built successfully in 90 days or more, we disable it to reduce the number of wasted executions of broken jobs.
We can relabel a set of slaves with a given group of labels - useful if we want to update the configuration of our EC2 slaves, we can
remove labels from the old slaves so that no more builds are sent to them, then wait for builds to finish before terminating them.
We monitor our standard slave pool for builds that failed with remoting errors that can affect other builds, and disconnect / reconnect the
slaves (a reconnect usually works, otherwise we leave the slave offline for analysis).
We monitor custom build slaves and send notifications to the owners if they get disconnected.
We scan all enabled jobs to make sure that the number of builds to keep is set (the default is to keep all builds, which can chew up disk
space fast).
We make sure that all jobs have the “restrict where this build can be run” field set, so that Linux builds don’t end up on a Windows slave
for example.

17. •
•
•
•
Examples
Disable old or broken jobs
Relabel a set of slaves
Monitor slave uptime
Disconnect bad slaves
Tweet @garethbowles with feedback!
Monday, October 28, 13
Here are a few examples of the work we do with system Groovy scripts.
If a job hasn’t built successfully in 90 days or more, we disable it to reduce the number of wasted executions of broken jobs.
We can relabel a set of slaves with a given group of labels - useful if we want to update the configuration of our EC2 slaves, we can
remove labels from the old slaves so that no more builds are sent to them, then wait for builds to finish before terminating them.
We monitor our standard slave pool for builds that failed with remoting errors that can affect other builds, and disconnect / reconnect the
slaves (a reconnect usually works, otherwise we leave the slave offline for analysis).
We monitor custom build slaves and send notifications to the owners if they get disconnected.
We scan all enabled jobs to make sure that the number of builds to keep is set (the default is to keep all builds, which can chew up disk
space fast).
We make sure that all jobs have the “restrict where this build can be run” field set, so that Linux builds don’t end up on a Windows slave
for example.

18. •
•
•
•
•
Examples
Disable old or broken jobs
Relabel a set of slaves
Monitor slave uptime
Disconnect bad slaves
Set number of builds to keep
Tweet @garethbowles with feedback!
Monday, October 28, 13
Here are a few examples of the work we do with system Groovy scripts.
If a job hasn’t built successfully in 90 days or more, we disable it to reduce the number of wasted executions of broken jobs.
We can relabel a set of slaves with a given group of labels - useful if we want to update the configuration of our EC2 slaves, we can
remove labels from the old slaves so that no more builds are sent to them, then wait for builds to finish before terminating them.
We monitor our standard slave pool for builds that failed with remoting errors that can affect other builds, and disconnect / reconnect the
slaves (a reconnect usually works, otherwise we leave the slave offline for analysis).
We monitor custom build slaves and send notifications to the owners if they get disconnected.
We scan all enabled jobs to make sure that the number of builds to keep is set (the default is to keep all builds, which can chew up disk
space fast).
We make sure that all jobs have the “restrict where this build can be run” field set, so that Linux builds don’t end up on a Windows slave
for example.

19. •
•
•
•
•
•
Examples
Disable old or broken jobs
Relabel a set of slaves
Monitor slave uptime
Disconnect bad slaves
Set number of builds to keep
Check build labels
Tweet @garethbowles with feedback!
Monday, October 28, 13
Here are a few examples of the work we do with system Groovy scripts.
If a job hasn’t built successfully in 90 days or more, we disable it to reduce the number of wasted executions of broken jobs.
We can relabel a set of slaves with a given group of labels - useful if we want to update the configuration of our EC2 slaves, we can
remove labels from the old slaves so that no more builds are sent to them, then wait for builds to finish before terminating them.
We monitor our standard slave pool for builds that failed with remoting errors that can affect other builds, and disconnect / reconnect the
slaves (a reconnect usually works, otherwise we leave the slave offline for analysis).
We monitor custom build slaves and send notifications to the owners if they get disconnected.
We scan all enabled jobs to make sure that the number of builds to keep is set (the default is to keep all builds, which can chew up disk
space fast).
We make sure that all jobs have the “restrict where this build can be run” field set, so that Linux builds don’t end up on a Windows slave
for example.

20. How do I figure out what to write ?
Tweet @garethbowles with feedback!
Monday, October 28, 13
Figuring out where to start writing a script can be tricky, but there are lots of resources available.

21. Resources
Tweet @garethbowles with feedback!
Monday, October 28, 13
There are a lot of sample scripts out there that may well do what you want. The Scriptler plugin gives you access to a hosted script
catalog. There’s also a GitHub repo with many examples - I find this easier to update so that’s where I’ve put our examples.
You can try out your code using the Groovy script console provided by the Groovy plugin. Make sure you know what you’re doing !
The online JavaDoc for the Jenkins Java API is the source of truth, but can be tricky to navigate.
We wrote a simple object inspector method that shows you what type of object you’re working with.

22. •
Resources
Sample scripts - Scriptler plugin (https://
wiki.jenkins-ci.org/display/JENKINS/Scriptler
+Plugin) or GitHub repo (https://github.com/
jenkinsci/jenkins-scripts)
Tweet @garethbowles with feedback!
Monday, October 28, 13
There are a lot of sample scripts out there that may well do what you want. The Scriptler plugin gives you access to a hosted script
catalog. There’s also a GitHub repo with many examples - I find this easier to update so that’s where I’ve put our examples.
You can try out your code using the Groovy script console provided by the Groovy plugin. Make sure you know what you’re doing !
The online JavaDoc for the Jenkins Java API is the source of truth, but can be tricky to navigate.
We wrote a simple object inspector method that shows you what type of object you’re working with.

23. Resources
•
Sample scripts - Scriptler plugin (https://
wiki.jenkins-ci.org/display/JENKINS/Scriptler
+Plugin) or GitHub repo (https://github.com/
jenkinsci/jenkins-scripts)
•
Script console in Jenkins - be careful !
Tweet @garethbowles with feedback!
Monday, October 28, 13
There are a lot of sample scripts out there that may well do what you want. The Scriptler plugin gives you access to a hosted script
catalog. There’s also a GitHub repo with many examples - I find this easier to update so that’s where I’ve put our examples.
You can try out your code using the Groovy script console provided by the Groovy plugin. Make sure you know what you’re doing !
The online JavaDoc for the Jenkins Java API is the source of truth, but can be tricky to navigate.
We wrote a simple object inspector method that shows you what type of object you’re working with.

24. Resources
•
Sample scripts - Scriptler plugin (https://
wiki.jenkins-ci.org/display/JENKINS/Scriptler
+Plugin) or GitHub repo (https://github.com/
jenkinsci/jenkins-scripts)
•
•
Script console in Jenkins - be careful !
API JavaDoc: http://javadoc.jenkins-ci.org/
overview-summary.html
Tweet @garethbowles with feedback!
Monday, October 28, 13
There are a lot of sample scripts out there that may well do what you want. The Scriptler plugin gives you access to a hosted script
catalog. There’s also a GitHub repo with many examples - I find this easier to update so that’s where I’ve put our examples.
You can try out your code using the Groovy script console provided by the Groovy plugin. Make sure you know what you’re doing !
The online JavaDoc for the Jenkins Java API is the source of truth, but can be tricky to navigate.
We wrote a simple object inspector method that shows you what type of object you’re working with.

25. Resources
•
Sample scripts - Scriptler plugin (https://
wiki.jenkins-ci.org/display/JENKINS/Scriptler
+Plugin) or GitHub repo (https://github.com/
jenkinsci/jenkins-scripts)
•
•
Script console in Jenkins - be careful !
•
Object inspector method
API JavaDoc: http://javadoc.jenkins-ci.org/
overview-summary.html
Tweet @garethbowles with feedback!
Monday, October 28, 13
There are a lot of sample scripts out there that may well do what you want. The Scriptler plugin gives you access to a hosted script
catalog. There’s also a GitHub repo with many examples - I find this easier to update so that’s where I’ve put our examples.
You can try out your code using the Groovy script console provided by the Groovy plugin. Make sure you know what you’re doing !
The online JavaDoc for the Jenkins Java API is the source of truth, but can be tricky to navigate.
We wrote a simple object inspector method that shows you what type of object you’re working with.

26. Resources
•
Sample scripts - Scriptler plugin (https://
wiki.jenkins-ci.org/display/JENKINS/Scriptler
+Plugin) or GitHub repo (https://github.com/
jenkinsci/jenkins-scripts)
•
•
Script console in Jenkins - be careful !
•
Object inspector method
API JavaDoc: http://javadoc.jenkins-ci.org/
overview-summary.html
Tweet @garethbowles with feedback!
Monday, October 28, 13
There are a lot of sample scripts out there that may well do what you want. The Scriptler plugin gives you access to a hosted script
catalog. There’s also a GitHub repo with many examples - I find this easier to update so that’s where I’ve put our examples.
You can try out your code using the Groovy script console provided by the Groovy plugin. Make sure you know what you’re doing !
The online JavaDoc for the Jenkins Java API is the source of truth, but can be tricky to navigate.
We wrote a simple object inspector method that shows you what type of object you’re working with.

27. Object Inspector Method
Tweet @garethbowles with feedback!
Monday, October 28, 13
Here’s the handy object inspector method that display an object’s type, properties and methods. Once yuo know the type you can look in
the JavaDoc or search existing code to find out how to do what you want.

28. Be careful about testing !
Tweet @garethbowles with feedback!
Monday, October 28, 13
System Groovy scripts can do some serious damage ! You often want to loop over every job or build or slave, so be careful to test your
code.
I learned the hard way when I accidentally kicked off every job on our main Jenkins master.
We’ve made a few other mistakes too.

29. Tweet @garethbowles with feedback!
Monday, October 28, 13
Here’s another one where we accidentally disabled all the jobs in the system and then had to figure out how to identify the jobs that
should really be disabled. We keep this one around as a cautionary tale.

30. Developing scripts
Tweet @garethbowles with feedback!
Monday, October 28, 13
Always test your scripts on a local Jenkins instance; it’s pretty easy to create dummy jobs, builds and slaves.
For short scripts you can edit within Jenkins, but longer ones benefit from using a code editor or IDE with syntax highlighting and code
completion - way fewer missing brackets !
Once you’re done, check the code into SCM and have your Jenkins job sync it, so that you can keep the change history and you have a
backup of the code outside of Jenkins.

31. Developing scripts
•
Use a local Jenkins instance (java -jar
jenkins.war)
Tweet @garethbowles with feedback!
Monday, October 28, 13
Always test your scripts on a local Jenkins instance; it’s pretty easy to create dummy jobs, builds and slaves.
For short scripts you can edit within Jenkins, but longer ones benefit from using a code editor or IDE with syntax highlighting and code
completion - way fewer missing brackets !
Once you’re done, check the code into SCM and have your Jenkins job sync it, so that you can keep the change history and you have a
backup of the code outside of Jenkins.

32. Developing scripts
•
Use a local Jenkins instance (java -jar
jenkins.war)
•
Use the script console or inline code in the job
config. while developing
Tweet @garethbowles with feedback!
Monday, October 28, 13
Always test your scripts on a local Jenkins instance; it’s pretty easy to create dummy jobs, builds and slaves.
For short scripts you can edit within Jenkins, but longer ones benefit from using a code editor or IDE with syntax highlighting and code
completion - way fewer missing brackets !
Once you’re done, check the code into SCM and have your Jenkins job sync it, so that you can keep the change history and you have a
backup of the code outside of Jenkins.

33. Developing scripts
•
Use a local Jenkins instance (java -jar
jenkins.war)
•
Use the script console or inline code in the job
config. while developing
•
Code editor or IDE for longer scripts
Tweet @garethbowles with feedback!
Monday, October 28, 13
Always test your scripts on a local Jenkins instance; it’s pretty easy to create dummy jobs, builds and slaves.
For short scripts you can edit within Jenkins, but longer ones benefit from using a code editor or IDE with syntax highlighting and code
completion - way fewer missing brackets !
Once you’re done, check the code into SCM and have your Jenkins job sync it, so that you can keep the change history and you have a
backup of the code outside of Jenkins.

34. Developing scripts
•
Use a local Jenkins instance (java -jar
jenkins.war)
•
Use the script console or inline code in the job
config. while developing
•
•
Code editor or IDE for longer scripts
Check the script into SCM when it’s ready
Tweet @garethbowles with feedback!
Monday, October 28, 13
Always test your scripts on a local Jenkins instance; it’s pretty easy to create dummy jobs, builds and slaves.
For short scripts you can edit within Jenkins, but longer ones benefit from using a code editor or IDE with syntax highlighting and code
completion - way fewer missing brackets !
Once you’re done, check the code into SCM and have your Jenkins job sync it, so that you can keep the change history and you have a
backup of the code outside of Jenkins.

35. Awesome ! Where can I get
this stuff ?
•
•
https://github.com/jenkinsci/jenkins-scripts
http://scriptlerweb.appspot.com/catalog/list
Tweet @garethbowles with feedback!
Monday, October 28, 13
Here’s where you can get lots of System Groovy script goodness. I just added a bunch of our scripts that I mentioned in this talk.

36. Jenkins User Conference
Palo Alto , Oct 23 2013
Thank You To Our Sponsors
Platinum
Gold
Silver
Tweet @garethbowles with feedback!
Monday, October 28, 13
#jenkinsconf

37. Thank You !
Email: gbowles@{gmail,netflix}.com
Twitter: @garethbowles
Linkedin: www.linkedin.com/in/garethbowles
Want to do Jenkins development for your
day job ? We’re hiring ! http://
jobs.netflix.com
Tweet @garethbowles with feedback!
Monday, October 28, 13
Here’s how to get in touch.
We’re hiring for lots of positions, but in particular we’re looking for someone to own Jenkins at Netflix - all the way from keeping the
infrastructure running to writing custom plugins and fixing the Jenkins core. If you want to take Jenkins in new directions as your full
time job - with a heavy dose of cloud thrown in - come and find a Netflixer.
thanks again