Conduct a few internal pen tests and you’re bound to come across Jenkins, the world’s most popular build automation server. When you encounter it, what do you do? Go beyond a 5-minute Google search and checking for open script consoles. This talk dives into various ways to exploit Jenkins and how to move laterally into sensitive systems.

1. 1
Intro to Pentesting Jenkins
February 6th, 2020
Brian D. Hysell

2. 2
• Consultant @ Synopsys Software Integrity Group
– Application Pentesting, Network Pentesting, Secure Code Review / Guidelines / Training, etc.
– Mostly just grep for passwords, rack up hotel status, write reports
• Previously: Consultant @ Accenture Security Threat & Vulnerability Management
• Like TV, travel, baking
About Me

3. 3
Intro to Jenkins
• About Jenkins
• Visual Tour
• Why Care about Jenkins?
• Finding Instances
• Initial Access

4. 4
About Jenkins
• Most popular CI/CD system – probably 50-
70% marketshare
– Open source
– Rich plugin ecosystem
– Long history of security vulnerabilities
• Forked(-ish) from project called Hudson
• Used for all sorts of common dev (& often
ops) tasks:
– Running builds, tests
– Publishing built artifacts (NRM, Artifactory, etc.)
– Deploying software
– Checking uptime/health
– Stopping/starting/restarting servers
– Applying patches

5. 5
Dashboard (https://gazelle.ihe.net/jenkins/)
Visual Tour

6. 6
Project
Visual Tour

7. 7
Workspace
Visual Tour

8. 8
Build
Visual Tour

9. 9
Securing Jenkins (internal or external) should be a major concern across red, purple, and blue
teams:
• Supply-Chain Integrity: Compromise could lead to backdoored apps deployed to production or
customer environments
• Intellectual Property Protection: Access to Jenkins can give an (possibly malicious) user
source code, binaries, and knowledge of infrastructure/architecture
• Privileged Access Management: Instances often contain credentials for source & artifact
repositories, test & production systems, etc.
• Segmentation: Jenkins supports distributed executors which may span different environments
(dev/prod, PCI/non-PCI, different AD domains, etc.)
Why Care about Jenkins?

10. 10
Finding Instances
• Usually port 8080, also check 8443,80,443,
other alternates like 9090,8888
• Nmap Script Engine (NSE) (see right)
• HTTP screenshot tools like Aquatone or
WitnessMe
Script Name Info
http-title Look for titles containing
“[Jenkins]”, e.g., “Sign In
[Jenkins]” or “Dashboard
[Jenkins]” (← anonymous
read!)
http-server-header Jenkins will usually use
Jetty
http-enum In case of context path
issues
http-headers Look for X-Hudson* and X-
Jenkins* headers
http-devframework Includes fingerprints for
Jenkins (based on headers)
broadcast-jenkins-discover See “Auto-discovering
Jenkins on the network”

11. 11
Initial Access
• Some level of anonymous access is common,
especially internally
– Typically read-only
– Sometimes administrative!
• Check if sign-ups are enabled
• Auth bypass: discussed in next section
• Conventional techniques: password spraying,
brute force, credential stuffing
• Get creative; I have found Jenkins creds in:
– Build logs of another Jenkins instance
– Job env vars on another Jenkins instance
– Python scripts in company Artifactory install
– Entry point of a Docker container running on a
compromised box
• Look for both passwords and API keys
From http://tuyenvuong.me/programming/deploy-web-application-jenkins-capistrano/ &
https://commons.wikimedia.org/wiki/File:Emojione_1F630.svg

12. 12
Scripting & Exploitation
• Selected RCE Attack Vectors
• Script Console
• Build Steps
• Dynamic Routing

13. 13
Selected RCE Attack Vectors
Vector CVE Access Level
Script Console N/A • Overall/RunScripts enabled for anonymous; or
• Compromised credentials for user with
Overall/RunScripts
Build Step N/A • Compromised credentials for user with Job/Build
and Job/Configure permissions
Script Security Plugin
doCheckScript method
CVE-2019-1003029,
CVE-2019-1003005
(POC)
• Overall/Read access enabled for anonymous; or
• Compromised credentials; or
• Authentication bypass CVE-2018-1000861
Jenkins Remoting
(deserialization)
CVE-2015-8103 (POC1) • Jenkins CLI port open
1 Protips: https://github.com/NickstaDB/ysoserial/tree/templatesimpl-reverse-shell &
https://github.com/BrianOfTheCosmos/broodwich/blob/master/deserialization/src/main/java/party/itistimeto/broodwich/deserialization/CommonsCollectionsThreePayload.java

14. 14
Script Console
• Script console
– Run arbitrary Groovy code: read/modify Jenkins
internals, read/write files, run external commands
– Requires Overall/RunScripts permission – should
be restricted to admins, but could be applied to
lower-privileged users, even anonymous (!)
• REST API is convenient:
https://support.cloudbees.com/hc/en-
us/articles/217509228-execute-groovy-with-a-
rest-call
– On older versions (<2.96), need Jenkins-Crumb
header from /crumbIssuer/api/json:
https://wiki.jenkins.io/display/JENKINS/Remote+a
ccess+API#RemoteaccessAPI-CSRFProtection
• Jenkins CLI can also be nice (useful for other
scripting / enumeration tasks as well)
• Generic Java RCE tip: make sure to use
Process.waitFor()
From https://2019.pass-the-salt.org/files/slides/12-Hacking_Jenkins.pdf
From https://issues.jenkins-ci.org/browse/JENKINS-46526

15. 15
Build Steps
• See if you can create or edit (and then run -- separate
permissions!) projects/builds
• Builds can contain “Execute shell” or “Execute
Windows batch command” steps with code of your
choice
• Can choose which node to run build on, including
master:
https://stackoverflow.com/questions/9841156/stick-
jobs-to-the-master
• Be careful:
– If you create a project, delete it after building (if able)
– If you have to edit an existing project’s build steps:
– Back up config (can do with Jenkins CLI)
– Disable email, Slack, etc. notifications
– Don’t deploy anything
– Check relationships to other projects, triggered builds,
etc.
– Etc.
Overview
From https://docs.bitnami.com/installer/apps/jenkins/configuration/use-custom-executable/

16. 16
Build Steps
• Many plugins allow execution of user-defined Groovy code,
usually as part of build
– E.g., Groovy plugin, workflow-cps (“Pipeline: Groovy”) plugin
– Other examples:
https://wiki.jenkins.io/display/JENKINS/Script+Security+Support+i
n+Plugins
• Most integrate with Script Security plugin: scripts must be
approved by admin or run in sandbox
– E.g., Groovy plugin’s “Execute system Groovy script” build step,
“Use Groovy Sandbox” option
• Sandbox blacklists various classes/methods; see
https://github.com/jenkinsci/script-security-
plugin/blob/master/src/main/resources/org/jenkinsci/plugins/script
security/sandbox/whitelists/
• Sandbox escapes frequently identified; check regression tests in
script-security repo for POC:
– SECURITY-1579: https://github.com/jenkinsci/script-security-
plugin/commit/415b6e2f3fa0c2e4bd2f9c4a589a9e1fc9cbac8b
– SECURITY-1658: https://github.com/jenkinsci/script-security-
plugin/commit/0e7da14171ed1d03ff72f6910392e630b40a8590
• Sandbox escapes occasionally applicable with only Overall/Read
(!), see https://jenkins.io/security/advisory/2017-04-10/
Sandbox Escape
Hint: Cause an exception to print output, e.g.,
java.nio.file.Paths.get(expression + "0") → InvalidPathException

17. 17
Dynamic Routing
• Jenkins uses a web framework called Stapler
– Same original author as Jenkins (née Hudson)
• Stapler parses URLs & calls methods in an
EL-like manner against a root object fetched
with
ServletContext.getAttribute("app")
• hudson.WebAppMain.contextInitialized
(): “Creates the sole instance of
jenkins.model.Jenkins and register it to
the ServletContext”
• TLDR: URLs in Jenkins are chains of getter
and doer method calls against a Jenkins
singleton or objects returned higher in the
chain
Introduction
From https://jenkins.io/doc/developer/handling-requests/routing/
Note: if, like I did, you’re wondering about the inconsistency between Jenkins#getLog() and Hudson#getJob()
(i.e., which is actually root?): Hudson is a subclass of Jenkins, and it’s an object of that type that
hudson.WebAppMain.contextInitialized() registers as root

18. 18
• Exploit URL:
/descriptorByName/org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScript/checkScript?sandbox
=True&value=[groovy code]
– (Jenkins implements DescriptorByNameOwner) ServletContext.getAttribute("app").
– (SecureGroovyScript)
getDescriptorByName("org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScr
ipt").
– (FormValidation) doCheckScript("[groovy code]", true)
• Part of Script Security plugin: not core Jenkins, but very common
• Prerequisites: vulnerable plugin & Jenkins versions, read access (including anonymous)
• Brian’s super-duper top-secret protip: public class x{public
x(){org.kohsuke.stapler.Stapler.getCurrentResponse().getWriter().println("hello world")}}
CVE-2019-1003029/CVE-2019-1003005 (doCheckScript)
Dynamic Routing

19. 19
• Exploit URL: /securityRealm/user/admin/descriptorByName/…
– (Jenkins) ServletContext.getAttribute("app").
– (HudsonPrivateSecurityRealm) getSecurityRealm().
– (User implements DescriptorByNameOwner) getUser("admin").
– (Descriptor) getDescriptorByName("...")
• https://devco.re/blog/2019/01/16/hacking-Jenkins-part1-play-with-dynamic-routing-en/
• Poorly-documented aspect of bypass: getUser(String) is method of HudsonPrivateSecurityRealm
(default implementation), not of SecurityRealm abstract class
CVE-2018-1000861 (Authentication Bypass Exploit)
Dynamic Routing

20. 20
Unauth RCE with CVE-2019-1003029/CVE-2019-1003005 (doCheckScript)
Dynamic Routing
Diagram from: https://2019.pass-the-salt.org/files/slides/12-Hacking_Jenkins.pdf
See also: https://github.com/orangetw/awesome-jenkins-rce-2019, https://github.com/orangetw/awesome-
jenkins-rce-2019/pull/2

21. 21
Post-Exploitation & Lateral Movement
• Embracing Groovy
• Lateral Scripting
• Scraping Jenkins
• Raiding Stored Credentials
• “Persistence” with API Keys
• Password Cracking
• Miscellaneous Post-Exploitation

22. 22
Embracing Groovy
Operation Code
Spawn reverse shell https://gist.github.com/frohoff/fed1ffaab9b9beeb1c76
List directory
contents (non-
recursive)
new File('.').eachFile { println(it) }
Print file contents println new File('/path/to/file').getText('UTF-
8')
Nslookup println
InetAddress.getByName('google.com').getHostAddr
ess()
• Groovy is a JVM-based scripting language
• Essentially a superset of Java: supports Java
syntax and APIs, plus its own syntax
extensions and standard library
• Some things you might normally do by
dropping into an OS shell you can do straight
in Groovy
Basics

23. 23
Embracing Groovy
Jenkins Internals
Programmatically add API token
From https://support.cloudbees.com/hc/en-us/articles/115003090592-How-to-re-
generate-my-Jenkins-user-token
List all admin users
From https://stackoverflow.com/a/34274738
If only these could be combined somehow… 😉
More script samples:
• https://github.com/cloudbees/jenkins-scripts/
• https://github.com/samrocketman/jenkins-script-console-scripts

24. 24
Lateral Scripting
• Script console (when you have permission) &
build steps can be run both on Jenkins
instance itself and on build nodes/agents
• Usually connected through either:
– Master → agent SSH connection
– Agent → master TCP connection
• Agents may be different:
– OS (e.g., Linux master with [potentially domain-
joined] Windows agent for C# projects)
– Environment/segment (e.g., master in dev,
agent in / ACL’d to prod [CDE?!] for
deployment)
If you have Overall/RunScripts access, the “Script Console” menu item will show up for a specific machine when you
go to its page (i.e., click it)

25. 25
Scraping Jenkins
• Internal Jenkins instances very often have
Overall/Read permission set for anonymous users
– Compromised credentials, sign-ups apply here too, of
course
• “Just” read access to Jenkins can be a goldmine
– Build logs
– Environment variables
– Project workspace (source code)
– Build artifacts
• REST API is easiest, no need to parse HTML
• Tools:
– https://github.com/DolosGroup/Jenkins-Pillage: build
logs, env vars; only lists URLs
– https://github.com/lc/jenkinz: build logs
– https://github.com/gquere/pwn_jenkins/blob/master/dum
p_builds/jenkins_dump_builds.py: build logs, env vars
– Jenkins CLI client + Bash
Overview & Tools
From https://medium.com/@aseem.shrey/mind-your-logs-how-a-build-log-from-a-jenkins-leaked-everything-
603cf07fa85

26. 26
• No magic, just search/grep for things like:
– “-p”, “--password”
– “Authorization: Basic”
– “net use” and similar (Windows creds)
– “jenkins-cli.jar” (Jenkins API keys)
– eyJ… (JWT)
• Have found lots of creds this way: domain users, Jenkins API keys w/ Overall/RunScripts, SSH
creds, creds to other parts of deployment infra
• Cannot emphasize enough how useful searching through this info can be for lateral movement
• Don’t forget to check workspaces too
– Artifacts if you’re desperate
Finding Credentials
Scraping Jenkins

27. 27
Raiding Stored Credentials
• Jenkins has a built-in mechanism to store credentials used by build pipelines
• Symmetrically encrypted with key stored on disk
• With script access: run a Groovy script like https://github.com/DolosGroup/Jenkins-
Pillage/blob/master/groovy/decrypt-credentials.groovy
• With file system access: grab secrets/master.key, secrets/hudson.util.Secret, and
credentials.xml (or other files containing encrypted data)
– https://github.com/gquere/pwn_jenkins#files-to-copy-after-compromission
– https://github.com/gquere/pwn_jenkins/blob/master/offline_decryption/jenkins_offline_decrypt.py
– https://gist.githubusercontent.com/menski/8f9980999ed43246b9b2/raw/ebac1b809ac4c10d4aa2c49b
8bb2a6c37dc4808a/jenkins-decrypt.py
– https://github.com/tweksteen/jenkins-decrypt/blob/master/decrypt.py
• N.B.: many plugins erroneously store passwords in plain-text config files and/or print them into
the value attributes of password inputs on the config page
Built-in Storage

28. 28
Raiding Stored Credentials
Plugins
From https://plugins.jenkins.io/mask-passwords

29. 29
“Persistence” with API Keys
• Jenkins <= 2.128: API keys are stored in users/<USERNAME>/config.xml under same
symmetric encryption scheme as credentials (may also apply to newer versions with legacy
tokens carried over in upgrades)
– Decrypt offline with aforementioned scripts
– Or read value and then decrypt with println(hudson.util.Secret.decrypt("{...}"))
– Or do 100% in-memory with script & Jenkins internal classes
• Newer versions are hashed, but support multiple tokens per user
– See previous slide for adding new tokens

30. 30
Password Cracking
• Potentially helpful for lateral movement to other Jenkins instances, other dev infrastructure
(Artifactory, Nexus Repository Manager, GitHub Enterprise, JIRA, etc.)
• Hashes stored in $JENKINS_HOME/users/[username]/config.xml
• Modern versions of Jenkins use bcrypt
– Potentially still worth it to try with a targeted wordlist + rules: jenkins, [company name], scm,
password, etc.
• Older versions & users that have not logged in since upgrade (I think?) use salted SHA256
– Format: salt:sha256("password{salt}") (from https://www.philipp.haussleiter.de/2011/11/jenkins-
password-hash-format/)
– Crack by replacing all salts with {salt} and using password.salt as format
– echo "salt:hash" | awk -F: '{print $2":{"$1"}"}'

31. 31
Miscellaneous Post-Exploitation
• In theory, even when Jenkins does not creds to
directly deploy artifacts, you may be able to
move laterally by backdooring them
– E.g., push backdoored JAR to private Maven repo
& wait for it to be downloaded / deployed by
another person / system
– Probably too risky & too time-consuming for most
pentests; difficult to control scope
– This & similar attacks should still be kept in mind
while threat modeling, though
• Jenkins can serve arbitrary static files out of
$JENKINS_HOME/userContent – may be good
for exfiltration
• Regular user session or remember me cookies
and API keys can (Usually? Always?) be used
interchangeably
– Regular user session can access API
– API key can access UI

32. 32
Conclusion
• Real-World Attack Paths
• Recommendations for Security & Ops Engineers

33. 33
• Searched for “jenkins” in an artifact store
• Found several Python scripts containing Jenkins usernames and API keys
• Hostnames in script incorrect/down, so sprayed against known Jenkins instances
• Discovered one valid instance-username-key combination
• Enumerated projects with Jenkins CLI; had configure access to one
• Backed up config; replaced with system Groovy build step: sandbox escape, add & print API key
for all admin users
• Used admin key to access “Configure System”; Mask Passwords Plugin config contained
password for master deployment service account in clear text
Example #1
Real-World Attack Paths

34. 34
• Found Jenkins instance A with anonymous read enabled
• Scraped build logs & found API creds for Jenkins instance B
• Used script console (over REST API) to grab SSH private key from agent X (connected to B)
• SSH’d into Jenkins instance C, then from instance C into agent Y (on PCI network)
• Using key on agent Y, SSH’d into prod PCI hosts
Example #2
Real-World Attack Paths

35. 35
• Inventory Jenkins systems and patch, patch, patch!
• Apply least privilege via matrix-based authorization (preferably project-based matrix)
– Never have anonymous script console access
– Even “Logged-in users can do anything” is too risky
– Anonymous read access may not be terribly risky per se – but disabling is an easy win
• Don’t leak creds in build logs, environment variables, workspaces*, artifacts*, etc.
– *Dependent on your org’s maturity WRT credential management
• Pay requisite attention to Jenkins instances with privileged access:
– Access to prod, CDE, publicly-released builds, etc.
– Access to master deployment keys/passwords/accounts 
Recommendations for Security & Ops Engineers

36. 36
Questions?
BrianHysell
BrianOfTheCosmos