Presentation section from Splunk Live content

1. Splunk for JMX
Damien Dallimore
Developer Evangelist

2. Copyright©2013,SplunkInc.
Splunk for JMX
2
• ConnecttoanylocalorremoteJVM'sJMXserver,Hotspot/JRockit/IBMJ9/OpenJDK/AzulZing
• QueryanyMBeanrunningonthatserver
• ExtractanyMBeanattributes(simple,compositeortabular)
• InvokeMBeanoperations
• Writeattributesandoperationresultsoutinadefaultkey/valueformat,orpluginyourowncustomformat,forSPLUNK
indexingandsearching
• TransporteventsoverSTDOUT(default),TCP,Syslog,SplunkRESTendpointordirecttofile.
• DeclareclustersofJVM'sforlargerscaleJVMdeployments
• Runson*NixandWindows
• OutoftheboxdashboardsforcommonJVMMBeans
• FreelyavailablefromSplunkBase,allsourcecodeisonGitHub

3. Copyright©2013,SplunkInc.
Connectivity Options
3
• Remote JMX interface
– rmi (JSR160StandardImplementationandMX4J'sJSR160Implementation)
– iiop(JSR160StandardImplementationandMX4J'sJSR160Implementation)
• Direct Process attachment
– ConnectdirectlytoalocallyrunningJVMprocess
• MX4J HTTP connectors (requires MX4J in the target JVM also)
– soap,soap+sssl
– hessian,hessian+ssl
– burlap,burlap+ssl

4. Copyright©2013,SplunkInc.
Setup and Configuration
4
• The main goalof the app was to make it as simple and intuitive as possible to connect to
your JVMs and start Splunking JMX data
• Enableyour targetJVM’s remoteJMX interface, test connectivitywith JConsole
• InstallSplunk for JMX
• ExtractSplunkforJMXtarballtoSPLUNK_HOME/etc/apps
• RestartSplunk
• Atthesetupscreen,choosea scriptedinputforyourplatform(Nix/Windows)
• Setup your JMX configurationfile
• Thedefaultconfig.xmlfileispreconfiguredforcommonJVMMBeans
• BrowseyourJVM(usingJConsole)forotherMBeansthatyouwishtopollandconfigurethese
• Youcanhaveasmanyconfigfilesasyourequire,andyoumightsettheseuptofireoffatdifferent
scheduledfrequencies

5. Copyright©2013,SplunkInc.
Configuration Examples - Simple
5

6. Copyright©2013,SplunkInc.
Configuration Examples - Wildcards
6
MBean Object name format “domain:key=value,key2=value2”
Around 25KBytes per dump on Hotspot JVMs
Don’t know the domain or properties name or have many Mbeans
with the same attributes ?
* and ? wildcards are supported in the Mbean name

7. Copyright©2013,SplunkInc.
Configuration Examples - Clusters
7
• Define clusters of JVM’s that share the
same MBean definitions
• Note , in these examples, for brevity I am
using “dumpAllAttributes” , but in
production you’d want to pick and
choose specific MBean attributes you
are interested in, and perhaps split
definitions over multiple files run at
varying frequencies

8. Copyright©2013,SplunkInc.
Configuration Examples - Operations
8
• Invoke JMX operations that return a value or simply
perform some action on the target JVM
• Operation definitions can take parameters
Use Case 1 : your developers might code a JMX
operation that returns a CSV or JSON formatted
snapshot of some metrics for Splunking
Use Case 2 : dynamically trigger HPROF dumps.
The “com.sun.management:type=HotSpotDiagnostic”
Mbean exposes a “dumpHeap” operation

9. Copyright©2013,SplunkInc.
Configuration Examples - Connecting
9
• IP Address with credentials
• Hostname
• Static Process ID
• Process ID lookup from file
• Process ID lookup from command output
• Raw JMX Service URL
• MX4J HTTP Connector

10. Copyright©2013,SplunkInc.
Custom Formatters/Transports
10
• The Splunk for JMX configuration is user extensible
• You can code and configure your own Formatters and Transports
Formatters
• Takes the raw MBean polled output and formats it for Splunking
• A Java implementation of the "com.dtdsoftware.splunk.formatter.Formatter" interface
• If the optional formatter declaration is omitted, then the default formatter will be used
Transports
• Takes the formatted output and transports it to a destination
• A Java implementation of the "com.dtdsoftware.splunk.transport.Transport" interface
• If the optional transport declaration is omitted, then the default transport(STD out) will be used

11. Copyright©2013,SplunkInc.
Formatter Examples
11

12. Copyright©2013,SplunkInc.
Transport Examples
12

13. Copyright©2013,SplunkInc.
Deployment Architectures 1
13
• Simplest scenario
• Monolithic Splunk installation
• Splunk for JMX polling 1 or more remote/local JVMs via
the remote JMX interface
• There is support for many target JVM’s in the
configuration schema but to really scale out, you need
a more advanced Splunk architecture

14. Copyright©2013,SplunkInc.
Deployment Architectures 2
14
Splunk UF running locally
with target JVM
Splunk Indexer
Cluster
Splunk Search Head
Pool
Load Balancer
• Run Splunk UF locally with target
JVM.Can connect use remote JMX
interface or direct process attachment.
• Each tier scales out horizontally.
• Can overcome firewall issues that are
sometimes inherent with Java RMI
• Deploy Splunk for JMX components and
configurations with Splunk Deployment
Server, Puppet or Chef.

15. Demo

16. Copyright©2013,SplunkInc.
Contact me
16
Email : ddallimore@splunk.com
Twitter : @damiendallimore
Skype : damien.dallimore
Github : damiendallimore
Splunkbase : damiend
Slideshare : http://www.slideshare.net/damiendallimore
Blogs : http://blogs.splunk.com/dev
Web : http://dev.splunk.com