The document discusses using Splunk to instrument Oracle Applications. Splunk is presented as a solution to federate telemetry and log data from different systems in various locations to help troubleshoot issues that may have root causes buried in difficult to find locations. The key components of Splunk - forwarders, indexers, and search heads - are described. Features like interactive searching, field extractions, data classification, and charts are highlighted. A use case is provided for monitoring garbage collection using a script to parse jstat output and index it in Splunk. Steps for implementing such a script are outlined.
7. 11/17/2016 7www.datavail.com
The Problem…
Root causes are quite often buried and difficult to find.
What if there was a neat way to federate the telemetry from all your key
pieces? Including the contents of your critical logs?
10. 11/17/2016 10www.datavail.com
What is Splunk?
Splunk is an enterprise application.
Splunk was made to monitor.
Splunk is agnostic.
Splunk is a framework!
11. 11/17/2016 11www.datavail.com
What Splunk isn’t
Not a collection of purchased lock-downed modules (Solar Winds etc.)
It’s not an application that lives on a database.
It’s not an out of the box solution.
12. 11/17/2016 12www.datavail.com
Splunk’s Pieces
Forwarder
• Installed on the server
• Light weight
• Has a watch list
• Basic filtering
• Basic classification
• Looks for file changes
• Looks for new files
• Sends it all to the Indexer
16. 11/17/2016 16www.datavail.com
Splunk’s Features
Interactive Searching
• Has robust SPL to search with.
• Combination of grep, regex, and custom functions like eval, average,
• Has grouping and de-dupe functions
• Searches can be saved as carts or reports.
19. 11/17/2016 19www.datavail.com
Splunk’s Features
REGEX Field Extractions
Splunk extracts your KPIs using REGEX quickly parsing through your
collected data to identify the information you are most interested in.
(?i) ORA-(?P<OracleAlertError>.+)
26. 11/17/2016 26www.datavail.com
Scripted Inputs For GC Monitoring
Use Cases
• Access data that is not available as an ordinary file.
• Access data that cannot be sent using TCP or UDP.
• Stream data from command-line tools, such as SQLPlus.
• Reformat complex data so you can more easily parse the data into events and fields.
• Attach a timestamp to transient data such as iostat.
Methods of Implementation
• Shell Scripts
• Batch Files
• Python/Perl Scripts
• Command Line Output
• Anything that writes to STDOUT
Methods of Capture
• Direct from STDOUT
• Write a file to be indexed
27. 11/17/2016 27www.datavail.com
Forwarder Recap
The forwarder is the beginning of
classification.
Source, source types, hosts, and
filenames are collected and sent
to the indexer.
Forwarders are self contained.
They do not have to be ‘installed’.
They can run as any OS user.
Forwarders run on Windows,
*NIX, and OSX
[monitor:///.../opmn]
disabled = false
sourcetype = OPMNLogs
index=euebp
ignoreOlderThan = 7d
[monitor:///.../Apache/access_log*]
disabled = false
sourcetype = ApacheAccessLogs
index=euebp
ignoreOlderThan = 7d
[script://$SPLUNK_HOME/etc/apps/<appNam
e>/bin/rcat.sh]
disabled = false
host = rmanhost
index = main interval = 30 #frequency to run
the script, in seconds
sourcetype = RMAN
28. 11/17/2016 28www.datavail.com
Garbage Collection Monitor Use Case
Based On jstat
jstat –gcutil <ospid>
Use perl to parse for pid and format into this.
11-03-2016 07:05 OSPid=28733 Proc=forms-c4ws_server1 S0=0.00 S1=8.99 E=56.21 O=66.80 P=85.68 YGC=83
YGCT=13.092 FGC=21 FGCT=80.713 GCT=93.80511-03-2016 07:05
30. 11/17/2016 30www.datavail.com
Activate the Script
Inputs.conf Stanza
[script://$SPLUNK_HOME/etc/apps/<appName>/bin/jstat.sh]
disabled = false
host = rmanhost
index = main interval = 30 #frequency to run the script, in
seconds
sourcetype = RMAN
33. 11/17/2016 33www.datavail.com
Splunk Installation Reqs
Splunk can be downloaded for free. You will be limited to 500mb of
ingested data a day. This is actually can be sufficient for one environment
if you are judicious about what you log.
Splunk for *NIX can be downloaded in tar ball format. This can be
unpacked in any directory and does not have to be installed by or run by
root.
Splunk stores everything in its self contained path, so you just have to
delete the directory to remove it.
You can install Splunk directly on the system you wish to monitor (not
always a good idea)
Splunk uses ports 8000 and higher for the browser and the forwarders so
again, no root user is needed.
34. 11/17/2016 34www.datavail.com
Existing Splunk?
If your company already uses Splunk you can ask your admin for the
following.
• Your own index (The index is actually the directory structure and files
Splunk uses to store the data).
• Additional forwarders, or…
• If your server already has a forwarder on it, you just need to get your log
locations added to inputs.conf and sent to your index.
• Your own Application. An application is just a collection of settings like
searches, chart and report descriptions and etc. This way you won’t
interfere with the network and security guys.