5. Traditional Data Sources
§ Captures events from log files in real time
§ Runs scripts to gather system metrics, connect
to APIs and databases
§ Listens to syslog and gathers Windows events
§ Universally indexes any data format so it
doesn’t need adapters
5
Windows
• Registry
• Event logs
• File system
• sysinternals
Linux/Unix
• Configurations
• Syslog
• File system
• Ps, iostat, top
Virtualization
• Hypervisor
• Guest OS
• Guest Apps
Applications
• Web logs
• Log4J, JMS, JMX
• .NET events
• Code and scripts
Databases
• Configurations
• Audit/query logs
• Tables
• Schemas
Network
• Configurations
• syslog
• SNMP
• netflow
10. Stream = Better Insights for *
Solution Area Contextual Data Wire Data Enriched View
Application
Management
application logs,
monitoring data,
metrics, events
protocol conversations on
database performance, DNS
lookups, client data, business
transaction paths…
Measure application response
times, deeper insights for root-
cause diagnostics, trace tx
paths, establish baselines…
IT Operations application logs,
monitoring data,
metrics, events
payload data including process
times, errors, transaction
traces, ICA latency, SQL
statements, DNS records…
Analyze traffic volume, speed
and packets to identify
infrastructure performance
issues, capacity constraints,
changes; establish baselines…
10
11. Stream = Better Insights for *
Solution Area Contextual Data Wire Data Enriched View
Security app + infra logs,
monitoring data,
events
protocol identification,
protocol headers, content
and payload information,
flow records
Build analytics and context for
incident response, threat
detection, monitoring and
compliance
Digital
Intelligence
website activity,
clickstream data,
metrics
browser-level customer
interactions
Customer Experience – analyze
website and application bottlenecks to
improve customer experience and
online revenues
Customer Support (online, call center)
– faster root cause analysis and
resolution of customer issues with
website or apps
11
12. Scripted Inputs
12
§ Send data to Splunk via a custom script
§ Splunk indexes anything written to stdout
§ Splunk handles scheduling
§ Supports shell, Python scripts, WIN batch, PowerShell
§ Any other utility that can format and stream data
Streaming Mode
§ Splunk executes script and indexes stdout
§ Checks for any running instances
Write to File Mode
§ Splunk launches script which produces
output file, no need for external scheduler
§ Splunk monitors output file
13. Use Cases for Scripted Inputs
13
§ Alternative to file-base or network-based inputs
§ Stream data from command-line tools, such as vmstat and iostat
§ Poll a web service, API or database and process the results
§ Reformat complex or binary data for easier parsing into events and fields
§ Maintain data sources with slow or resource-intensive startup
procedures
§ Provide special or complex handling for transient or unstable inputs
§ Scripts that manage passwords and credentials
§ Wrapper scripts for command line inputs that contain special characters
15. Configure Database Inputs
15
§ DB Connect App
§ Real-time, scalable integration with relational DBs
§ Browse and navigate schemas and tables before data import
§ Reliable scheduled import
§ Seamless installation and UI configuration
§ Supports connection pooling and caching
§ “Tail” tables or import entire tables
§ Detect and import new/updated rows using timestamps or unique IDs
§ Supports many RDBMS flavors
§ AWS RDS Aurora, AWS RedShift, IBM DB2 for Linux, Informix, MemSQL, MS SQL, MySQL,
Oracle, PostgreSQL, SAP SQL Anywhere (aka Sybase SA), Sybase ASE and IQ, Teradata
18. Modular Inputs
18
§ Create your own custom inputs
§ Scripted input with structure and intelligence
§ First class citizen in the Splunk management interface
§ Appears under Settings > Data Inputs
§ Benefits over simple scripted input
§ Instance control: launch a single or multiple instances
§ Input validation
§ Support multiple platforms
§ Stream data as text or XML
§ Secure access to mod input scripts via REST endpoints
28. Agenda
§ Tags – categorize and add meaning to data
§ Field Aliases – simplify search and correlation
§ Calculated Fields – shortcut complex/repetitive computations
§ Event Types – group common events and share knowledge
§ Lookups – augment data with additional external fields
28
54. § top – limit
§ rare – same options as top
§ timechart – parameters
§ stats – functions (sum, avg, list, values, sparkline)
§ sort – inline ascending or descending
§ addcoltotals
§ addtotals
Doing More with Basic Search Commands
54
55. § Commands have parameters or qualifiers
§ top and rare have similar syntax
§ Each search command has its own syntax – show inline help
Find Most and Least Active Customers
Using the top + rare Commands
... | top limit=20 clientip
... | rare limit=20 clientip
IPs with the
most visits
IPs with the
least visits
56. § Sort inline descending or ascending
56
... | stats count by clientip | sort - count
... | stats count by clientip | sort + count
Number of requests by
customer - descending
Number of requests by
customer - ascending
Sort the Number of Customer Requests
Using the sort Command
57. § Show Search Command Reference Docs
§ Functions for eval + where
§ Functions for stats + chart and timechart
§ Invoke a function
§ Rename inline
57
... | stats sum(bytes) by clientip | sort - sum(bytes)
... | stats sum(bytes) as totalbytes by clientip | sort - totalbytes
Total payload by
customer - descending
Total payload by
customer - ascending
Determine Total Customer Payload
Using functions + rename command
60. § Add columns
§ Sum specific columns
60
... | stats count by clientip, action
2 cols: clientip + action
... | stats sum(bytes) as totalbytes, avg(bytes) as avgbytes,
count as totalevents by clientip | addcoltotals totalbytes,
totalevents
Sum totalbytes and
totalevents colums
Building a Table of Customer Activity
Add Columns and Sum Columns
61. 61
... | stats sum(bytes) as totalbytes, sum(other) as totalother
by clientip | addtotals fieldname=totalstuff
For each row, add
totalbytes + totalother
A better example:
physical memory + virtual memory =
total memory
Building a Table of Customer Activity
Sum Across Rows
62. 62
... | stats sparkline(count) as trendline by clientip
In context of
larger event set
... | stats sparkline(count) as trendline sum(bytes) by clientip
Inline in tables
Trend Individual Customer Activity
Sparklines in Action
63. Advanced Search Commands
Command Short Description Hints
transaction Group events by a common field value. Convenient, but resource intensive.
cluster Cluster similar events together. Can be used on _raw or field.
associate Identifies correlations between fields. Calculates entropy btn field values.
correlate Calculates the correlation between
different fields.
Evaluates relationship of all fields in
a result set.
contingency Builds a contingency table for two fields. Computes co-occurrence, or % two
fields exist in same events.
anomalies Computes an unexpectedness score for
an event.
Computes similarity of event (X) to a
set of previous events (P).
anomalousvalue Finds and summarizes irregular, or
uncommon, search results.
Considers frequency of occurrence
or number of stdev from the mean