Besides seeing the newest features in Splunk Enterprise and learning the best practices for data models and pivot, we will show you how to use a handful of search commands that will solve most search needs. Learn these well and become a ninja.
2. 2
Safe Harbor Statement
During the course of this presentation,we may make forward looking statements regarding future events
or the expected performance of the company. We caution you that such statements reflect our current
expectations and estimates based on factors currently known to us and that actual events or results could
differ materially. For important factors that may cause actual results to differ from those contained in our
forward-looking statements, please review our filings with the SEC. The forward-looking statements
made in this presentation are being made as of the time and date of its live presentation. If reviewed
after its live presentation, this presentation may not contain current or accurate information. We do not
assume any obligation to update any forward looking statements we may make. In addition, any
information about our roadmap outlines our general product direction and is subject to change at any
time without notice. It is for informational purposes only and shall not be incorporated into any contract
or other commitment. Splunk undertakes no obligation either to develop the features or functionality
described orto includeany suchfeatureor functionalityina futurerelease.
3. 3
Agenda
What’s new in 6.3
– Breakthrough Performance and Scale
– Advanced Analysis and Visualization
– High Volume Event Collection
– Enterprise-Scale Platform
Harness the power of search
– The 5 Search Commands That Can Solve Most Problems
4. 4
Splunk Enterprise & Cloud 6.3
*Breakthrough
Performance & Scale
Doubles performance
and lowers TCO
Advanced Analysis
& Visualization
High Volume Event
Collection
Enterprise-Scale
Platform
Supports DevOps and IoT
data analysis at scale
Simplifies analysis of
large datasets
Enterprise
management and
integration
*Not applicable for Cloud.
5. 5
Breakthrough Performance and Scale
Vertical Scaling Maximizes Use of CPU Power Through:
– Indexer Parallelization
– Search Parallelization
Improved Search Performance and System Capacity Through:
– Intelligent Job Scheduling
5
8. 8
Summary of Parallelization Settings
Setting Description Setting name / location
Default
Value
Max
Recmd
Value
Impact
Batch mode search
parallelization
Allows a batch mode search to open
additional search pipelines on each
indexer.
limits.conf
batch_search_max_pipeline
1 2
Multiplies the number of
search pipelines per batch
mode search per indexer.
Parallel
summarization for
data models
Allows the scheduler to run
concurrent data model acceleration
searches on the indexers.
datamodels.conf
acceleration.max_concurrent
2 2
Multiplies the number of
scheduled acceleration
searches per data model per
indexer.
Parallel
summarization for
report accelerations
Allows the scheduler to run
concurrent report acceleration
searches on the indexers.
savedsearches.conf
auto_summarize.max_concurrent
1 2
Multiplies the number of
scheduled acceleration
searches per search per
indexer.
Index parallelization
Allows concurrent data processing
pipelines on indexers and
forwarders.
server.conf
parallelIngestionPipelines
1 2
Multiplies the number of
pipelines per indexer.
http://docs.splunk.com/Documentation/Splunk/latest/Capacity/Parallelization
9. 9
Intelligent Job Scheduling
• Adds better priority scoring and
search windows for much improved
saved search scheduling
• Reduces # of skipped searches
• Re-run failed searches during
downtime
9
6.3
6.2
10. 10
Splunk Enterprise & Cloud 6.3
Breakthrough
Performance & Scale
Doubles performance
and lowers TCO
Advanced Analysis
& Visualization
High Volume Event
Collection
Enterprise-Scale
Platform
Supports DevOps and IoT
data analysis at scale
Simplifies analysis of
large datasets
Enterprise
management and
integration
11. 11
Single Value Display
At-a-glance, single-value indicators with useful context
No JS coding / CSS styling necessary!
Configurable sparkline
Value rangemap, custom thresholds
Trend up/down - reversible
Great for Operation Centers and
War Rooms
11
12. 12
Anomaly Detection
Incorporates Z-Score, IQR & histogram methodologies in a single command
Detect and summarize anomalies
Return anomalous values and
outliers
3 commands in one
Easy-to-use
Configurable threshold
1
13. 13
Choropleth maps
Visualize how a metric varies across a (custom) geographic area
50 States plus World Countries built in
3 Different Color Modes
– Sequential
– Divergent
– Categorical
Custom Polygon Definitions
– Use KMZs and also make your own!
– Shapester App!
Point-in Polygon lookups
13
15. 15
Splunk Enterprise & Cloud 6.3
Breakthrough
Performance & Scale
Doubles performance
and lowers TCO
Advanced Analysis
& Visualization
High Volume Event
Collection
Enterprise-Scale
Platform
Supports DevOps and IoT
data analysis at scale
Simplifies analysis of
large datasets
Enterprise
management and
integration
16. curl -k https://<host>:8088/services/collector -H 'Authorization: Splunk <token>' -d
'{"event":"Hello Event Collector"}'
Applications IoT Devices
Agentless, direct data onboarding via a standard developer API
HTTP Event Collector
18. 18
Splunk Enterprise & Cloud 6.3
Breakthrough
Performance & Scale
Doubles performance
and lowers TCO
Advanced Analysis
& Visualization
High Volume Event
Collection
Enterprise-Scale
Platform
Supports DevOps and IoT
data analysis at scale
Simplifies analysis of
large datasets
Enterprise management
and integration
19. 19
Distributed Management Console - II
New topology views, status, and alerting for Splunk deployments
• Visualizes Search Head/Indexer matrix
with KPI and performance overlays
• Search Head clustering replication
and scheduler views
• Forwarder views with status and
performance data
• Index and metadata storage utilization
• System health alerting
19
20. 20
Custom Alert Actions
Use Splunk Alerts to trigger & automate workflows
• Allows packaged integration with
third-party applications
• Simple admin/user configuration
• Developers can build, package, and
publish alert actions within an app
• Growing list of integrations available
20
25. 25
search and filter | munge | report | cleanup
Search Processing Language
sourcetype=access*
| eval KB=bytes/1024
| stats sum(KB) dc(clientip)
| rename sum(KB) AS "Total MB" dc(clientip) AS "Unique Customers"
26. 26
Five Commands That Will Solve Most Data Questions
eval - Modify or Create New Fields and Values
stats - Calculate Statistics Based on Field Values
eventstats - Add Summary Statistics to Search Results
streamstats - Cumulative Statistics for Each Event
transaction - Group Related Events Spanning Time
30. 32
stats – Calculate Statistics Based on Field Values
Examples
• Calculate stats and rename
sourcetype=access*
| eval KB=bytes/1024
| stats sum(KB) AS “Total KB”
• Multiple statistics
sourcetype=access*
| eval KB=bytes/1024
| stats sum(KB) avg(KB)
• By another field
sourcetype=access*
| eval KB=bytes/1024
| stats sum(KB) avg(KB) by clientip
31. 33
stats – Calculate Statistics Based on Field Values
Examples
• Calculate stats and rename
sourcetype=access*
| eval KB=bytes/1024
| stats sum(KB) as “Total KB”
• Multiple statistics
sourcetype=access*
| eval KB=bytes/1024
| stats sum(KB) avg(KB)
• By another field
sourcetype=access*
| eval KB=bytes/1024
| stats sum(KB) avg(KB) by clientip
32. 34
stats – Calculate Statistics Based on Field Values
Examples
• Calculate statistics
sourcetype=access*
| eval KB=bytes/1024
| stats sum(KB) AS "Total KB”
• Multiple statistics
sourcetype=access*
| eval KB=bytes/1024
| stats avg(KB) sum(KB)
• By another field
sourcetype=access*
| eval KB=bytes/1024
| stats sum(KB) avg(KB) by clientip
33. 36
eventstats – Add Summary Statistics to Search Results
Examples
• Overlay Average
sourcetype=access*
| eventstats avg(bytes) AS avg_bytes
| timechart latest(avg_bytes) avg(bytes)
• Moving Average
sourcetype=access*
| eventstats avg(bytes) AS avg_bytes by date_hour
| timechart latest(avg_bytes) avg(bytes)
• By created field
sourcetype=access*
| eval http_response = if(status == 200, "OK", "Error”)
| eventstats avg(bytes) AS avg_bytes by http_response
| timechart latest(avg_bytes) avg(bytes) by http_response
34. 37
Examples
• Overlay Average
sourcetype=access*
| eventstats avg(bytes) AS avg_bytes
| timechart latest(avg_bytes) avg(bytes)
• Moving Average
sourcetype=access*
| eventstats avg(bytes) AS avg_bytes by date_hour
| timechart latest(avg_bytes) avg(bytes)
• By created field
sourcetype=access*
| eval http_response = if(status == 200, "OK", "Error”)
| eventstats avg(bytes) AS avg_bytes by http_response
| timechart latest(avg_bytes) avg(bytes) by http_response
eventstats – Add Summary Statistics to Search Results
35. 38
eventstats – Add Summary Statistics to Search Results
Examples
• Overlay Average
sourcetype=access*
| eventstats avg(bytes) AS avg_bytes
| timechart latest(avg_bytes) avg(bytes)
• Moving Average
sourcetype=access*
| eventstats avg(bytes) AS avg_bytes by date_hour
| timechart latest(avg_bytes) avg(bytes)
• By created field
sourcetype=access*
| eval http_response = if(status == 200, "OK", "Error”)
| eventstats avg(bytes) AS avg_bytes by http_response
| timechart latest(avg_bytes) avg(bytes) by http_response
36. 40
streamstats – Cumulative Statistics for Each Event
Examples
• Cumulative Sum
sourcetype=access*
| reverse
| streamstats sum(bytes) as bytes_total
| timechart max(bytes_total)
• Cumulative Sum by Field
sourcetype=access*
| reverse
| streamstats sum(bytes) as bytes_total by status
| timechart max(bytes_total) by status
• Moving Average
sourcetype=access*
| timechart avg(bytes) as avg_bytes
| streamstats avg(avg_bytes) AS moving_avg_bytes window=10
| timechart latest(moving_avg_bytes) latest(avg_bytes)
37. 41
streamstats – Cumulative Statistics for Each Event
Examples
• Cumulative Sum
sourcetype=access*
| timechart sum(bytes) as bytes
| streamstats sum(bytes) as cumulative_bytes
| timechart max(cumulative_bytes)
• Cumulative Sum by Field
sourcetype=access*
| reverse
| streamstats sum(bytes) as bytes_total by status
| timechart max(bytes_total) by status
• Moving Average
sourcetype=access*
| timechart avg(bytes) as avg_bytes
| streamstats avg(avg_bytes) AS moving_avg_bytes window=10
| timechart latest(moving_avg_bytes) latest(avg_bytes)
38. 42
streamstats – Cumulative Statistics for Each Event
Examples
• Cumulative Sum
sourcetype=access*
| timechart sum(bytes) as bytes
| streamstats sum(bytes) as cumulative_bytes
| timechart max(cumulative_bytes)
• Cumulative Sum by Field
sourcetype=access*
| reverse
| streamstats sum(bytes) as bytes_total by status
| timechart max(bytes_total) by status
• Moving Average
sourcetype=access*
| timechart avg(bytes) as avg_bytes
| streamstats avg(avg_bytes) AS moving_avg_bytes
window=10
| timechart latest(moving_avg_bytes) latest(avg_bytes)
39. 44
transaction – Group Related Events Spanning Time
Examples
• Group by Session ID
sourcetype=access*
| transaction JSESSIONID
• Calculate Session Durations
sourcetype=access*
| transaction JSESSIONID
| stats min(duration) max(duration) avg(duration)
• Stats is Better
sourcetype=access*
| stats min(_time) AS earliest max(_time) AS latest by JSESSIONID
| eval duration=latest-earliest
| stats min(duration) max(duration) avg(duration)
40. 45
transaction – Group Related Events Spanning Time
Examples
• Group by Session ID
sourcetype=access*
| transaction JSESSIONID
• Calculate Session Durations
sourcetype=access*
| transaction JSESSIONID
| stats min(duration) max(duration) avg(duration)
• Stats is Better
sourcetype=access*
| stats min(_time) AS earliest max(_time) AS latest by JSESSIONID
| eval duration=latest-earliest
| stats min(duration) max(duration) avg(duration)
41. 46
transaction – Group Related Events Spanning Time
Examples
• Group by Session ID
sourcetype=access*
| transaction JSESSIONID
• Calculate Session Durations
sourcetype=access*
| transaction JSESSIONID
| stats min(duration) max(duration) avg(duration)
• Stats is Better
sourcetype=access*
| stats min(_time) AS earliest max(_time) AS latest by JSESSIONID
| eval duration=latest-earliest
| stats min(duration) max(duration) avg(duration)
42. 47
Learn Them Well and Become a Ninja
eval - Modify or Create New Fields and Values
stats - Calculate Statistics Based on Field Values
eventstats - Add Summary Statistics to Search Results
streamstats - Cumulative Statistics for Each Event
transaction - Group Related Events Spanning Time
See many more examples and neat tricks at docs.splunk.com and answers.splunk.com
Here is what you need for this presentation:
Link to videos on box: <coming soon>
You should have the following installed:
6.3 Overview
OI Demo 3.1 – Get it from the Technical Enablement Portal under SE tools –> Demos https://splunk--c.na2.visual.force.com/apex/LMS_TechnicalEnablementPortal
NOTE: Configure your role to search the oidemo index by default, otherwise you will have to type “index=oidemo” for the examples later on.
There is a lot to cover in this presentation! Try to go quickly and at a pretty high level. When you get through the presentation judge the audience’s interest and go deeper in whichever section. For example, if they want to know more about Choropleths and polygons spend some time there, or if they want to go deeper on the search commands talk through the extra examples.
Splunk safe harbor statement.
Previously, Splunk made use of available CPU cores to execute multiple simultaneous searches while indexing data. Release 6.3 vertical scaling uses allows both individual searches and the data indexing process to execute more efficiently by using multiple CPU cores per task. For systems with available CPU cores, the benefits are broad performance improvements in search processing, report generation, data on-boarding capacity and data forwarding efficiency.
We didn’t want to just make the searches faster, but also smarter. That is why we created an intelligent job scheduler.
Let’s take a look at these features.
Indexer Parallelization helped Cisco UCS achieve 4x the data ingestion (doing pure indexing)
This is an eye chart, BUT it summarizes the parallelization parameters and how to enable them.
This scheduler optimizes which scheduled searches are run and when. Instead of just telling searches when to start, you can tell them a window to run by. It’s like saying you need to get to work by 8am and now Splunk can tell you when to start your journey so you aren’t stuck in traffic.
Continues Scheduled Searches (CSSs)
Problem in 6.2: Continuous Scheduled Searches (CSSs) are missed due to Splunk downtime creating data gaps
Solution in 6.3: By remembering last execution time, missed CSSs are run as soon as Splunk comes back up to fill in data gaps
Schedule Window is an option when scheduling your search. It’s that easy to use!
When combined with 6.3 parallel search capabilities, you may see even more of a reduction or elimination of skipped searches AND increased capacity of job execution
For infrequent searches (hourly, daily, etc.) use schedule windows.
Use the built-in scheduler performance reports (under Activity > System Activity > Scheduler) to monitor performance: lots of skipped searches or high lag is bad.
Release 6.3 improves big data analysis and visualization.
I’m going to talk about and show you:
Single Value Display
Anomaly Detection Command
Geospatial mapping and choropleths
New SPL command that offers histogram based approach for detecting anomalies. Also includes the capabilities of existing anomalousvalue & outlier SPL commands. Options include Histogram, Z-Score and IQR.
Use Splunk 6.3 Overview App.
Go to Single Value Visualization and explain components. Edit in panel and show how to turn on and off, change the sparkline granularity using timechart span=1h, 1m, etc.
Go to Anomaly Detection example. Explain story of using vehicle data. Imagine thousands of cars in a fleet and hundreds of attributes per car to look for anomalies in. You can’t chart all of this at once over all time. Anomaly detection is a great starting point. Then you can chart the findings to investigate further or alert on the results.
Go to Choropleth Maps and explain the different options. Now we’re going to create our own using an app called Shapester built by one of our Splunkers. Go to splunkbase, d/l shapester and load up the app. (Can have this preloaded to save time – but mention how easy it is to install). Create some custom polygons such as Sales Regions (East West Central) and use OI Demo data to show sales by region. See search below:
TBD
See video for more details
Release 6.3 includes the new HTTP Event Collector that directly onboards data from applications, DevOps and IoT devices in real-time, scaling to millions of events per second
This new data input makes it simple and fast to collect data from any application and the world of IoT – at massive scale and speed. Think about it, your phones sent data directly into Splunk without using a forwarder.
Application developers can use a standard API or logging libraries directly.
For example, if you’re using AWS Lambda or containers like Docker, you can push events directly to Splunk.
IoT devices can use the same direct method, and there is a growing list of IoT collection services already. Like xively, and Citrix Octoblu.
And it scales to millions of events per second
Use Splunk 6.3 Overview App for tutorial.
Set up HEC, show test using Curl command. (Use 6.3 Over App Tutorial)
Do Splunk Shake Demo!
Reference: TBD
Interactive, topology-oriented display with mouse-overs for status
Today, a large Splunk deployment can include 100’s of individual system components. The new Distributed Management Console (DMC) provides a complete monitoring console, including topology views, system status, and health alerting, for all components of an on-premise deployment. DMC creates a single interface to view the status, performance, capacity, and interconnectivity of these components, allowing the admin to optimize solution operation and efficiency.
Custom Alert Actions provide the ability to use Splunk Alerts to trigger custom actions or pre-packaged integrations with 3rd party products such as trouble ticketing or support systems. Developers can build and publish integrations or custom action packages that users or admins can use via a simple menu within the Splunk Alert Interface. Splunk and partners provide a growing set of integrations including, ServiceNow, xMatters, Webhooks and more. Previously these integrations were complex, ad-hoc efforts requiring custom scripts. The new scheme makes it simple for partners (and customers) to create and contribute out-of-the-box integration templates, and for customers to use them via a simple pull-down menu.
Provide Quick Overview of each. Mention You can learn more in the overview app that can be downloaded from Splunkbase
Use OI Demo 3.1
Go to Settings Alert Actions. Discuss the capability of d/l custom alert actions from SplunkBase. Go to hipchat and show how configured to run.
Go to IoT DataCenter dashboard and point out the use of anomalydetection being used to detect Power anomalies. Open in search and talk about how the usual “Save as: Alert” process would work. Show the new dropdown of triggers at the bottom. “Imagine changing the colors of lights in the NOC when a critical event occurs using the Phillips Hue plugin, etc.”
Go to Settings -> Searches, Reports, Alerts and enable” OI Demo Anomaly Detection Alert” --- Go to hipchat room OIDemo3 Alerts and alert should show up in 1 minute or less. Show how can pass tokens.
For more information, or to try out the features yourself. Check out the overview app which explains each of the features and includes code samples and examples where applicable.
<This section should take ~15 minutes>Search is the most powerful part of Splunk.
The Splunk search language is very expressive and can perform a wide variety of tasks ranging from filtering to data, to munging, and reporting. The results can be used to answer questions, visualize results, or even send to a third party application in whatever format they require.
Although there are 135 documented search commands; however, most questions can be answered by using just a handful.
These are the five commands you should get very familiar with. If you know how to use these well, you will be able to solve most data questions that come your way. Let’s take a quick look at each of these.
<Walk through the examples with a demo. Hidden slides are available as backup. NOTE: Each of the grey boxes is clickable. If you are running Splunk on port 8000 you won’t have to type in the searches, this will save time.>
Note: Chart is just stats visualized. Timechart is just stats by _time visualized.
sourcetype=access*
| eval KB=bytes/1024
| stats sum(KB) AS "Sum of KB"
sourcetype=access*
| stats values(useragent) avg(bytes) max(bytes) by clientip
sourcetype=access*
| stats values(useragent) avg(bytes) max(bytes) by clientip
Eventstats let’s you add statistics about the entire search results and makes the statistics available as fields on each event.
<Walk through the examples with a demo. Hidden slides are available as backup>
Eventstats let’s you add statistics about the entire search results and makes the statistics available as fields on each event.
Let’s use eventstats to create a timechart of the average bytes on top of the overall average.
index=* sourcetype=access*
| eventstats avg(bytes) AS avg_bytes
| timechart latest(avg_bytes) avg(bytes)
We can turn this into a moving average simply by adding “by date_hour” to calculate the average per hour instead of the overall average.
index=* sourcetype=access*
| eventstats avg(bytes) AS avg_bytes by date_hour
| timechart latest(avg_bytes) avg(bytes)
Streamstats calculates statistics for each event at the time the event is seen. So for example, if I had an event with a temperature reading I could use streamstats to create a new field to tell me the temperature difference between the event and one or more previous events. Similar to the delta command, but more powerful. In this example, I’m going to take the bytes field of my access logs and see how much total data is being transferred code over time.
To create a cumulative sum:
sourcetype=access*
| timechart sum(bytes) as bytes
| streamstats sum(bytes) as cumulative_bytes
| timechart max(cumulative_bytes)
sourcetype=access*
| reverse
| streamstats sum(bytes) as bytes_total by status
| timechart max(bytes_total) by status
sourcetype=access*
| timechart avg(bytes) as avg_bytes
| streamstats avg(avg_bytes) AS moving_avg_bytes window=10
| timechart latest(moving_avg_bytes) latest(avg_bytes)
Bonus: This could also be completed using the trendline command with the simple moving average (sma) parameter:
sourcetype=access*
| timechart avg(bytes) as avg_bytes
| trendline sma10(avg_bytes) as moving_average_bytes
| timechart latest(avg_bytes) latest(moving_average_bytes)
Double Bonus: Cumulative sum by period
sourcetype=access*
| timechart span=15m sum(bytes) as cumulative_bytes by status
| streamstats global=f sum(cumulative_bytes) as bytes_total
A transaction is any group of related events that span time. It’s quite useful for finding overall durations. For example, how long did it take a user to complete a transaction. This really shows the power of Splunk. Think about it, if you are sending all your data to splunk then you have data from multiple subsystems (think database, webserver, and app server), you can see the overall time it’s taking AND how long each subsystem is taking. So many customers are using this to quickly pinpoint whether slowness is because of the network, database, or app server.
NOTE: Many transactions can be re-created using stats. Transaction is easy but stats is way more efficient and it’s a mapable command (more work will be distributed to the indexers).
sourcetype=access*
| stats min(_time) AS earliest max(_time) AS latest by JSESSIONID
| eval duration=latest-earliest
| stats min(duration) max(duration) avg(duration)
There is much more each of these commands can be used for. Check out answers.splunk.com and docs.splunk.com for many more examples.
<If you have time, feel free to show one of your favorite commands or a neat use case of a command. The cluster command is provided here as an example >
“There are over 135 splunk commands, the five you have just seen are incredibly powerful. Here is another to add to your arsenal.”
You can use the cluster command to learn more about your data and to find common and/or rare events in your data. For example, if you are investigating an IT problem and you don't know specifically what to look for, use the cluster command to find anomalies. In this case, anomalous events are those that aren't grouped into big clusters or clusters that contain few events. Or, if you are searching for errors, use the cluster command to see approximately how many different types of errors there are and what types of errors are common in your data.
Decrease the threshold of similarity and see the change in results
sourcetype=access* | cluster field=bc_uri showcount=t t=0.1| table cluster_count bc_uri _raw | sort -cluster_count