SlideShare ist ein Scribd-Unternehmen logo

SplunkLive! Data Models 101

•Als PPTX, PDF herunterladen•
•
Splunk
Splunk
Technologie
1 von 50
Copyright © 2013 Splunk Inc. Data Models 101
Legal Notices During the course of this presentation, we may make forward-looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward-looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release. Splunk, Splunk>, Splunk Storm, Listen to Your Data, SPL and The Engine for Machine Data are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. ©2013 Splunk Inc. All rights reserved.
Search is hard.
Analytics Big Picture Pivot Build complex reports without the search language Data Model Provides more meaningful representation of underlying raw machine data Analytics Store Acceleration technology delivers up to 1000x faster analytics over Splunk 5 4
Operational Intelligence Across the Enterprise [10/11/12 18:57:04 000000b0 UTC] Raw Data IT professional Create and share data models Accelerate data models and custom searches with the analytics store Create reports with pivot Analytics Store Developer Leverage data models to abstract data Leverage pivot in custom apps Data Model Pivot Analyst Create reports using pivot based on data models created by IT
Pivot is a query builder.
Data Models 101
Source Data set Source Source
Success Sourcetype Failure Warning
Source Business division Source Data set Source Business division Source
Technology 1 Common model Technology 2 Technology 3
Context
Splunk Search Language search and filter | munge | report | clean-up sourcetype=access_combined source = "/home/ssorkin/banner_access.log.2013.6.gz" | eval unique=(uid + useragent) | stats dc(unique) by os_name | rename dc(unique) as "Unique Visitors" os_name as "Operating System"
Hurdles index=main source=*/banner_access* uri_path=/js/*/*/login/* guid=* useragent!=*KTXN* useragent!=*GomezAgent* clientip!=206.80.3.67 clientip!=198.144.207.62 clientip!=97.65.63.66 clientip!=175.45.37.78 clientip!=209.119.210.194 clientip!=212.36.37.138 clientip!=204.156.84.0/24 clientip!=216.221.226.0/24 clientip!=207.87.200.162 | rex field=uri_path "/js/(?<t>[^/]*)/(?<v>[^/]*)/login/(?<l>[^/]*)” | eval license = case(l LIKE "prod%" AND t="pro", "enterprise", l LIKE "trial%" AND t="pro", "trial", t="free", "free”) | rex field=v "^(?<vers>d.d)” | bin span=1d _time as day | stats values(vers) as vers min(day) as min_day min(eval(if(vers=="5.0", _time, null()))) as min_day_50 dc(day) as days values(license) as license by guid | eval type = if(match(vers,"4.*"), "upgrade", "not upgrade") + "/" + if(days > 1, "repeat", "not repeat")| search license=enterprise | eval _time = min_day_50| timechart count by type| streamstats sum(*) as * • Simple searches easy… Multi-stage munging/reporting is hard! • Need to understand data’s structure to construct search • Non-technical users may not have data source domain knowledge • Splunk admins do not have end-user search context
Data Model Goals • Make it easy to share/reuse domain knowledge • Admins/power users build data models • Non-technical users interact with data via pivot UI
Data Models 101
What is a Data Model? A data model is a search-time mapping of data onto a hierarchical structure Encapsulate the knowledge needed to build a search Pivot reports are build on top of data models Data-independent Screenshot here
A Data Model is a Collection of Objects Screenshot here
Objects Have Constraints and Attributes Screenshot here
Child Objects Inherit Constraints and Attributes Screenshot here
Child Objects Inherit Constraints and Attributes
Building Data Models
Three Root Object Types Event – MapstoSplunkevents – Requiresconstraints andattributes
Three Root Object Types Event – MapstoSplunkevents – Requiresconstraints andattributes Search – MapstoarbitrarySplunksearch(may includegenerating,transformingand reportingsearchcommands) – Requiressearchstringattributes • Transaction – Mapsto groupsof Splunkeventsor groupsof Splunksearchresults – Requiresobjectsto group,fields/ conditionstogroupby,andattributes
Three Root Object Types Event – MapstoSplunkevents – Requiresconstraints andattributes Search – MapstoarbitrarySplunksearch(may includegenerating,transformingand reportingsearchcommands) Requiressearchstringattributes Transaction – Mapsto groupsof Splunkeventsor groupsof Splunksearchresults – Requiresobjectsto group,fields/ conditionstogroupby,andattributes
Object Attributes Auto-extracted – default and predefined fields Eval expression – a new field based on an expression that you define Lookup – leverage an existing lookup table Regular expression – extract a new field based on regex Geo IP – add geolocation fields such as latitude, longitude, country, etc.
Object Attributes Set field types Configure various flags Note: Child object configuration can differ from parent
Best Practices Use event objects as often as possible – Benefit from data model acceleration Resist the urge to use search objects instead of event objects!! – Event based searches can be optimized better Minimize object hierarchy depth when possible – Constraint based filtering is less efficient deeper down the tree Event object with deepest tree (and most matching results) first – Model-wide acceleration only for first event object and its descendants
Warnings! Object constraints and attributes cannot contain pipes or subsearches A transaction object requires at least one event or search object in the data model Lookups used in attributes must be globally visible (or at least visible to the app using the data model) No versioning on data models (and objects)!
From Data Models to Reports
Using the UI Subhead Count of http_success events, split by useragent events fields
Under the Hood: Object Search String Generation Event Object Syntax: <constraints search> | <my attribute definitions> Example: sourcetype=access_* OR sourcetype=iis* uri=* uri_path=* status=* clientip=* referer=* useragent=*
Under the Hood: Object Search String Generation Search Object Syntax: <base search> | <my attribute definitions> Example: _time=* host=* source=* sourcetype=* uri=* status<600 clientip=* referer=* useragent=* (sourcetype=access_* OR source=*.log) | eval userid=clientip | stats first(_time) as earliest, last(_time) as latest, list(uri_path) as uri_list by userid | earliest=* latest=* uri_list=*
Under the Hood: Object Search String Generation Transaction Object Syntax: <objects to group search> | transaction <group by fields> <group by params> | <my attribute definitions> Example: sourcetype=access_* uri=* uri_path=* status=* clientip=* referer=* useragent=* | transaction clientip useragent | eval landingpage=mvindex(uri_path,1) | eval exitpage=mvindex(uri_path,-1)
Under the Hood: Object Search String Generation Child Object Syntax: <parent object search> | search <my constraints> | <my attribute definitions> Example: sourcetype=access_* uri=* uri_path=* status=* clientip=* referer=* useragent=* status=2* | <my attribute definitions>
Using the Splunk Search Language Object Search String | datamodel <modelname> <objectID> search Example: | datamodel WebIntelligence HTTP_Request search Behind the scenes: sourcetype=access_* OR sourcetype=iis* uri=* uri_path=* status=* clientip=* referer=* useragent=*
Under the hood: Pivot Search String Generation Pivot search = object search + filters + reporting + formatting Example: (sourcetype=access_* OR sourcetype=iis*) status=2* uri=* uri_path=* status=* clientip=* referer=* useragent=* | stats count AS "Count of HTTP_Sucess" by ”useragent" | sort limit=0 "useragent" | fields - _span | fields "useragent" "Count of HTTP_Success" | fillnull "Count of HTTP_Success" | fields "useragent" *
Using the Splunk Search Language Pivot Search String | pivot <modelname> <objectID> [statsfns, rowsplit, colsplit, filters, …] Example: | pivot WebIntelligence HTTP_Request count(HTTP_Request) AS "Count of HTTP_Request" SPLITROW status AS "status" SORT 0 status Behind the scenes: sourcetype=access_* OR sourcetype=iis* uri=* uri_path=* status=* clientip=* referer=* useragent=* | stats count AS "Count of HTTP_Request" by "status" | sort limit=0 "status" | fields - _span | fields "status", "Count of HTTP_Request" | fillnull "Count of HTTP_Request" | fields "status" *
Warnings • | datamodel and | pivot are generating commands – They must be at the beginning of the search string • Use objectIDs NOT user-visible object names
Managing Data Models
Data Model on Disk Each data model is a separate JSON file Lives in <myapp>/local/data/models (or <myapp>/default/data/models for pre-installed models) Has associated conf stanzas and metadata
Editing Data Model JSON At your own risk! Models edited via the UI are validated Manually edited data models: NOT SUPPORTED Exception: installing a new model by adding the file to <myapp>/<local OR default>/data/models is probably okay
Deleting a Data Model Use the UI for appropriate cleanup Potential for bad state if manually deleting model on disk
Interacting With a Data Model Use data model builder and pivot UI – safest option! Use REST API – for developers (see docs for details) Use | datamodel and | pivot Splunk search commands
Permissions Data models have permissions just like other Splunk objects Edit permissions through the UI
Data Model Acceleration Admin or power user Backend magic Acceleration Non-technical user Run search using on-disk acceleration Run a pivot report No acceleration Kick off ad-hoc acceleration and run search
Model-Wide Acceleration Only accelerates first eventbased object and descendants Does not accelerate search and transaction-based objects Pivot search: | tstats count AS "Count of HTTP_Success" from datamodel="WebIntelligence" where (nodename="HTTP_Request") (nodename="HTTP_Request.HTTP_Success") prestats=true | stats count AS "Count of HTTP_Success”
Ad-Hoc Object Acceleration Kick off acceleration on pivot page (re) load for non-accelerated models and search/transaction objects Amortize cost of ad-hoc acceleration over repeated pivoting on same object Pivot search: | tstats count AS "Count of HTTP_Success" from sid=1379116434.663 prestats=true | stats count AS "Count of HTTP_Success”
Acceleration Disclaimers Works with search-head pooling – we collect on indexers Cannot edit accelerated models
Thank You

Empfohlen

SplunkLive 2011 Beginners Session
SplunkLive 2011 Beginners SessionSplunkLive 2011 Beginners Session
SplunkLive 2011 Beginners Session
Splunk
 
Splunk Data Onboarding Overview - Splunk Data Collection Architecture
Splunk Data Onboarding Overview - Splunk Data Collection ArchitectureSplunk Data Onboarding Overview - Splunk Data Collection Architecture
Splunk Data Onboarding Overview - Splunk Data Collection Architecture
Splunk
 
Splunk Overview
Splunk OverviewSplunk Overview
Splunk Overview
Splunk
 
Worst Splunk practices...and how to fix them
Worst Splunk practices...and how to fix themWorst Splunk practices...and how to fix them
Worst Splunk practices...and how to fix them
Splunk
 
Splunk Enterprise Security
Splunk Enterprise SecuritySplunk Enterprise Security
Splunk Enterprise Security
Splunk
 
Getting started with Splunk
Getting started with SplunkGetting started with Splunk
Getting started with Splunk
Splunk
 
Splunk for Enterprise Security featuring User Behavior Analytics
Splunk for Enterprise Security featuring User Behavior Analytics Splunk for Enterprise Security featuring User Behavior Analytics
Splunk for Enterprise Security featuring User Behavior Analytics
Splunk
 
Splunk-Presentation
Splunk-Presentation Splunk-Presentation
Splunk-Presentation
PrasadThorat23
 

Empfohlen

SplunkLive 2011 Beginners Session
SplunkLive 2011 Beginners SessionSplunkLive 2011 Beginners Session
SplunkLive 2011 Beginners Session
Splunk
 
Splunk Data Onboarding Overview - Splunk Data Collection Architecture
Splunk Data Onboarding Overview - Splunk Data Collection ArchitectureSplunk Data Onboarding Overview - Splunk Data Collection Architecture
Splunk Data Onboarding Overview - Splunk Data Collection Architecture
Splunk
 
Splunk Overview
Splunk OverviewSplunk Overview
Splunk Overview
Splunk
 
Worst Splunk practices...and how to fix them
Worst Splunk practices...and how to fix themWorst Splunk practices...and how to fix them
Worst Splunk practices...and how to fix them
Splunk
 
Splunk Enterprise Security
Splunk Enterprise SecuritySplunk Enterprise Security
Splunk Enterprise Security
Splunk
 
Getting started with Splunk
Getting started with SplunkGetting started with Splunk
Getting started with Splunk
Splunk
 
Splunk for Enterprise Security featuring User Behavior Analytics
Splunk for Enterprise Security featuring User Behavior Analytics Splunk for Enterprise Security featuring User Behavior Analytics
Splunk for Enterprise Security featuring User Behavior Analytics
Splunk
 
Splunk-Presentation
Splunk-Presentation Splunk-Presentation
Splunk-Presentation
PrasadThorat23
 
Splunk for Enterprise Security and User Behavior Analytics
Splunk for Enterprise Security and User Behavior Analytics Splunk for Enterprise Security and User Behavior Analytics
Splunk for Enterprise Security and User Behavior Analytics
Splunk
 
Cyber Threat Hunting - Hunting in Memory at Scale
Cyber Threat Hunting - Hunting in Memory at ScaleCyber Threat Hunting - Hunting in Memory at Scale
Cyber Threat Hunting - Hunting in Memory at Scale
Infocyte
 
Splunk Tutorial for Beginners - What is Splunk | Edureka
Splunk Tutorial for Beginners - What is Splunk | EdurekaSplunk Tutorial for Beginners - What is Splunk | Edureka
Splunk Tutorial for Beginners - What is Splunk | Edureka
Edureka!
 
Splunk Architecture overview
Splunk Architecture overviewSplunk Architecture overview
Splunk Architecture overview
Alex Fok
 
QRadar, ArcSight and Splunk
QRadar, ArcSight and Splunk QRadar, ArcSight and Splunk
QRadar, ArcSight and Splunk
M sharifi
 
Splunk 101
Splunk 101Splunk 101
Splunk 101
Splunk
 
Splunk Overview
Splunk OverviewSplunk Overview
Splunk Overview
Splunk
 
Splunk Architecture
Splunk ArchitectureSplunk Architecture
Splunk Architecture
Kishore Chaganti
 
Getting Started with Splunk (Hands-On)
Getting Started with Splunk (Hands-On) Getting Started with Splunk (Hands-On)
Getting Started with Splunk (Hands-On)
Splunk
 
Getting Started with Splunk Enterprise
Getting Started with Splunk EnterpriseGetting Started with Splunk Enterprise
Getting Started with Splunk Enterprise
Splunk
 
Splunk
SplunkSplunk
Splunk
Douglas Bernardini
 
SplunkLive! Getting Started with Splunk Enterprise
SplunkLive! Getting Started with Splunk EnterpriseSplunkLive! Getting Started with Splunk Enterprise
SplunkLive! Getting Started with Splunk Enterprise
Splunk
 
Splunk - универсальная платформа для работы с любыми данными
Splunk - универсальная платформа для работы с любыми даннымиSplunk - универсальная платформа для работы с любыми данными
Splunk - универсальная платформа для работы с любыми данными
CleverDATA
 
Best Practices for Splunk Deployments
Best Practices for Splunk DeploymentsBest Practices for Splunk Deployments
Best Practices for Splunk Deployments
Splunk
 
Splunk for IT Operations
Splunk for IT OperationsSplunk for IT Operations
Splunk for IT Operations
Splunk
 
Exploring Frameworks of Splunk Enterprise Security
Exploring Frameworks of Splunk Enterprise SecurityExploring Frameworks of Splunk Enterprise Security
Exploring Frameworks of Splunk Enterprise Security
Splunk
 
ArcSight Basics.ppt
ArcSight Basics.pptArcSight Basics.ppt
ArcSight Basics.ppt
neoalt
 
Splunk Architecture | Splunk Tutorial For Beginners | Splunk Training | Splun...
Splunk Architecture | Splunk Tutorial For Beginners | Splunk Training | Splun...Splunk Architecture | Splunk Tutorial For Beginners | Splunk Training | Splun...
Splunk Architecture | Splunk Tutorial For Beginners | Splunk Training | Splun...
Edureka!
 
Splunk for ITOps
Splunk for ITOpsSplunk for ITOps
Splunk for ITOps
Splunk
 
Splunk Dashboarding & Universal Vs. Heavy Forwarders
Splunk Dashboarding & Universal Vs. Heavy ForwardersSplunk Dashboarding & Universal Vs. Heavy Forwarders
Splunk Dashboarding & Universal Vs. Heavy Forwarders
Harry McLaren
 
Data Models Breakout Session
Data Models Breakout SessionData Models Breakout Session
Data Models Breakout Session
Splunk
 
Splunk Ninja: New Features, Pivot and Search Dojo
Splunk Ninja: New Features, Pivot and Search Dojo Splunk Ninja: New Features, Pivot and Search Dojo
Splunk Ninja: New Features, Pivot and Search Dojo
Splunk
 

Weitere ähnliche Inhalte

Was ist angesagt?

Splunk for Enterprise Security and User Behavior Analytics
Splunk for Enterprise Security and User Behavior Analytics Splunk for Enterprise Security and User Behavior Analytics
Splunk for Enterprise Security and User Behavior Analytics
Splunk
 
Splunk Overview
Splunk OverviewSplunk Overview
Splunk Overview
Splunk
 
SplunkLive! Getting Started with Splunk Enterprise
SplunkLive! Getting Started with Splunk EnterpriseSplunkLive! Getting Started with Splunk Enterprise
SplunkLive! Getting Started with Splunk Enterprise
Splunk
 

Was ist angesagt? (20)

Splunk for Enterprise Security and User Behavior Analytics
Splunk for Enterprise Security and User Behavior Analytics Splunk for Enterprise Security and User Behavior Analytics
Splunk for Enterprise Security and User Behavior Analytics
 
Cyber Threat Hunting - Hunting in Memory at Scale
Cyber Threat Hunting - Hunting in Memory at ScaleCyber Threat Hunting - Hunting in Memory at Scale
Cyber Threat Hunting - Hunting in Memory at Scale
 
Splunk Tutorial for Beginners - What is Splunk | Edureka
Splunk Tutorial for Beginners - What is Splunk | EdurekaSplunk Tutorial for Beginners - What is Splunk | Edureka
Splunk Tutorial for Beginners - What is Splunk | Edureka
 
Splunk Architecture overview
Splunk Architecture overviewSplunk Architecture overview
Splunk Architecture overview
 
QRadar, ArcSight and Splunk
QRadar, ArcSight and Splunk QRadar, ArcSight and Splunk
QRadar, ArcSight and Splunk
 
Splunk 101
Splunk 101Splunk 101
Splunk 101
 
Splunk Overview
Splunk OverviewSplunk Overview
Splunk Overview
 
Splunk Architecture
Splunk ArchitectureSplunk Architecture
Splunk Architecture
 
Getting Started with Splunk (Hands-On)
Getting Started with Splunk (Hands-On) Getting Started with Splunk (Hands-On)
Getting Started with Splunk (Hands-On)
 
Getting Started with Splunk Enterprise
Getting Started with Splunk EnterpriseGetting Started with Splunk Enterprise
Getting Started with Splunk Enterprise
 
Splunk
SplunkSplunk
Splunk
 
SplunkLive! Getting Started with Splunk Enterprise
SplunkLive! Getting Started with Splunk EnterpriseSplunkLive! Getting Started with Splunk Enterprise
SplunkLive! Getting Started with Splunk Enterprise
 
Splunk - универсальная платформа для работы с любыми данными
Splunk - универсальная платформа для работы с любыми даннымиSplunk - универсальная платформа для работы с любыми данными
Splunk - универсальная платформа для работы с любыми данными
 
Best Practices for Splunk Deployments
Best Practices for Splunk DeploymentsBest Practices for Splunk Deployments
Best Practices for Splunk Deployments
 
Splunk for IT Operations
Splunk for IT OperationsSplunk for IT Operations
Splunk for IT Operations
 
Exploring Frameworks of Splunk Enterprise Security
Exploring Frameworks of Splunk Enterprise SecurityExploring Frameworks of Splunk Enterprise Security
Exploring Frameworks of Splunk Enterprise Security
 
ArcSight Basics.ppt
ArcSight Basics.pptArcSight Basics.ppt
ArcSight Basics.ppt
 
Splunk Architecture | Splunk Tutorial For Beginners | Splunk Training | Splun...
Splunk Architecture | Splunk Tutorial For Beginners | Splunk Training | Splun...Splunk Architecture | Splunk Tutorial For Beginners | Splunk Training | Splun...
Splunk Architecture | Splunk Tutorial For Beginners | Splunk Training | Splun...
 
Splunk for ITOps
Splunk for ITOpsSplunk for ITOps
Splunk for ITOps
 
Splunk Dashboarding & Universal Vs. Heavy Forwarders
Splunk Dashboarding & Universal Vs. Heavy ForwardersSplunk Dashboarding & Universal Vs. Heavy Forwarders
Splunk Dashboarding & Universal Vs. Heavy Forwarders
 

Andere mochten auch

Data Mining with Splunk
Data Mining with SplunkData Mining with Splunk
Data Mining with Splunk
David Carasso
 
SplunkLive! Washington DC May 2013 - Splunk Security Workshop
SplunkLive! Washington DC May 2013 - Splunk Security WorkshopSplunkLive! Washington DC May 2013 - Splunk Security Workshop
SplunkLive! Washington DC May 2013 - Splunk Security Workshop
Splunk
 
Morgenbriefing: Fra user insights til user experience 2011
Morgenbriefing: Fra user insights til user experience 2011Morgenbriefing: Fra user insights til user experience 2011
Morgenbriefing: Fra user insights til user experience 2011
Sofia Honoré
 
Splunk Enterprise for IT Troubleshooting
Splunk Enterprise for IT Troubleshooting Splunk Enterprise for IT Troubleshooting
Splunk Enterprise for IT Troubleshooting
Splunk
 

Andere mochten auch (20)

Data Models Breakout Session
Data Models Breakout SessionData Models Breakout Session
Data Models Breakout Session
 
Splunk Ninja: New Features, Pivot and Search Dojo
Splunk Ninja: New Features, Pivot and Search Dojo Splunk Ninja: New Features, Pivot and Search Dojo
Splunk Ninja: New Features, Pivot and Search Dojo
 
SplunkSummit 2015 - A Quick Guide to Search Optimization
SplunkSummit 2015 - A Quick Guide to Search OptimizationSplunkSummit 2015 - A Quick Guide to Search Optimization
SplunkSummit 2015 - A Quick Guide to Search Optimization
 
Conf2014_SplunkSecurityNinjutsu
Conf2014_SplunkSecurityNinjutsuConf2014_SplunkSecurityNinjutsu
Conf2014_SplunkSecurityNinjutsu
 
Data Mining with Splunk
Data Mining with SplunkData Mining with Splunk
Data Mining with Splunk
 
SplunkLive! Washington DC May 2013 - Splunk Security Workshop
SplunkLive! Washington DC May 2013 - Splunk Security WorkshopSplunkLive! Washington DC May 2013 - Splunk Security Workshop
SplunkLive! Washington DC May 2013 - Splunk Security Workshop
 
Morgenbriefing: Fra user insights til user experience 2011
Morgenbriefing: Fra user insights til user experience 2011Morgenbriefing: Fra user insights til user experience 2011
Morgenbriefing: Fra user insights til user experience 2011
 
Sitecore 7; What's in it for you?
Sitecore 7; What's in it for you? Sitecore 7; What's in it for you?
Sitecore 7; What's in it for you?
 
Splunk Enterprise for IT Troubleshooting Hands-On
Splunk Enterprise for IT Troubleshooting Hands-OnSplunk Enterprise for IT Troubleshooting Hands-On
Splunk Enterprise for IT Troubleshooting Hands-On
 
Digital analytics lecture1
Digital analytics lecture1Digital analytics lecture1
Digital analytics lecture1
 
Sitecore Symposium: DMS Where is the data at?
Sitecore Symposium: DMS Where is the data at?Sitecore Symposium: DMS Where is the data at?
Sitecore Symposium: DMS Where is the data at?
 
Splunk Enterprise for IT Troubleshooting
Splunk Enterprise for IT Troubleshooting Splunk Enterprise for IT Troubleshooting
Splunk Enterprise for IT Troubleshooting
 
Splunk: How to Design, Build and Map IT Services
Splunk: How to Design, Build and Map IT ServicesSplunk: How to Design, Build and Map IT Services
Splunk: How to Design, Build and Map IT Services
 
Running Splunk on AWS
Running Splunk on AWSRunning Splunk on AWS
Running Splunk on AWS
 
What's New in Splunk Cloud and Enterprise 6.5
What's New in Splunk Cloud and Enterprise 6.5What's New in Splunk Cloud and Enterprise 6.5
What's New in Splunk Cloud and Enterprise 6.5
 
Enrich a 360-degree Customer View with Splunk and Apache Hadoop
Enrich a 360-degree Customer View with Splunk and Apache HadoopEnrich a 360-degree Customer View with Splunk and Apache Hadoop
Enrich a 360-degree Customer View with Splunk and Apache Hadoop
 
Making Pretty Charts in Splunk
Making Pretty Charts in SplunkMaking Pretty Charts in Splunk
Making Pretty Charts in Splunk
 
Understanding the Sitecore Architecture
Understanding the Sitecore ArchitectureUnderstanding the Sitecore Architecture
Understanding the Sitecore Architecture
 
Machine Data 101
Machine Data 101Machine Data 101
Machine Data 101
 
Introducing Splunk – The Big Data Engine
Introducing Splunk – The Big Data EngineIntroducing Splunk – The Big Data Engine
Introducing Splunk – The Big Data Engine
 

Ähnlich wie SplunkLive! Data Models 101

Analytics with splunk - Advanced
Analytics with splunk - AdvancedAnalytics with splunk - Advanced
Analytics with splunk - Advanced
jenny_splunk
 
SplunkLive! Analytics with Splunk Enterprise - Part 2
SplunkLive! Analytics with Splunk Enterprise - Part 2SplunkLive! Analytics with Splunk Enterprise - Part 2
SplunkLive! Analytics with Splunk Enterprise - Part 2
Splunk
 
CMGT410 v19Project Charter TemplateCMGT410 v19Page 2 of 3P.docx
CMGT410 v19Project Charter TemplateCMGT410 v19Page 2 of 3P.docxCMGT410 v19Project Charter TemplateCMGT410 v19Page 2 of 3P.docx
CMGT410 v19Project Charter TemplateCMGT410 v19Page 2 of 3P.docx
mccormicknadine86
 
SplunkLive! What's New in Splunk 6 Session
SplunkLive! What's New in Splunk 6 SessionSplunkLive! What's New in Splunk 6 Session
SplunkLive! What's New in Splunk 6 Session
Splunk
 
SplunkLive! Analytics with Splunk Enterprise - Part 1
SplunkLive! Analytics with Splunk Enterprise - Part 1SplunkLive! Analytics with Splunk Enterprise - Part 1
SplunkLive! Analytics with Splunk Enterprise - Part 1
Splunk
 

Ähnlich wie SplunkLive! Data Models 101 (20)

Analytics with splunk - Advanced
Analytics with splunk - AdvancedAnalytics with splunk - Advanced
Analytics with splunk - Advanced
 
SplunkLive! Analytics with Splunk Enterprise - Part 2
SplunkLive! Analytics with Splunk Enterprise - Part 2SplunkLive! Analytics with Splunk Enterprise - Part 2
SplunkLive! Analytics with Splunk Enterprise - Part 2
 
Data models pivot with splunk break out session
Data models pivot with splunk break out sessionData models pivot with splunk break out session
Data models pivot with splunk break out session
 
SplunkLive! Munich 2018: Data Onboarding Overview
SplunkLive! Munich 2018: Data Onboarding OverviewSplunkLive! Munich 2018: Data Onboarding Overview
SplunkLive! Munich 2018: Data Onboarding Overview
 
SplunkLive! Zurich 2018: Integrating Metrics and Logs
SplunkLive! Zurich 2018: Integrating Metrics and LogsSplunkLive! Zurich 2018: Integrating Metrics and Logs
SplunkLive! Zurich 2018: Integrating Metrics and Logs
 
SplunkLive! Frankfurt 2018 - Data Onboarding Overview
SplunkLive! Frankfurt 2018 - Data Onboarding OverviewSplunkLive! Frankfurt 2018 - Data Onboarding Overview
SplunkLive! Frankfurt 2018 - Data Onboarding Overview
 
CMGT410 v19Project Charter TemplateCMGT410 v19Page 2 of 3P.docx
CMGT410 v19Project Charter TemplateCMGT410 v19Page 2 of 3P.docxCMGT410 v19Project Charter TemplateCMGT410 v19Page 2 of 3P.docx
CMGT410 v19Project Charter TemplateCMGT410 v19Page 2 of 3P.docx
 
Business Analytics Paradigm Change
Business Analytics Paradigm ChangeBusiness Analytics Paradigm Change
Business Analytics Paradigm Change
 
Cognos framework manager
Cognos framework managerCognos framework manager
Cognos framework manager
 
Advanced Use Cases for Analytics Breakout Session
Advanced Use Cases for Analytics Breakout SessionAdvanced Use Cases for Analytics Breakout Session
Advanced Use Cases for Analytics Breakout Session
 
Getting Started with Splunk Enterprise
Getting Started with Splunk EnterpriseGetting Started with Splunk Enterprise
Getting Started with Splunk Enterprise
 
SplunkLive! What's New in Splunk 6 Session
SplunkLive! What's New in Splunk 6 SessionSplunkLive! What's New in Splunk 6 Session
SplunkLive! What's New in Splunk 6 Session
 
Azure_Purview.pdf
Azure_Purview.pdfAzure_Purview.pdf
Azure_Purview.pdf
 
SplunkLive! Analytics with Splunk Enterprise - Part 1
SplunkLive! Analytics with Splunk Enterprise - Part 1SplunkLive! Analytics with Splunk Enterprise - Part 1
SplunkLive! Analytics with Splunk Enterprise - Part 1
 
(ATS6-APP01) Unleashing the Power of Your Data with Discoverant
(ATS6-APP01) Unleashing the Power of Your Data with Discoverant(ATS6-APP01) Unleashing the Power of Your Data with Discoverant
(ATS6-APP01) Unleashing the Power of Your Data with Discoverant
 
Splunk Ninjas: New Features, Pivot, and Search Dojo
Splunk Ninjas: New Features, Pivot, and Search DojoSplunk Ninjas: New Features, Pivot, and Search Dojo
Splunk Ninjas: New Features, Pivot, and Search Dojo
 
SplunkLive! Frankfurt 2018 - Integrating Metrics & Logs
SplunkLive! Frankfurt 2018 - Integrating Metrics & LogsSplunkLive! Frankfurt 2018 - Integrating Metrics & Logs
SplunkLive! Frankfurt 2018 - Integrating Metrics & Logs
 
SplunkLive! Frankfurt 2018 - Get More From Your Machine Data with Splunk AI
SplunkLive! Frankfurt 2018 - Get More From Your Machine Data with Splunk AISplunkLive! Frankfurt 2018 - Get More From Your Machine Data with Splunk AI
SplunkLive! Frankfurt 2018 - Get More From Your Machine Data with Splunk AI
 
Microsoft Purview
Microsoft PurviewMicrosoft Purview
Microsoft Purview
 
Splunk Enterprise 6.3 - Splunk Tech Day
Splunk Enterprise 6.3 - Splunk Tech DaySplunk Enterprise 6.3 - Splunk Tech Day
Splunk Enterprise 6.3 - Splunk Tech Day
 

Mehr von Splunk

SOC, Amore Mio! | Security Webinar
SOC, Amore Mio! | Security WebinarSOC, Amore Mio! | Security Webinar
SOC, Amore Mio! | Security Webinar
Splunk
 

Mehr von Splunk (20)

.conf Go 2023 - Data analysis as a routine
.conf Go 2023 - Data analysis as a routine.conf Go 2023 - Data analysis as a routine
.conf Go 2023 - Data analysis as a routine
 
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
 
.conf Go 2023 - Navegando la normativa SOX (Telefónica)
.conf Go 2023 - Navegando la normativa SOX (Telefónica).conf Go 2023 - Navegando la normativa SOX (Telefónica)
.conf Go 2023 - Navegando la normativa SOX (Telefónica)
 
.conf Go 2023 - Raiffeisen Bank International
.conf Go 2023 - Raiffeisen Bank International.conf Go 2023 - Raiffeisen Bank International
.conf Go 2023 - Raiffeisen Bank International
 
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett .conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
 
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär).conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
 
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu....conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
 
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever....conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
 
.conf go 2023 - De NOC a CSIRT (Cellnex)
.conf go 2023 - De NOC a CSIRT (Cellnex).conf go 2023 - De NOC a CSIRT (Cellnex)
.conf go 2023 - De NOC a CSIRT (Cellnex)
 
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
 
Splunk - BMW connects business and IT with data driven operations SRE and O11y
Splunk - BMW connects business and IT with data driven operations SRE and O11ySplunk - BMW connects business and IT with data driven operations SRE and O11y
Splunk - BMW connects business and IT with data driven operations SRE and O11y
 
Splunk x Freenet - .conf Go Köln
Splunk x Freenet - .conf Go KölnSplunk x Freenet - .conf Go Köln
Splunk x Freenet - .conf Go Köln
 
Splunk Security Session - .conf Go Köln
Splunk Security Session - .conf Go KölnSplunk Security Session - .conf Go Köln
Splunk Security Session - .conf Go Köln
 
Data foundations building success, at city scale – Imperial College London
Data foundations building success, at city scale – Imperial College London Data foundations building success, at city scale – Imperial College London
Data foundations building success, at city scale – Imperial College London
 
Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...
Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...
Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...
 
SOC, Amore Mio! | Security Webinar
SOC, Amore Mio! | Security WebinarSOC, Amore Mio! | Security Webinar
SOC, Amore Mio! | Security Webinar
 
.conf Go 2022 - Observability Session
.conf Go 2022 - Observability Session.conf Go 2022 - Observability Session
.conf Go 2022 - Observability Session
 
.conf Go Zurich 2022 - Keynote
.conf Go Zurich 2022 - Keynote.conf Go Zurich 2022 - Keynote
.conf Go Zurich 2022 - Keynote
 
.conf Go Zurich 2022 - Platform Session
.conf Go Zurich 2022 - Platform Session.conf Go Zurich 2022 - Platform Session
.conf Go Zurich 2022 - Platform Session
 
.conf Go Zurich 2022 - Security Session
.conf Go Zurich 2022 - Security Session.conf Go Zurich 2022 - Security Session
.conf Go Zurich 2022 - Security Session
 

Kürzlich hochgeladen

Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
Enterprise Knowledge
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
Joaquim Jorge
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
Earley Information Science
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
Principled Technologies
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
UK Journal
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
 

Kürzlich hochgeladen (20)

Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
Evaluating the top large language models.pdf
Evaluating the top large language models.pdfEvaluating the top large language models.pdf
Evaluating the top large language models.pdf
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 

SplunkLive! Data Models 101

  • 1. Copyright © 2013 Splunk Inc. Data Models 101
  • 2. Legal Notices During the course of this presentation, we may make forward-looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward-looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release. Splunk, Splunk>, Splunk Storm, Listen to Your Data, SPL and The Engine for Machine Data are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. ©2013 Splunk Inc. All rights reserved.
  • 4. Analytics Big Picture Pivot Build complex reports without the search language Data Model Provides more meaningful representation of underlying raw machine data Analytics Store Acceleration technology delivers up to 1000x faster analytics over Splunk 5 4
  • 5. Operational Intelligence Across the Enterprise [10/11/12 18:57:04 000000b0 UTC] Raw Data IT professional Create and share data models Accelerate data models and custom searches with the analytics store Create reports with pivot Analytics Store Developer Leverage data models to abstract data Leverage pivot in custom apps Data Model Pivot Analyst Create reports using pivot based on data models created by IT
  • 6. Pivot is a query builder.
  • 13. Splunk Search Language search and filter | munge | report | clean-up sourcetype=access_combined source = "/home/ssorkin/banner_access.log.2013.6.gz" | eval unique=(uid + useragent) | stats dc(unique) by os_name | rename dc(unique) as "Unique Visitors" os_name as "Operating System"
  • 14. Hurdles index=main source=*/banner_access* uri_path=/js/*/*/login/* guid=* useragent!=*KTXN* useragent!=*GomezAgent* clientip!=206.80.3.67 clientip!=198.144.207.62 clientip!=97.65.63.66 clientip!=175.45.37.78 clientip!=209.119.210.194 clientip!=212.36.37.138 clientip!=204.156.84.0/24 clientip!=216.221.226.0/24 clientip!=207.87.200.162 | rex field=uri_path "/js/(?<t>[^/]*)/(?<v>[^/]*)/login/(?<l>[^/]*)” | eval license = case(l LIKE "prod%" AND t="pro", "enterprise", l LIKE "trial%" AND t="pro", "trial", t="free", "free”) | rex field=v "^(?<vers>d.d)” | bin span=1d _time as day | stats values(vers) as vers min(day) as min_day min(eval(if(vers=="5.0", _time, null()))) as min_day_50 dc(day) as days values(license) as license by guid | eval type = if(match(vers,"4.*"), "upgrade", "not upgrade") + "/" + if(days > 1, "repeat", "not repeat")| search license=enterprise | eval _time = min_day_50| timechart count by type| streamstats sum(*) as * • Simple searches easy… Multi-stage munging/reporting is hard! • Need to understand data’s structure to construct search • Non-technical users may not have data source domain knowledge • Splunk admins do not have end-user search context
  • 15. Data Model Goals • Make it easy to share/reuse domain knowledge • Admins/power users build data models • Non-technical users interact with data via pivot UI
  • 17. What is a Data Model? A data model is a search-time mapping of data onto a hierarchical structure Encapsulate the knowledge needed to build a search Pivot reports are build on top of data models Data-independent Screenshot here
  • 18. A Data Model is a Collection of Objects Screenshot here
  • 19. Objects Have Constraints and Attributes Screenshot here
  • 20. Child Objects Inherit Constraints and Attributes Screenshot here
  • 21. Child Objects Inherit Constraints and Attributes
  • 23. Three Root Object Types Event – MapstoSplunkevents – Requiresconstraints andattributes
  • 24. Three Root Object Types Event – MapstoSplunkevents – Requiresconstraints andattributes Search – MapstoarbitrarySplunksearch(may includegenerating,transformingand reportingsearchcommands) – Requiressearchstringattributes • Transaction – Mapsto groupsof Splunkeventsor groupsof Splunksearchresults – Requiresobjectsto group,fields/ conditionstogroupby,andattributes
  • 25. Three Root Object Types Event – MapstoSplunkevents – Requiresconstraints andattributes Search – MapstoarbitrarySplunksearch(may includegenerating,transformingand reportingsearchcommands) Requiressearchstringattributes Transaction – Mapsto groupsof Splunkeventsor groupsof Splunksearchresults – Requiresobjectsto group,fields/ conditionstogroupby,andattributes
  • 26. Object Attributes Auto-extracted – default and predefined fields Eval expression – a new field based on an expression that you define Lookup – leverage an existing lookup table Regular expression – extract a new field based on regex Geo IP – add geolocation fields such as latitude, longitude, country, etc.
  • 27. Object Attributes Set field types Configure various flags Note: Child object configuration can differ from parent
  • 28. Best Practices Use event objects as often as possible – Benefit from data model acceleration Resist the urge to use search objects instead of event objects!! – Event based searches can be optimized better Minimize object hierarchy depth when possible – Constraint based filtering is less efficient deeper down the tree Event object with deepest tree (and most matching results) first – Model-wide acceleration only for first event object and its descendants
  • 29. Warnings! Object constraints and attributes cannot contain pipes or subsearches A transaction object requires at least one event or search object in the data model Lookups used in attributes must be globally visible (or at least visible to the app using the data model) No versioning on data models (and objects)!
  • 30. From Data Models to Reports
  • 31. Using the UI Subhead Count of http_success events, split by useragent events fields
  • 32. Under the Hood: Object Search String Generation Event Object Syntax: <constraints search> | <my attribute definitions> Example: sourcetype=access_* OR sourcetype=iis* uri=* uri_path=* status=* clientip=* referer=* useragent=*
  • 33. Under the Hood: Object Search String Generation Search Object Syntax: <base search> | <my attribute definitions> Example: _time=* host=* source=* sourcetype=* uri=* status<600 clientip=* referer=* useragent=* (sourcetype=access_* OR source=*.log) | eval userid=clientip | stats first(_time) as earliest, last(_time) as latest, list(uri_path) as uri_list by userid | earliest=* latest=* uri_list=*
  • 34. Under the Hood: Object Search String Generation Transaction Object Syntax: <objects to group search> | transaction <group by fields> <group by params> | <my attribute definitions> Example: sourcetype=access_* uri=* uri_path=* status=* clientip=* referer=* useragent=* | transaction clientip useragent | eval landingpage=mvindex(uri_path,1) | eval exitpage=mvindex(uri_path,-1)
  • 35. Under the Hood: Object Search String Generation Child Object Syntax: <parent object search> | search <my constraints> | <my attribute definitions> Example: sourcetype=access_* uri=* uri_path=* status=* clientip=* referer=* useragent=* status=2* | <my attribute definitions>
  • 36. Using the Splunk Search Language Object Search String | datamodel <modelname> <objectID> search Example: | datamodel WebIntelligence HTTP_Request search Behind the scenes: sourcetype=access_* OR sourcetype=iis* uri=* uri_path=* status=* clientip=* referer=* useragent=*
  • 37. Under the hood: Pivot Search String Generation Pivot search = object search + filters + reporting + formatting Example: (sourcetype=access_* OR sourcetype=iis*) status=2* uri=* uri_path=* status=* clientip=* referer=* useragent=* | stats count AS "Count of HTTP_Sucess" by ”useragent" | sort limit=0 "useragent" | fields - _span | fields "useragent" "Count of HTTP_Success" | fillnull "Count of HTTP_Success" | fields "useragent" *
  • 38. Using the Splunk Search Language Pivot Search String | pivot <modelname> <objectID> [statsfns, rowsplit, colsplit, filters, …] Example: | pivot WebIntelligence HTTP_Request count(HTTP_Request) AS "Count of HTTP_Request" SPLITROW status AS "status" SORT 0 status Behind the scenes: sourcetype=access_* OR sourcetype=iis* uri=* uri_path=* status=* clientip=* referer=* useragent=* | stats count AS "Count of HTTP_Request" by "status" | sort limit=0 "status" | fields - _span | fields "status", "Count of HTTP_Request" | fillnull "Count of HTTP_Request" | fields "status" *
  • 39. Warnings • | datamodel and | pivot are generating commands – They must be at the beginning of the search string • Use objectIDs NOT user-visible object names
  • 41. Data Model on Disk Each data model is a separate JSON file Lives in <myapp>/local/data/models (or <myapp>/default/data/models for pre-installed models) Has associated conf stanzas and metadata
  • 42. Editing Data Model JSON At your own risk! Models edited via the UI are validated Manually edited data models: NOT SUPPORTED Exception: installing a new model by adding the file to <myapp>/<local OR default>/data/models is probably okay
  • 43. Deleting a Data Model Use the UI for appropriate cleanup Potential for bad state if manually deleting model on disk
  • 44. Interacting With a Data Model Use data model builder and pivot UI – safest option! Use REST API – for developers (see docs for details) Use | datamodel and | pivot Splunk search commands
  • 45. Permissions Data models have permissions just like other Splunk objects Edit permissions through the UI
  • 46. Data Model Acceleration Admin or power user Backend magic Acceleration Non-technical user Run search using on-disk acceleration Run a pivot report No acceleration Kick off ad-hoc acceleration and run search
  • 47. Model-Wide Acceleration Only accelerates first eventbased object and descendants Does not accelerate search and transaction-based objects Pivot search: | tstats count AS "Count of HTTP_Success" from datamodel="WebIntelligence" where (nodename="HTTP_Request") (nodename="HTTP_Request.HTTP_Success") prestats=true | stats count AS "Count of HTTP_Success”
  • 48. Ad-Hoc Object Acceleration Kick off acceleration on pivot page (re) load for non-accelerated models and search/transaction objects Amortize cost of ad-hoc acceleration over repeated pivoting on same object Pivot search: | tstats count AS "Count of HTTP_Success" from sid=1379116434.663 prestats=true | stats count AS "Count of HTTP_Success”
  • 49. Acceleration Disclaimers Works with search-head pooling – we collect on indexers Cannot edit accelerated models

Hinweis der Redaktion

  1. Splunk 6 takes large-scalemachine data analytics to the next level by introducing three breakthrough innovations:Pivot – opens up the power of Splunk search to non-technical users with an easy-to-use drag and drop interface to explore, manipulate and visualize data Data Model – defines meaningful relationships in underlying machine data and making the data more useful to broader base of non-technical usersAnalytics Store – patent pending technology that accelerates data models by delivering extremely high performance data retrieval for analytical operations, up to 1000x faster than Splunk 5Let’s dig into each of these new features in more detail.
  2. How does theAnalytics Store, Data Model and Pivot benefit users across the enterprise?Lets start with the IT Professional – this includes the Splunk Administrator or an advanced Splunk user that is familiar with SPL.Using Splunk 6 they can:Create data modelsShare data models with other users – delivering a consistent view of the dataAccelerate data models using the Analytics StoreCreate reports using Pivot (although being power users, they may prefer using SPL directly!)Next we have the enterprise developer.Using Splunk 6 they can:Leverage data models built by IT, making searches more portable (using common Data Models ensures predictability of results)Leverage the Pivot interface in custom enterprise appsFinally, there are additional users that can now benefit – for example, the business or data analyst. Using Splunk 6 they can:Create reports, dashboards, charts and other visualizations using the Pivot interface and based on data models that provide an abstracted view of the raw data. Splunk 6 is not meant to replace existing BI and Business Analytics tools, but it does provide new visibility, insights and intelligence from operational data that can be used by business analysts to augment these tools. Data from Splunk software can also be leveraged directly using the Splunk API and SDKs and integrated into existing business analytics tools. For example, the recently announced Pentaho Business Analytics for Splunk® Enterprise (http://apps.splunk.com/app/1554), enables business users to utilize Pentaho to rapidly visualize and gain additional insights from Splunk’s machine data platform using existing in-house skills.
  3. -The Splunk search language is very expressive. - Can perform a wide variety of tasks ranging from filtering to data munging and reporting- There are various search commands for complex transformations and statistics (e.g. correlation, prediction etc)
  4. What does the search do?Basically, first it normalizes the individual accesses, which should be representable as a model object.Next it aggregates by guid to create an &quot;instance&quot; object, which should be representable in a DM.It calculates a field on that instance object, &quot;type&quot;.Then it builds a timechart. of those, using a special &quot;_time&quot; value.Low overhead to start but learning curve quickly gets steepObtaining website usage metrics should not require understanding Apache vs IIS formatAdmins won’t know apriori what questions are being asked of the data…so they can’t provide canned dashboards for all scenariosBackup search for example: eventtype=pageview | eval stage_2=if(searchmatch(&quot;uri=/download*&quot;), _time, null()) | eval stage_1=if(searchmatch(&quot;uri=/product*&quot;), _time, null()) | eval stage_3=if(searchmatch(&quot;uri=*download_track*&quot;), _time, null()) | stats min(stage_*) as stage_*  by cookie | search stage_1=* | where isnull(stage_2) OR stage_2 &gt;= stage_1 | where isnull(stage_3) OR stage_3 &gt;= stage_2 | eval stage = case(isnull(stage_2), &quot;stage_1&quot;, isnull(stage_3), &quot;stage_2&quot;, 1==1, &quot;stage_3&quot;) | stats count by stage | reverse | accum count as cumulative_count |  reverse | streamstats current=f max(cumulative_count) as stage_1_count last(cumulative_count) as prev_count
  5. What are the important “things” in your data?E.g. WebIntelligence might haveHTTPAccessHTTPSuccessUser SessionHow are they related?There’s more than one “right” way to define your objects
  6. Constraints filter down to a set of a dataAttributes are the fields and knowledge associated with the objectBoth are inherited!
  7. A child object is a type of its parent object: e.g. An HTTP_Success object is a type of HTTP_AccessAdding a child object is essentially a way of adding a filter on the parentsA parent-child relationship makes it easy to do queries like “What percentage of my HTTP_Access events are HTTP_Success events?”
  8. Constraints are essentially the search broken down into a hierarchy, attributes are the associated fields and knowledge
  9. Arbitrary searches that include transforming commands to define the dataset that they representFix example here? TODO
  10. Enable the creation of objects that represent transactionsUse fields that have already been added to the model via event or search objects
  11. This is how we capture knowledge
  12. Required: Only events that contain this field will be returned in PivotOptional: The field doesn&apos;t have to appear in every event Hidden: The field will not be displayed to Pivot users when they select the object in PivotUse this for fields that are only being used to define another attribute, such as an eval expression Hidden &amp; Required: Only events that contain this field will be returned, and the field will be hidden from use in Pivot
  13. Be careful about lookup permissions – must be available in the context where you want to use them
  14. Divanny will help make this slide MUCH prettier
  15. This could be more slides,more details
  16. This could be more slides,more details