15. Turning Machine Data Into Business Value
15
Platform for Machine Data
Application
Delivery
IT
Operations
Security,
Compliance
and Fraud
Business
Analytics Industrial
Data and
Internet of
Things
16. HA/DR Admin Data Security Apps SDKs/APIsScale
Collect
Data
Index
Data
Enrich
Data
Search &
Explore
Analyze
& Predict
Report &
Visualize
Alert &
Action
16
Fully Integrated Enterprise Platform
17. Technology Telecommunications Travel and Leisure
Education
Healthcare
Energy and Utilities
Manufacturing
Financial Services and Insurance
Media
Proven at 11,000+ Customers in 100 Countries
More Than 80 of the Fortune 100
Retail
Cloud and Online Services
Government
18. Splunk Capabilities for IoT
Data Ingest
at Scale
Partner
Ecosystem
Advanced Analytics
and Visualization
Sense and
Respond
19. IoT and Industrial Machine Data
DevelopVisualize PredictAlertSearch
Engineers Data
Analysts
Security
Analysts
Business
Users
Native Inputs
TCP, UDP, Logs, Scripts, Wire, Mobile
SDKs and APIs
Java, JS, C#, Python, Ruby, PHP
Modular Inputs
MQTT, AMQP, COAP, REST, JMS
HTTP Event Collector
Token Authenticated Events
Real-time
Technology Partnerships
Kepware, ThingWorx, Cisco, Palo Alto
Maintenance
Info
Asset
Info
Data
Stores
External Lookups/Enrichment
OT
Industrial Assets
IT
Consumer and
Mobile Devices
20. HTTP Event Collector
Supports DevOps and IoT data analysis needs at scale
20
• Standard API and logging libraries send events directly to Splunk
• Libraries integrated with popular platforms and services
Scales to Millions of Events/Second
AWS IoT
EVENT COLLECTOR
API
21. Advanced Analytics
Splunk ML Extensions
• New/Enhanced Commands: Data Sampling,
Fit, Apply, Summary, Predict
• Access to Python Data Science Library
• Model storage and export to production
Splunk ML App
• Step-wise guidance to create, test and deploy
custom ML models
• Purpose-built visualizations
• Sample data and best practices
Optimizing enterprise operations with predictive ML analytics
23. How VW Visualizes Connected Car Data
VW Data Labs
Connected Car program
Post-sales big data
visualization
Customer loyalty &
retention
IoT analysis & prediction of
customer needs
24. Sense and Respond
Use Splunk Alerts and Custom Alert Actions to trigger & automate workflows
● Allows packaged integration with
third-party applications
● Simple admin/user configuration
● Developers can build, package, and
publish alert actions within an app
● Growing list of integrations available
24
25. Splunk’s IoT and Industrial Partner Ecosystem
25
SDKs UI
Ingest and Platforms
IoT and ICS SecurityAdvanced Analytics and ML Custom User Interfaces
Services and Delivery
28. Zebra Technologies Corporation
28
Founded in 1969, Zebra offers customers a complete end-to-end
solution – from mobile computers and scanners to specialty
printers, RFID, software and services – for identifying, tracking
and managing critical assets, people and transactions.
Headquarters: Lincolnshire, IL
Offices: 122 offices across 81 countries
Employees: 7000+ globally
Market Capitalization: $5,640M1
Revenue: $2,275M 1
Profit (EBITDA): $439M 1
Mobile Computing
Printers
Data Capture (barcode scanners)
RFID
Location solutions
Wireless LAN
About
Global
presence
Financials
Products
2,000+ NFL players
tagged and tracked
since 2014, generating
more that 68+ billion
bytes of player
position data
1 Zebra corporate fact sheet, summer 2015
29. Challenge for Zebra Printer Division
29
Limited data analysis capabilities . . .
• Data not being collected for long-term use
• Reports not being generated
• Data not shareable across departments
• Changes to traditional database took a long time
• Minimal analytics capabilities (e.g., Excel)
• Unable to scale to increasing data volumes
Reactive mindset: Teams were reacting to
issues after they occurred
Little continuous improvement: Company
was unable to leverage data to improve
• Product design
• Manufacturing
• Repair and service
Resulted in . . .
Goal is to design a system that:
• Captures complete device and test history
• Is flexible and can be easily modified
• Is capable of robust analytics, trending, & alerting capabilities
• Can easily share data to enable better business decisions
30. Solution developed by Zebra
30
Test History
History of the test performed by
the test utility on the device
Test Data
Any data deemed necessary to
log and store
Device (Printer) Data
Raw data output of device
(4 data formats)
1 manufacturing center
8 service centers
3 reconfig centers
30+ applications
600+ computers globally
Allows for real-time view of production data to address low yields
and drive quality improvements
Communicates product performance over time, leveraging statistical
methods to baseline product performance
Anomaly detection analytics utilized to capture known defect
patterns that prevent product shipment
Dashboard
Reports
Analytics
31. Value realized by Zebra
31
Quick access to test history and device data allows
for enhanced customer experience
Faster customer
resolution
Enhanced product
development
Reduced
return rate
Improved business
decisions
Reduced cost
Data being captured early in development drives
improvements in design
Anomaly detection analytics used to capture known
defect patterns that prevent product shipment
Data being easily shared across Quality, Product,
Technical Support, and Services enables data-driven
decision making process
Quantified cost reduction of $90,000 per year
through media reduction alone
1
2
3
4
5
32. Planes, Trains and Automobiles
(and Coke, Zombies, Floods, Buildings, Crops and Medicine)
33. Data-driven Refreshment
Aggregate machine data from
Coca-Cola freestyle® machines
Insights into customer
interactions and decisions
Reduced Downtime and
Increased Consumer Satisfaction
Vending machine performance and
diagnostics
35. 35
How Gatwick Airport Ensures Better
Passenger Experience With Splunk Cloud
On-time efficiency & dramatic queue reduction with 925 flights per day
Real-time, predictive airfield analytics - mobile app & CEO’s apple watch
Data from airport gates, board pass scans, x-ray, travel, passenger flow
36.
37.
38. Saving The US Rail Industry A Billion Dollars And
250 Million Acres Of Trees in CO2
Train sensor data
Fuel savings
Better trained drivers
39. 24 Hour DB Hackathon
Highlight defect impact &
rail construction issues
Predictive maintenance
& reduced disruption
Transport, infrastructure,
environment & journey data
How Deutsche Bahn Analyzed
Tracks in 24 Hours
40.
41. Top ten types of notifications about issues
that have occurred in the transportation infrastructure
42. What kinds of defects occur and quantify how big the deviations are
Correlated with materials of the track sleepers: concrete (“Beton”) vs. wood (“Holz”)
Track deviations with indication notifications. Blue bars denote “no known issues”
43. Width of Sankey bar shows amount of track deviations between different destinations
Fulda and Frankfurt has high track deviations (it is the widest bar)
Indicates the need for upcoming repair, maintenance and possible renewal
45. Robot Analytics to Reduce Costs
in the Supply Chain
4%Increased
Throughput per
Distribution Center
Aggregate machine data
from robots
Failure pattern detection
and reporting
Preventative maintenance
scheduling
48. +
Content browsed,
purchased and
watched. All tracked by
time and MAC address
Customer
behavior
analytics
Customer
profile and MAC
address / device
assignments
Understanding Customer Behavior
49. IoT WORKLOADS
Blurring the Lines Between Digital & Physical
Security
Ops
Center
Business
Ops
Center
IT Ops
Center
CLOUD WORKLOADS ENTERPRISE IT
WORKLOADS
ADVANCED ANALYTICS
RAPID SOLUTION DEVELOPMENT
DATA INGEST AT SCALE
At Splunk, our mission is to make machine data accessible, usable and valuable to everyone. And this overarching mission is what drives our company and product priorities.
Splunk products are being used for data volumes ranging from gigabytes to hundreds of terabytes per day. Splunk software and cloud services reliably collects and indexes machine data, from a single source to tens of thousands of sources. All in real time. Once data is in Splunk Enterprise, you can search, analyze, report on and share insights form your data. The Splunk Enterprise platform is optimized for real-time, low-latency and interactivity, making it easy to explore, analyze and visualize your data. This is described as Operational Intelligence.
The insights gained from machine data support a number of use cases and can drive value across your organization.
Splunk provides an open, fully integrated platform. That means you can collect, index, analyze, report and predict on machine-generated data from a single product. It’s enterprise-ready with high availability and disaster recovery features, role-based access control and scales to index hundreds of terabytes per day. It’s an open platform with over 500 Splunk Apps available and allows for custom development.
More than 10,000 customers in 100 countries have purchased the enterprise license of Splunk. This includes a majority of the Fortune 100. Enterprises, service providers and government agencies in 100 countries use Splunk to improve service levels, reduce IT operations costs, mitigate security risks and drive new levels of operational visibility.
As they gain new visibility into their real-time and historical machine data, Splunk’s customers are finding answers and solving the most challenging issues facing IT and the business.
Now you can onboard data directly from any application or device– opening up new types of machine data to the benefits of Splunk analysis.
The new Event Collector makes it simple and efficient to collect this data, scaling to millions of events per second, using a developer-friendly, standard HTTP/JSON API and logging libraries
And NO FORWARDERS.
Today it is possible to send data directly to Splunk using Modular Inputs or a TCP connection, however this is not an efficient or scalable solution. While log files and forwarders provide an efficient mechanism for typical log and syslog files, use of files and forwarders is not possible or necessarily a desired data collection method for the world of custom applications DevOps, Docker, and other packaged application environments. The same is true for the world of IoT event data, where devices/apps need have no local storage, and even intermediate event collection systems and partners would prefer to use a real-time interface to Splunk rather than create specific log files and use forwarders.
The HTTP Event Collector (EC) uses a standard API and high-volume Splunk endpoint to allow events to be directly sent/collected at extreme velocity. The HTTP/JSON API is a developer-standard, whose simple but powerful functionality will be attractive to DevOps and custom application developers and operations managers. Without requiring new system configuration, log creation or administration support, developers can instrument their applications to understand usage flows, performance, error conditions and more. The interface/functionality is also a fit for IoT software developers to connect their devices either directly or via intermediate collection services. The data volumes supported by Splunk are ideal for the transactional and diagnostic data of devices such as Point-Of-Sale systems, vending machines, gaming consoles, automobiles and other devices/systems – opening up a new world of machine data to the benefits of Splunk analysis
An ecommerce saas provider uses EC to analyze the performance of their Docker apps
A large video game company uses EC to monitor security of a popular sports game. 16B events collected in first 2 months
NY Air Brake – Optimizing fuel efficiency per route
Large Telecom – Main use case is to forecast possible telecom network outages - “intelligent performance-based alarming”. Happy ITSI customer who wants to do more free form ML with more data.
Another Large Telecom – Proactive cell tower outage prediction and avoidance
Online real-estate marketplace and app – First use case is website app troubleshooting; Second is identifying fraudulent web visitors.
Electronics company specializing in video games – Predicting and avoiding online services outages
Financial Services company – Detecting lateral movement inside network
Another Financial Services company – ML services to app developers and support
Large food and beverage company – Real-time inventory forecasting
Industry: Automotive/Manufacturing
Use case: IoT and Analytics
More can be found at: http://www.techworld.com/news/big-data/volkswagen-group-uses-splunk-underpin-iot-experiment-3575744/
Splunk have been working with VW’s data labs on their connected car and IoT program. Splunk is being used to analyse sensor data from VW’s E-Up cars. Splunk’s ability to search, alert, report and analyse IoT data allowed VW to see a fleet of E-Up cars at CeBIT and analyse metrics such as speed, RPM, battery level, range, temperature, when the doors were open and when windscreen wipers were on. This data could also be combined with location, maps and smart watches to show where vehicles had been, heatmaps, and driver heart rate
Custom Alert Actions provide the ability to use Splunk Alerts to trigger custom actions or pre-packaged integrations with 3rd party products such as trouble ticketing or support systems. Developers can build and publish integrations or custom action packages that users or admins can use via a simple menu within the Splunk Alert Interface. Splunk and partners provide a growing set of integrations including, ServiceNow, xMatters, Webhooks and more. Previously these integrations were complex, ad-hoc efforts requiring custom scripts. The new scheme makes it simple for partners (and customers) to create and contribute out-of-the-box integration templates, and for customers to use them via a simple pull-down menu.
There are many free add-ons and Apps for Splunk software that simplify the connection and collection of data from both industrial systems and the Internet of Things. These include:
Rest API Modular Input: Poll local and remote REST APIs and index the responses.
Amazon Kinesis Modular Input: Index data from Amazon Kinesis, a fully managed service for real-time streaming data.
Apache Kafka Modular Input: Index messages from Apache Kafka messaging brokers, including clusters managed by Zookeeper.
DB Connect 2: Integrate structured data sources with your Splunk real-time machine data collection.
Universal Forwarder for Linux (ARM – Raspberry Pi): Dedicated Splunk package for Linux and ARM based systems where data needs to be collected directly from embedded devices such as the Raspberry Pi.
MQTT Modular Input: Index messages from MQTT, a machine-to-machine connectivity protocol, by subscribing Splunk software to MQTT Broker Topics.
AMQP Modular Input: Index data from message queues provided by AMQP brokers.
JMS Modular Input: Poll and index message queues and topics from messaging queues and topics, including MQTT messages, provided by message providers, including TibcoEMS, Weblogic JMS and ActiveMQ.
Protocol Data Inputs: Recieve data via a number of different data protocols such as TCP , TCP(s) ,HTTP(s) PUT/POST/File Upload , UDP , Websockets , SockJS.
Splunk App for Stream: Capture, filter and index real-time streaming wire data and network events.
COAP Modular Input: Index messages from a COAP (Constrained Application Protocol) Server.
SNMP Modular Input: Collect data by polling SNMP attributes and catching SNMP traps from datacenter infrastructure devices providing cooling and power distribution.
In addition, Splunk has a powerful ecosystem of technology partners.
Kepware Technologies – Connects Splunk software with thousands of industrial devices communicating on over a hundred proprietary industrial protocols. Stream real-time data to Splunk from industrial control systems, including SCADA.
Carvoyant – Connected car platform, integration with Splunk software allows enterprises to monitor their automobile fleets, including geo-location, engine parameters and diagnostics.
B&B SmartWorx – Intelligent sensors and gateways. Integration with Splunk (Splunk App) will include sensor data collection (via MQTT), and gateway and sensor network diagnostics and cyber security.
Bluvision– Intelligent beacons. Integration with Splunk (Splunk App) will include beacon data collection (via Websockets). Powerful retail applications.
ThingWorx (PTC) – The leading IoT Application Development Platform. Seamless data exchange between ThingWorx applications and Splunk Enterprise and Splunk Cloud, and ThingWorx customers can access Splunk search and analytics through the ThingWorx mashup builder.
Buddy.com – Cloud services for connected devices. Integration (Splunk App) will allow Splunk to stream data from any device connected to the buddy platform.
Octoblu (Citrix) – IoT developer platform. Has created libraries that allow any Octoblu-enabled device to stream its data to Splunk software and allows those same devices to use Splunk search and analytics to inform their own decisions and logic
Red Balloon Security – Security platform for the defense of embedded systems in the enterprise (IP Phones, Printers, switches and routers, etc). Uses proprietary firmware level protection and appliance-based endpoint monitoring, and is integrating (Splunk TA/ES Compliant CIM) with Splunk software to allow Enterprise Security monitoring of threats to embedded enterprise devices.
Bayshore Networks – Content-aware cyber security platform for industrial networks. Is integrating (Splunk TA/ES Compliant CIM) with Splunk software to allow Enterprise Security monitoring of threats to SCADA and other industrial networks.
Foxguard Solutions– Cyber security and compliance solutions for industrial networks built with Splunk. NERC-CIP compliance specialists.
UltraElectronics-3eti – Cyber security platform for industrial networks. Building ES compliant TA to allow collection and analysis of security relevant ICS data in Splunk.
Distrix – Software defined networking for industrial networks and the internet of things. Simplifies connectivity and delivers and enhances data over extremely complex networks. Distrix’s SDN supports Splunk to Splunk communication, and can enhance other data, including timestamping and meta-data enrichment, for ingestion in Splunk.
Prelert – Anomaly detection app for Splunk Enterprise. Valuable app for management of sensors and devices where rapid identification of anomalies in sensor readings or operations are critical.
Predikto – Leverages the power of Predictive Analytics enabling organizations to use their data to predict future asset failures.
N3N – Custom, advanced user interfaces for Splunk specializing in isometric views of industrial facilities.
R Project App – harness the power of R statistical processing language directly from Splunk interfaces and search processing language.
D3.js – Data driven documents for powerful user experiences.
HTML5 – Advanced web interfaces and applications for browser and mobile based user experience.
Coca-cola uses Splunk to understand connected vending machines, including the Coca-Cola Freestyle. They are able to better understand patterns in consumption as well as the performance of the machines, both of which ultimately lead to better customer satisfaction.
Splunk have been working with VW’s data labs on their connected car and IoT program. Splunk is being used to analyse sensor data from VW’s E-Up cars. Splunk’s ability to search, alert, report and analyse IoT data allowed VW to see a fleet of E-Up cars at CeBIT and analyse metrics such as speed, RPM, battery level, range, temperature, when the doors were open and when windscreen wipers were on. This data could also be combined with location, maps and smart watches to show where vehicles had been, heatmaps, and driver heart rate
Visuals need to be worked on
New York Air Brake’s Train Dynamic Systems Division is using Splunk to manage inter-train forces, the “slinky factor” inherent in large freight trains with 6 inches of flex between cars. With splunk, they are able to produce insight and reports allowing the owners of the locomotives they manage to better train the engineers, and better manage the acceleration and braking of the trains throughout thousand mile journeys. Managing this data with Splunk, they can produce 5-10% fuel savings for customers. For their largest customers this can mean a billion dollars in savings a year.
Energy efficiency is one of the most critical factors in any building project. It's estimated that commercial buildings consume nearly 20% of all energy in the U.S., the majority of which is used for lighting and indoor climate control. From site selection and layout to building materials and mechanical systems, energy efficiency is a primary goal at every stage in a building's lifecycle.
In early 2012, McKenney's was enlisted to assist Gulf Power and its partner Chevron Energy Solutions in implementing a new energy management system at Eglin Air Force Base through Gulf Power's GSA contract services in Florida. At 724 square miles, Eglin is one of the largest military bases in the world and includes hundreds of buildings and a base population of about 17,000.
The Enterprise Intelligence Group leveraged its in-house experience with Splunk to provide Eglin with continuous collection and aggregation of data from most of the energy management, building control systems and utility metering and monitoring systems on base, helping to monitor and analyze tens of thousands of sensors and data inputs from HVAC systems in more than 100 Eglin buildings.
The new Eglin energy management system (EMS) will leverage the Splunk to provide dashboards that will help base maintenance staff to assess building performance and energy efficiency, generate automated Air Force/DoD energy usage reports, compare current energy usage with historical data, and enable the deployment of load shedding and load shifting strategies to take advantage of favorable electric rates. The project is projected to save about $2.5 million annually, with a payback period of less than three years.
Comcast Corporation (Nasdaq: CMCSA, CMCSK) (www.comcast.com) is one of the world’s leading media, entertainment and communications companies.
Comcast has many different Splunk use cases. One of their use cases involves taking data from the set-top boxes to gain real time insights in to customer interaction with content served up by the set top box. Each set top box has a media access control (MAC) address that is unique and is associated with a specific customer. The set top box is capturing all customer interaction with device including which content the customer searched for, what the date of search was, what search results were displayed (this information is recorded a unique identifiers called IDA numbers) and what content was purchased. However, the set top box does not have any information on the customer including their profile. That information is stored in the billing system. Comcast is using Splunk to correlate data across set top boxes and billing systems to gain real-time business insights.
Using the correlation criteria of MAC address, content displayed in search and time of purchase, Comcast is gaining a broad range of business insights into their customers. For example, these insights are helping Comcast understand revenues driven by search. By overlaying this information with geo location data, they are able to improve content mix and drive higher monetization. These insights are also helping Comcast improve content promotion based on region.
Comcast is using the Splunk and Hadoop integration to visualize Comcast setbox log information. The setbox data comes to Hadoop, get pre processed and moved to Splunk for visualization.
Hadoop Input = High volume of data from many systems along a complex workflow, Developers expressing artistic prerogative on log formats, Many different data sources and formats
Splunk Output = Drive operational intelligence, Improve user experience, Troubleshooting, root cause analysis, Track and measure success, Reports, alarms