Learn more: http://hortonworks.com/hdf/ Log data can be complex to capture, typically collected in limited amounts and difficult to operationalize at scale. HDF expands the capabilities of log analytics integration options for easy and secure edge analytics of log files in the following ways: More efficient collection and movement of log data by prioritizing, enriching and/or transforming data at the edge to dynamically separate critical data. The relevant data is then delivered into log analytics systems in a real-time, prioritized and secure manner. Cost-effective expansion of existing log analytics infrastructure by improving error detection and troubleshooting through more comprehensive data sets. Intelligent edge analytics to support real-time content-based routing, prioritization, and simultaneous delivery of data into Connected Data Platforms, log analytics and reporting systems for comprehensive coverage and retention of Internet of Anything data.

1. Optimizing Log
Analytics from the Edge
April 2016
© Hortonworks Inc. 2011 – 2015. All Rights Reserved

2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
About Hortonworks
Customer Momentum
~800 customers (as of Feb 10, 2016)
Publicly traded on NASDAQ: HDP
Hortonworks Data Platform
Completely open multi-tenant platform
for any app and any data
Consistent enterprise services for security,
operations, and governance
Partner for Customer Success
Leader in open-source community, focused
on innovation to meet enterprise needs
Unrivaled Hadoop support subscriptions
Founded in 2011
Original 24 architects, developers,
operators of Hadoop from Yahoo!
800+
E M P L O Y E E S
1500+
E C O S Y S T E M
PA R T N E R S

3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
EMBRACE AN
OPEN APPROACH
MASTER THE
VALUE OF DATA
EVERY BUSINESS
IS A DATA BUSINESS

4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
DATA
AT REST
DATA
IN MOTION
ACTIONABLE
INTELLIGENCE
MODERN DATA APPLICATIONS
Actionable
Intelligence from
Connected Data
Platforms
Capturing perishable
insights from data in motion
Ensuring rich, historical insights on
data at rest
Necessary for modern data
applications
Hortonworks
DataFlow
Hortonworks
Data Platform

5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Optimizing Log Ingest with
Hortonworks DataFlow

6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Why Hortonworks DataFlow?
Because even the best data scientists
and most powerful platforms need
the right data to analyze

7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Store Data
Process and Analyze
Data
Acquire Data
Perception of DataFlows: Easy, Definitive
Dataflow

8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Reality of Dataflows: Complex, Convoluted
Store Data
Process and Analyze
Data
Acquire Data
Store DataStore Data
Store Data
Store Data
Acquire Data
Acquire Data
Acquire Data
Dataflow

9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
HDF has 130+ Processors - Multiple for Log Analytics
HTTP
Syslog
Email
HTML
Image
Hash Encrypt
Extract
TailMerge
Evaluate
Duplicate Execute
Scan
GeoEnrich
Replace
ConvertSplit
Translate
HL7
FTP
UDP
XML
SFTP
Route Content
Route Context
Route Text
Control Rate
Distribute Load
AMQP

10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Log Analytics Systems Today
LOG
ANALYTICS
PLATFORMNetwork
Device Logs
• Not all data can be captured
• Not all captured data is valuable
• Transport all data

11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Cost Effectively Expand Storage Options of Log Data
LOG
ANALYTICS
PLATFORM
Network
Device Logs
HDP
HDF
3. Cost effectively
expand collection and
grow timescale of logs
collected
2. Content-based routing
based on dynamic
evaluation of content,
attributes, priority
1. Integrate and
enrich logs across
data centers and
security zones

12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Efficiently Expand Log Ingestion from the Edge
LOG
ANALYTICS
PLATFORM
Network
Device Logs
HDF
HDF
HDF
HDPHDF
• Expand collection to new sources of machine data
• Edge analytics to transform, enrich and prioritize content based routing
• Capture and transport only valuable data

13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Expand Analytics and Reporting Options with HDP
LOG
ANALYTICS
PLATFORM
Network
Device Logs
HDF
HDF
HDF
HDPHDF
ODBC interface
traditional BI tools
Easy access to log analytics data
through traditional BI tools
Give data scientists better
tooling – Spark, Storm etc

14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Expand to small scale, remote systems
LOG
ANALYTICS
PLATFORM
Network
Device Logs
HDF
HDF
HDF
HDPHDF

15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Optimize Log Analytics with Content Based Routing
LOG
ANALYTICS
PLATFORM
Edge analytics for cost-effective
and efficient movement of
machine data
HDF
Intelligent, content based
routing, transformation
and enrichment
Send data to alternative
systems based on value,
content, priority
HDP
HDF
HDF
HDF

16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Splunk Optimization:
Using HDP as Data Refinery

17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Splunk Hadoop Connect
17
 Reliable bi-directional integration
Import
Browse
Export
Splunk Hadoop Connect
>2000 downloads
HA Indexes and
Storage
Commodity
Servers
Hadoop
(MapReduce &
HDFS)
Report &
analyze
Custom
dashboards
Monitor
and alert
Ad hoc
search

18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Splunk, Hunk & Hortonworks
YARN Ready Partner
Certified on Hortonworks Data Platform
Existing Sandbox tutorial

19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Splunk, Part of the Modern Data Architecture
• Bi-directional data integration
between Splunk & HDP
• Collect data from across the
organization, deliver it to Hadoop
for refining data and batch
analytics
• Output of Hadoop jobs can be
imported into Splunk Enterprise
for rapid analysis and visualization
• Archiving from Splunk Enterprise
to Hadoop

20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Splunk, Part of the Modern Data Architecture
• Bi-directional data integration
between Splunk & HDP
• Collect data from across the
organization, deliver it to
Hadoop for refining data and
batch analytics
• Output of Hadoop jobs can be
imported into Splunk Enterprise
for rapid analysis and
visualization
• Archiving from Splunk Enterprise
to Hadoop

21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hunk + Hortonworks
21
Explore, analyze and visualize data in
HDP from one integrated platform
Simply point Hunk at your HDP cluster(s)
and start exploring data immediately
Search data, change perspectives and
preview results as MapReduce jobs run
INTERACTIVE
EXPLORATION
RICH DEVELOPER
ENVIRONMENT
Build big data apps on data in HDP using
standard web languages and frameworks
FULL-FEATURED
ANALYTICS
FAST TO DEPLOY
AND DRIVE VALUE

22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Augment Splunk Deployment with Hortonworks Data Platform
Heavy Indexer
Universal
Forwarders
HDP
Enables
Splunk Storage
• Expansion to more data than previously feasible
• Archive data from Splunk into Hadoop
• Query archived Splunk data in Hadoop
• Focus Splunk infrastructure on what really matters

23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Find out how you much can optimize
your log analytics infrastructure today.
Contact sales@hortonworks.com