Weitere ähnliche Inhalte Ähnlich wie Rescue your Big Data from Downtime with HP Operations Bridge and Apache Hadoop (20) Mehr von Hortonworks (20) Kürzlich hochgeladen (20) Rescue your Big Data from Downtime with HP Operations Bridge and Apache Hadoop1. We’ll get started soon…
Q&A box is available for your questions
Webinar will be recorded for future viewing
Thank you for joining!
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
2. Rescue your Big Data from Downtime with
HP Operations Bridge and Apache Hadoop
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
We do Hadoop.
3. Your speakers…
Ajay Singh, Director of Technical Channels
Hortonworks
Saumya Upadhyaya, Product Manager
HP
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
5. Traditional systems under pressure
RDBMS EDW MPP
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
DATA SYSTEM APPLICATIONS
Business
Analytics
Custom
Applications
Packaged
Applications
• Silos of Data
• Costly to Scale
• Constrained Schemas
Clickstream
Geolocation
Sentiment, Web Data
Sensor. Machine Data
Unstructured docs,
emails
Server logs
SOURCES
Existing Sources
(CRM, ERP,…)
New Data Types
…and difficult to
manage new data
6. HDP is deeply integrated in the data center
YARN
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
DEV
&
DATA
TOOLS
OPERATIONAL
TOOLS
INFRASTRUCTURE
SOURCES
EXISTING
Systems
Clickstream
Web
&Social
GeolocaBon
Sensor
&
Machine
Server
Logs
Unstructured
DATA SYSTEM
RDBMS
EDW
MPP
HANA
APPLICATIONS
BusinessObjects BI
Deep Partnerships
Hortonworks engages
in deep engineered relationships
with the leaders in the data center,
such as HP, Microsoft, Redhat, SAP,
SAS & Teradata
Broad Partnerships
Over 600 partners work with us to
certify their applications to work with
Hadoop so they can extend big data
to their users
HDP 2.1
Governance
& Integration
Security
Operations
Data Access
Data Management
7. 1. Unlock New Applications from New Types of Data
INDUSTRY USE CASE Sentiment
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
& Web
Clickstream
& Behavior
Machine
& Sensor Geographic Server Logs Structured &
Unstructured
Financial Services
New Account Risk Screens ✔ ✔
Trading Risk ✔
Insurance Underwriting ✔ ✔ ✔
Telecom
Call Detail Records (CDR) ✔ ✔
Infrastructure Investment ✔ ✔
Real-time Bandwidth Allocation ✔ ✔ ✔
Retail
360° View of the Customer ✔ ✔ ✔
Localized, Personalized Promotions ✔
Website Optimization ✔
Manufacturing
Supply Chain and Logistics ✔
Assembly Line Quality Assurance ✔
Crowd-sourced Quality Assurance ✔
Healthcare
Use Genomic Data in Medial Trials ✔ ✔ ✔
Monitor Patient Vitals in Real-Time
Pharmaceuticals
Recruit and Retain Patients for Drug Trials ✔ ✔
Improve Prescription Adherence ✔ ✔ ✔ ✔
Oil & Gas
Unify Exploration & Production Data ✔ ✔ ✔ ✔
Monitor Rig Safety in Real-Time ✔ ✔ ✔
Government
ETL Offload/Federal Budgetary Pressures ✔ ✔
Sentiment Analysis for Gov. Programs ✔
8. 2. Or to realize a dramatic cost savings…
EDW Optimization
✚
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
OPERATIONS
50%
ANALYTICS
20%
ETL PROCESS
30%
OPERATIONS
50% ANALYTICS
50%
Current Reality
EDW at capacity: some usage
from low value workloads
Older data archived, unavailable
for ongoing exploration
Source data often discarded
Augment w/ Hadoop
Free up EDW resources from
low value tasks
Keep 100% of source data and historical
data for ongoing exploration
Mine data for value after loading it
because of schema-on-read
Commodity Compute & Storage
Hadoop Enables Scalable Compute &
Storage at a Compelling Cost Structure
Cloud Storage
HADOOP
Engineered System
MPP
SAN
NAS
$0 $20,000 $40,000 $60,000 $80,000 $180,000
Fully-loaded Cost Per Raw TB of Data (Min–Max Cost)
Hadoop
Parse, Cleanse
Apply Structure, Transform
Storage Costs/Compute
Costs from $19/GB to
$0.23/GB
9. 3. Data Lake: An architectural shift
Unlocking the Data Lake
SCALE SCOPE
RDBM
S
MPP
EDW
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
Data Lake
Enabled by YARN
• Single data repository,
shared infrastructure
• Multiple biz apps
accessing all the data
• Enable a shift from
reactive to proactive
interactions
• Gain new insight across
the entire enterprise
New Analytic Apps
or IT Optimization
HDP 2.1
Governance
& Integration
Security
Operations
Data Access
YARN
Data Management
10. HDP is deeply integrated in the data center
YARN
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
DEV
&
DATA
TOOLS
OPERATIONAL
TOOLS
INFRASTRUCTURE
SOURCES
EXISTING
Systems
Clickstream
Web
&Social
GeolocaBon
Sensor
&
Machine
Server
Logs
Unstructured
DATA SYSTEM
RDBMS
EDW
MPP
HANA
APPLICATIONS
BusinessObjects BI
Deep Partnerships
Hortonworks engages
in deep engineered relationships
with the leaders in the data center,
such as Microsoft, Teradata, Redhat,
HP, SAS & SAP
Broad Partnerships
Over 600 partners work with us to
certify their applications to work with
Hadoop so they can extend big data
to their users
HDP 2.1 Governance
& Integration
Security
Operations
Data Access
Data Management
12. The 1st Generation of Hadoop: Batch
HADOOP 1.0
Built for Web-Scale Batch Apps
Single
App
INTERACTIVE
Single
App
BATCH
HDFS
Single
App
BATCH
HDFS
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
• All other usage patterns
must leverage that same
infrastructure
• Forces the creation of silos
for managing mixed
workloads
Single
App
ONLINE
Single
App
BATCH
HDFS
13. 2006 2009
Hadoop
w/
MapReduce
MapReduce
Largely
Batch
Processing
HDFS
MR-‐279:
YARN
1
°
°
°
°
°
(Hadoop
Distributed
File
System)
°
°
°
°
°
N
Hadoop2 & YARN based Architecture
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
YARN: Data Operating System
1
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
N°
HDFS
(Hadoop Distributed File System)
Silo’d clusters
Largely batch system
Difficult to integrate
Hadoop 2 & YARN
Batch Interactive Real-Time
Architected &
led development
of YARN to enable
the Modern Data
Architecture
October 23, 2013
14. A Blueprint for Enterprise Hadoop
Load data
and manage
according
to policy
PRESENTATION & APPLICATION
ENTERPRISE MGMT & SECURITY
DATA ACCESS SECURITY
Access your data simultaneously in multiple ways
(batch, interactive, real-time) Provide layered
YARN Data Operating System
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
Deploy and
effectively
manage the
platform
Store and process all of your Corporate Data Assets
approach to
security through
Authentication,
Authorization,
Accounting, and
Data Protection
DATA MANAGEMENT
GOVERNANCE
&
INTEGRATION
OPERATIONS
Enable both existing and new application to
provide value to the organization
Empower existing operations and
security tools to manage Hadoop
Provide deployment choice across physical, virtual,
cloud
DEPLOYMENT OPTIONS
15. HDP delivers a comprehensive data management platform
BATCH, INTERACTIVE & REAL-TIME SECURITY
DATA ACCESS
TezTez
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
YARN is the architectural
center of HDP
• Enables batch, interactive
and real-time workloads
• Single SQL engine for both batch
and interactive
• Enable existing ISV apps to plug
directly into Hadoop via YARN
Provides comprehensive
enterprise capabilities
• Governance
• Security
• Operations
The widest range of
deployment options
• Linux & Windows
• On premise & cloud
HDP 2.1
Hortonworks Data Platform
Provision,
Manage &
Monitor
Ambari
Zookeeper
Scheduling
Oozie
Data Workflow,
Lifecycle &
Governance
Falcon
Sqoop
Flume
NFS
WebHDFS
In-Memory
Spark
YARN: Data Operating System
DATA MANAGEMENT
GOVERNANCE
&
INTEGRATION
Authentication
Authorization
Accounting
Data Protection
Storage: HDFS
Resources: YARN
Access: Hive, …
Pipeline: Falcon
Cluster: Knox
OPERATIONS
Script
Pig
Search
Solr
SQL
Hive
HCatalog
NoSQL
HBase
Accumulo
Stream
Storm
Others
ISV
Engines
1 ° ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° ° °
°
°
N
HDFS
(Hadoop Distributed File System)
Deployment Choice
Linux Windows On-Premise Cloud
17. How do you Operate a Hadoop Cluster?
Apache Ambari is a
platform to provision,
manage and monitor
Hadoop clusters
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
18. Innovation at the Speed of Community
An active 100% open source Community:
• 90+ Contributors / 45+ Committers
• 270+ Ambari User Group Members
Graduates to Apache Top Level Project
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
2013
Dec
19. Apache Ambari Themes
Operate
Hadoop
at
Scale
Deliver
the
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
core
opera@onal
capabili@es
to
provision,
manage
and
monitor
Hadoop
clusters
at
scale.
Integrate
with
the
Enterprise
Robust
API
for
integra@on
with
exisBng
enterprise
systems,
such
as
Teradata
Viewpoint,
HP
OperaBons
Bridge
and
MicrosoY
SCOM.
Extend
for
the
Ecosystem
Provide
extensible
pla[orm
for
Customers,
Partners
and
the
Community
using
Stacks
and
Views.
20. Hadoop Operations Platform
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
Apache Ambari is a 100% open source platform for
provisioning, managing and monitoring Apache Hadoop
clusters
AMBARI
WEB
HP
Opera)ons
Bridge
compute
&
. . .
storage . . .
. . compute
&
storage
.
EXTEND .
AMBARI
REST
API
AMBARI
SERVER
PROVISION
|
MANAGE
|
MONITOR
IntegraBon
with
ExisBng
Tools
OPERATE
AMBARI
STACKS
21. Cluster Operations
Start,
Stop
and
Smoke
Test
Services
Apply
configuraBon
changes
and
inject
custom
properBes
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
23. Host Level Visibility + Controls
View System Metrics and Host Information
Manage Host Components, including start/stop/decommission
“Add Hosts” to expand cluster capacity
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
24. Cluster Heatmaps
Gain health visibility of entire cluster
Quickly identify host outliers and potential issues
Includes both System and Hadoop Metric Heatmaps
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
25. Hadoop Metrics
Hadoop Service metrics such as HDFS, MapReduce and HBase
Visibility info DFS capacity, file operations, RPC latency and JVM
performance
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
26. Hadoop + System Monitoring
• Watch key Hadoop operational metrics for
all services
• Master service availability, CPU utilization,
network latency, storage capacity
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
© Hortonworks Inc. 2014
• Host-level alerting
• Component service processes and storage
availability
28. Extensibility Features
Goal: Extend Ambari without hard-coding in Ambari
• To add new Services (ISV or otherwise) beyond HDP Stack
• To customize a Stack for customer specific environments
• To use Ambari for automating cluster installations
• To share best practices on layout and cluster configuration
• To extend and customize the Ambari Web UI
• Add new capabilities, customize existing capabilities
Stacks
Blueprints
Views
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
29. Bringing it all together
HP Operations Bridge
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
30. No context = high risk
Traditional Private Managed Public
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 30 without notice.
High Opex
Point Tools
Uncorrelated
Silos
Slow Triage
Inaccurate
Reports
Inefficient
Sea of red
What’s going wrong?
31. The typical challenge – application siloes
Application performance
monitored by customers 2
No link between
infrastructure and
application worlds
4
Can’t do automated
remediation – not
totally sure of root
cause
5 • Slow resolution
Can’t consolidate
performance data
to drive planning
Silo-ed health management and
1 Silo-ed view of what’s happening
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 31 without notice.
• Not business aligned
• Not people efficient
• Poor planning & optimization
• Monitoring duplication
3
32. 3 Steps to deliver
business SLAs
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
33. Solution: ACT - Aggregate
• Single pane of glass across performance, events & topology
• Consolidate the data across all point tools & applications
Single pane of glass
Access from anywhere
Role driven
Simple navigation
Drill down to detail
Integrations
Biz Impact aware
Traditional to New Style of IT
Depth via Management Packs
Breadth via Connectors &
HP portfolio
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 33 without notice.
Event Detection &
Enrichment
Event Correlation &
Business Impact
Incident Submission
Recovery & Closure
Investigation &
Diagnosis
Resolution
Closed Loop Management
Integrated in IT processes
Bi-Directional integration of
HP & 3rd party tools
Downtime & Assets synchronized via
common repository
www.hp.com/go/hpln
34. Visibility into your Hortonworks Hadoop Environment
Leverage Apache Ambari & OpsBridge interfaces for integration
BSM Connector for Apache Ambari
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 34 without notice.
Topology Events
35. Solution: ACT - Control
• Correlate data to find root cause via the RTSM
• Automated actions & monitoring
Monitoring
Self-maintaining monitoring
Tool & Application Agnostic1
SME self-service portal
Centralized Config & Reports
Remediation
Link to CI or root cause
Custom Actions or
Run Book Automation2
No Operator involvement
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 35 without notice.
Predictive3
Automated baselining
Identify anomalies
Proactive alerting
Across BSM data (HP & 3rd party)
1) SiS, ArcSight, OM Agents, Oracle, MS-SQL, HP Cloud, Vertica, Hadoop
2) Enabled through integration i.e. HP Operations Orchestration
3) Enabled through integration with Service Health Analyzer
36. Simplify IT Operations
Out of the box monitoring deployment, orchestration & ticketing integrations
Customizable Dashboard
Topology Events
Save expensive expert personnel hours by
• Identifying actionable events through correlation
• Automate remediation of frequent, batch events
• Automate ticket creation to the right audience
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 36 without notice.
Service Desk
Orchestrate
Analyze
37. Solution: ACT - Team
• Collaborate across silos & teams
• Gamify to engage operators and unlock efficiencies
Achievements
• Motivate Users
• Guide desired behavior
• Explore the product
Experts Board
• Identify experts per area
• Improve collaboration
• Increase efficiency
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 37 without notice.
More to Come…
• Exploration Map
• Custom rewards
• Challenges
• Social features
38. What Hortonworks & HP Software are doing
Managing Big Data for efficient Digital Business
New HP Operations Bridge for Apache AMBARI
• Native AMBARI API integration
• Provisioning, managing and monitoring Apache Hadoop
clusters
• Supports HortonWorks Data Platform i.e. HDFS, Yarn, Hbase,
…
• Events and topology from the above
Gain even more value capabilities using
HP Operations Bridge Hadoop Management Pack
• Events, topology, metrics for Hadoop HDFS and MapReduce
• Management cockpit dashboards
• OOTB SHR Content report packs
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 38 without notice.
NEW
BSM Connector for Apache Ambari
Download free at HP Live Network
39. Applying Predictive IT Operations to Hadoop
Automated Predictive Operations1 and HortonWorks Data Platform integrations
HP Operations Bridge
Topology Events
Automated
Remediation
Automated
Tickets
Operations
Analytics
HP Operations
BSriimdpglifey Unify, Automate
your WebScale
IT
Save expensive expert personnel hours by
• Identifying actionable events through correlation
• Automate remediation of frequent, batch events
• Automate ticket creation to the right audience
1 –Requires OpsBridge used in BSM configuration and Hadoop Management pack
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 39 without notice.
40. Wide Coverage of Tools & Application Coverage
Out of the box | 46+ domains covered today | expanding rapidly
AIX CentOS Cluster Debian HPUX HyperV IBM LPAR KVM
RedHat Oracle Ent.
Citrix
Linux Solaris Solaris Zone Suse Ubuntu VMWare
vSphere Windows XEN
Infrastructure
Applications Oracle DB Oracle RAC SQL Server SAP
Sybase MS Exchange IBM
Websphere
WMQ Siebel Documentum
Apache
Webserver MS IIS SunOne iPlanet TIBCO
JBOSS MS Lync MS
Sharepoint MS Biztalk Oracle AS Informix Peoplesoft Tuxedo
Big Data & Cloud HP Cloud Amazon Web
SAP HANA
DB
Vertica DB
Weblogic
Services
AD
Ticketing Remedy Radia
integrations HP ITSM Service Now
Impact
Manager
AppDynamics Aternity CA Spectrum
CollectD SIM IBM Tivoli MS SCOM
3rd Party
Integrations
Solarwinds VC
Nagios OEM Ops Zabbix ZenOSS xMatters
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 40 without notice.
Supported with
OMi
Supported with
OM only
DB2
Domino
MS SCOM
Apache
Ambari
41. HP Software and Hortonworks
Managing Big Data for efficient Digital Business
Demo
Remedial action
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 41 without notice.
Predicted anomaly
alert slow
response
marketing analysis
Operations
Bridge
+
HDP AMBARI
Operator notices
Hbase alert via
AMBARI
integration
Dependency
confirmed in
Service tree
ticket opened
Hadoop admin
sees ticket +
context
taken
(manual or
automatic)
Marketing Analysis
Performance
v
v
42. HP Operations Bridge allows you to “ACT”
Simplifying your complex world
Aggregate
• Single pane of glass across performance, events & topology
• Consolidate the data across all point tools & applications
Control
• Correlate data to find root cause via the RTSM
• Automate actions & monitoring
Team
• Collaborate across silos & teams
• Gamify to engage operators and unlock efficiencies
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 42 without notice.
43. Tangible benefits
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 43 without notice.
30% reduction in help
desk call
60%+ reduction in
cumulative monthly
downtime
90% reduction in
Time to Restore
Service
30% reduced cost
• Single consolidated view of domain managers
• model-driven, X-domain reporting & forecasting
60% increased availability
• Advanced business aware & model driven
correlation
• Faster identification of root cause
• More efficient operations
90% improved efficiency
• Single dashboard frees staff time for innovation
• consolidated information
• high degree of automation
44. Get more information
Check out the OpsBridge Web Page
• www.hp.com/go/OpsBridge
Check out the latest OpsBridge Integration Content
• https://hpln.hp.com/group/operations-bridge
Check out the latest webinars / hangouts
• Operations Bridge Hangout, Operations Bridge Webinar
• Webinar “Learn how DIRECTV Embraced deep-dive tools for end-to-end
visibility across IT” (link)
• Webinar “Look No Hands - How Automation benefits IT Operations in
Data Centers at Sykehuspartner and PlayTech” (Coming on Nov 19,
2014)
• OpsBridge announcements at HP Discover (Coming on Dec 17,
2014)
Check out the latest #HPBSM blogs
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change 44 without notice.
45. Next Steps…
More about HP & Hortonworks
http://hortonworks.com/partner/HP/
Download the Hortonworks Sandbox
Learn Hadoop
Build Your Analytic App
Try Hadoop 2
Contact us: events@hortonworks.com
© Hortonworks Inc. 2011 – 2014. All Rights Reserved