On Demand HDP Clusters using Cloudbreak and Ambari

On-Demand HDP Clusters using Cloudbreak and Ambari
Karthik Karuppaiya Narendra Bidari
Sr. Engineering Manager, CPE Sr. So0ware Engineer, CPE
Dublin Hadoop Summit 2016 – Karthik Karuppaiya & Narendra Bidari

Agenda
IntroducIon 1
Big Data PlaJorm Challenges 2
What is the soluIon? 3
Self Service AnalyIcs 4
Going Hybrid Cloud using Cloudbreak 5

IngesIng Data 6

IntroducFon
Symantec
-  Symantec is the world leader in providing security so0ware for both enterprises
and end users
-  There are 1000’s of Enterprises and more than 400 million devices (Pcs, Tablets and
Phones) that rely on Symantec to help them secure their assets from aXacks,
including their data centers, emails and other sensiIve data
Cloud PlaHorm Engineering (CPE)
-  Build consolidated cloud infrastructure and plaJorm services for next generaIon
data powered Symantec applicaIons
-  A big data plaJorm for batch and stream analyIcs integrated with Openstack
-  Open source components as building blocks
-  Hadoop and Openstack
-  Bridge feature gaps and contribute back

Big Data PlaHorm Challenge
• Hundreds of millions of users generaIng Billions of events every
day from across the globe
• Hundreds of Big Data ApplicaIon Developers developing 1000s
of applicaIons
• At 12 PB and 500+ nodes, Cloud PlaJorm Engineering AnalyIcs
team built the largest security data lake at Symantec

• Great! Now Developers can start building applicaIons on our Big Data Lake
• 100s of developers start building applicaIons using diﬀerent big data tools

• Product team developers wants quick changes, latest versions
• PlaJorm team wants stability!
• Soon, frustraIon prevails


What is the SoluFon?
• Build and use your own liXle cluster for development
• Copy subset of data for development purposes
• Tear down the cluster a0er development is complete
• Repeat and Rinse


What is the SoluFon?
• But Building clusters are hard and Ime consuming
• Too many services to install and conﬁgure
• Developers are not interested in building and managing clusters


What is the SoluFon? – Self Service
• What if we make it really easy to build clusters?
• Abstract all the deployment complexiIes and enable developers to get their own
cluster in one click of a buXon

What is the SoluFon? – Extend Ambari
• What about the services that are not supported by Ambari out of the box?
• We write our own Ambari custom stack

Self Service AnalyFcs (SSA) Clusters
• RESTful web services to allow creaIon and management of custom clusters
• Select from pre-deﬁned Ambari Blueprints
• Spins up VMs on our private Openstack cloud
• Installs HDP stack speciﬁed as part of Ambari blueprint
• Dashing dashboard to monitor and manage (start/stop/kill) clusters

Environment
• Private cloud on Openstack
• HDP 2.3.2
• Ambari 2.1.2
• Ability to seamlessly support public cloud like AWS


SSA Architecture


SSA Services


SSA Demo

SSA Pros and Cons
•  Pros
– Automates spinning up clusters
– Gets cluster up and running with HDP in minutes
– Users can spin up/kill clusters at will
– Central Dashboard to manage clusters
– Customizable to cater to our private cloud
•  Cons
– Tightly coupled with our speciﬁc private cloud infrastructure
– Not portable to work with other public cloud vendors

Next Gen SSA
•  This is all great! But, we are out of capacity on our private openstack cloud.
•  Just use the same so0ware to set up clusters on AWS and Google Cloud – the same
code should work right?

Next Gen SSA – Cloudbreak
•  Cloudbreak
– Cloudbreak helps to simplify the provisioning of HDP clusters in cloud environments
– Supports mulIple clouds including AWS, Google, Azure and Openstack
– Uses Apache Ambari for HDP installaIon and management
– Has a nice UI to build and manage clusters
– Supports automated cluster scaling

Hybrid Cloud Using Cloudbreak - Gaps
•  Cloudbreak’s out of the box support for openstack is limited
–  No support for Keystone v3 – private cloud uses keystone v3
–  No support for native Openstack APIs
–  Supports only Heat template for cluster provisioning
•  Now we have two diﬀerent systems – one for private openstack cloud and one for
AWS


Symantec’s ContribuFon to Cloudbreak
•  Keystone v3 support
–  Cloudbreak 1.2 – released 03/2016
•  NaIve Openstack API support (without using Heat template)
– Code development is done, will be contributed back soon.

Cloudbreak – Keystonre V3 Screenshot

Cloudbreak – Keystone V3 Project Scope Screenshot

Where is my data?
•  Great! I now have a shiny new cluster – but what do I do with no data?

Data IngesFon Service
•  Built as a Storm-Trident Topology
•  Support various sources like Kaja, RabbitMQ, HDFS, Cassandra, etc
•  Pluggable interface to add more sources
•  Sample the data, for tesIng purposes.
•  Generalized template provides ability to add data transformaIon
•  Ability to monitor data transfer and control rate of transfer
•  Trying to achieve Exactly once message Delivery

Data IngesFon Service Architecture

Data IngesFon Service Demo

Summary and Future Work
•  A journey towards one click cluster deployment
•  Cloudbreak - one tool for all cloud
-  Enable Cloudbreak to support our version of Openstack
-  Enable Cloudbreak to support Baremetal cluster provisioning
-  Single large YARN cluster for variety of compute and storage loads
•  Open source – use and contribute
-  Work with community to address gaps
•  SSA and Data IngesIon code already opensourced
-  hXps://github.com/symantec/

Thank You!

Q & A

Karthik Karuppaiya
karthik_karuppaiya@symantec.com

Narendra Bidari
narendra_bidari@symantec.com


On Demand HDP Clusters using Cloudbreak and Ambari

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie On Demand HDP Clusters using Cloudbreak and Ambari

Ähnlich wie On Demand HDP Clusters using Cloudbreak and Ambari (20)

Mehr von DataWorks Summit/Hadoop Summit

Mehr von DataWorks Summit/Hadoop Summit (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

On Demand HDP Clusters using Cloudbreak and Ambari