Simplified Cluster Operation & Troubleshooting

1. Simplified Cluster Operation &
Troubleshooting
Alejandro Fernandez + Jayush Luniya

2. Speakers
Alejandro Fernandez
Sr. Software Engineer @ Hortonworks
Apache Ambari PMC
alejandro@apache.org
Jayush Luniya
Staff Engineer @ Hortonworks
Apache Ambari PMC
jluniya@apache.org

3. What is Apache Ambari?
Apache Ambari is the open-source platform to
provision, manage and monitor Hadoop clusters

4. New Enterprise Features
Ambari 2.4
• New Services: Log Search, Zeppelin, Hive LLAP
• Role Based Access Control
• Management Packs
• Grafana UI for Ambari Metrics System
• New Views: Zeppelin, Storm

5. Apache Ambari Jiras
April
2015
1690
1864
277
379
797
206
488
July - Sept
2015
Dec 2015 –
Feb 2016
Today
v2.0
v2.1
v2.2
v2.4
1542 and
growing

6. Deploy
Secure/L
DAP
Smart
Configs
Monitor
Upgrade
Scale,
Extend,
Analyze
Simply Operations - Lifecycle
Ease-of-Use Deploy

7. Deploy On Premise
Ambari UI wizard handles all of these
combinations and makes recommendations
based on host specs.

8. Deploy On The Cloud
Certified environments
Sysprepped VMs
Hundreds of similar clusters

9. Deploy with Blueprints
• Systematic way of defining a cluster
• Export existing cluster into blueprint
/api/v1/clusters/:clusterName?format=blueprint
Configs Topology Hosts Cluster

10. Create a cluster with Blueprints
{
"configurations" : [
{
"hdfs-site" : {
"dfs.datanode.data.dir" : "/hadoop/1,
/hadoop/2,/hadoop/3"
}
}
],
"host_groups" : [
{
"name" : "master-host",
"components" : [
{ "name" : "NAMENODE” },
{ "name" : "RESOURCEMANAGER” },
…
],
"cardinality" : "1"
},
{
"name" : "worker-host",
"components" : [
{ "name" : "DATANODE" },
{ "name" : "NODEMANAGER” },
…
],
"cardinality" : "1+"
},
],
"Blueprints" : {
"stack_name" : "HDP",
"stack_version" : "2.5"
}
}
{
"blueprint" : "my-blueprint",
"host_groups" :[
{
"name" : "master-host",
"hosts" : [
{
"fqdn" : "master001.ambari.apache.org"
}
]
},
{
"name" : "worker-host",
"hosts" : [
{
"fqdn" : "worker001.ambari.apache.org"
},
{
"fqdn" : "worker002.ambari.apache.org"
},
…
{
"fqdn" : "worker099.ambari.apache.org"
}
]
}
]
}
1. POST /api/v1/blueprints/my-blueprint 2. POST /api/v1/clusters/my-cluster

11. Create a cluster with Blueprints
{
"configurations" : [
{
"hdfs-site" : {
"dfs.datanode.data.dir" : "/hadoop/1,
/hadoop/2,/hadoop/3"
}
}
],
"host_groups" : [
{
"name" : "master-host",
"components" : [
{ "name" : "NAMENODE” },
{ "name" : "RESOURCEMANAGER” },
…
],
"cardinality" : "1"
},
{
"name" : "worker-host",
"components" : [
{ "name" : "DATANODE" },
{ "name" : "NODEMANAGER” },
…
],
"cardinality" : "1+"
},
],
"Blueprints" : {
"stack_name" : "HDP",
"stack_version" : "2.5"
}
}
{
"blueprint" : "my-blueprint",
"host_groups" :[
{
"name" : "master-host",
"hosts" : [
{
"fqdn" : "master001.ambari.apache.org"
}
]
},
{
"name" : "worker-host",
"hosts" : [
{
"fqdn" : "worker001.ambari.apache.org"
},
{
"fqdn" : "worker002.ambari.apache.org"
},
…
{
"fqdn" : "worker099.ambari.apache.org"
}
]
}
]
}
1. POST /api/v1/blueprints/my-blueprint 2. POST /api/v1/clusters/my-cluster

12. Create a cluster with Blueprints
{
"configurations" : [
{
"hdfs-site" : {
"dfs.datanode.data.dir" : "/hadoop/1,
/hadoop/2,/hadoop/3"
}
}
],
"host_groups" : [
{
"name" : "master-host",
"components" : [
{ "name" : "NAMENODE” },
{ "name" : "RESOURCEMANAGER” },
…
],
"cardinality" : "1"
},
{
"name" : "worker-host",
"components" : [
{ "name" : "DATANODE" },
{ "name" : "NODEMANAGER” },
…
],
"cardinality" : "1+"
},
],
"Blueprints" : {
"stack_name" : "HDP",
"stack_version" : "2.5"
}
}
{
"blueprint" : "my-blueprint",
"host_groups" :[
{
"name" : "master-host",
"hosts" : [
{
"fqdn" : "master001.ambari.apache.org"
}
]
},
{
"name" : "worker-host",
"hosts" : [
{
"fqdn" : "worker001.ambari.apache.org"
},
{
"fqdn" : "worker002.ambari.apache.org"
},
…
{
"fqdn" : "worker099.ambari.apache.org"
}
]
}
]
}
1. POST /api/v1/blueprints/my-blueprint 2. POST /api/v1/clusters/my-cluster

13. Create a cluster with Blueprints
{
"configurations" : [
{
"hdfs-site" : {
"dfs.datanode.data.dir" : "/hadoop/1,
/hadoop/2,/hadoop/3"
}
}
],
"host_groups" : [
{
"name" : "master-host",
"components" : [
{ "name" : "NAMENODE” },
{ "name" : "RESOURCEMANAGER” },
…
],
"cardinality" : "1"
},
{
"name" : "worker-host",
"components" : [
{ "name" : "DATANODE" },
{ "name" : "NODEMANAGER” },
…
],
"cardinality" : "1+"
},
],
"Blueprints" : {
"stack_name" : "HDP",
"stack_version" : "2.5"
}
}
{
"blueprint" : "my-blueprint",
"host_groups" :[
{
"name" : "master-host",
"hosts" : [
{
"fqdn" : "master001.ambari.apache.org"
}
]
},
{
"name" : "worker-host",
"hosts" : [
{
"fqdn" : "worker001.ambari.apache.org"
},
{
"fqdn" : "worker002.ambari.apache.org"
},
…
{
"fqdn" : "worker099.ambari.apache.org"
}
]
}
]
}
1. POST /api/v1/blueprints/my-blueprint 2. POST /api/v1/clusters/my-cluster

14. Blueprints for Large Scale
• Kerberos, secure out-of-the-box
• High Availability is setup initially for
NameNode, YARN, Hive, Oozie, etc
• Host Discovery allows Ambari to
automatically install services for a Host
when it comes online
• Stack Advisor recommendations

15. POST /api/v1/clusters/MyCluster/hosts
[
{
"blueprint" : "single-node-hdfs-test2",
"host_groups" :[
{
"host_group" : "slave",
"host_count" : 3,
"host_predicate" : "Hosts/cpu_count>1”
}, {
"host_group" : "super-slave",
"host_count" : 5,
"host_predicate" : "Hosts/cpu_count>2&
Hosts/total_mem>3000000"
}
]
}
]
Blueprint Host Discovery

16. Kerberos
Available since Ambari 2.0
• Ambari manages Kerberos principals and keytabs
• Works with existing MIT KDC or Active Directory
• Once Kerberized, handles
• Adding hosts
• Adding components to existing hosts
• Adding services
• Moving components to different hosts

17. Management Packs - Motivation
• Release Management
o Ambari core and stacks released together
o Stack changes require Ambari release
o Decouple stack and Ambari core releases
• Add-on Services
o Release vehicle for 3rd party services
o Self contained release artifacts

18. Management Packs – Release Trains

19. Management Packs
• Generalized release artifact for stacks, add-on
services, views, etc
• Decouples stack releases from Ambari core
release
• Tarballs with metadata for applicability and
content
• Stack is an overlay of multiple management
packs

20. Overlay of Management Packs

21. Management Pack++
Short Term Goals (Ambari 2.4)
• Retrofit in Stack Processing Framework
• Enable 3rd party to ship add-on services
• Command line support
Long Term Goals (Future)
• Management Pack Framework
• Deliver Views
• Rest API support

22. Role Based Access Control (RBAC)
As Ambari & organizations grow,
so do security needs
Ambari integrates with external
authentication systems & LDAP

23. RBAC Terms
• Roles have permissions,
e.g., add services to cluster
• Roles are applied to Resources
e.g., Ambari, particular Cluster, particular View
• Users belong to groups
• A group has a role
• Users can also have additional roles

24. New RBAC Roles
allAmbari Admin
Cluster Admin except manage permissions
Cluster Op except add services, Kerberos,
manage Alerts, & upgrades
Service Admin except alter cluster topology
or install components
Service Op except change configs
Read-Only only view

25. Background: Upgrade Terminology
Manual
Upgrade
The user follows instructions to upgrade
the stack
Incurs downtime

26. Background: Upgrade Terminology
Manual
Upgrade
The user follows instructions to upgrade
the stack
Incurs downtime
Rolling
Upgrade
Automated
Upgrades one component
per host at a time
Preserves cluster operation
and minimizes service impact

27. Background: Upgrade Terminology
Express
Upgrade
Automated
Runs in parallel across hosts
Incurs downtime
Manual
Upgrade
The user follows instructions to upgrade
the stack
Incurs downtime
Rolling
Upgrade
Automated
Upgrades one component
per host at a time
Preserves cluster operation
and minimizes service impact

28. Automated Upgrade: Rolling or Express
Check
Prerequisites
Review the
prereqs to
confirm
your cluster
configs are
ready
Prepare
Take
backups of
critical
cluster
metadata
Perform
Upgrade
Perform the
HDP
upgrade.
The steps
depend on
upgrade
method:
Rolling or
Express
Register +
Install
Register the
HDP
repository
and install
the target
HDP version
on the
cluster
Finalize
Finalize the
upgrade,
making the
target
version the
current
version

29. Process: Rolling Upgrade
ZooKeeper
Ranger
Core
Masters
Core Slaves
Hive
Oozie
Falcon
Clients
Kafka
Knox
Storm
Slider
Flume
Finalize or
Downgrade
HDFS, YARN, MR, Tez,
HBase, Pig. Hive, etc.
HDFS
YARN
HBas
e

30. Grafana for Ambari Metrics
• Grafana as a “Native UI” for
Ambari Metrics
• Pre-built Dashboards
Host-level, Service-level
• Supports HTTPS
• System Home, Servers
• HDFS Home, NameNodes,
DataNodes
• YARN Home, Applications,
Job History Server
• HBase Home,
Performance, Misc
FEATURES DASHBOARDS

31. Grafana includes pre-built
dashboards for visualizing the most
important cluster metrics.

32. The HDFS NameNode
dashboard highlights
file system activity.

33. Future of Ambari
• Cloud features
• Multiple instances of same service at different
versions, e.g., Spark 1.6 and Spark 2.0
• YARN assemblies
• Component & Patch Upgrades: upgrade
individual components in the same stack
version, e.g., just DN and RM in HDP 2.4.*.*
with zero downtime