This presentation is from BigData November Bangalore MeetUp by Varun Vasudev. technology.inmobi.com/events/bigdata-meetup Talk Outline: - Overview of YARN - New YARN Innovation in Hadoop 2.6 - Rolling upgrades - Added fault tolerance - CPU scheduling in Capacity Scheduler - C-Group isolation - Node labels - Support for long running services

1. Page1 © Hortonworks Inc. 2014
What’s new in YARN
Hortonworks. We do Hadoop.

2. Page2 © Hortonworks Inc. 2014
Speaker
Varun Vasudev
Hortonworks
Work on Hadoop YARN
vvasudev@hortonworks.com

3. Page3 © Hortonworks Inc. 2014
Agenda
• Overview of YARN
• New YARN Innovation in Hadoop 2.6
– Rolling upgrades
– Added fault tolerance
– CPU scheduling in Capacity Scheduler
– C-Group isolation
– Node labels
– Support for long running services
• Q & A

4. Page4 © Hortonworks Inc. 2014
Overview of YARN

5. Page5 © Hortonworks Inc. 2014
What is YARN?
YARN : Data Operating System
°1 ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° °
°
°°
° ° ° ° ° ° °
° ° ° ° ° ° N
Batch
MapReduce
Batch & Interactive
Tez
Real-Time
Slider
Direct
Java
.NET
Scripting
Pig
SQL
Hive
Cascading
Java
Scala
NoSQL
HBase
Accumulo
Stream
Storm
Other
ISV
Other
ISV
Others
Spark
Other ISV

6. Page6 © Hortonworks Inc. 2014
What’s new in Hadoop YARN 2.6?

7. Page7 © Hortonworks Inc. 2014
YARN in Hadoop 2.6: What’s New
• Security: Kerberos Token Renewal
• Log Aggregation: View logs for running applications
• Fault Tolerance: AM Retry and Container keep alive
• Service Registry: Directory for services running in YARN
Long
Running
Services
Support
THEME
• CPU Scheduling in Capacity Scheduler
• CPU Isolation through CGroups
• Node Labels for scheduling constraints
Workload
Scheduling
THEME
• YARN Rolling Upgrades support
• Work Preserving Restart
• Timeline Server support in Secure clusters
Reliable and
Secure
Operations
THEME

8. Page8 © Hortonworks Inc. 2014
New in Hadoop YARN 2.6:
Long Running Service Support

9. Page9 © Hortonworks Inc. 2014
YARN Supports Multiple Workloads
YARN : Data Operating System
°1 ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° °
°
°°
° ° ° ° ° ° °
° ° ° ° ° ° N
HDFS
(Hadoop Distributed File System)
Batch
MapReduce
Batch & Interactive
Tez
Real-Time
Slider
Direct
Java
.NET
Scripting
Pig
SQL
Hive
Cascading
Java
Scala
NoSQL
HBase
Accumulo
Stream
Storm
Other
ISV
Other
ISV
Application
Others
Spark
Other ISV

10. Page10 © Hortonworks Inc. 2014
Enhancements to Support Long Running Services
• YARN-941: YARN updates Kerberos token for a Long Running Service
after the token expires
Security
Capability
• YARN-2468: Aggregate and capture logs during the lifetime of a Long
Running Service
Log
Aggregation
Capability
• YARN-1489: When ApplicationMaster(AM) restarts, do not kill all
associated containers – reconnect to the AM
• YARN-611/YARN-614: Tolerate AM failures for Long Running Services
Fault
Tolerance
Capability
• YARN-913: Service Registry that publishes host and ports that each
service comes up on
Service
Registry
Capability

11. Page11 © Hortonworks Inc. 2014
New in Hadoop YARN 2.6:
Workload Scheduling

12. Page12 © Hortonworks Inc. 2014
Node Labels: Apply Node Constraints
A
App
L1 L1 L1
Deploy/Allocate
B
App
L1 L1 L1
Isolate
A A A
nodes
labels

13. Page13 © Hortonworks Inc. 2014
CPU Scheduling in Capacity Scheduler
What
• Admin tells YARN how much CPU capacity is available in cluster
• Applications specify CPU capacity needed for each container
• YARN Capacity Scheduler schedules application taking CPU capacity availability into account
Why
• Applications (for example Storm, HBase, Machine Learning) need predictable access to CPU
as a resource
• CPU has become bottleneck instead of memory in certain clusters (128 GB RAM, 6 CPUs)

14. Page14 © Hortonworks Inc. 2014
CGroup Isolation
What
• Admin enables CGroups for CPU Isolation for all YARN application workloads
Why
• Applications need guaranteed access to CPU resources
• To ensure SLAs, need to enforce CPU allocations given to an Application container

15. Page15 © Hortonworks Inc. 2014
New in Hadoop YARN 2.6:
Reliable and Secure Operations

16. Page16 © Hortonworks Inc. 2014
YARN Rolling Upgrades Support
• No Service Disruption
– Highly Available Resource Manager
– Job submission automatically retries during failover
• No Service Degradation
– Resource Manager Restart with Work Preservation
– Node Manager Restart with Container Preservation
• Preserve State
– Preserve Application Queue
– Preserve Application History
– Preserve Node/Container state

17. Page17 © Hortonworks Inc. 2014
Secure Timeline Server Support
App Timeline Server
AMBARI
Custom App
Monitoring
Client
HTTPs
HTTPs

18. Page18 © Hortonworks Inc. 2014
Q & A