Weitere ähnliche Inhalte
Ähnlich wie Apache Hadoop YARN: Past, Present and Future (20)
Mehr von DataWorks Summit/Hadoop Summit (20)
Kürzlich hochgeladen (20)
Apache Hadoop YARN: Past, Present and Future
- 1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache Hadoop YARN:
Past, Present and
Future
Dublin, April 2016
Varun Vasudev
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
About myself
⬢ Apache Hadoop contributor since 2014
⬢ Apache Hadoop committer
⬢ Currently working for Hortonworks
⬢ vvasudev@apache.org
- 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Introduction to Apache Hadoop YARN
YARN: Data Operating System
(Cluster Resource Management)
1 ° ° ° ° ° ° °
° ° ° ° ° ° ° °
Script
Pig
SQL
Hive
TezTez
Java
Scala
Cascading
Tez
° °
° °
° ° ° ° °
° ° ° ° °
Others
ISV
Engines
HDFS
(Hadoop Distributed File System)
Stream
Storm
Search
Solr
NoSQL
HBase
Accumulo
Slider Slider
BATCH, INTERACTIVE & REAL-TIME DATA ACCESS
In-Memory
Spark
YARN
The Architectural
Center of Hadoop
• Common data platform, many applications
• Support multi-tenant access & processing
• Batch, interactive & real-time use cases
- 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Introduction to Apache Hadoop YARN
⬢ Architectural center of big data workloads
⬢ Enterprise adoption accelerating
–Secure mode becoming more widespread
–Multi-tenant support
–Diverse workloads
⬢ SLAs
–Tolerance for slow running jobs decreasing
–Consistent performance desired
- 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Past – Apache Hadoop 2.6, 2.7
- 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache Hadoop YARN
ResourceManager
(active)
ResourceManager
(standby)
NodeManager1
NodeManager2
NodeManager3
NodeManager4
Resources: 128G, 16 vcores
Label: SAS
- 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Scheduler
Inter queue pre-emption
Application
Queue B – 25%
Queue C – 25%
Label: SAS (exclusive)
Queue A – 50%
FIFO
ResourceManager
(active)
Application, Queue A, 4G, 1 vcore
Reservation for application
User
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Node 1
NodeManager128G, 16 vcores
Launch Applicaton 1 AMAM process
Launch AM process via
ContainerExecutor – DCE, LCE, WSCE.
Monitor/isolate memory and cpu
Application Lifecycle
ResourceManager
(active)
Request containers
Allocate containers
Container 1 process
Container 2 process
Launch containers on node using
DCE, LCE, WSCE. Monitor/isolate
memory and cpu
History Server(ATS – leveldb,
JHS - HDFS)
HDFS
Log aggregation
- 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Operational support
⬢ Support added for work preserving restarts in the RM and the NM
⬢ Support added for rolling upgrades and downgrades from 2.6 onwards
- 10. 1
0
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Recent releases
⬢ 2.6 and 2.7 maintenance releases are carried out
–Only blockers and critical fixes are added
⬢ Apache Hadoop 2.7
–2.7.3 should be out soon
–2.7.2 released in January, 2016
–2.7.1 released in July, 2015
⬢ Apache Hadoop 2.6
–2.6.4 released in February, 2016
–2.6.3 released in December, 2015
–2.6.2 released in October, 2015
- 11. 1
1
© Hortonworks Inc. 2011 – 2016. All Rights Reserved1
1
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Present – Apache Hadoop 2.8
- 12. 1
2
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
YARN
ResourceManager
(active)
ResourceManager
(standby)
NodeManager1
NodeManager2
NodeManager3
NodeManager4
Resources: 128G, 16 vcores
Auto-calculate node resources
Label: SAS
Dynamic NodeManager
resource configuration
- 13. 1
3
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
NodeManager resource management
⬢ Options to report NM resources based on node hardware
–YARN-160
–Restart of the NM required to enable feature
⬢ Alternatively, admins can use the rmadmin command to update the node’s resources
–YARN-291
–Looks at the dynamic-resource.xml
–No restart of the NM or the RM required
- 14. 1
4
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
YARN Scheduler
Inter queue pre-emption
Improvements to pre-emption
Application
Queue B – 25%
Queue C – 25%
Label: SAS (non-exclusive)
Queue A – 50%
Priority/FIFO, Fair
ResourceManager
(active)
Application, Queue A, 4G, 1 vcore
Support for application priority
Reservation for application
Support for cost based placement
agent
User
- 15. 1
5
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Scheduler
⬢ Support for application priority within a queue
–YARN-1963
–Users can specify application priority
–Specified as an integer, higher number is higher priority
–Application priority can be updated while it’s running
⬢ Improvements to reservations
–YARN-2572
–Support for cost based placement agent added in addition to greedy
⬢ Queue allocation policy can be switched to fair sharing
–YARN-3319
–Containers allocated on a fair share basis instead of FIFO
- 16. 1
6
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Scheduler
⬢ Support for non-exclusive node labels
–YARN-3214
–Improvement over partition that existed earlier
–Better for cluster utilization
⬢ Improvements to pre-emption
- 17. 1
7
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Node 1
NodeManager
Support added for graceful
decomissioning
128G, 16 vcores
Launch Applicaton 1 AMAM process/Docker container(alpha)
Launch AM via ContainerExecutor –
DCE, LCE, WSCE. Monitor/isolate
memory and cpu. Support added for
disk and network isolation via
CGroups(alpha)
Application Lifecycle
ResourceManager
(active)
Request containers
Allocate containers
Support added to resize containers. Container 1 process/Docker
container(alpha)
Container 2 process/Docker
container(alpha)
Launch containers on node using DCE,
LCE, WSCE. Monitor/isolate memory and
cpu. Support added for disk and network
isolation using Cgroups(alpha).
History Server(ATS 1.5– leveldb
+ HDFS, JHS - HDFS)
HDFS
Log aggregation
- 18. 1
8
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Application Lifecycle
⬢ Graceful decommissioning of NodeManagers
–YARN-914
–Drains a node that’s being decommissioned to allow running containers to finish
⬢ Resource isolation support for disk and network
–YARN-2619, YARN-2140
–Containers get a fair share of disk and network resources using CGroups
–Alpha feature
⬢ Docker support in LinuxContainerExecutor
–YARN-3853
–Support to launch Docker containers alongside process containers
–Alpha feature
–Talk by Sidharta Seethana at 12:20 tomorrow in Liffey A
- 19. 1
9
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Application Lifecycle
⬢ Support for container resizing
–YARN-1197
–Allows applications to change the size of an existing container
⬢ ATS 1.5
–YARN-4233
–Store timeline events on HDFS
–Better scalability and reliability
- 20. 2
0
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Operational support
⬢ Improvements to existing tools(like yarn logs)
⬢ New tools added(yarn top)
⬢ Improvements to the RM UI to expose more details about running applications
- 21. 2
1
© Hortonworks Inc. 2011 – 2016. All Rights Reserved2
1
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Future
- 22. 2
2
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Drivers for changes
⬢ Changing workload types
–Workloads have moved from batch to batch + interactive
–Workloads will change to batch + interactive + services
⬢ Big data workloads continue to evolve
–Spark on YARN the most popular way to run Spark in production
⬢ Containerization has taken off
–Docker becoming extremely popular
⬢ Improve ease of operations
–Easier to debug application failures/poor performance
–Make overall cluster management easier
–Improve existing tools such as yarn logs, yarn top, etc
- 23. 2
3
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache Hadoop YARN
ResourceManager
(active)
ResourceManager
(standby)
NodeManager1
NodeManager2
NodeManager3
NodeManager4
Resources: 128G, 16 vcores
Add support for arbitrary resource types
Label: SAS
Add support for
federation – allow YARN
to scale
New RM UI
- 24. 2
4
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Future work
⬢ Support for arbitrary resource types and resource profiles
–YARN-3926
–Admins can add arbitrary resource types for scheduling
–Users can specify resource profile name instead of individual resources
⬢ YARN federation
–YARN-2915
–Allows YARN to scale out to tens of thousands of nodes
–Cluster of clusters which appear as a single cluster to an end user
⬢ New RM UI
–YARN-3368
–Enhanced usability
–Easier to add new features
- 25. 2
5
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Scheduler
Inter queue pre-emption
Support for intra queue pre-emption
Application
Queue B – 25%
Queue C – 25%
Label: SAS (non-exclusive)
Queue A – 50%
Priority/FIFO, Fair
ResourceManager
(active)
Application, Queue A
Add support for resource profiles
Reservation for application
User
New scheduler API
Schedule based on actual resource usage
- 26. 2
6
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Future work
⬢ New scheduler features
–YARN-4902
–Support richer placement strategies such as affinity, anti-affinity
⬢ Support pre-emption within a queue
–YARN-4781
⬢ More improvements to pre-emption
–YARN-4108, YARN-4390
⬢ Scheduling based on actual resource usage
–YARN-1011
–Nodes report actual memory and cpu usage to the scheduler to make better decisions
- 27. 2
7
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Node 1
NodeManager
Add distributed scheduling
128G, 16 vcores
Launch Applicaton 1 AMAM process/Docker container
Launch AM process via
ContainerExecutor – DCE, LCE, WSCE.
Monitor/isolate memory and cpu.
Support for disk and network isolation
Application Lifecycle
ResourceManager
(active)
Request containers
Allocate containers
New scheduler API to allow far more
powerful placement strategies
Container 1 process/Docker
container. Support container restart.
Container 2 process/Docker
container. Support container restart.
Launch containers on node using DCE,
LCE, WSCE. Monitor/isolate memory and
cpu. Support for disk and network
isolation.
History Server(ATS v2 - HBase,
JHS - HDFS)
HDFS
Log aggregation
DNS sevice
- 28. 2
8
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Future work
⬢ Distributed scheduling
–YARN-2877, YARN-4742
–NMs run a local scheduler
–Allows faster scheduling turnaround
⬢ Better support for disk and network isolation
–Tied to supporting arbitrary resource types
⬢ Enhance Docker support
–YARN-3611
–Support to mount volumes
–Isolate containers using CGroups
- 29. 2
9
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Future work – support for services
⬢ YARN-4692
⬢ Container restart
–YARN-3988
–Allow container restart without losing allocation
⬢ Service discovery via DNS
–YARN-4757
–Running services can be discovered via DNS
⬢ Allocation re-use
–YARN-4726
–Allow AMs to stop a container but not lose resources on the node
–Required for application upgrades
- 30. 3
0
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Future work
⬢ ATS v2
–YARN-2928
–Run timeline service on Hbase
–Support for more data, better performance
⬢ Also in the pipeline
–Switch to Java 8 with Hadoop 3.0
–Add support for GPU isolation
–Better tools to detect limping nodes
- 31. 3
1
© Hortonworks Inc. 2011 – 2016. All Rights Reserved3
1
© Hortonworks Inc. 2011 – 2016. All Rights Reserved
Thank you!