Presentation slides from Deploying Long-running services on Apache Hadoop YARN using Apache Slider Meetup. Presenter: Thomas Liu, Software Engineer, Hortonworks Inc.

1. © Hortonworks Inc. 2015
Deploying Docker applications on YARN
via Slider
HW Slider team
Presenter: Thomas Liu
Page 1

2. © Hortonworks Inc. 2015
Slider is an in-incubation project with one goal:
Make it easy to deploy and manage
applications in a Hadoop cluster with YARN
Page 2

3. © Hortonworks Inc.
Project Status
• In ASF incubation
• 0.70-incubating release this month
• Growing set of application packages
• Working in simplifying packaging
• HDP 2.2 added Slider component for easy on-boarding of
long-lived service on YARN
http://slider.incubator.apache.org/

4. © Hortonworks Inc.
An application consists of
“Application Package”
JSON configuration files: YARN resources + config
Data persisted in HDFS (May)
Placement history in HDFS
Page 4

5. © Hortonworks Inc. 2015
Slider: architecture
1. AppMaster
2. AgentProvider
3. Agent
4. AppPackage
5. CLI
Slider
App
config
files
HDFS
YARN Resource Manager
“The RM”
HDFS
YARN Node Manager
Agent Component
HDFS
YARN Node Manager
Agent Component
Slider Application Master
Slider CLI

6. © Hortonworks Inc. 2015
Slider App Master runs the application
HDFS
YARN Node Manager
HDFS
YARN Node Manager
HDFS
YARN Resource Manager
HDFS
YARN Node Manager
Slider Application Master
Redis Server
Nodejs Server

7. © Hortonworks Inc.
On-boarding Dockerized Application on YARN via Slider
• Application definition – metainfo.json (contains docker image name)
• Application instance configuration (if any) – app_config.json
• Application resource definition – resources.json
No Application bundle and lifecycle scripts
Page 7

8. © Hortonworks Inc. 2015
Demo
Dockerized node.js app using dockerized redis for persistence
Page 8

9. © Hortonworks Inc.
metainfo.json – structure of the application
{
"schemaVersion": "2.1",
"application": {
"name": "NODEJS-REDIS",
"components": [
{
"name": "NODEJS",
"type": "docker",
"dockerContainers": [
{
"name": "nodejs”,
"image": "rsahahw/centos-node-redis”,
"ports": [{"containerPort" : "8000” }]
}
]
},
{
"name": "REDIS",
"type": "docker",
"dockerContainers": [
{
"name": "redis",
"image": "tutum/redis",
"ports": [{
"containerPort" : "6379",
"hostPort": "6379”
...

10. © Hortonworks Inc
resources.json – resource requirement of the application
{
"schema" : "http://example.org/specification/v2.0.0",
"metadata" : {
},
"global" : {
},
"components": {
"NODEJS": {
"yarn.role.priority": "1",
"yarn.component.instances": "1",
"yarn.memory": "512"
},
"REDIS": {
"yarn.role.priority": "2",
"yarn.component.instances": "1",
"yarn.memory": "512"
}
}
}

11. © Hortonworks Inc.
app_config.json – instance specific configuration of the application
{
"schema": "http://example.org/specification/v2.0.0",
"metadata": {
},
"global": {
},
"components": {
"NODEJS": {
"nodejs.options":"-d -e REDIS_HOST=${REDIS_HOST}",
"nodejs.statusCommand":"/usr/bin/docker ps"
},
"REDIS": {
"redis.options":"-d -e REDIS_PASS=**None**",
"redis.statusCommand":"/usr/bin/docker ps"
}
}
}

12. © Hortonworks Inc
Commands: create, start, stop, destroy
$ slider create dockerapp1 --resources resources.json -–template
appConfig.json -–metainfo metainfo.json
$ slider list
$ slider stop dockerapp1
$ slider destroy dockerapp1

13. © Hortonworks Inc.
How to get started
• Use
–Pre-requisite – Hadoop 2.6 or above [ or HDP 2.2 ]
–Build Slider [ link ]
–Author slider configuration files for your dockerized app or use
examples
• Contribute
–Explore Slider @ http://slider.incubator.apache.org/
–Few ideas
– Aligning Docker security model with YARN’s
– Docker images with multiple initialization steps
– Active resource monitoring docker containers by YARN

14. © Hortonworks Inc. 2015
Q&A