Deep learning workloads are computing intensive, and training these type of models is better done with specialized hardware like GPUs. Luciano Resende outlines a pattern for building deep learning models using the Jupyter Notebook’s interactive development in commodity hardware and leveraging platforms and services such as Fabric for Deep Learning (FfDL) for cost-effective full dataset training of deep learning models.

1. Scaling Notebooks for
Deep Learning
workloads
Luciano Resende
JupyterCon - 2018
1© 2018 IBM Corporation

2. About me - Luciano Resende
2
Open Source AI Platform Architect – IBM – CODAIT
• Senior Technical Staff Member at IBM, contributing to open source for over 10 years
• Currently contributing to : Jupyter Notebook ecosystem, Apache Bahir, Apache
Toree, Apache Spark among other projects related to AI/ML platforms
lresende@us.ibm.com
https://www.linkedin.com/in/lresende
@lresende1975
https://github.com/lresende
© 2018 IBM Corporation

3. Jupyter Notebooks
Overview
3© 2018 IBM Corporation

4. Jupyter Notebooks
© 2018 IBM Corporation 4
Notebooks are interactive
computational
environments, in which
you can combine code
execution, rich text,
mathematics, plots and
rich media.

5. Jupyter Notebooks
© 2018 IBM Corporation 5
• Notebook UI runs on the browser
• The Notebook Server serves the
’Notebooks’
• Kernels interpret/execute cell
contents
– Are responsible for code execution
– Abstracts different languages
– 1:1 relationship with Notebook
– Runs and consume resources as long as
notebook is running

6. Jupyter Notebooks
and Deep Learning
6© 2018 IBM Corporation

7. Deep Learning workloads
© 2018 IBM Corporation 7
Resource Intensive workloads
Requires expensive hardware (GPUs)
Long Running training jobs
- Simple MNIST takes over one hour
WITHOUT a decent GPU
- Other non complex deep learning model
training can easily take over a day
WITH GPUs

8. Jupyter & Kubernetes
© 2018 IBM Corporation 8
Kubernetes Platform
- Containers provides a flexible way to
deploy applications and are here to stay
- Kubernetes enables easy management
of containerized applications and
resources with the benefit of Elasticity
and Quality of Services
Source: https://github.com/Langhalsdino/Kubernetes-GPU-Guide

9. Enterprise Gateway & Kubernetes
© 2018 IBM Corporation
Supported Platforms
FfDL
Before Enterprise Gateway After Enterprise Gateway
Before Jupyter Enterprise Gateway …
• Resources required for all kernels needs to
be allocated during Notebook Server pod
creation
• Resources limited to what is physically
available on the host node that runs all
kernels and associated Spark drivers
After Jupyter Enterprise Gateway …
• Gateway pod very lightweight
• Kernels in their own pod, isolation
• Kernel pods built from community images:
Spark-on-K8s, TensorFlow, Keras, etc.

10. Jupyter Enterprise Gateway - Kubernetes
© 2018 IBM Corporation 10
Container images defined in kernelspec
Community image
Kernel
Spark on K8
Kernel
Distributed
File
System
Vanilla Kernels
Spark based kernels
Gateway
nb2kg
nb2kg

11. Deep Learning Platforms
© 2018 IBM Corporation 11
Prohibited costs
- Deep Learning resources are prohibitive in
costs to be locked/idle during interactive
development
Deep Learning Platforms
- We have seen the rise of Deep Learning
platforms that leverage containers and
Kubernetes as the basis of their
infrastructure
- Kubernetes enables Deep Learning
platforms to easily share and restrict
accelerated hardware
Fabric for Deep Learning
IBM Watson Studio
Deep Learning as a service
Batch
oriented
developm
ent

12. Deep Learning Workspace
March 30 2018 / © 2018 IBM Corporation 12
Streamline Data Science user experience when
coming from Notebook/Interactive
development interfaces
• Current process include multiple steps, one
being decomposing the notebook into an
application that needs to be submitted as a
zip to the deep learning runtime which
becomes a show stopper for data scientists
to adopt FfDL and DLaas

13. March 30 2018 / © 2018 IBM Corporation 13
Streamline the Deep Learning application
lifecycle
• Run local notebook experiments, with small data
samples and seamlessly validate experiments on
Deep Learning environments
• IBM Cloud DLaaS, FfDL (open source), KubeFlow
(open source)
Simplify productionalization of Model training
and serving from Notebooks
• Enable running/scheduling notebooks on
production environments as batch jobs
• Results can be made available via updated
notebook, or exported to html, pdf and a few
other formats.
Interactive development
lifecycle done on
commodity hardware
with sampled data
Training on full dataset
gets scheduled as
batch jobs on deep
learning infrastructure
Deep Learning Workspace

14. 14March 30 2018 / © 2018 IBM Corporation

15. March 30 2018 / © 2018 IBM Corporation 15
• User select where to run the
experiment
• Job is packaged and submitted on
behalf of user
• User has access to Job Console to
monitor experiment
Deep Learning Workspace

16. March 30 2018 / © 2018 IBM Corporation
Deep Learning Extensions
Notebook Extension (button)
https://github.com/lresende/enterprise_scheduler_extension
Notebook Scheduler (backend)
https://github.com/lresende/enterprise_scheduler
16
Enable interactive development on
commodity hardware while easily
enabling model training on Deep
Learning infrastructure with GPU based
hardware.
Deep Learning Workspace

17. 17March 30 2018 / © 2018 IBM Corporation

18. BACKUP
SLIDES
18© 2018 IBM Corporation

19. Jupyter Enterprise Gateway
19© 2018 IBM Corporation

20. Jupyter Enterprise
Gateway
© 2018 IBM Corporation
Jupyter Enterprise Gateway at IBM Code
https://developer.ibm.com/code/openprojects/jupyter-enterprise-gateway/
Jupyter Enterprise Gateway source code at GitHub
https://github.com/jupyter-incubator/enterprise_gateway
Jupyter Enterprise Gateway Documentation
http://jupyter-enterprise-gateway.readthedocs.io/en/latest/
Supported Kernels
Supported Platforms
20
A lightweight, multi-tenant, scalable
and secure gateway that enables
Jupyter Notebooks to share resources
across an Apache Spark or Kubernetes
cluster for Enterprise/Cloud use cases
Spectrum Conductor
+

21. Jupyter Enterprise Gateway Features
© 2018 IBM Corporation
Gather
Data
Analyze
Data
Machine
Learning
Deep
Learning
Deploy
Model
Maintain
Model
Python
Data Science
Stack
Fabric for
Deep Learning
(FfDL)
Mleap +
PFA
Scikit-LearnPandas
Apache
Spark
Apache
Spark
Jupyter
Model
Asset
eXchange
Keras +
Tensorflow
21
16
32
48
64
0
10
20
30
40
50
60
70
80
4 Nodes 8 Nodes 12 Nodes 16 NodesMaxKernels(4GBHeap)
Cluster Size (32GB Nodes)
MAXIMUM NUMBER OF
SIMULTANEOUS KERNELS
Optimized Resource Allocation
– Utilize resources on all cluster nodes by running kernels as Spark
applications in YARN Cluster Mode.
– Pluggable architecture to enable support for additional Resource Managers
Enhanced Security
– End-to-End secure communications
• Secure socket communications
• Encrypted HTTP communication using SSL
Multiuser support with user impersonation
– Enhance security and sandboxing by enabling user impersonation when
running kernels (using Kerberos).
– Individual HDFS home folder for each notebook user.
– Use the same user ID for notebook and batch jobs.
Kernel
Kernel
Kernel
Kernel
Kernel
Kernel
Kernel
Kernel
Kernel

22. Jupyter Enterprise Gateway – YARN
© 2018 IBM Corporation 22
YARN Cluster
YARN
Workers
Gateway Node
Jupyter Enterprise Gateway
• Multitenancy
• Remote kernel lifecycle management via process proxies
Spark Executors
Spark Executors
Spark Executors
Yarn Container
Jupyter Kernel
Spark Driver
Impersonation:
Alice’s kernel runs
under Alice’s user ID.
Spark Executors
Spark Executors
Spark Executors
Yarn Container
Jupyter Kernel
Spark Driver
SecurityLayer
nb2kg
nb2kg
Spark Executors
Spark Executors
Spark Executors
Yarn Container
Jupyter Kernel
Spark Driver
Bob
Alice

23. Enterprise Gateway & Kubernetes
© 2018 IBM Corporation
Supported Platforms
FfDL
Before Enterprise Gateway After Enterprise Gateway
Before Jupyter Enterprise Gateway …
• Resources required for all kernels needs to
be allocated during Notebook Server pod
creation
• Resources limited to what is physically
available on the host node that runs all
kernels and associated Spark drivers
After Jupyter Enterprise Gateway …
• Gateway pod very lightweight
• Kernels in their own pod, isolation
• Kernel pods built from community images:
Spark-on-K8s, TensorFlow, Keras, etc.

24. Jupyter Enterprise Gateway - Kubernetes
© 2018 IBM Corporation 24
Container images defined in kernelspec
Community image
Kernel
Spark on K8
Kernel
Distributed
File
System
Vanilla Kernels
Spark based kernels
Gateway
nb2kg
nb2kg

25. The Secret Sauce:
Process Proxies mixed with Kernel Launchers
© 2018 IBM Corporation
Process Proxy:
• Abstracts kernel process represented by Jupyter
framework
• Pluggable class definition identified in kernelspec
(kernel.json)
• Manages kernel lifecycle
Kernel Launcher:
• Embeds target kernel
• Listens on gateway communication port
• Conveys interrupt requests (via local signal)
• Could be extended for additional communications
{
"language": "python",
"display_name": "Spark - Python (Kubernetes Mode)",
"process_proxy": {
"class_name":
"enterprise_gateway.services.processproxies.k8s.KubernetesProcessP
roxy",
"config": {
"image_name": "elyra/kubernetes-kernel-py:dev",
"executor_image_name": "elyra/kubernetes-kernel-py:dev”,
"port_range" : "40000..42000"
}
},
"env": {
"SPARK_HOME": "/opt/spark",
"SPARK_OPTS": "--master k8s://https://${KUBERNETES_SERVICE_HOST
--deploy-mode cluster --name …",
…
},
"argv": [
"/usr/local/share/jupyter/kernels/spark_python_kubernetes/bin/run.
sh",
"{connection_file}",
"--RemoteProcessProxy.response-address",
"{response_address}",
"--RemoteProcessProxy.spark-context-initialization-mode",
"lazy"
]
}

26. 26March 30 2018 / © 2018 IBM Corporation