Born in academia, Jupyter notebooks are prevalent in both learning and research environments throughout the scientific community. Due to the widespread adoption of big data, AI, and deep learning frameworks, notebooks are also finding their way into the enterprise, which introduces a different set of requirements. Alan Chin and Luciano Resende explain how to introduce Jupyter Enterprise Gateway into new and existing notebook environments to enable a “bring your own notebook” model while simultaneously optimizing resources consumed by the notebook kernels running across managed clusters within the enterprise. Along the way, they detail how to use different frameworks with Enterprise Gateway to meet the needs of data scientists operating within the AI and deep learning ecosystems.

1. Scaling Jupyter with
Jupyter Enterprise Gateway
Luciano Resende
Alan Chin
CODAIT - IBM

2. About me – Alan Chin
Sr. Software Engineer – Build and Infrastructure – CODAIT
• Over 3 years working with Open Source Projects (Apache SystemML, Apache Spark,
Apache Ambari
• Currently Contributing to the Jupyter Enterprise Gateway Project
akchin@us.ibm.com
https://www.linkedin.com/in/alankchin/
@AlanChin11
https://github.com/akchinSTC
IBM Developer / © 2019 IBM Corporation 2

3. About me - Luciano Resende
Open Source AI Platform Architect – IBM – CODAIT
• Senior Technical Staff Member at IBM, contributing to open source for over 10 years
• Currently contributing to : Jupyter Notebook ecosystem, Apache Bahir, Apache
Toree, Apache Spark among other projects related to AI/ML platforms
lresende@us.ibm.com
https://www.linkedin.com/in/lresende
@lresende1975
https://github.com/lresende
IBM Developer / © 2019 IBM Corporation 3

4. IBM Open Source Contributions
IBM Developer / © 2019 IBM Corporation
Learn
Open Source @ IBM
Program touches
78,000
IBMers annually
Consume
Virtually all
IBM products
contain some
open source
• 40,363 pkgs
Per Year
Contribute
• >62K OS Certs per
year
• ~10K IBM commits
per month
• 1500+ GitHub repos
Connect
> 1000
active IBM
Contributors
Working in key OS
projects
4

5. IBM Open Source
Participation
IBM generated open source innovation
• 137 IBM Open Code projects w/1500+ Github projects
• Projects that have graduated into full open governance:
Jupyter Enterprise Gateway, Node-Red, OpenWhisk,
Apache SystemML, Blockchain Fabric
• https://developer.ibm.com/code/open/code/
Community
• IBM focused on 18 strategic communities
• Drive open governance in “Centers of Gravity”
• IBM Leaders drive key technologies and assure freedom
of action
The IBM OS Way is now open sourced
• Training, Recognition, Tooling
• Organization, Consuming, Contributing
5IBM Developer / © 2019 IBM Corporation

6. Center for Open Source
Data and AI
Technologies
6
CODAIT aims to make AI solutions
dramatically easier to create, deploy,
and manage in the enterprise
Relaunch of the Spark Technology
Center (STC) to reflect expanded
mission
6IBM Developer / © 2019 IBM Corporation
CODAIT
codait.org
codait (French)
= coder/coded
https://m.interglot.com/fr/en/codait

7. Jupyter Notebooks
7IBM Developer / © 2019 IBM Corporation

8. Jupyter Notebooks
Notebooks are interactive
computational environments,
in which you can combine
code execution, rich text,
mathematics, plots and rich
media.
8IBM Developer / © 2019 IBM Corporation

9. Jupyter Notebook Platform Architecture
Notebook UI runs on the browser
The Notebook Server serves the
‘Notebooks’
Kernels interpret/execute cell contents
Are responsible for code execution
Abstracts different languages
1:1 relationship with Notebook
Runs and consume resources as long as
notebook is running
9IBM Developer / © 2019 IBM Corporation

10. Jupyter Notebook
Interactive Workloads
10IBM Developer / © 2019 IBM Corporation

11. Analytics Workloads
• Large amount of data
• Shared across organization in Data
Lakes
• Multiple workload types
Data cleansing
Data Warehouse
Machine Learning and Insights
11IBM Developer / © 2019 IBM Corporation

12. AI / Deep Learning Workloads
Resource intensive workloads
Requires expensive hardware (GPU,
TPU)
Long Running training jobs
Simple MNIST takes over one hour
WITHOUT a decent GPU
Other non complex deep learning
model training can easily take over a
day WITH GPUs
12IBM Developer / © 2019 IBM Corporation

13. Local Development Environment
IBM Developer / © 2019 IBM Corporation 13

14. Analytic and AI
Platforms
Large pool of shared computing
resources
- Enterprise Cloud, Public Cloud or Hybrid
- Shared Data (Data Lakes/Object Storage)
Distributed Consumers
- Notebooks running local (users laptop)
or as a service (e.g. Jupyter Hub)
Different Resource Utilization Patterns
- High number of idle resources
IBM Developer / © 2019 IBM Corporation 14

15. Jupyter Notebook Stack
Limitations
Kernel
Kernel
Kernel
Kernel
Kernel
Scalability
- Jupyter Kernels running as local process
- Resources are limited by what is available
on the one single node that runs all Kernels
and associated Spark drivers
Security
- Single user sharing the same privileges
- Users can see and control each other process
using Jupyter administrative utilities
8 8 8 8
0
10
20
30
40
50
60
70
80
4 Nodes 8 Nodes 12 Nodes 16 Nodes
MaxKernels(4GBHeap)
Cluster Size (32GB Nodes)
MAXIMUM NUMBER OF SIMULTANEOUS KERNELS
IBM Developer / © 2019 IBM Corporation 15

16. Jupyter Enterprise Gateway
16IBM Developer / © 2019 IBM Corporation

17. Jupyter Enterprise Gateway website
https://Jupyter.org/enterprise_gateway/
Jupyter Enterprise Gateway source code at GitHub
https://github.com/jupyter-incubator/enterprise_gateway
Jupyter Enterprise Gateway Documentation
http://jupyter-enterprise-gateway.readthedocs.io/en/latest/
Supported Kernels
Supported Platforms
Jupyter Enterprise Gateway
Spectrum Conductor
+
A lightweight, multi-tenant,
scalable and secure gateway
that enables Jupyter
Notebooks to share resources
across an Apache Spark or
Kubernetes cluster for
Enterprise/Cloud use cases
IBM Developer / © 2019 IBM Corporation 17
+

18. Jupyter Enterprise Gateway Features
Optimized Resource Allocation
Utilize resources on all cluster nodes by running kernels
as Spark applications in YARN Cluster Mode.
Pluggable architecture to enable support for additional
Resource Managers
Enhanced Security
End-to-End secure communications
- Secure socket communications
- Encrypted HTTP communication using SSL
Multiuser support with user
impersonation
Enhance security and sandboxing by enabling user
impersonation when running kernels (using Kerberos).
Individual HDFS home folder for each notebook user.
Use the same user ID for notebook and batch jobs.
Kernel
Kernel Kernel
Kernel
Kernel
Kernel
Kernel
16
32
48
64
0
10
20
30
40
50
60
70
80
4 Nodes 8 Nodes 12 Nodes 16 Nodes
MaxKernels(4GBHeap)
Cluster Size (32GB Nodes)
MAXIMUM NUMBER OF SIMULTANEOUS KERNELS
IBM Developer / © 2019 IBM Corporation 18

19. Jupyter Notebooks
and Kubernetes
19IBM Developer / © 2019 IBM Corporation

20. Deep Learning Workloads
Resource Intensive workloads
Requires expensive hardware (GPU,
TPU)
Long Running training jobs
- Simple MNIST takes over one hour
WITHOUT a decent GPU
- Other non complex deep learning model
training can easily take over a day WITH
GPUs
IBM Developer / © 2019 IBM Corporation 20

21. Jupyter & Kubernetes
Kubernetes Platform
- Containers provides a flexible way to
deploy applications and are here to stay
- Containers simplify management of
complicated and heterogenous AI/Deep
Learning infrastructure
- Kubernetes enables easy management of
containerized applications and resources
with the benefit of Elasticity and Quality of
Services
Source: https://github.com/Langhalsdino/Kubernetes-GPU-Guide
IBM Developer / © 2019 IBM Corporation 21

22. Enterprise Gateway
& Kubernetes
Supported Platforms
Before Jupyter Enterprise Gateway …
- Resources required for all kernels needs to
be allocated during Notebook Server pod
creation
- Resources limited to what is physically
available on the host node that runs all
kernels and associated Spark drivers
After Jupyter Enterprise Gateway …
- Gateway pod very lightweight
- Kernels in their own pod, isolation
- Kernel pods built from community images:
Spark-on-K8s, TensorFlow, Keras, etc.
FfDL
Before Enterprise Gateway After Enterprise Gateway
IBM Developer / © 2019 IBM Corporation 22

23. Bob
Alice
Jupyter
Enterprise
Gateway
Bob
Alice
Container images defined in kernelspec
Community image
Kernel
Spark on Kubernetes
Kernel
Jupyter Enterprise Gateway - Kubernetes
IBM Developer / © 2019 IBM Corporation 23

24. Bob
Alice
Jupyter
Enterprise
Gateway
Bob
Alice
Container images defined in kernelspec
JupyterHub will provision
custom images containing
Notebook + NB2KG
extension
JupyterLab
Jupyter
Notebook
Community image
Kernel
Spark on Kubernetes
Kernel
Jupyter Enterprise Gateway - Kubernetes

25. IBM Developer / © 2019 IBM Corporation 25

26. Jupyter & Kubernetes
• Multi-user Enterprise Gateway pod
• Each kernel launched on it’s own pod
• Kernel pod namespace is configurable
IBM Developer / © 2019 IBM Corporation 26

27. Configuration
Jupyter Kernels are configured by
kernelspecs
- Each kernel has a correspondent kernelspec
- Stored in one of the Jupyter data path
directory
- $ jupyter kernelspec list
/…/anaconda3/share/jupyter/kernels/python2/kernel.jsom
IBM Developer / © 2019 IBM Corporation 27

28. Configurations
Process Proxy:
• Abstracts kernel process represented by Jupyter
framework
• Pluggable class definition identified in kernelspec
(kernel.json)
• Manages kernel lifecycle
Kernel Launcher:
• Embeds target kernel
• Listens on gateway communication port
• Conveys interrupt requests (via local signal)
• Could be extended for additional communications
{
"language": "python",
"display_name": "Spark - Python (Kubernetes Mode)",
"process_proxy": {
"class_name":
"enterprise_gateway.services.processproxies.k8s.KubernetesProcessProxy",
"config": {
"image_name": "elyra/kubernetes-kernel-py:dev",
"executor_image_name": "elyra/kubernetes-kernel-py:dev”,
"port_range" : "40000..42000"
}
},
"env": {
"SPARK_HOME": "/opt/spark",
"SPARK_OPTS": "--master k8s://https://${KUBERNETES_SERVICE_HOST --deploy-
mode cluster --name …",
…
},
"argv": [
"/usr/local/share/jupyter/kernels/spark_scala_yarn_cluster/bin/run.sh",
"--RemoteProcessProxy.kernel-id",
"{kernel_id}",
"--RemoteProcessProxy.response-address",
"{response_address}",
"--RemoteProcessProxy.port-range",
"{port_range}",
"--RemoteProcessProxy.spark-context-initialization-mode",
"lazy"
]
}
IBM Developer / © 2019 IBM Corporation 28

29. Spectrum Conductor
+
Supported
Runtime
Platforms
J U P Y T E R E N T E R P R I S E G A T E W A Y
Remote
Kernel Manager
Distributed
Process Proxy
YARN Cluster
Process Proxy
Kubernetes
Process Proxy
Conductor Cluster
Process Proxy
J U P Y T E R N O T E B O O K UI
NB2KG Extension
J U P Y T E R K E R N E L G A T E W A Y
J U P Y T E R N O T E B O O K
FfDL
P R O G R A M M A T I C A P I
Docker
Process Proxy
Jupyter Enterprise Gateway Components
IBM Developer / © 2019 IBM Corporation 29
+
With Notebook
6.0, the NB2KG
capabilities have
been integrated
into the Notebook
server.
For
programmatically
access, we have a
experimental
Enterprise
Gateway client
that enable
request a kernel
and submit code
very simply.

30. Summary
IBM Developer / © 2019 IBM Corporation 30

31. Interactive Workloads
across Kubernetes Cluster
+
• Enable support to
remote kernels in order
to scale Notebook
across entire cluster
• Multitenant with support
for user impersonation
leveraging Kerberos
• Base container image
becomes a choice (e.g.
Python with Tensorflow)
J U P Y T E R
E N T E R P R I S E G A T E W A Y
S U P P O R T E D
K E R N E L S
S U P P O R T E D
R U N T I M E S
IBM Developer / © 2019 IBM Corporation 31
+

32. Other resources
Jupyter Enterprise Gateway
https://Jupyter.org/enterprise_gateway/
Jupyter Enterprise Gateway source code at GitHub
https://github.com/jupyter/enterprise_gateway
Jupyter Enterprise Gateway Documentation
http://jupyter-enterprise-gateway.readthedocs.io/en/latest/
Jupyter Enterprise Gateway Gitter
https://gitter.im/jupyter/enterprise_gateway
Jupyter Blog
https://blog.jupyter.org/
IBM Developer / © 2019 IBM Corporation 32
Stable Release - EG 1.2.0
(Analytics Workload with Spark running
YARN cluster mode support)
pip install jupyter_enterprise_gateway
Beta Release - EG 2.0.0 RC1
Introduce support for AI Workloads on
Kubernetes
pip install --pre jupyter_enterprise_gateway
STAR
US
&
FORK
US
ON
GITHUB

33. Thank you!
@lresende1975
@AlanChin11
IBM Developer / © 2019 IBM Corporation 33