Jupyter Notebooks provide an interactive computational environment, in which you can combine Python code, rich text, mathematics, plots and rich media. It provides a convenient way for data analysts to explore, capture and share their research. Numerous options exist for working with Jupyter Notebooks, including running a Jupyter Notebook instance locally or by using a Jupyter Notebook hosting service. This talk will provide a quick tour of some of the more well known options available for running Jupyter Notebooks. It will then look at custom options for hosting Jupyter Notebooks yourself using public or private cloud infrastructure. An in-depth look at how you can run Jupyter Notebooks in OpenShift will be presented. This will cover how you can directly deploy a Jupyter Notebook server image, as well as how you can use Source-to-Image (S2I) to create a custom application for your requirements by combining an existing Jupyter Notebook server image with your own notebooks, additional code and research data. Specific use cases around Jupyter Notebooks which will be explored will include individual use, team use within an organisation, and class room environments for teaching. Other issues which will be covered include importing of notebooks and data into an environment, storing data using persistent volumes and other forms of centralised storage. As an example of the possibilities of using Jupyter Notebooks with a cloud, it will be shown how you can easily use OpenShift to set up a distributed parallel computing cluster using ‘ipyparallel’ and use it in conjunction with a Jupyter Notebook.

1. Data analytics in the
cloud with Jupyter
Notebooks
Graham Dumpleton
Graham.Dumpleton@gmail.com

2. http://jupyter.org/

3. Python Data Science Handbook / 04.12-Three-Dimensional-Plotting

4. Python Data Science Handbook / 04.13-Geographic-Data-With-Basemap

5. https://blog.data.gov.sg/how-we-caught-the-circle-line-rogue-train-with-data-79405c86ab6a

7. Who’s Using It?
Individuals
Collaborators
Teachers

8. Getting Started
pip3 install jupyter
jupyter notebook

9. Empty Workspace

10. Upload Notebooks

11. Local File System
$ ls notebooks/01*.ipynb
notebooks/01.00-IPython-Beyond-Normal-Python.ipynb
notebooks/01.01-Help-And-Documentation.ipynb
notebooks/01.02-Shell-Keyboard-Shortcuts.ipynb
notebooks/01.03-Magic-Commands.ipynb
notebooks/01.04-Input-Output-History.ipynb
notebooks/01.05-IPython-And-Shell-Commands.ipynb
notebooks/01.06-Errors-and-Debugging.ipynb
notebooks/01.07-Timing-and-Profiling.ipynb
notebooks/01.08-More-IPython-Resources.ipynb

12. Browsing Files

13. Interacting with a Notebook

14. Status of Notebooks

15. Installing Packages

16. Positives
• Save notebooks/data locally.
• Python virtual environments.
• Select Python version you want.
• Install required Python packages.

17. Negatives
• Operating system differences.
• Python distribution differences.
• Python version differences.
• Package index differences.
• PyPi (pip) vs Anaconda (conda)
• Effort to setup and maintain.

18. Docker Images
https://github.com/jupyter/docker-stacks

20. Running Docker Image
docker run -it --rm -p 8888:8888
jupyter/minimal-notebook

21. Positives
• Pre-created images.
• Bundled operating system packages.
• Known Python distribution/vendor.
• Bundled Python packages.
• Docker images are read only.
• Don’t need to maintain the image.

22. Negatives (1)
• More effort to customise experience.
• Build a custom Docker image to extend.
• Install extra packages each time you run it.
• Images can be very large.
• Multiple Python versions.
• Packages that you do not need.

23. Negatives (2)
• Access to and saving your notebooks/data.
• Need to mount persistent storage volumes.
• Ensuring access is done securely.

24. tmpnb.org
https://tmpnb.org/

25. Azure Notebooks
https://notebooks.azure.com/

26. Binder Service
http://mybinder.org/

27. Positives
• Somebody else looks after everything.

28. Negatives
• Shared resource.
• Outside of your control.
• Reliability.
• Customisation.
• Software versions.
• Information security.

29. JupyterHub
https://jupyterhub.readthedocs.io

30. Positives
• Can customise however you want.
• Modify code for service.
• Use custom images.

31. Negatives
• Dedicated infrastructure.
• Effort to understand and set it up.
• Effort to keep it running.

32. Many Options to
Choose From

33. OpenShift

34. Deployments

35. Docker Image

36. Image Stream

37. Notebook Storage

38. Attaching Storage

39. Shared Storage

40. Positives
• Use existing features of OpenShift
• No special storage backends required.
• No custom provisioning applications.
• Cluster can still be used for other applications.
• Simply set quotas and users do what they want.

41. Source-to-Image

42. Positives
• Easily build custom images.
• Pre-populated with required Python packages.
• Pre-populated with required Jupyter Notebooks.
• Pre-populated with required data files.
• Direct to application, or to create images.

43. Service Catalog

44. Templates (builder)

45. Templates (cluster)

46. Templates (notebook)

47. IPyParallel Cluster

48. Parallel Computing

49. Positives
• Templates enable complex deployments.
• Don’t need something like JupyterHub.

50. Challenges
• Custom base images and builders.
• Learning curve for writing templates.

51. Command Line
oc new-app stats101-notebook-template
--param STUDENT_NUMBER=1
--param CLASS_NUMBER=1234
oc new-app stats101-notebook-template
--param STUDENT_NUMBER=2
--param CLASS_NUMBER=1234
…
oc delete all --selector class=1234

52. REST API
import powershift.endpoints as endpoints
client = endpoints.Client()
projects = client.oapi.v1.projects.get()
def public_address(route):
host = route.spec.host
path = route.spec.path or '/'
if route.spec.tls:
return 'https://%s%s' % (host, path)
return 'http://%s%s' % (host, path)
routes = client.oapi.v1.namespaces(namespace='stats101').routes.get()
for route in routes.items:
print(' route=%r' % public_address(route))

53. Positives
• Easily trigger multiple deployments using CLI.
• REST API also available for custom front ends.

54. Resources
• S2I enabled Jupyter Notebook images
• https://github.com/getwarped/jupyter-notebooks
• OpenShift versions of Jupyter Project images
• https://github.com/getwarped/jupyter-stacks
• Python REST API client for OpenShift
• https://github.com/getwarped/powershift