A PyCon TW 2016 speech about Jupyter Kernels

1. Jupyter Kernel
How to Speak in Another Language

2. How to Make a
Jupyter Kernel:

3. 1: 5 ømq sockets

4. 2: Implement Protocol

5. 3. Holy Shit,
YOU RUINED EVERYTHING,
LIKE EVERYTHING YOU DO

6. 4. This Kernel
Is Like Your Life

7. 5. Begin a Lot of Things,
Left Undone

8. 6. And Nobody Loves You

9. Understanding
Jupyter Kernel

10. Agenda
• Jupyter, IPython, notebook, console, client, kernel
• The Interactive Computing Protocol
• Implementing a Kernel
• Live Demo

11. About this guy
• 廖偉涵 Adrian Liaw
• github.com/adrianliaw
• High School Homeschooler
• A Pythonista
• Student of Udacity’s
Data Analyst Nanodegree

12. About this guy
• Amateur Pianist
• Classical Music Lover
• Coffee Shop Lover

14. IPYTHON
ENHANCED PYTHON SHELL

15. IPYTHON NOTEBOOK
WEB-BASED NOTEBOOK ENVIRONMENT

16. The Big Split™

18. Jupyter Notebook
(With R Kernel)

19. -Project Jupyter
“Project Jupyter was born out of the
IPython Project in 2014 as it evolved
to support interactive data science
and scientific computing across all
programming languages.”

22. CLIENT KERNEL
ØMQ Sockets

23. CLIENT KERNEL
Execute “10 + 3”

24. CLIENT KERNEL
Result: 13

25. CLIENT
KERNEL
CLIENT

26. What’s Inside IPython
• The Interactive IPython Shell (No ØMQ)
• Magic Commands
• Auto Word Completer
• Beautiful Traceback
• Shell History Management

27. Uh, And Kernel?

28. IPyKernel
• Jupyter’s “Kernel 0”
• ØMQ Handlers
• github.com/ipython/ipykernel

29. What’s Inside Jupyter
• Web-based Notebook Interface & nbconvert
• Interactive Qt Console
• Terminal-based Console
• Spec for Notebook Document Format
• Spec for Interactive Computing Protocol

30. CLIENT KERNEL
ØMQ Sockets

31. Interactive Computing
Protocol
• i.e. Jupyter Messaging Protocol
• Communication Between Kernel and Client
• Based on ØMQ and JSON

32. DEALER ROUTER
CLIENT KERNEL
Shell
SUB PUBIOPub
DEALER ROUTER
stdin
DEALER ROUTER
Control
REQ REP
Heartbeat

33. Connection File
{
"transport": "tcp",
"ip": "127.0.0.1",
"shell_port": 56166,
"iopub_port": 56167,
"stdin_port": 56168,
"control_port": 56169,
"hb_port": 56170,
"signature_scheme": "hmac-sha256",
"key": "<hashed key>"
}

34. Message Format
{
'header' : {
'msg_id' : str,
'username' : str,
'session' : str,
'date': str,
'msg_type' : str,
'version' : '5.0',
},
'parent_header' : dict,
'metadata' : dict,
'content' : dict,
}

35. http://bit.ly/JupyterMessaging

36. {
"header": {
"msg_type": "kernel_info_request",
...
},
"content": {},
}
CLIENT KERNEL
Shell

37. CLIENT KERNEL
Shell
{
"header": {
"msg_type": "kernel_info_reply",
...
},
"parent_header": {...},
"content": {
"banner": "Python 3.5.1 ... IPython 4.0.0 ...",
"implementation": "ipython",
"implementation_version": "4.0.0",
"language_info": {
"name": "python",
"version": "3.5.1",
...
},
},
}

38. CLIENT KERNELShell
{
"header": {
"msg_type": "execute_request",
...
},
"content": {
"code": "print('HAI')",
"silent": False,
"store_history": True,
},
}

39. CLIENT KERNELIOPub
{
"header": {"msg_type": "stream", ...},
"parent_header": {...},
"content": {
"name": "stdout",
"text": "HAIn",
},
}

40. CLIENT KERNELShell
{
"header": {
"msg_type": "execute_reply",
...
},
"parent_header": {...},
"content": {
"execution_count": 1,
"status": "ok",
"user_expressions": {},
},
}

41. CLIENT KERNELShell
{
"header": {"msg_type": "execute_request", ...},
"content": {
"code": "pd.read_csv('foo.csv')",
"silent": False,
"store_history": True,
},
}

42. CLIENT KERNELIOPub
{
"header": {"msg_type": "execute_result", ...},
"parent_header": {...},
"content": {
"execution_count": 5,
"data": {
"text/plain": " col1 col2n A 3n....",
"text/html": "<div><table>...</table></div>",
},
},
}

43. CLIENT KERNELShell
{
"header": {
"msg_type": "execute_reply",
...
},
"parent_header": {...},
"content": {
"execution_count": 5,
"status": "ok",
"user_expressions": {},
},
}

46. CLIENT KERNELShell
{
"header": {"msg_type": "execute_request", ...},
"content": {
"code": "read.csv('table.csv')",
"silent": False,
"store_history": True,
},
}

47. CLIENT KERNELIOPub
{
"header": {"msg_type": "execute_result", ...},
"parent_header": {...},
"content": {
"execution_count": 1,
"data": {
"text/plain": " col1 col2n A 3n....",
"text/html": "<table>...</table>",
},
},
}

48. CLIENT KERNELShell
{
"header": {
"msg_type": "execute_reply",
...
},
"parent_header": {...},
"content": {
"execution_count": 1,
"status": "ok",
"user_expressions": {},
},
}

49. MAKING A KERNEL

50. Kernel Types
• Native Kernel
• Python Wrapper Kernel
• REPL Wrapper Kernel

51. IRKernel
11. zmqctx <<- zmq.ctx.new()
12. sockets <<- list(
13. hb = zmq.socket(zmqctx, .pbd_env$ZMQ.ST$REP),
14. iopub = zmq.socket(zmqctx, .pbd_env$ZMQ.ST$PUB),
15. control = zmq.socket(zmqctx, .pbd_env$ZMQ.ST$ROUTER),
16. stdin = zmq.socket(zmqctx, .pbd_env$ZMQ.ST$ROUTER),
17. shell = zmq.socket(zmqctx, .pbd_env$ZMQ.ST$ROUTER))

52. IRKernel
IRuby
IHaskell
cling
IJulia
IScala
Lua Kernel
Clojure-Kernel
ITorch
IJavascript
ipykernel

53. Wrapper Kernel
• Python-driven
• ipykernel.kernelbase.Kernel
• http://bit.ly/WrapperKernel

54. from ipykernel.kernelbase import Kernel
class MyKernel(Kernel):
def do_execute
def do_complete
def do_history
def do_shutdown

55. Hy Kernel
Redis Kernel
Xonsh
C Kernel
Matlab Kernel
SAS Kernel

56. REPL Wrapper
from pexpect.replwrap
import REPLWrapper

57. Octave Kernel
Jupyter Powershell
Bash Kernel
Mongo Kernel
Scilab Kernel

58. LIVE DEMO

59. Q&A

60. Reference
• P.3, P.4, P.17, P.20, P.50 images: Project Jupyter
• P.6, P.7, P.8 images: Molg H.
• P.16: Project Jupyter
• P.11 image: Toomore Chiang
• P.12 image: Virginia Cheng
• P.13 image: ipython
• P.45 image: R