~10min dive to Python Asynchronous IO HTML version (recommended): https://dl.dropboxusercontent.com/u/1565687/speak/Python3%20AsyncIO%20Horizon/index.html

1. 19 Dec 2013
@lukmdo
Python Asynchronous I/O
asynchronous landscape refresher
Photo by Jake Brown used under CC BY 2.0/“Mono+Balance”

2. Agenda ⏰10 minutes
!
•
Background (can skip)
•
Problem statement
•
“Current state”
•
Python 3 async horizon (by examples)

3. Warmup
!
•
CPU IO (CPU bound)
•
Pure IO (IO bound)
•
C10k
•
Concurrency vs. parallelism

4. Warmup
!
•
Lazy schedule a task for later
•
Avoid waiting for IO (i.e. 2x DB queries)
•
Solve embarrassingly parallel problems
•
Control and manage status of task

5. PEP index brief
PEP
Name
Status
Doc
PEP 255
Simple Generators
2.2 ✔
»»
PEP 342
Coroutines via Enhanced Generators
2.5 ✔
»
PEP 380
Syntax for Delegating to a Subgenerator
3.3 ✔
»
PEP 3148
Futures - execute computations asynchronously
3.2 ✔
»
PEP 3153
Asynchronous IO support
3.3 ✔
»
PEP 3156
Asynchronous IO Support Rebooted: the
"asyncio" Module
3.4 ✔
»

6. PEP index brief
PEP
Name
PEP 219
Stackless Python
❙❙
PEP 220
Coroutines, Generators, Continuations
✗
PEP 319
Python Synchronize/Asynchronize Block
✗
PEP 3145
PEP 3152
Asynchronous I/O For subprocess.Popen
Cofunctions
Status
❙❙
❙❙

7. stackless
In the Class
concurrent.futures
threading
tornado
greenlet
subpocess
eventlet
celery
twisted
pyuv
gevent
shed asyncio
asyncore

8. In the Class
asyncio

9. stackless
In the Class
concurrent.futures
threading
tornado
greenlet
subpocess
eventlet
celery
twisted
pyuv
gevent
shed asyncio
asyncore

10. In the Class
concurrent.futures
threading
tornado
subpocess
celery
twisted
asyncio

11. Slang*
•
future ~ result wrapper
•
coro != generator function (but close)
•
task = coro wrapped in Future
•
result = yield from future/coro
(to wait for result)

12. new* old style concurrency
def update_foo(data):
data1 = run_query1(data)
data2 = run_query2(data)
!
data3 = process(data1, data2)
run_update_query(data3)
run_update_cache(data3)
!
return ["OK"]

13. new* old style concurrency
def update_foo(data):
data1 = run_query1(data)
data2 = run_query2(data)
!
data3 = process(data1, data2)
run_update_query(data3)
run_update_cache(data3)
!
return ["OK"]

14. new* old style concurrency

15. new* old style concurrency

16. new* old style concurrency
@asyncio.coroutine
def update_foo(data, loop):
data1 = yield from run_query1(data)
data2 = yield from run_query2(data)
!
data3 = yield from process(data1, data2)
yield from run_update_query(data3)
loop.call_soon(update_cache, data3)
!
return ["OK"]

17. new* old style concurrency
@asyncio.coroutine
def update_foo(data, loop):
data1 = yield from run_query1(data)
data2 = yield from run_query2(data)
!
data3 = yield from process(data1, data2)
yield from run_update_query(data3)
loop.call_soon(update_cache, data3)
!
return ["OK"]

18. new* old style concurrency
@asyncio.coroutine
def update_foo(data, loop):
data1 = yield from run_query1(data)
data2 = yield from run_query2(data)
!
data3 = yield from process(data1, data2)
yield from run_update_query(data3)
loop.call_soon(update_cache, data3)
!
return ["OK"]

19. new* old style concurrency

20. new* old style concurrency

21. new* old style concurrency

22. new* old style concurrency
@asyncio.coroutine
def update_foo(data, loop):
future1 = asyncio.async(run_query1(data))
future2 = asyncio.async(run_query2(data))
!
results = yield from asyncio.gather(
future1, future2)
data3 = yield from process(*results)
yield from run_update_query(data3)
!
loop.call_soon(update_cache, data3)
!
return ["OK"]

23. new* old style concurrency
@asyncio.coroutine
def update_foo(data, loop):
future1 = asyncio.async(run_query1(data))
future2 = asyncio.async(run_query2(data))
!
results = yield from asyncio.gather(
future1, future2)
data3 = yield from process(*results)
yield from run_update_query(data3)
!
loop.call_soon(update_cache, data3)
!
return ["OK"]

24. new* old style concurrency
@asyncio.coroutine
def update_foo(data, loop):
future1 = asyncio.async(run_query1(data))
future2 = asyncio.async(run_query2(data))
!
results = yield from asyncio.gather(
future1, future2)
data3 = yield from process(*results)
yield from run_update_query(data3)
!
loop.call_soon(update_cache, data3)
!
return ["OK"]

25. new* old style concurrency

26. new* old style concurrency

27. new* old style concurrency

28. new* old style concurrency
def api_calls():
data1 = call_api1()
data2 = call_api2()
data3 = call_api3()
!
return [data1, data2, data3]

29. new* old style concurrency
def api_calls():
data1 = call_api1()
data3 = call_api2()
data3 = call_api3()
!
return [data1, data2, data3]

30. new* old style concurrency

31. new* old style concurrency

32. new* old style concurrency
@asyncio.coroutine
def api_calls():
future1 = asyncio.Task(call_api1())
future2 = asyncio.Task(call_api2())
future3 = asyncio.Task(call_api3())
!
return ( yield from asyncio.gather(
future1, future2, future3) )

33. new* old style concurrency
@asyncio.coroutine
def api_calls():
future1 = asyncio.Task(call_api1())
future2 = asyncio.Task(call_api2())
future3 = asyncio.Task(call_api3())
!
return ( yield from asyncio.gather(
future1, future2, future3) )

34. new* old style concurrency

35. new* old style concurrency

36. new* old style concurrency

37. new* old style concurrency
from aiohttp import request
!
@asyncio.coroutine
def call_api():
url = "URL"
response = yield from request('GET', url)
return (yield from response.read())

38. Takeaways
•
concurrency patterns
•
asyncio bootstrap
•
early adopters

39. Links
•
Video by BDFL Tulip: Async I/O for Python 3
•
Video by BDFL PyCon US 2013 Keynote
•
Code by Nikolay Kim aiohttp
•
Code by google appengine-pipeline
•
Code by google ndb tasklets
Photo by Jake Brown used under CC BY 2.0/“Mono+Tile”