1. Concurrency with Multiprocessing in Python
PhillyPug & Philly.rb RedSnake Meeting
February 8th, 2011
Gavin M. Roy
myYearbook.com

2. green threads in threading

3. from threading import Thread
import time
class MyThread(Thread):
def run(self):
print "%s running" % self.name
time.sleep(5)
print "%s done" % self.name
for x in xrange(0,10):
thread = MyThread()
thread.start()
thread.join(0)

4. gmr-0x04:pika gmr$ python threads.py
MyThread-1 running
MyThread-2 running
MyThread-3 running
MyThread-4 running
MyThread-5 running
MyThread-6 running
MyThread-7 running
MyThread-8 running
MyThread-9 running
MyThread-10 running
MyThread-1 done
MyThread-2 done
MyThread-3 done
MyThread-4 done
MyThread-5 done
MyThread-6 done
MyThread-7 done
MyThread-8 done
MyThread-9 done
MyThread-10 done

5. threading
• Locks
• Reentrant Locks
Reentrant knows who
own the locks and the
recursion level
• Conditions
Additional classes like
Queue.Queue
• Semaphores
• Events
• Timers

7. This GIL From David Beeazley’s GIL Visualization
http://www.dabeaz.com/GIL/gilvis/fourthread.html

8. enter multiprocessing

9. multiprocessing module
• All the things threading has • Connections
• Exchanging Objects • Managers TCP Server coordinating
shared objects
• Queues and Pipes • SyncManager Process Sync
• Shared State • Logging
• Pipes and Queues
• Pools

10. from multiprocessing import Process
import time
class MyThread(Process):
def run(self):
print "%s running" % self.name
time.sleep(5)
print "%s done" % self.name
for x in xrange(0,10):
thread = MyThread()
thread.start()
thread.join(0)

11. gmr-0x04:pika gmr$ python processes.py
Thread-1 running
Thread-2 running
Thread-3 running
Thread-4 running
Thread-5 running
Thread-6 running
Thread-7 running
Thread-8 running
Thread-9 running
Thread-10 running
Thread-1 done
Thread-2 done
Thread-3 done
Thread-4 done
Thread-5 done
Thread-6 done
Thread-7 done
Thread-8 done
Thread-9 done
Thread-10 done

12. multiprocessing.reduction
Photo By Susan NYC: http://www.flickr.com/photos/en321/33868864/

13. # Process #1
from multiprocessing.reduction import reduce_handle
import socket
# Create a socket
sock = socket.socket(socket.AF_INET,
socket.SOCK_STREAM, 0)
# Do stuff here, client or server wise
# Create the pickled socket handle
handle = reduce_handle(sock.fileno)
# Process #2
from multiprocessing.reduction import rebuild_handle
import socket
# In other process
fd = rebuild_handle(handle)
sock = socket.fromfd(fd, socket.AF_INET,
socket.SOCK_STREAM)
# Now I can read and write from the socket too