SlideShare ist ein Scribd-Unternehmen logo
1 von 21
Downloaden Sie, um offline zu lesen
Intro
       Threads
           GIL
      Processes
         Async




Concurrency in Python

      Konrad Delong


March 23 2010, Amsterdam




  Konrad Delong   Concurrency in Python
Intro
                          Threads
                              GIL
                         Processes
                            Async


me




     IT student (Erasmus)
     Python
     Thesis




                     Konrad Delong   Concurrency in Python
Intro
                             Threads
                                 GIL
                            Processes
                               Async


concurrency



  In computer science, concurrency is a property of systems in which
  several computations are executing simultaneously, and
  potentially interacting with each other. The computations may be
  executing on multiple cores in the same chip, preemptively
  time-shared threads on the same processor, or executed on
  physically separated processors.
  (en.wikipedia.org)




                        Konrad Delong   Concurrency in Python
Intro
                       Threads
                           GIL
                      Processes
                         Async


threads




      thread
      threading




                  Konrad Delong   Concurrency in Python
Intro
                         Threads
                             GIL
                        Processes
                           Async


thread




  import thread
  thread.start_new_thread(function, args[, kwargs])
  thread.get_ident()
  thread.allocate_lock()




                    Konrad Delong   Concurrency in Python
Intro
                         Threads
                             GIL
                        Processes
                           Async


threading


  from threading import (Thread, Condition,
                         Lock, RLock, Semaphore)
  class MyThread(Thread):
      def run(self)
          pass
  t = MyThread()
  # or
  t = Thread(target=function, args=[], kwargs={})
  # and then
  t.start()
  t.join()


                    Konrad Delong   Concurrency in Python
Intro
           Threads
               GIL
          Processes
             Async




GIL




      Konrad Delong   Concurrency in Python
Intro
                            Threads
                                GIL
                           Processes
                              Async


How it works?




     Interpreter in only one thread
     Released every check (== 100 ticks) and on blocking
     operations




                       Konrad Delong   Concurrency in Python
Intro
                         Threads
                             GIL
                        Processes
                           Async


check


  if (--_Py_Ticker < 0) {
      _Py_Ticker = _Py_CheckInterval; // default 100
      if (interpreter_lock) {
          /* Give another thread a chance */
          PyThread_release_lock(interpreter_lock);
          /* Other threads may run now */
          PyThread_acquire_lock(interpreter_lock, 1);
      }
  }



                    Konrad Delong   Concurrency in Python
Intro
                           Threads
                               GIL
                          Processes
                             Async


Why GIL is bad?




     No gains on multicore
     Latency in responding to events




                      Konrad Delong   Concurrency in Python
Intro
                            Threads
                                GIL
                           Processes
                              Async


Why GIL is not that bad?




     Performance on single core
     Safe environment for C extentions
     Other solutions than threads exist




                       Konrad Delong   Concurrency in Python
Intro
                           Threads
                               GIL
                          Processes
                             Async


Remove GIL?




     Greg Stein, python-safethread
     PyPy, Unladen Swallow
     current policy on dropping GIL in cPython




                      Konrad Delong   Concurrency in Python
Intro
                             Threads
                                 GIL
                            Processes
                               Async


Other solutions?




      Processes instead of threads
      Jython/IronPython
      Async




                        Konrad Delong   Concurrency in Python
Intro
                         Threads
                             GIL
                        Processes
                           Async


multiprocessing module


  from multiprocessing import (Process, Condition,
                               Lock, RLock, Semaphore)
  class MyProcess(Process):
      def run(self)
          pass
  t = MyProcess()
  # or
  t = Process(target=function, args=[], kwargs={})
  # and then
  t.start()
  t.join()


                    Konrad Delong   Concurrency in Python
Intro
                          Threads
                              GIL
                         Processes
                            Async


just like ...


   from threading import (Thread, Condition,
                          Lock, RLock, Semaphore)
   class MyThread(Thread):
       def run(self)
           pass
   t = MyThread()
   # or
   t = Thread(target=function, args=[], kwargs={})
   # and then
   t.start()
   t.join()


                     Konrad Delong   Concurrency in Python
Intro
                             Threads
                                 GIL
                            Processes
                               Async


not entirely the same




      no implicit shared memory (global vars, imported modules,
      other)
      IPC (serializable data)
      separate machines IPC




                        Konrad Delong   Concurrency in Python
Intro
                         Threads
                             GIL
                        Processes
                           Async


Queues



  def foo(queue):
      print queue.get()
  q = Queue()
  t = Process(target=foo, args=[q])
  t.start()
  q.put([1, "a"])

  # will print [1, "a"]




                    Konrad Delong   Concurrency in Python
Intro
                         Threads
                             GIL
                        Processes
                           Async


Shared memory




  from multiprocessing import Value, Array

  num = Value(’d’, 0.0)
  arr = Array(’i’, range(10))




                    Konrad Delong   Concurrency in Python
Intro
                              Threads
                                  GIL
                             Processes
                                Async


Async




        callbacks
        no blocking
        still single CPU, but better scaling (memory management)
        explicit(Twisted, Tornado) vs hidden




                         Konrad Delong   Concurrency in Python
Intro
               Threads
                   GIL
              Processes
                 Async


Reactor




          Konrad Delong   Concurrency in Python
Intro
                              Threads
                                  GIL
                             Processes
                                Async


Hidden async




     Kamaelia
     select, epoll, kqueue
     Eventlet, Gevent, PyEvent




                       Konrad Delong     Concurrency in Python

Weitere ähnliche Inhalte

Was ist angesagt?

[Harvard CS264] 10a - Easy, Effective, Efficient: GPU Programming in Python w...
[Harvard CS264] 10a - Easy, Effective, Efficient: GPU Programming in Python w...[Harvard CS264] 10a - Easy, Effective, Efficient: GPU Programming in Python w...
[Harvard CS264] 10a - Easy, Effective, Efficient: GPU Programming in Python w...npinto
 
[Harvard CS264] 03 - Introduction to GPU Computing, CUDA Basics
[Harvard CS264] 03 - Introduction to GPU Computing, CUDA Basics[Harvard CS264] 03 - Introduction to GPU Computing, CUDA Basics
[Harvard CS264] 03 - Introduction to GPU Computing, CUDA Basicsnpinto
 
20110319 parameterized algorithms_fomin_lecture01-02
20110319 parameterized algorithms_fomin_lecture01-0220110319 parameterized algorithms_fomin_lecture01-02
20110319 parameterized algorithms_fomin_lecture01-02Computer Science Club
 
Sequence Learning with CTC technique
Sequence Learning with CTC techniqueSequence Learning with CTC technique
Sequence Learning with CTC techniqueChun Hao Wang
 
Essentials of Multithreaded System Programming in C++
Essentials of Multithreaded System Programming in C++Essentials of Multithreaded System Programming in C++
Essentials of Multithreaded System Programming in C++Shuo Chen
 
Java 7 - New Features - by Mihail Stoynov and Svetlin Nakov
Java 7 - New Features - by Mihail Stoynov and Svetlin NakovJava 7 - New Features - by Mihail Stoynov and Svetlin Nakov
Java 7 - New Features - by Mihail Stoynov and Svetlin NakovSvetlin Nakov
 
Java session13
Java session13Java session13
Java session13Niit Care
 
Sentence-to-Code Traceability Recovery with Domain Ontologies
Sentence-to-Code Traceability Recovery with Domain OntologiesSentence-to-Code Traceability Recovery with Domain Ontologies
Sentence-to-Code Traceability Recovery with Domain OntologiesShinpei Hayashi
 
2009 CSBB LAB 新生訓練
2009 CSBB LAB 新生訓練2009 CSBB LAB 新生訓練
2009 CSBB LAB 新生訓練Abner Huang
 
Where destructors meet threads
Where destructors meet threadsWhere destructors meet threads
Where destructors meet threadsShuo Chen
 
Os Reindersfinal
Os ReindersfinalOs Reindersfinal
Os Reindersfinaloscon2007
 
Presentation of the open source CFD code Code_Saturne
Presentation of the open source CFD code Code_SaturnePresentation of the open source CFD code Code_Saturne
Presentation of the open source CFD code Code_SaturneRenuda SARL
 
Parallel program design
Parallel program designParallel program design
Parallel program designZongYing Lyu
 
A benchmark evaluation for incremental pattern matching in graph transformation
A benchmark evaluation for incremental pattern matching in graph transformationA benchmark evaluation for incremental pattern matching in graph transformation
A benchmark evaluation for incremental pattern matching in graph transformationIstvan Rath
 
Transformer Zoo (a deeper dive)
Transformer Zoo (a deeper dive)Transformer Zoo (a deeper dive)
Transformer Zoo (a deeper dive)Grigory Sapunov
 
Detecting Occurrences of Refactoring with Heuristic Search
Detecting Occurrences of Refactoring with Heuristic SearchDetecting Occurrences of Refactoring with Heuristic Search
Detecting Occurrences of Refactoring with Heuristic SearchShinpei Hayashi
 
Welcome to International Journal of Engineering Research and Development (IJERD)
Welcome to International Journal of Engineering Research and Development (IJERD)Welcome to International Journal of Engineering Research and Development (IJERD)
Welcome to International Journal of Engineering Research and Development (IJERD)IJERD Editor
 

Was ist angesagt? (20)

[Harvard CS264] 10a - Easy, Effective, Efficient: GPU Programming in Python w...
[Harvard CS264] 10a - Easy, Effective, Efficient: GPU Programming in Python w...[Harvard CS264] 10a - Easy, Effective, Efficient: GPU Programming in Python w...
[Harvard CS264] 10a - Easy, Effective, Efficient: GPU Programming in Python w...
 
[Harvard CS264] 03 - Introduction to GPU Computing, CUDA Basics
[Harvard CS264] 03 - Introduction to GPU Computing, CUDA Basics[Harvard CS264] 03 - Introduction to GPU Computing, CUDA Basics
[Harvard CS264] 03 - Introduction to GPU Computing, CUDA Basics
 
Advancements in Neural Vocoders
Advancements in Neural VocodersAdvancements in Neural Vocoders
Advancements in Neural Vocoders
 
20110319 parameterized algorithms_fomin_lecture01-02
20110319 parameterized algorithms_fomin_lecture01-0220110319 parameterized algorithms_fomin_lecture01-02
20110319 parameterized algorithms_fomin_lecture01-02
 
Sequence Learning with CTC technique
Sequence Learning with CTC techniqueSequence Learning with CTC technique
Sequence Learning with CTC technique
 
Essentials of Multithreaded System Programming in C++
Essentials of Multithreaded System Programming in C++Essentials of Multithreaded System Programming in C++
Essentials of Multithreaded System Programming in C++
 
Java 7 - New Features - by Mihail Stoynov and Svetlin Nakov
Java 7 - New Features - by Mihail Stoynov and Svetlin NakovJava 7 - New Features - by Mihail Stoynov and Svetlin Nakov
Java 7 - New Features - by Mihail Stoynov and Svetlin Nakov
 
Java session13
Java session13Java session13
Java session13
 
Sentence-to-Code Traceability Recovery with Domain Ontologies
Sentence-to-Code Traceability Recovery with Domain OntologiesSentence-to-Code Traceability Recovery with Domain Ontologies
Sentence-to-Code Traceability Recovery with Domain Ontologies
 
javarmi
javarmijavarmi
javarmi
 
2009 CSBB LAB 新生訓練
2009 CSBB LAB 新生訓練2009 CSBB LAB 新生訓練
2009 CSBB LAB 新生訓練
 
Where destructors meet threads
Where destructors meet threadsWhere destructors meet threads
Where destructors meet threads
 
Os Reindersfinal
Os ReindersfinalOs Reindersfinal
Os Reindersfinal
 
Presentation of the open source CFD code Code_Saturne
Presentation of the open source CFD code Code_SaturnePresentation of the open source CFD code Code_Saturne
Presentation of the open source CFD code Code_Saturne
 
Parallel program design
Parallel program designParallel program design
Parallel program design
 
A benchmark evaluation for incremental pattern matching in graph transformation
A benchmark evaluation for incremental pattern matching in graph transformationA benchmark evaluation for incremental pattern matching in graph transformation
A benchmark evaluation for incremental pattern matching in graph transformation
 
Transformer Zoo (a deeper dive)
Transformer Zoo (a deeper dive)Transformer Zoo (a deeper dive)
Transformer Zoo (a deeper dive)
 
Ayse
AyseAyse
Ayse
 
Detecting Occurrences of Refactoring with Heuristic Search
Detecting Occurrences of Refactoring with Heuristic SearchDetecting Occurrences of Refactoring with Heuristic Search
Detecting Occurrences of Refactoring with Heuristic Search
 
Welcome to International Journal of Engineering Research and Development (IJERD)
Welcome to International Journal of Engineering Research and Development (IJERD)Welcome to International Journal of Engineering Research and Development (IJERD)
Welcome to International Journal of Engineering Research and Development (IJERD)
 

Andere mochten auch

Ruby 1.9 Fibers
Ruby 1.9 FibersRuby 1.9 Fibers
Ruby 1.9 FibersKevin Ball
 
Introduction to Python Celery
Introduction to Python CeleryIntroduction to Python Celery
Introduction to Python CeleryMahendra M
 
Async programming and python
Async programming and pythonAsync programming and python
Async programming and pythonChetan Giridhar
 
Parallel Programming in Python: Speeding up your analysis
Parallel Programming in Python: Speeding up your analysisParallel Programming in Python: Speeding up your analysis
Parallel Programming in Python: Speeding up your analysisManojit Nandi
 
Distributed and concurrent programming with RabbitMQ and EventMachine Rails U...
Distributed and concurrent programming with RabbitMQ and EventMachine Rails U...Distributed and concurrent programming with RabbitMQ and EventMachine Rails U...
Distributed and concurrent programming with RabbitMQ and EventMachine Rails U...Paolo Negri
 

Andere mochten auch (6)

Python For Large Company?
Python For Large Company?Python For Large Company?
Python For Large Company?
 
Ruby 1.9 Fibers
Ruby 1.9 FibersRuby 1.9 Fibers
Ruby 1.9 Fibers
 
Introduction to Python Celery
Introduction to Python CeleryIntroduction to Python Celery
Introduction to Python Celery
 
Async programming and python
Async programming and pythonAsync programming and python
Async programming and python
 
Parallel Programming in Python: Speeding up your analysis
Parallel Programming in Python: Speeding up your analysisParallel Programming in Python: Speeding up your analysis
Parallel Programming in Python: Speeding up your analysis
 
Distributed and concurrent programming with RabbitMQ and EventMachine Rails U...
Distributed and concurrent programming with RabbitMQ and EventMachine Rails U...Distributed and concurrent programming with RabbitMQ and EventMachine Rails U...
Distributed and concurrent programming with RabbitMQ and EventMachine Rails U...
 

Ähnlich wie Concurrency in Python

Python For Scientists
Python For ScientistsPython For Scientists
Python For Scientistsaeberspaecher
 
Pycon11: Python threads: Dive into GIL!
Pycon11: Python threads: Dive into GIL!Pycon11: Python threads: Dive into GIL!
Pycon11: Python threads: Dive into GIL!Chetan Giridhar
 
GIL - Concurrency & Parallelism in Python
GIL - Concurrency & Parallelism in PythonGIL - Concurrency & Parallelism in Python
GIL - Concurrency & Parallelism in PythonPiyus Gupta
 
Understanding how concurrency work in os
Understanding how concurrency work in osUnderstanding how concurrency work in os
Understanding how concurrency work in osGenchiLu1
 
Unmanaged Parallelization via P/Invoke
Unmanaged Parallelization via P/InvokeUnmanaged Parallelization via P/Invoke
Unmanaged Parallelization via P/InvokeDmitri Nesteruk
 
Scaling Django with gevent
Scaling Django with geventScaling Django with gevent
Scaling Django with geventMahendra M
 
Python Deserialization Attacks
Python Deserialization AttacksPython Deserialization Attacks
Python Deserialization AttacksNSConclave
 
How To Make Your Component Compliant
How To Make Your Component CompliantHow To Make Your Component Compliant
How To Make Your Component CompliantJan Gregersen
 
开源沙龙第一期 Python intro
开源沙龙第一期 Python intro开源沙龙第一期 Python intro
开源沙龙第一期 Python introfantasy zheng
 
Using SWIG to Control, Prototype, and Debug C Programs with Python
Using SWIG to Control, Prototype, and Debug C Programs with PythonUsing SWIG to Control, Prototype, and Debug C Programs with Python
Using SWIG to Control, Prototype, and Debug C Programs with PythonDavid Beazley (Dabeaz LLC)
 
Beating the (sh** out of the) GIL - Multithreading vs. Multiprocessing
Beating the (sh** out of the) GIL - Multithreading vs. MultiprocessingBeating the (sh** out of the) GIL - Multithreading vs. Multiprocessing
Beating the (sh** out of the) GIL - Multithreading vs. MultiprocessingGuy K. Kloss
 
Python multithreading
Python multithreadingPython multithreading
Python multithreadingJanu Jahnavi
 
Python multithreading
Python multithreadingPython multithreading
Python multithreadingJanu Jahnavi
 
Pyjion - a JIT extension system for CPython
Pyjion - a JIT extension system for CPythonPyjion - a JIT extension system for CPython
Pyjion - a JIT extension system for CPythonAnthony Shaw
 
The Joy of SciPy, Part I
The Joy of SciPy, Part IThe Joy of SciPy, Part I
The Joy of SciPy, Part IDinu Gherman
 
Python internals and how they affect your code - kasra ahmadvand
Python internals and how they affect your code - kasra ahmadvandPython internals and how they affect your code - kasra ahmadvand
Python internals and how they affect your code - kasra ahmadvandirpycon
 
Startup Camp - Git, Python, Django session
Startup Camp - Git, Python, Django sessionStartup Camp - Git, Python, Django session
Startup Camp - Git, Python, Django sessionJuraj Michálek
 
Multiprocessing with python
Multiprocessing with pythonMultiprocessing with python
Multiprocessing with pythonPatrick Vergain
 
Overview of Python - Bsides Detroit 2012
Overview of Python - Bsides Detroit 2012Overview of Python - Bsides Detroit 2012
Overview of Python - Bsides Detroit 2012Tazdrumm3r
 
Python and Pytorch tutorial and walkthrough
Python and Pytorch tutorial and walkthroughPython and Pytorch tutorial and walkthrough
Python and Pytorch tutorial and walkthroughgabriellekuruvilla
 

Ähnlich wie Concurrency in Python (20)

Python For Scientists
Python For ScientistsPython For Scientists
Python For Scientists
 
Pycon11: Python threads: Dive into GIL!
Pycon11: Python threads: Dive into GIL!Pycon11: Python threads: Dive into GIL!
Pycon11: Python threads: Dive into GIL!
 
GIL - Concurrency & Parallelism in Python
GIL - Concurrency & Parallelism in PythonGIL - Concurrency & Parallelism in Python
GIL - Concurrency & Parallelism in Python
 
Understanding how concurrency work in os
Understanding how concurrency work in osUnderstanding how concurrency work in os
Understanding how concurrency work in os
 
Unmanaged Parallelization via P/Invoke
Unmanaged Parallelization via P/InvokeUnmanaged Parallelization via P/Invoke
Unmanaged Parallelization via P/Invoke
 
Scaling Django with gevent
Scaling Django with geventScaling Django with gevent
Scaling Django with gevent
 
Python Deserialization Attacks
Python Deserialization AttacksPython Deserialization Attacks
Python Deserialization Attacks
 
How To Make Your Component Compliant
How To Make Your Component CompliantHow To Make Your Component Compliant
How To Make Your Component Compliant
 
开源沙龙第一期 Python intro
开源沙龙第一期 Python intro开源沙龙第一期 Python intro
开源沙龙第一期 Python intro
 
Using SWIG to Control, Prototype, and Debug C Programs with Python
Using SWIG to Control, Prototype, and Debug C Programs with PythonUsing SWIG to Control, Prototype, and Debug C Programs with Python
Using SWIG to Control, Prototype, and Debug C Programs with Python
 
Beating the (sh** out of the) GIL - Multithreading vs. Multiprocessing
Beating the (sh** out of the) GIL - Multithreading vs. MultiprocessingBeating the (sh** out of the) GIL - Multithreading vs. Multiprocessing
Beating the (sh** out of the) GIL - Multithreading vs. Multiprocessing
 
Python multithreading
Python multithreadingPython multithreading
Python multithreading
 
Python multithreading
Python multithreadingPython multithreading
Python multithreading
 
Pyjion - a JIT extension system for CPython
Pyjion - a JIT extension system for CPythonPyjion - a JIT extension system for CPython
Pyjion - a JIT extension system for CPython
 
The Joy of SciPy, Part I
The Joy of SciPy, Part IThe Joy of SciPy, Part I
The Joy of SciPy, Part I
 
Python internals and how they affect your code - kasra ahmadvand
Python internals and how they affect your code - kasra ahmadvandPython internals and how they affect your code - kasra ahmadvand
Python internals and how they affect your code - kasra ahmadvand
 
Startup Camp - Git, Python, Django session
Startup Camp - Git, Python, Django sessionStartup Camp - Git, Python, Django session
Startup Camp - Git, Python, Django session
 
Multiprocessing with python
Multiprocessing with pythonMultiprocessing with python
Multiprocessing with python
 
Overview of Python - Bsides Detroit 2012
Overview of Python - Bsides Detroit 2012Overview of Python - Bsides Detroit 2012
Overview of Python - Bsides Detroit 2012
 
Python and Pytorch tutorial and walkthrough
Python and Pytorch tutorial and walkthroughPython and Pytorch tutorial and walkthrough
Python and Pytorch tutorial and walkthrough
 

Mehr von konryd

Za co nie lubi Cię Twój hoster
Za co nie lubi Cię Twój hosterZa co nie lubi Cię Twój hoster
Za co nie lubi Cię Twój hosterkonryd
 
Objective C
Objective CObjective C
Objective Ckonryd
 
Bazaar Mercurial
Bazaar MercurialBazaar Mercurial
Bazaar Mercurialkonryd
 
Wstęp do Subversion
Wstęp do SubversionWstęp do Subversion
Wstęp do Subversionkonryd
 
Haskell
HaskellHaskell
Haskellkonryd
 
Erlang
ErlangErlang
Erlangkonryd
 
Wstęp do Ruby\'ego
Wstęp do Ruby\'egoWstęp do Ruby\'ego
Wstęp do Ruby\'egokonryd
 
Wprowadzenie do Pythona
Wprowadzenie do PythonaWprowadzenie do Pythona
Wprowadzenie do Pythonakonryd
 

Mehr von konryd (9)

Za co nie lubi Cię Twój hoster
Za co nie lubi Cię Twój hosterZa co nie lubi Cię Twój hoster
Za co nie lubi Cię Twój hoster
 
Objective C
Objective CObjective C
Objective C
 
Bazaar Mercurial
Bazaar MercurialBazaar Mercurial
Bazaar Mercurial
 
Wstęp do Subversion
Wstęp do SubversionWstęp do Subversion
Wstęp do Subversion
 
Scala
ScalaScala
Scala
 
Haskell
HaskellHaskell
Haskell
 
Erlang
ErlangErlang
Erlang
 
Wstęp do Ruby\'ego
Wstęp do Ruby\'egoWstęp do Ruby\'ego
Wstęp do Ruby\'ego
 
Wprowadzenie do Pythona
Wprowadzenie do PythonaWprowadzenie do Pythona
Wprowadzenie do Pythona
 

Concurrency in Python

  • 1. Intro Threads GIL Processes Async Concurrency in Python Konrad Delong March 23 2010, Amsterdam Konrad Delong Concurrency in Python
  • 2. Intro Threads GIL Processes Async me IT student (Erasmus) Python Thesis Konrad Delong Concurrency in Python
  • 3. Intro Threads GIL Processes Async concurrency In computer science, concurrency is a property of systems in which several computations are executing simultaneously, and potentially interacting with each other. The computations may be executing on multiple cores in the same chip, preemptively time-shared threads on the same processor, or executed on physically separated processors. (en.wikipedia.org) Konrad Delong Concurrency in Python
  • 4. Intro Threads GIL Processes Async threads thread threading Konrad Delong Concurrency in Python
  • 5. Intro Threads GIL Processes Async thread import thread thread.start_new_thread(function, args[, kwargs]) thread.get_ident() thread.allocate_lock() Konrad Delong Concurrency in Python
  • 6. Intro Threads GIL Processes Async threading from threading import (Thread, Condition, Lock, RLock, Semaphore) class MyThread(Thread): def run(self) pass t = MyThread() # or t = Thread(target=function, args=[], kwargs={}) # and then t.start() t.join() Konrad Delong Concurrency in Python
  • 7. Intro Threads GIL Processes Async GIL Konrad Delong Concurrency in Python
  • 8. Intro Threads GIL Processes Async How it works? Interpreter in only one thread Released every check (== 100 ticks) and on blocking operations Konrad Delong Concurrency in Python
  • 9. Intro Threads GIL Processes Async check if (--_Py_Ticker < 0) { _Py_Ticker = _Py_CheckInterval; // default 100 if (interpreter_lock) { /* Give another thread a chance */ PyThread_release_lock(interpreter_lock); /* Other threads may run now */ PyThread_acquire_lock(interpreter_lock, 1); } } Konrad Delong Concurrency in Python
  • 10. Intro Threads GIL Processes Async Why GIL is bad? No gains on multicore Latency in responding to events Konrad Delong Concurrency in Python
  • 11. Intro Threads GIL Processes Async Why GIL is not that bad? Performance on single core Safe environment for C extentions Other solutions than threads exist Konrad Delong Concurrency in Python
  • 12. Intro Threads GIL Processes Async Remove GIL? Greg Stein, python-safethread PyPy, Unladen Swallow current policy on dropping GIL in cPython Konrad Delong Concurrency in Python
  • 13. Intro Threads GIL Processes Async Other solutions? Processes instead of threads Jython/IronPython Async Konrad Delong Concurrency in Python
  • 14. Intro Threads GIL Processes Async multiprocessing module from multiprocessing import (Process, Condition, Lock, RLock, Semaphore) class MyProcess(Process): def run(self) pass t = MyProcess() # or t = Process(target=function, args=[], kwargs={}) # and then t.start() t.join() Konrad Delong Concurrency in Python
  • 15. Intro Threads GIL Processes Async just like ... from threading import (Thread, Condition, Lock, RLock, Semaphore) class MyThread(Thread): def run(self) pass t = MyThread() # or t = Thread(target=function, args=[], kwargs={}) # and then t.start() t.join() Konrad Delong Concurrency in Python
  • 16. Intro Threads GIL Processes Async not entirely the same no implicit shared memory (global vars, imported modules, other) IPC (serializable data) separate machines IPC Konrad Delong Concurrency in Python
  • 17. Intro Threads GIL Processes Async Queues def foo(queue): print queue.get() q = Queue() t = Process(target=foo, args=[q]) t.start() q.put([1, "a"]) # will print [1, "a"] Konrad Delong Concurrency in Python
  • 18. Intro Threads GIL Processes Async Shared memory from multiprocessing import Value, Array num = Value(’d’, 0.0) arr = Array(’i’, range(10)) Konrad Delong Concurrency in Python
  • 19. Intro Threads GIL Processes Async Async callbacks no blocking still single CPU, but better scaling (memory management) explicit(Twisted, Tornado) vs hidden Konrad Delong Concurrency in Python
  • 20. Intro Threads GIL Processes Async Reactor Konrad Delong Concurrency in Python
  • 21. Intro Threads GIL Processes Async Hidden async Kamaelia select, epoll, kqueue Eventlet, Gevent, PyEvent Konrad Delong Concurrency in Python