SlideShare ist ein Scribd-Unternehmen logo
1 von 20
Downloaden Sie, um offline zu lesen
About me :)
●   Computer Programmer,
●   Coding in Python for last 3 years,
●   Part of the Team at HP that developed an early
    warning software that parses over 40+TB of
    data annually to find problems before they
    happen, (Coded in Python)
●   Skills in Django, PyQt,
●   Http://uptosomething.in [ Homepage ]
Python Performance 101

   Performance : Why it matters ?

   Performance : Measurement

   Performance : Low hanging fruits

   Python Performance : Interpreter

   Python Performance : Outsourcing to C/C++
Performance : Measurement



Reading cProfile output
Ncalls : for the number of calls,
Tottime : for the total time spent in the given function (and excluding time
made in calls to sub-functions),
Percall : is the quotient of tottime divided by ncalls
Cumtime : is the total time spent in this and all subfunctions (from invocation
till exit). This figure is accurate even for recursive functions.
Percall : is the quotient of cumtime divided by primitive calls
filename:lineno(function) : provides the respective data of each function
Performance : Measurement
$ apt-get install python graphviz
$ sudo apt-get install python graphviz
$ wget http://gprof2dot.jrfonseca.googlecode.com/git/gprof2dot.py
$ python -m cProfile -o out.pstats ex21.py
$ python gprof2dot.py -f pstats out.pstats | dot -Tpng -o output.png
$ gimp output.png
Performance : Measurement
RunPythonRun : http://www.vrplumber.com/programming/runsnakerun/
Python Performance : Low Hanging Fruits
●   String concatenation Benchmark ( http://sprocket.io/blog/2007/10/string-concatenation-performance-in-
    python/ )
    add: a + b + c + d

    add equals: a += b; a += c; a += d

    format strings: ‘%s%s%s%s’ % (a, b, c, d)

    named format strings:‘%(a)s%(b)s%(c)s%(d)s’ % {‘a’: a, ‘b’: b, ‘c’: c, ‘d’: d}”

    join: ”.join([a,b,c,d])

    #!/usr/bin/python
    # benchmark various string concatenation methods. Run each 5*1,000,000 times
    # and pick the best time out of the 5. Repeats for string lengths of
    # 4, 16, 64, 256, 1024, and 4096. Outputs in CSV format via stdout.
    import timeit
     
    tests = {
      'add': "x = a + b + c + d",
      'join': "x = ''.join([a,b,c,d])",
      'addequals': "x = a; x += b; x += c; x += d",
      'format': "x = '%s%s%s%s' % (a, b, c, d)",
      'full_format': "x = '%(a)s%(b)s%(c)s%(d)s' % {'a': a, 'b': b, 'c': c, 'd': d}"
    }
     
    count = 1
    for i in range(6):
      count = count * 4
      init = "a = '%s'; b = '%s'; c = '%s'; d = '%s'" % 
             ('a' * count, 'b' * count, 'c' * count, 'd' * count)
     
      for test in tests:
        t = timeit.Timer(tests[test], init)
        best = min(t.repeat(5, 1000000))
        print "'%s',%s,%s" % (test, count, best)
Python Performance : Low Hanging Fruits




Simple addition is the fastest string concatenation for small strings, followed by add equals.

”.join() is the fastest string concatenation for large strings.

* named format is always the worst performer.

* using string formatting for joins is equally as good as add equals for large strings, but for small strings it’s mediocre.
Python Performance : Low Hanging Fruits
newlist = []                                                                newlist = map(str.upper, oldlist)
for word in oldlist:
    newlist.append(word.upper())



                                   newlist = [s.upper() for s in oldlist]




upper = str.upper
newlist = []
append = newlist.append
                                                                              I wouldn't do this
for word in oldlist:
    append(upper(word))




                              Exception for branching
                                                                              wdict = {}
wdict = {}
                                                                              for word in words:
for word in words:
                                                                                  try:
    if word not in wdict:
                                                                                      wdict[word] += 1
        wdict[word] = 0
                                                                                  except KeyError:
    wdict[word] += 1
                                                                                      wdict[word] = 1
Python Performance : Low Hanging Fruits

Function call overhead

  import time                                                           import time
  x = 0                                                                 x = 0
  def doit1(i):                                                         def doit2(list):
      global x                                                              global x
      x = x + i                                                             for i in list:
  list = range(100000)                                                          x = x + i
  t = time.time()                                                       list = range(100000)
  for i in list:                                                        t = time.time()
      doit1(i)                                                          doit2(list)
  print "%.3f" % (time.time()-t)                                        print "%.3f" % (time.time()-t)




                                   >>> t = time.time()
                                   >>> for i in list:
                                   ... doit1(i)
                                   ...
                                   >>> print "%.3f" % (time.time()-t)
                                   0.758
                                   >>> t = time.time()
                                   >>> doit2(list)
                                   >>> print "%.3f" % (time.time()-t)
                                   0.204
Python Performance : Low Hanging Fruits

Xrange vs range

Membership testing with sets and dictionaries is much faster, O(1), than searching
sequences, O(n).
When testing "a in b", b should be a set or dictionary instead of a list or tuple.

Lists perform well as either fixed length arrays or variable length stacks. However, for queue
applications using pop(0) or insert(0,v)), collections.deque() offers superior O(1)
performance because it avoids the O(n) step of rebuilding a full list for each insertion or
deletion.

In functions, local variables are accessed more quickly than global variables, builtins, and
attribute lookups. So, it is sometimes worth localizing variable access in inner-loops.

http://wiki.python.org/moin/PythonSpeed

http://wiki.python.org/moin/PythonSpeed/PerformanceTips
Python : Multi-core Architecture
●   In CPython, the global interpreter lock, or GIL, is a mutex that prevents multiple native threads from
    executing Python bytecodes at once. This lock is necessary mainly because CPython's memory
    management is not thread-safe. (However, since the GIL exists, other features have grown to depend on the
    guarantees that it enforces.) More here http://wiki.python.org/moin/GlobalInterpreterLock
●   Use Multi Processing to overcome GIL

    from multiprocessing import Process, Queue

    def f(iq,oq):
     if not iq.empty():
       values = iq.get()
       oq.put(sum(values))

    if __name__ == '__main__':
      inputQueue = Queue()
      outputQueue = Queue()
      values = range(0,1000000)
      processOne = Process(target=f, args=(inputQueue,outputQueue))
      processTwo = Process(target=f, args=(inputQueue,outputQueue))
      inputQueue.put(values[0:len(values)/2])
      inputQueue.put(values[len(values)/2:])
      processOne.start()
      processTwo.start()
      processOne.join()
      processTwo.join()
      outputOne = outputQueue.get()
      outputTwo = outputQueue.get()
      print sum([outputOne, outputTwo])
Python : Multi-core Architecture
●   IPL encapsulated. Queue, Pipe, Lock.
●   Use logging module to log multiprocess i.e. SocketHandler,
●   Good practise is to have maximum 2 * No of cores processes
    spawned.
●   Debugging is a little painful as cProfile has to be attached to each
    process and then you dump the stats output of it and one can join
    them all. Still a little painful.
Python : Interpreter
CPython - the default install everyone uses
Jython - Python on the JVM, currently targets Python 2.5, true
concurrency, strong JVM integration. About even with CPython speed-
wise, maybe a bit slower.
IronPython - Python on the CLR, currently targets 2.6, with a 2.7 pre-
release available, true concurrency, good CLR integration. Speed
comparison with CPython varies greatly depending on which feature you're
looking at.
PyPy - Python on RPython (a static subset of python), currently targets
2.5, with a branch targeting 2.7, has a GIL, and a JIT, which can result in
huge performance gains (see http://speed.pypy.org/).
Unladen Swallow - a branch of CPython utilizing LLVM to do just in time
compilation. Branched from 2.6, although with the acceptance of PEP
3146 it is slated for merger into py3k.
Source: Alex Gaynor @ Quora
Python : Interpreter

PyPy
Http://pypy.org
PyPy is a fast, compliant alternative implementation of the Python language (2.7.1). It has several
advantages and distinct features:
Speed: thanks to its Just-in-Time compiler, Python programs often run faster on PyPy.
(What is a JIT compiler?)
Memory usage: large, memory-hungry Python programs might end up taking less space than they
do in CPython.
Compatibility: PyPy is highly compatible with existing python code. It supports ctypes and can run
popular python libraries like twisted and django.
Sandboxing: PyPy provides the ability to run untrusted code in a fully secure way.
Stackless: PyPy can be configured to run in stackless mode, providing micro-threads for massive
concurrency.
Source : http://pypy.org
Python : Interpreter
●   Unladen swallow
    An optimization branch of CPython, intended to
    be fully compatible and significantly faster.
    http://code.google.com/p/unladen-swallow/
●   Mandate is to merge the codebase with Python
    3.x series.
●   It's a google sponsered project.
●   Know to be used @ Youtube which is in Python.
Python : Interpreter Benchmarks




Source: http://morepypy.blogspot.com/2009/11/some-benchmarking.html
Python : Interpreter Benchmarks




Source: http://morepypy.blogspot.com/2009/11/some-benchmarking.html
Python Performance 101
Python : Outsourcing to C/C++
●   Ctypes
●   SWIG
Python : Outsourcing to C/C++
●    $ sudo apt-get install libboost-python-dev
●    $ sudo apt-get install python-dev
●    $ sudo apt-get install swig
    /*hellomodule.c*/

    #include <stdio.h>

    void say_hello(const char* name) {
        printf("Hello %s!n", name);
    }
    /*hello.i*/

    %module hello
    extern void say_hello(const char* name);




    $ swig -python hello.i
    $ gcc -fpic -c hellomodule.c hello_wrap.c -I/usr/include/python2.7/
    $ gcc -shared hellomodule.o hello_wrap.o -o _hello.so

    >>> import hello
    >>> hello.say_hello("World")
    Hello World!

Más contenido relacionado

Was ist angesagt?

ITT 2015 - Saul Mora - Object Oriented Function Programming
ITT 2015 - Saul Mora - Object Oriented Function ProgrammingITT 2015 - Saul Mora - Object Oriented Function Programming
ITT 2015 - Saul Mora - Object Oriented Function ProgrammingIstanbul Tech Talks
 
Python profiling
Python profilingPython profiling
Python profilingdreampuf
 
Scala - where objects and functions meet
Scala - where objects and functions meetScala - where objects and functions meet
Scala - where objects and functions meetMario Fusco
 
Python and sysadmin I
Python and sysadmin IPython and sysadmin I
Python and sysadmin IGuixing Bai
 
Python fundamentals - basic | WeiYuan
Python fundamentals - basic | WeiYuanPython fundamentals - basic | WeiYuan
Python fundamentals - basic | WeiYuanWei-Yuan Chang
 
The best language in the world
The best language in the worldThe best language in the world
The best language in the worldDavid Muñoz Díaz
 
Python легко и просто. Красиво решаем повседневные задачи
Python легко и просто. Красиво решаем повседневные задачиPython легко и просто. Красиво решаем повседневные задачи
Python легко и просто. Красиво решаем повседневные задачиMaxim Kulsha
 
Python高级编程(二)
Python高级编程(二)Python高级编程(二)
Python高级编程(二)Qiangning Hong
 
Programming Under Linux In Python
Programming Under Linux In PythonProgramming Under Linux In Python
Programming Under Linux In PythonMarwan Osman
 
Are we ready to Go?
Are we ready to Go?Are we ready to Go?
Are we ready to Go?Adam Dudczak
 
Design Patterns - Compiler Case Study - Hands-on Examples
Design Patterns - Compiler Case Study - Hands-on ExamplesDesign Patterns - Compiler Case Study - Hands-on Examples
Design Patterns - Compiler Case Study - Hands-on ExamplesGanesh Samarthyam
 
Tuga it 2016 - What's New In C# 6
Tuga it 2016 - What's New In C# 6Tuga it 2016 - What's New In C# 6
Tuga it 2016 - What's New In C# 6Paulo Morgado
 
T3chFest 2016 - The polyglot programmer
T3chFest 2016 - The polyglot programmerT3chFest 2016 - The polyglot programmer
T3chFest 2016 - The polyglot programmerDavid Muñoz Díaz
 
What's new in C# 6 - NetPonto Porto 20160116
What's new in C# 6  - NetPonto Porto 20160116What's new in C# 6  - NetPonto Porto 20160116
What's new in C# 6 - NetPonto Porto 20160116Paulo Morgado
 
Implementing virtual machines in go & c 2018 redux
Implementing virtual machines in go & c 2018 reduxImplementing virtual machines in go & c 2018 redux
Implementing virtual machines in go & c 2018 reduxEleanor McHugh
 
«Отладка в Python 3.6: Быстрее, Выше, Сильнее» Елизавета Шашкова, JetBrains
«Отладка в Python 3.6: Быстрее, Выше, Сильнее» Елизавета Шашкова, JetBrains«Отладка в Python 3.6: Быстрее, Выше, Сильнее» Елизавета Шашкова, JetBrains
«Отладка в Python 3.6: Быстрее, Выше, Сильнее» Елизавета Шашкова, JetBrainsit-people
 
[Let'Swift 2019] 실용적인 함수형 프로그래밍 워크샵
[Let'Swift 2019] 실용적인 함수형 프로그래밍 워크샵[Let'Swift 2019] 실용적인 함수형 프로그래밍 워크샵
[Let'Swift 2019] 실용적인 함수형 프로그래밍 워크샵Wanbok Choi
 

Was ist angesagt? (20)

ITT 2015 - Saul Mora - Object Oriented Function Programming
ITT 2015 - Saul Mora - Object Oriented Function ProgrammingITT 2015 - Saul Mora - Object Oriented Function Programming
ITT 2015 - Saul Mora - Object Oriented Function Programming
 
Python profiling
Python profilingPython profiling
Python profiling
 
Scala - where objects and functions meet
Scala - where objects and functions meetScala - where objects and functions meet
Scala - where objects and functions meet
 
Python and sysadmin I
Python and sysadmin IPython and sysadmin I
Python and sysadmin I
 
Why Learn Python?
Why Learn Python?Why Learn Python?
Why Learn Python?
 
Python fundamentals - basic | WeiYuan
Python fundamentals - basic | WeiYuanPython fundamentals - basic | WeiYuan
Python fundamentals - basic | WeiYuan
 
The best language in the world
The best language in the worldThe best language in the world
The best language in the world
 
Python легко и просто. Красиво решаем повседневные задачи
Python легко и просто. Красиво решаем повседневные задачиPython легко и просто. Красиво решаем повседневные задачи
Python легко и просто. Красиво решаем повседневные задачи
 
Python高级编程(二)
Python高级编程(二)Python高级编程(二)
Python高级编程(二)
 
Programming Under Linux In Python
Programming Under Linux In PythonProgramming Under Linux In Python
Programming Under Linux In Python
 
Are we ready to Go?
Are we ready to Go?Are we ready to Go?
Are we ready to Go?
 
Design Patterns - Compiler Case Study - Hands-on Examples
Design Patterns - Compiler Case Study - Hands-on ExamplesDesign Patterns - Compiler Case Study - Hands-on Examples
Design Patterns - Compiler Case Study - Hands-on Examples
 
Tuga it 2016 - What's New In C# 6
Tuga it 2016 - What's New In C# 6Tuga it 2016 - What's New In C# 6
Tuga it 2016 - What's New In C# 6
 
T3chFest 2016 - The polyglot programmer
T3chFest 2016 - The polyglot programmerT3chFest 2016 - The polyglot programmer
T3chFest 2016 - The polyglot programmer
 
What's new in C# 6 - NetPonto Porto 20160116
What's new in C# 6  - NetPonto Porto 20160116What's new in C# 6  - NetPonto Porto 20160116
What's new in C# 6 - NetPonto Porto 20160116
 
Implementing virtual machines in go & c 2018 redux
Implementing virtual machines in go & c 2018 reduxImplementing virtual machines in go & c 2018 redux
Implementing virtual machines in go & c 2018 redux
 
Java VS Python
Java VS PythonJava VS Python
Java VS Python
 
«Отладка в Python 3.6: Быстрее, Выше, Сильнее» Елизавета Шашкова, JetBrains
«Отладка в Python 3.6: Быстрее, Выше, Сильнее» Елизавета Шашкова, JetBrains«Отладка в Python 3.6: Быстрее, Выше, Сильнее» Елизавета Шашкова, JetBrains
«Отладка в Python 3.6: Быстрее, Выше, Сильнее» Елизавета Шашкова, JetBrains
 
Python tour
Python tourPython tour
Python tour
 
[Let'Swift 2019] 실용적인 함수형 프로그래밍 워크샵
[Let'Swift 2019] 실용적인 함수형 프로그래밍 워크샵[Let'Swift 2019] 실용적인 함수형 프로그래밍 워크샵
[Let'Swift 2019] 실용적인 함수형 프로그래밍 워크샵
 

Andere mochten auch

Machine Learning for NLP
Machine Learning for NLPMachine Learning for NLP
Machine Learning for NLPbutest
 
Inside the ANN: A visual and intuitive journey to understand how artificial n...
Inside the ANN: A visual and intuitive journey to understand how artificial n...Inside the ANN: A visual and intuitive journey to understand how artificial n...
Inside the ANN: A visual and intuitive journey to understand how artificial n...XavierArrufat
 
Redis And python at pycon_2011
Redis And python at pycon_2011Redis And python at pycon_2011
Redis And python at pycon_2011sunilar0ra
 
Natural Language Processing and Machine Learning for Discovery
Natural Language Processing and Machine Learning for DiscoveryNatural Language Processing and Machine Learning for Discovery
Natural Language Processing and Machine Learning for Discoverymjbommar
 
NLTK - Natural Language Processing in Python
NLTK - Natural Language Processing in PythonNLTK - Natural Language Processing in Python
NLTK - Natural Language Processing in Pythonshanbady
 
Natural language processing
Natural language processingNatural language processing
Natural language processingYogendra Tamang
 
Introduction to Natural Language Processing
Introduction to Natural Language ProcessingIntroduction to Natural Language Processing
Introduction to Natural Language ProcessingPranav Gupta
 
Python Coroutines, Present and Future
Python Coroutines, Present and FuturePython Coroutines, Present and Future
Python Coroutines, Present and Futureemptysquare
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to RedisDvir Volk
 
Slideshare Powerpoint presentation
Slideshare Powerpoint presentationSlideshare Powerpoint presentation
Slideshare Powerpoint presentationelliehood
 
State of the Word 2011
State of the Word 2011State of the Word 2011
State of the Word 2011photomatt
 

Andere mochten auch (13)

Machine Learning for NLP
Machine Learning for NLPMachine Learning for NLP
Machine Learning for NLP
 
Inside the ANN: A visual and intuitive journey to understand how artificial n...
Inside the ANN: A visual and intuitive journey to understand how artificial n...Inside the ANN: A visual and intuitive journey to understand how artificial n...
Inside the ANN: A visual and intuitive journey to understand how artificial n...
 
Redis And python at pycon_2011
Redis And python at pycon_2011Redis And python at pycon_2011
Redis And python at pycon_2011
 
Natural Language Processing and Machine Learning for Discovery
Natural Language Processing and Machine Learning for DiscoveryNatural Language Processing and Machine Learning for Discovery
Natural Language Processing and Machine Learning for Discovery
 
NLTK - Natural Language Processing in Python
NLTK - Natural Language Processing in PythonNLTK - Natural Language Processing in Python
NLTK - Natural Language Processing in Python
 
Natural language processing
Natural language processingNatural language processing
Natural language processing
 
NLP
NLPNLP
NLP
 
Introduction to Natural Language Processing
Introduction to Natural Language ProcessingIntroduction to Natural Language Processing
Introduction to Natural Language Processing
 
Natural Language Processing
Natural Language ProcessingNatural Language Processing
Natural Language Processing
 
Python Coroutines, Present and Future
Python Coroutines, Present and FuturePython Coroutines, Present and Future
Python Coroutines, Present and Future
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
 
Slideshare Powerpoint presentation
Slideshare Powerpoint presentationSlideshare Powerpoint presentation
Slideshare Powerpoint presentation
 
State of the Word 2011
State of the Word 2011State of the Word 2011
State of the Word 2011
 

Ähnlich wie Python Performance 101

Pydiomatic
PydiomaticPydiomatic
Pydiomaticrik0
 
Swift for tensorflow
Swift for tensorflowSwift for tensorflow
Swift for tensorflow규영 허
 
GE8151 Problem Solving and Python Programming
GE8151 Problem Solving and Python ProgrammingGE8151 Problem Solving and Python Programming
GE8151 Problem Solving and Python ProgrammingMuthu Vinayagam
 
sonam Kumari python.ppt
sonam Kumari python.pptsonam Kumari python.ppt
sonam Kumari python.pptssuserd64918
 
Porque aprender haskell me fez um programador python melhor?
Porque aprender haskell me fez um programador python melhor?Porque aprender haskell me fez um programador python melhor?
Porque aprender haskell me fez um programador python melhor?UFPA
 
The Ring programming language version 1.5.3 book - Part 25 of 184
The Ring programming language version 1.5.3 book - Part 25 of 184The Ring programming language version 1.5.3 book - Part 25 of 184
The Ring programming language version 1.5.3 book - Part 25 of 184Mahmoud Samir Fayed
 
Real Time Big Data Management
Real Time Big Data ManagementReal Time Big Data Management
Real Time Big Data ManagementAlbert Bifet
 
Introduction to Python 01-08-2023.pon by everyone else. . Hence, they must be...
Introduction to Python 01-08-2023.pon by everyone else. . Hence, they must be...Introduction to Python 01-08-2023.pon by everyone else. . Hence, they must be...
Introduction to Python 01-08-2023.pon by everyone else. . Hence, they must be...DRVaibhavmeshram1
 
Python Workshop - Learn Python the Hard Way
Python Workshop - Learn Python the Hard WayPython Workshop - Learn Python the Hard Way
Python Workshop - Learn Python the Hard WayUtkarsh Sengar
 
Advanced Web Technology ass.pdf
Advanced Web Technology ass.pdfAdvanced Web Technology ass.pdf
Advanced Web Technology ass.pdfsimenehanmut
 
Python Interview Questions | Python Interview Questions And Answers | Python ...
Python Interview Questions | Python Interview Questions And Answers | Python ...Python Interview Questions | Python Interview Questions And Answers | Python ...
Python Interview Questions | Python Interview Questions And Answers | Python ...Simplilearn
 
Python-GTK
Python-GTKPython-GTK
Python-GTKYuren Ju
 
Clojure Intro
Clojure IntroClojure Intro
Clojure Introthnetos
 
Python 培训讲义
Python 培训讲义Python 培训讲义
Python 培训讲义leejd
 
Python GTK (Hacking Camp)
Python GTK (Hacking Camp)Python GTK (Hacking Camp)
Python GTK (Hacking Camp)Yuren Ju
 
Introduction to python
Introduction to pythonIntroduction to python
Introduction to pythonMarian Marinov
 

Ähnlich wie Python Performance 101 (20)

Pydiomatic
PydiomaticPydiomatic
Pydiomatic
 
Python idiomatico
Python idiomaticoPython idiomatico
Python idiomatico
 
Swift for tensorflow
Swift for tensorflowSwift for tensorflow
Swift for tensorflow
 
Introduction to python
Introduction to pythonIntroduction to python
Introduction to python
 
GE8151 Problem Solving and Python Programming
GE8151 Problem Solving and Python ProgrammingGE8151 Problem Solving and Python Programming
GE8151 Problem Solving and Python Programming
 
Biopython: Overview, State of the Art and Outlook
Biopython: Overview, State of the Art and OutlookBiopython: Overview, State of the Art and Outlook
Biopython: Overview, State of the Art and Outlook
 
sonam Kumari python.ppt
sonam Kumari python.pptsonam Kumari python.ppt
sonam Kumari python.ppt
 
Porque aprender haskell me fez um programador python melhor?
Porque aprender haskell me fez um programador python melhor?Porque aprender haskell me fez um programador python melhor?
Porque aprender haskell me fez um programador python melhor?
 
The Ring programming language version 1.5.3 book - Part 25 of 184
The Ring programming language version 1.5.3 book - Part 25 of 184The Ring programming language version 1.5.3 book - Part 25 of 184
The Ring programming language version 1.5.3 book - Part 25 of 184
 
Real Time Big Data Management
Real Time Big Data ManagementReal Time Big Data Management
Real Time Big Data Management
 
Introduction to Python 01-08-2023.pon by everyone else. . Hence, they must be...
Introduction to Python 01-08-2023.pon by everyone else. . Hence, they must be...Introduction to Python 01-08-2023.pon by everyone else. . Hence, they must be...
Introduction to Python 01-08-2023.pon by everyone else. . Hence, they must be...
 
Python Workshop - Learn Python the Hard Way
Python Workshop - Learn Python the Hard WayPython Workshop - Learn Python the Hard Way
Python Workshop - Learn Python the Hard Way
 
Advanced Web Technology ass.pdf
Advanced Web Technology ass.pdfAdvanced Web Technology ass.pdf
Advanced Web Technology ass.pdf
 
Python Basics
Python BasicsPython Basics
Python Basics
 
Python Interview Questions | Python Interview Questions And Answers | Python ...
Python Interview Questions | Python Interview Questions And Answers | Python ...Python Interview Questions | Python Interview Questions And Answers | Python ...
Python Interview Questions | Python Interview Questions And Answers | Python ...
 
Python-GTK
Python-GTKPython-GTK
Python-GTK
 
Clojure Intro
Clojure IntroClojure Intro
Clojure Intro
 
Python 培训讲义
Python 培训讲义Python 培训讲义
Python 培训讲义
 
Python GTK (Hacking Camp)
Python GTK (Hacking Camp)Python GTK (Hacking Camp)
Python GTK (Hacking Camp)
 
Introduction to python
Introduction to pythonIntroduction to python
Introduction to python
 

Último

Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Alkin Tezuysal
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3DianaGray10
 
How to release an Open Source Dataweave Library
How to release an Open Source Dataweave LibraryHow to release an Open Source Dataweave Library
How to release an Open Source Dataweave Libraryshyamraj55
 
The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)IES VE
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationKnoldus Inc.
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTopCSSGallery
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdfThe Good Food Institute
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applicationsnooralam814309
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud DataEric D. Schabell
 
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingFrancesco Corti
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveIES VE
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNeo4j
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxNeo4j
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and businessFrancesco Corti
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameKapil Thakar
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FESTBillieHyde
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0DanBrown980551
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...DianaGray10
 

Último (20)

Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
 
SheDev 2024
SheDev 2024SheDev 2024
SheDev 2024
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3
 
How to release an Open Source Dataweave Library
How to release an Open Source Dataweave LibraryHow to release an Open Source Dataweave Library
How to release an Open Source Dataweave Library
 
The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its application
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development Companies
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applications
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data
 
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is going
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4j
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and business
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First Frame
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FEST
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...
 

Python Performance 101

  • 1. About me :) ● Computer Programmer, ● Coding in Python for last 3 years, ● Part of the Team at HP that developed an early warning software that parses over 40+TB of data annually to find problems before they happen, (Coded in Python) ● Skills in Django, PyQt, ● Http://uptosomething.in [ Homepage ]
  • 2. Python Performance 101  Performance : Why it matters ?  Performance : Measurement  Performance : Low hanging fruits  Python Performance : Interpreter  Python Performance : Outsourcing to C/C++
  • 3. Performance : Measurement Reading cProfile output Ncalls : for the number of calls, Tottime : for the total time spent in the given function (and excluding time made in calls to sub-functions), Percall : is the quotient of tottime divided by ncalls Cumtime : is the total time spent in this and all subfunctions (from invocation till exit). This figure is accurate even for recursive functions. Percall : is the quotient of cumtime divided by primitive calls filename:lineno(function) : provides the respective data of each function
  • 4. Performance : Measurement $ apt-get install python graphviz $ sudo apt-get install python graphviz $ wget http://gprof2dot.jrfonseca.googlecode.com/git/gprof2dot.py $ python -m cProfile -o out.pstats ex21.py $ python gprof2dot.py -f pstats out.pstats | dot -Tpng -o output.png $ gimp output.png
  • 5. Performance : Measurement RunPythonRun : http://www.vrplumber.com/programming/runsnakerun/
  • 6. Python Performance : Low Hanging Fruits ● String concatenation Benchmark ( http://sprocket.io/blog/2007/10/string-concatenation-performance-in- python/ ) add: a + b + c + d add equals: a += b; a += c; a += d format strings: ‘%s%s%s%s’ % (a, b, c, d) named format strings:‘%(a)s%(b)s%(c)s%(d)s’ % {‘a’: a, ‘b’: b, ‘c’: c, ‘d’: d}” join: ”.join([a,b,c,d]) #!/usr/bin/python # benchmark various string concatenation methods. Run each 5*1,000,000 times # and pick the best time out of the 5. Repeats for string lengths of # 4, 16, 64, 256, 1024, and 4096. Outputs in CSV format via stdout. import timeit   tests = { 'add': "x = a + b + c + d", 'join': "x = ''.join([a,b,c,d])", 'addequals': "x = a; x += b; x += c; x += d", 'format': "x = '%s%s%s%s' % (a, b, c, d)", 'full_format': "x = '%(a)s%(b)s%(c)s%(d)s' % {'a': a, 'b': b, 'c': c, 'd': d}" }   count = 1 for i in range(6): count = count * 4 init = "a = '%s'; b = '%s'; c = '%s'; d = '%s'" % ('a' * count, 'b' * count, 'c' * count, 'd' * count)   for test in tests: t = timeit.Timer(tests[test], init) best = min(t.repeat(5, 1000000)) print "'%s',%s,%s" % (test, count, best)
  • 7. Python Performance : Low Hanging Fruits Simple addition is the fastest string concatenation for small strings, followed by add equals. ”.join() is the fastest string concatenation for large strings. * named format is always the worst performer. * using string formatting for joins is equally as good as add equals for large strings, but for small strings it’s mediocre.
  • 8. Python Performance : Low Hanging Fruits newlist = [] newlist = map(str.upper, oldlist) for word in oldlist: newlist.append(word.upper()) newlist = [s.upper() for s in oldlist] upper = str.upper newlist = [] append = newlist.append I wouldn't do this for word in oldlist: append(upper(word)) Exception for branching wdict = {} wdict = {} for word in words: for word in words: try: if word not in wdict: wdict[word] += 1 wdict[word] = 0 except KeyError: wdict[word] += 1 wdict[word] = 1
  • 9. Python Performance : Low Hanging Fruits Function call overhead import time import time x = 0 x = 0 def doit1(i): def doit2(list): global x global x x = x + i for i in list: list = range(100000) x = x + i t = time.time() list = range(100000) for i in list: t = time.time() doit1(i) doit2(list) print "%.3f" % (time.time()-t) print "%.3f" % (time.time()-t) >>> t = time.time() >>> for i in list: ... doit1(i) ... >>> print "%.3f" % (time.time()-t) 0.758 >>> t = time.time() >>> doit2(list) >>> print "%.3f" % (time.time()-t) 0.204
  • 10. Python Performance : Low Hanging Fruits Xrange vs range Membership testing with sets and dictionaries is much faster, O(1), than searching sequences, O(n). When testing "a in b", b should be a set or dictionary instead of a list or tuple. Lists perform well as either fixed length arrays or variable length stacks. However, for queue applications using pop(0) or insert(0,v)), collections.deque() offers superior O(1) performance because it avoids the O(n) step of rebuilding a full list for each insertion or deletion. In functions, local variables are accessed more quickly than global variables, builtins, and attribute lookups. So, it is sometimes worth localizing variable access in inner-loops. http://wiki.python.org/moin/PythonSpeed http://wiki.python.org/moin/PythonSpeed/PerformanceTips
  • 11. Python : Multi-core Architecture ● In CPython, the global interpreter lock, or GIL, is a mutex that prevents multiple native threads from executing Python bytecodes at once. This lock is necessary mainly because CPython's memory management is not thread-safe. (However, since the GIL exists, other features have grown to depend on the guarantees that it enforces.) More here http://wiki.python.org/moin/GlobalInterpreterLock ● Use Multi Processing to overcome GIL from multiprocessing import Process, Queue def f(iq,oq): if not iq.empty(): values = iq.get() oq.put(sum(values)) if __name__ == '__main__': inputQueue = Queue() outputQueue = Queue() values = range(0,1000000) processOne = Process(target=f, args=(inputQueue,outputQueue)) processTwo = Process(target=f, args=(inputQueue,outputQueue)) inputQueue.put(values[0:len(values)/2]) inputQueue.put(values[len(values)/2:]) processOne.start() processTwo.start() processOne.join() processTwo.join() outputOne = outputQueue.get() outputTwo = outputQueue.get() print sum([outputOne, outputTwo])
  • 12. Python : Multi-core Architecture ● IPL encapsulated. Queue, Pipe, Lock. ● Use logging module to log multiprocess i.e. SocketHandler, ● Good practise is to have maximum 2 * No of cores processes spawned. ● Debugging is a little painful as cProfile has to be attached to each process and then you dump the stats output of it and one can join them all. Still a little painful.
  • 13. Python : Interpreter CPython - the default install everyone uses Jython - Python on the JVM, currently targets Python 2.5, true concurrency, strong JVM integration. About even with CPython speed- wise, maybe a bit slower. IronPython - Python on the CLR, currently targets 2.6, with a 2.7 pre- release available, true concurrency, good CLR integration. Speed comparison with CPython varies greatly depending on which feature you're looking at. PyPy - Python on RPython (a static subset of python), currently targets 2.5, with a branch targeting 2.7, has a GIL, and a JIT, which can result in huge performance gains (see http://speed.pypy.org/). Unladen Swallow - a branch of CPython utilizing LLVM to do just in time compilation. Branched from 2.6, although with the acceptance of PEP 3146 it is slated for merger into py3k. Source: Alex Gaynor @ Quora
  • 14. Python : Interpreter PyPy Http://pypy.org PyPy is a fast, compliant alternative implementation of the Python language (2.7.1). It has several advantages and distinct features: Speed: thanks to its Just-in-Time compiler, Python programs often run faster on PyPy. (What is a JIT compiler?) Memory usage: large, memory-hungry Python programs might end up taking less space than they do in CPython. Compatibility: PyPy is highly compatible with existing python code. It supports ctypes and can run popular python libraries like twisted and django. Sandboxing: PyPy provides the ability to run untrusted code in a fully secure way. Stackless: PyPy can be configured to run in stackless mode, providing micro-threads for massive concurrency. Source : http://pypy.org
  • 15. Python : Interpreter ● Unladen swallow An optimization branch of CPython, intended to be fully compatible and significantly faster. http://code.google.com/p/unladen-swallow/ ● Mandate is to merge the codebase with Python 3.x series. ● It's a google sponsered project. ● Know to be used @ Youtube which is in Python.
  • 16. Python : Interpreter Benchmarks Source: http://morepypy.blogspot.com/2009/11/some-benchmarking.html
  • 17. Python : Interpreter Benchmarks Source: http://morepypy.blogspot.com/2009/11/some-benchmarking.html
  • 19. Python : Outsourcing to C/C++ ● Ctypes ● SWIG
  • 20. Python : Outsourcing to C/C++ ● $ sudo apt-get install libboost-python-dev ● $ sudo apt-get install python-dev ● $ sudo apt-get install swig /*hellomodule.c*/ #include <stdio.h> void say_hello(const char* name) { printf("Hello %s!n", name); } /*hello.i*/ %module hello extern void say_hello(const char* name); $ swig -python hello.i $ gcc -fpic -c hellomodule.c hello_wrap.c -I/usr/include/python2.7/ $ gcc -shared hellomodule.o hello_wrap.o -o _hello.so >>> import hello >>> hello.say_hello("World") Hello World!