SlideShare ist ein Scribd-Unternehmen logo
1 von 43
Downloaden Sie, um offline zu lesen
- Gayatri Nittala
About:
Optimization - what, when & where?
General Optimization strategies
Python specific optimizations
Profiling
Golden Rule:

    "First make it work.
          Then make it right.
               Then make it fast!"
                           - Kent Beck
The process:

 1.Get it right.
     2.Test it's right.
            3.Profile if slow.
                  4.Optimize.
                         5.Repeat from 2.

 test suites
 source control
Make it right & pretty!

 Good Programming :-)

    General optimization strategies
    Python optimizations
Optimization:
Aims to improve
Not perfect, the result
Programming with performance tips
When to start?
Need for optimization
  are you sure you need to do it at all?
  is your code really so bad?
         benchmarking
         fast enough vs. faster


Time for optimization
  is it worth the time to tune it?
  how much time is going to be spent running that
   code?
When to start?
Cost of optimization
   costly developer time
   addition of new features
   new bugs in algorithms
   speed vs. space


            Optimize only if necessary!
Where to start?
 Are you sure you're done coding?
 frosting a half-baked cake
 Premature optimization is the root of all evil!
                                           - Don Knuth
 Working, well-architected code is always a must
General strategies
  Algorithms - the big-O notation
  Architecture
  Choice of Data structures
  LRU techniques
  Loop invariant code out of loops
  Nested loops
  try...catch instead of if...else
  Multithreading for I/O bound code
  DBMS instead of flat files
General strategies
  Big – O – The Boss!

  performance of the algorithms
  a function of N - the input size to the algorithm
    O(1) - constant time
    O(ln n) - logarithmic

    O(n)   - linear
    O(n2) - quadratic
Common big-O’s
Order      Said to be Examples
           “…. time”
--------------------------------------------------
O(1)       constant       key in dict
                          dict[key] = value
                          list.append(item)
O(ln n)    logarithmic Binary search
O(n)       linear         item in sequence
                          str.join(list)
O(n ln n)                 list.sort()
O(n2)      quadratic      Nested loops (with constant time bodies)
Note the notation
  O(N2)                         O(N)
  def slow(it):                 def fast(it):
    result = []                   result = []
    for item in it:               for item in it:
       result.insert(0, item)       result.append(item)
       return result                result.reverse( )
                                  return result
  result = list(it)
Big-O’s of Python Building blocks
   lists - vectors
   dictionaries - hash tables
   sets - hash tables
Big-O’s of Python Building blocks
  Let, L be any list, T any string (plain or Unicode); D
   any dict; S any set, with (say) numbers as items
   (with O(1) hashing and comparison) and x any
   number:

  O(1) - len( L ), len(T), len( D ), len(S), L [i],
           T [i], D[i], del D[i], if x in D, if x in S,
           S .add( x ), S.remove( x ), additions or
           removals to/from the right end of L
Big-O’s of Python Building blocks
  O(N) - Loops on L, T, D, S, general additions or
          removals to/from L (not at the right end),
          all methods on T, if x in L, if x in T,
          most methods on L, all shallow copies

  O(N log N) - L .sort in general (but O(N) if L is
   already nearly sorted or reverse-sorted)
Right Data Structure
   lists, sets, dicts, tuples
   collections - deque, defaultdict, namedtuple
   Choose them based on the functionality
     search an element in a sequence
     append

     intersection

     remove from middle

     dictionary initializations
Right Data Structure
   my_list = range(n)
    n in my_list
   my_list = set(range(n))
    n in my_list

   my_list[start:end] = []
   my_deque.rotate(-end)
    for counter in (end-start):
      my_deque.pop()
Right Data Structure
  s = [('yellow', 1), ('blue', 2), ('yellow', 3), ('blue', 4), ('red', 1)]

   d = defaultdict(list)
    for k, v in s:
       d[k].append(v)
    d.items()
    [('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]

   d = {}
    for k, v in s:
       d.setdefault(k, []).append(v)
    d.items()
    [('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]
Python Performance Tips
   built-in modules
   string concatenation
   lookups and local variables
   dictionary initialization
   dictionary lookups
   import statements
   loops
Built-ins
  - Highly optimized
  - Sort a list of tuples by it’s n-th field


   def sortby(somelist, n):
      nlist = [(x[n], x) for x in somelist]
      nlist.sort()
      return [val for (key, val) in nlist]
  n = 1
   import operator
   nlist.sort(key=operator.itemgetter(n))
String Concatenation
   s = ""
     for substring in list:
         s += substring
   s = "".join(list)
   out = "<html>" + head + prologue + query + tail +
   "</html>"
   out = "<html>%s%s%s%s</html>" % (head,
   prologue, query, tail)
   out = "<html>%(head)s%(prologue)s%(query)s%
   (tail)s</html>" % locals()
Searching:
  using ‘in’
    O(1) if RHS is set/dictionary

    O(N) if RHS is string/list/tuple

  using ‘hasattr’
    if the searched value is an attribute

    if the searched value is not an attribute
Loops:
  list comprehensions
  map as for loop moved to c – if the body of the loop is a
   function call

    newlist = []
     for word in oldlist:
       newlist.append(word.upper())

    newlist = [s.upper() for s in oldlist]

    newlist = map(str.upper, oldlist)
Lookups and Local variables:
  evaluating function references in loops
  accessing local variables vs global variables



    upper = str.upper
     newlist = []
     append = newlist.append
     for word in oldlist:
       append(upper(word))
Dictionaries
  Initialization -- try... Except
  Lookups -- string.maketrans



Regular expressions:
  RE's better than writing a loop
  Built-in string functions better than RE's

  Compiled re's are significantly faster



    re.search('^[A-Za-z]+$', source)
    x = re.compile('^[A-Za-z]+$').search
     x(source)
Imports
  avoid import *
  use only when required(inside functions)

  lazy imports



exec and eval
  better to avoid
  Compile and evaluate
Summary on loop optimization - (extracted from an
                                  essay by Guido)
  only optimize when there is a proven speed bottleneck
  small is beautiful

  use intrinsic operations

  avoid calling functions written in Python in your inner
   loop
  local variables are faster than globals

  try to use map(), filter() or reduce() to replace an
   explicit for loop(map with built-in, for loop with inline)
  check your algorithms for quadratic behaviour

  and last but not least: collect data. Python's excellent
   profile module can quickly show the bottleneck in your
   code
Might be unintentional, better not to be intuitive!

The right answer to improve performance
          - Use PROFILERS
Spot it Right!
   Hotspots
   Fact and fake( - Profiler Vs Programmers intuition!)
   Threads
    IO operations

    Logging

    Encoding and Decoding

    Lookups

   Rewrite just the hotspots!
   Psyco/Pyrex
   C extensions
Profilers
   timeit/time.clock
   profile/cprofile
   Visualization
     RunSnakeRun
     Gprof2Dot

     PycallGraph
timeit
   precise performance of small code snippets.
   the two convenience functions - timeit and repeat
    timeit.repeat(stmt[, setup[, timer[, repeat=3[,
     number=1000000]]]])
    timeit.timeit(stmt[, setup[, timer[, number=1000000]]])



   can also be used from command line
      python -m timeit [-n N] [-r N] [-s S] [-t] [-c] [-h]
       [statement ...]
timeit
  import timeit

   timeit.timeit('for i in xrange(10): oct(i)', gc.enable()')
  1.7195474706909972

   timeit.timeit('for i in range(10): oct(i)', 'gc.enable()')
  2.1380978155005295

   python -m timeit -n1000 -s'x=0' 'x+=1'
  1000 loops, best of 3: 0.0166 usec per loop

   python -m timeit -n1000 -s'x=0' 'x=x+1'
  1000 loops, best of 3: 0.0169 usec per loop
timeit
  import timeit

   python -mtimeit "try:" "   str.__nonzero__" "except
    AttributeError:" " pass"
  1000000 loops, best of 3: 1.53 usec per loop

   python -mtimeit "try:" "   int.__nonzero__" "except
    AttributeError:" " pass"
  10000000 loops, best of 3: 0.102 usec per loop
timeit
  test_timeit.py

   def f():
       try:
         str.__nonzero__
       except AttributeError:
         pass

    if __name__ == '__main__':
       f()

   python -mtimeit -s "from test_timeit import f" "f()"
  100000 loops, best of 3: 2.5 usec per loop
cProfile/profile
   Deterministic profiling
   The run time performance
   With statistics
   Small snippets bring big changes!


      import cProfile
       cProfile.run(command[, filename])

      python -m cProfile myscript.py [-o output_file] [-s
       sort_order]
cProfile statistics
  E:pycon12>profile_example.py
   100004 function calls in 0.306 CPU seconds

   Ordered by: standard name
   ncalls tottime percall cumtime percall filename:lineno(function)
      1   0.014 0.014      0.014   0.014 :0(setprofile)
      1   0.000 0.000 0.292        0.292 <string>:1(<module>)
      1   0.000 0.000 0.306        0.306 profile:0(example())
      0   0.000            0.000             profile:0(profiler)
      1    0.162 0.162 0.292        0.292 profile_example.py:10(example)
   100000 0.130 0.000 0.130 0.000            profile_example.py:2(check)
Using the stats
   The pstats module
   View and compare stats
      import cProfile
       cProfile.run('foo()', 'fooprof')
       import pstats
       p = pstats.Stats('fooprof')

      p.strip_dirs().sort_stats(-1).print_stats()
      p.sort_stats('cumulative').print_stats(10)
      p.sort_stats('file').print_stats('__init__')
Visualization
   A picture is worth a thousand words!
   Other tools to visualize profiles
     kcachegrind
     RunSnakeRun

     GProf2Dot

     PyCallGraph

     PyProf2CallTree
RunSnakeRun
  E:pycon12>runsnake D:simulation_gui.profile
Don't be too clever.
Don't sweat it too much.
 Develop an instinct for the sort of code that
 Python runs well.
References
   http://docs.python.org
   http://wiki.python.org/moin/PythonSpeed/PerformanceTips/
   http://sschwarzer.com/download/optimization_europython2006.pdf
   http://oreilly.com/python/excerpts/python-in-a-nutshell/testing-
    debugging.html
Questions?

Más contenido relacionado

Was ist angesagt?

«Отладка в Python 3.6: Быстрее, Выше, Сильнее» Елизавета Шашкова, JetBrains
«Отладка в Python 3.6: Быстрее, Выше, Сильнее» Елизавета Шашкова, JetBrains«Отладка в Python 3.6: Быстрее, Выше, Сильнее» Елизавета Шашкова, JetBrains
«Отладка в Python 3.6: Быстрее, Выше, Сильнее» Елизавета Шашкова, JetBrainsit-people
 
ITT 2015 - Saul Mora - Object Oriented Function Programming
ITT 2015 - Saul Mora - Object Oriented Function ProgrammingITT 2015 - Saul Mora - Object Oriented Function Programming
ITT 2015 - Saul Mora - Object Oriented Function ProgrammingIstanbul Tech Talks
 
Python and sysadmin I
Python and sysadmin IPython and sysadmin I
Python and sysadmin IGuixing Bai
 
Python profiling
Python profilingPython profiling
Python profilingdreampuf
 
EuroPython 2016 - Do I Need To Switch To Golang
EuroPython 2016 - Do I Need To Switch To GolangEuroPython 2016 - Do I Need To Switch To Golang
EuroPython 2016 - Do I Need To Switch To GolangMax Tepkeev
 
Scala - where objects and functions meet
Scala - where objects and functions meetScala - where objects and functions meet
Scala - where objects and functions meetMario Fusco
 
«iPython & Jupyter: 4 fun & profit», Лев Тонких, Rambler&Co
«iPython & Jupyter: 4 fun & profit», Лев Тонких, Rambler&Co«iPython & Jupyter: 4 fun & profit», Лев Тонких, Rambler&Co
«iPython & Jupyter: 4 fun & profit», Лев Тонких, Rambler&CoMail.ru Group
 
Java 7, 8 & 9 - Moving the language forward
Java 7, 8 & 9 - Moving the language forwardJava 7, 8 & 9 - Moving the language forward
Java 7, 8 & 9 - Moving the language forwardMario Fusco
 
python beginner talk slide
python beginner talk slidepython beginner talk slide
python beginner talk slidejonycse
 
Haskell in the Real World
Haskell in the Real WorldHaskell in the Real World
Haskell in the Real Worldosfameron
 
The best language in the world
The best language in the worldThe best language in the world
The best language in the worldDavid Muñoz Díaz
 
Fun never stops. introduction to haskell programming language
Fun never stops. introduction to haskell programming languageFun never stops. introduction to haskell programming language
Fun never stops. introduction to haskell programming languagePawel Szulc
 
What's new in C# 6 - NetPonto Porto 20160116
What's new in C# 6  - NetPonto Porto 20160116What's new in C# 6  - NetPonto Porto 20160116
What's new in C# 6 - NetPonto Porto 20160116Paulo Morgado
 
Kotlin Slides from Devoxx 2011
Kotlin Slides from Devoxx 2011Kotlin Slides from Devoxx 2011
Kotlin Slides from Devoxx 2011Andrey Breslav
 
Idiomatic Kotlin
Idiomatic KotlinIdiomatic Kotlin
Idiomatic Kotlinintelliyole
 

Was ist angesagt? (20)

«Отладка в Python 3.6: Быстрее, Выше, Сильнее» Елизавета Шашкова, JetBrains
«Отладка в Python 3.6: Быстрее, Выше, Сильнее» Елизавета Шашкова, JetBrains«Отладка в Python 3.6: Быстрее, Выше, Сильнее» Елизавета Шашкова, JetBrains
«Отладка в Python 3.6: Быстрее, Выше, Сильнее» Елизавета Шашкова, JetBrains
 
ITT 2015 - Saul Mora - Object Oriented Function Programming
ITT 2015 - Saul Mora - Object Oriented Function ProgrammingITT 2015 - Saul Mora - Object Oriented Function Programming
ITT 2015 - Saul Mora - Object Oriented Function Programming
 
Python and sysadmin I
Python and sysadmin IPython and sysadmin I
Python and sysadmin I
 
Python profiling
Python profilingPython profiling
Python profiling
 
EuroPython 2016 - Do I Need To Switch To Golang
EuroPython 2016 - Do I Need To Switch To GolangEuroPython 2016 - Do I Need To Switch To Golang
EuroPython 2016 - Do I Need To Switch To Golang
 
Python
PythonPython
Python
 
Scala - where objects and functions meet
Scala - where objects and functions meetScala - where objects and functions meet
Scala - where objects and functions meet
 
Python tour
Python tourPython tour
Python tour
 
«iPython & Jupyter: 4 fun & profit», Лев Тонких, Rambler&Co
«iPython & Jupyter: 4 fun & profit», Лев Тонких, Rambler&Co«iPython & Jupyter: 4 fun & profit», Лев Тонких, Rambler&Co
«iPython & Jupyter: 4 fun & profit», Лев Тонких, Rambler&Co
 
Java 7, 8 & 9 - Moving the language forward
Java 7, 8 & 9 - Moving the language forwardJava 7, 8 & 9 - Moving the language forward
Java 7, 8 & 9 - Moving the language forward
 
Hammurabi
HammurabiHammurabi
Hammurabi
 
python beginner talk slide
python beginner talk slidepython beginner talk slide
python beginner talk slide
 
Haskell in the Real World
Haskell in the Real WorldHaskell in the Real World
Haskell in the Real World
 
The best language in the world
The best language in the worldThe best language in the world
The best language in the world
 
Fun never stops. introduction to haskell programming language
Fun never stops. introduction to haskell programming languageFun never stops. introduction to haskell programming language
Fun never stops. introduction to haskell programming language
 
Python Objects
Python ObjectsPython Objects
Python Objects
 
What's new in C# 6 - NetPonto Porto 20160116
What's new in C# 6  - NetPonto Porto 20160116What's new in C# 6  - NetPonto Porto 20160116
What's new in C# 6 - NetPonto Porto 20160116
 
MTL Versus Free
MTL Versus FreeMTL Versus Free
MTL Versus Free
 
Kotlin Slides from Devoxx 2011
Kotlin Slides from Devoxx 2011Kotlin Slides from Devoxx 2011
Kotlin Slides from Devoxx 2011
 
Idiomatic Kotlin
Idiomatic KotlinIdiomatic Kotlin
Idiomatic Kotlin
 

Ähnlich wie Profiling and optimization

sonam Kumari python.ppt
sonam Kumari python.pptsonam Kumari python.ppt
sonam Kumari python.pptssuserd64918
 
Learning notes of r for python programmer (Temp1)
Learning notes of r for python programmer (Temp1)Learning notes of r for python programmer (Temp1)
Learning notes of r for python programmer (Temp1)Chia-Chi Chang
 
Python高级编程(二)
Python高级编程(二)Python高级编程(二)
Python高级编程(二)Qiangning Hong
 
Python 101 language features and functional programming
Python 101 language features and functional programmingPython 101 language features and functional programming
Python 101 language features and functional programmingLukasz Dynowski
 
Porque aprender haskell me fez um programador python melhor?
Porque aprender haskell me fez um programador python melhor?Porque aprender haskell me fez um programador python melhor?
Porque aprender haskell me fez um programador python melhor?UFPA
 
Real Time Big Data Management
Real Time Big Data ManagementReal Time Big Data Management
Real Time Big Data ManagementAlbert Bifet
 
Effective Numerical Computation in NumPy and SciPy
Effective Numerical Computation in NumPy and SciPyEffective Numerical Computation in NumPy and SciPy
Effective Numerical Computation in NumPy and SciPyKimikazu Kato
 
An overview of Python 2.7
An overview of Python 2.7An overview of Python 2.7
An overview of Python 2.7decoupled
 
Swift for tensorflow
Swift for tensorflowSwift for tensorflow
Swift for tensorflow규영 허
 
Advanced Datastructures and algorithms CP4151unit1b.pdf
Advanced Datastructures and algorithms CP4151unit1b.pdfAdvanced Datastructures and algorithms CP4151unit1b.pdf
Advanced Datastructures and algorithms CP4151unit1b.pdfSheba41
 
An Overview Of Python With Functional Programming
An Overview Of Python With Functional ProgrammingAn Overview Of Python With Functional Programming
An Overview Of Python With Functional ProgrammingAdam Getchell
 
C++ STL (quickest way to learn, even for absolute beginners).pptx
C++ STL (quickest way to learn, even for absolute beginners).pptxC++ STL (quickest way to learn, even for absolute beginners).pptx
C++ STL (quickest way to learn, even for absolute beginners).pptxGauravPandey43518
 
Python bootcamp - C4Dlab, University of Nairobi
Python bootcamp - C4Dlab, University of NairobiPython bootcamp - C4Dlab, University of Nairobi
Python bootcamp - C4Dlab, University of Nairobikrmboya
 
19. algorithms and-complexity
19. algorithms and-complexity19. algorithms and-complexity
19. algorithms and-complexityshowkat27
 
Refactoring to Macros with Clojure
Refactoring to Macros with ClojureRefactoring to Macros with Clojure
Refactoring to Macros with ClojureDmitry Buzdin
 

Ähnlich wie Profiling and optimization (20)

sonam Kumari python.ppt
sonam Kumari python.pptsonam Kumari python.ppt
sonam Kumari python.ppt
 
Learning notes of r for python programmer (Temp1)
Learning notes of r for python programmer (Temp1)Learning notes of r for python programmer (Temp1)
Learning notes of r for python programmer (Temp1)
 
Python高级编程(二)
Python高级编程(二)Python高级编程(二)
Python高级编程(二)
 
Python 101 language features and functional programming
Python 101 language features and functional programmingPython 101 language features and functional programming
Python 101 language features and functional programming
 
Python for Dummies
Python for DummiesPython for Dummies
Python for Dummies
 
Porque aprender haskell me fez um programador python melhor?
Porque aprender haskell me fez um programador python melhor?Porque aprender haskell me fez um programador python melhor?
Porque aprender haskell me fez um programador python melhor?
 
Real Time Big Data Management
Real Time Big Data ManagementReal Time Big Data Management
Real Time Big Data Management
 
Effective Numerical Computation in NumPy and SciPy
Effective Numerical Computation in NumPy and SciPyEffective Numerical Computation in NumPy and SciPy
Effective Numerical Computation in NumPy and SciPy
 
An overview of Python 2.7
An overview of Python 2.7An overview of Python 2.7
An overview of Python 2.7
 
A tour of Python
A tour of PythonA tour of Python
A tour of Python
 
Swift for tensorflow
Swift for tensorflowSwift for tensorflow
Swift for tensorflow
 
NUMPY
NUMPY NUMPY
NUMPY
 
Hands on lua
Hands on luaHands on lua
Hands on lua
 
Advanced Datastructures and algorithms CP4151unit1b.pdf
Advanced Datastructures and algorithms CP4151unit1b.pdfAdvanced Datastructures and algorithms CP4151unit1b.pdf
Advanced Datastructures and algorithms CP4151unit1b.pdf
 
An Overview Of Python With Functional Programming
An Overview Of Python With Functional ProgrammingAn Overview Of Python With Functional Programming
An Overview Of Python With Functional Programming
 
C++ STL (quickest way to learn, even for absolute beginners).pptx
C++ STL (quickest way to learn, even for absolute beginners).pptxC++ STL (quickest way to learn, even for absolute beginners).pptx
C++ STL (quickest way to learn, even for absolute beginners).pptx
 
Introduction to python
Introduction to pythonIntroduction to python
Introduction to python
 
Python bootcamp - C4Dlab, University of Nairobi
Python bootcamp - C4Dlab, University of NairobiPython bootcamp - C4Dlab, University of Nairobi
Python bootcamp - C4Dlab, University of Nairobi
 
19. algorithms and-complexity
19. algorithms and-complexity19. algorithms and-complexity
19. algorithms and-complexity
 
Refactoring to Macros with Clojure
Refactoring to Macros with ClojureRefactoring to Macros with Clojure
Refactoring to Macros with Clojure
 

Último

2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdfThe Good Food Institute
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfTejal81
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch TuesdayIvanti
 
20140402 - Smart house demo kit
20140402 - Smart house demo kit20140402 - Smart house demo kit
20140402 - Smart house demo kitJamie (Taka) Wang
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud DataEric D. Schabell
 
UiPath Studio Web workshop series - Day 1
UiPath Studio Web workshop series  - Day 1UiPath Studio Web workshop series  - Day 1
UiPath Studio Web workshop series - Day 1DianaGray10
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationKnoldus Inc.
 
The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)IES VE
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTopCSSGallery
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosErol GIRAUDY
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and businessFrancesco Corti
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FESTBillieHyde
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024Brian Pichman
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingMAGNIntelligence
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0DanBrown980551
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2DianaGray10
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsDianaGray10
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIVijayananda Mohire
 

Último (20)

2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch Tuesday
 
20140402 - Smart house demo kit
20140402 - Smart house demo kit20140402 - Smart house demo kit
20140402 - Smart house demo kit
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data
 
UiPath Studio Web workshop series - Day 1
UiPath Studio Web workshop series  - Day 1UiPath Studio Web workshop series  - Day 1
UiPath Studio Web workshop series - Day 1
 
SheDev 2024
SheDev 2024SheDev 2024
SheDev 2024
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its application
 
The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development Companies
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenarios
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile Brochure
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and business
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FEST
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced Computing
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projects
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAI
 

Profiling and optimization

  • 2. About: Optimization - what, when & where? General Optimization strategies Python specific optimizations Profiling
  • 3. Golden Rule: "First make it work. Then make it right. Then make it fast!" - Kent Beck
  • 4. The process: 1.Get it right. 2.Test it's right. 3.Profile if slow. 4.Optimize. 5.Repeat from 2.  test suites  source control
  • 5. Make it right & pretty! Good Programming :-)  General optimization strategies  Python optimizations
  • 6. Optimization: Aims to improve Not perfect, the result Programming with performance tips
  • 7. When to start? Need for optimization  are you sure you need to do it at all?  is your code really so bad?  benchmarking  fast enough vs. faster Time for optimization  is it worth the time to tune it?  how much time is going to be spent running that code?
  • 8. When to start? Cost of optimization  costly developer time  addition of new features  new bugs in algorithms  speed vs. space Optimize only if necessary!
  • 9. Where to start?  Are you sure you're done coding? frosting a half-baked cake Premature optimization is the root of all evil! - Don Knuth  Working, well-architected code is always a must
  • 10. General strategies Algorithms - the big-O notation Architecture Choice of Data structures LRU techniques Loop invariant code out of loops Nested loops try...catch instead of if...else Multithreading for I/O bound code DBMS instead of flat files
  • 11. General strategies Big – O – The Boss! performance of the algorithms a function of N - the input size to the algorithm  O(1) - constant time  O(ln n) - logarithmic  O(n) - linear  O(n2) - quadratic
  • 12. Common big-O’s Order Said to be Examples “…. time” -------------------------------------------------- O(1) constant key in dict dict[key] = value list.append(item) O(ln n) logarithmic Binary search O(n) linear item in sequence str.join(list) O(n ln n) list.sort() O(n2) quadratic Nested loops (with constant time bodies)
  • 13. Note the notation O(N2) O(N) def slow(it): def fast(it): result = [] result = [] for item in it: for item in it: result.insert(0, item) result.append(item) return result result.reverse( ) return result result = list(it)
  • 14. Big-O’s of Python Building blocks  lists - vectors  dictionaries - hash tables  sets - hash tables
  • 15. Big-O’s of Python Building blocks Let, L be any list, T any string (plain or Unicode); D any dict; S any set, with (say) numbers as items (with O(1) hashing and comparison) and x any number: O(1) - len( L ), len(T), len( D ), len(S), L [i], T [i], D[i], del D[i], if x in D, if x in S, S .add( x ), S.remove( x ), additions or removals to/from the right end of L
  • 16. Big-O’s of Python Building blocks O(N) - Loops on L, T, D, S, general additions or removals to/from L (not at the right end), all methods on T, if x in L, if x in T, most methods on L, all shallow copies O(N log N) - L .sort in general (but O(N) if L is already nearly sorted or reverse-sorted)
  • 17. Right Data Structure  lists, sets, dicts, tuples  collections - deque, defaultdict, namedtuple  Choose them based on the functionality  search an element in a sequence  append  intersection  remove from middle  dictionary initializations
  • 18. Right Data Structure  my_list = range(n) n in my_list  my_list = set(range(n)) n in my_list  my_list[start:end] = []  my_deque.rotate(-end) for counter in (end-start): my_deque.pop()
  • 19. Right Data Structure s = [('yellow', 1), ('blue', 2), ('yellow', 3), ('blue', 4), ('red', 1)]  d = defaultdict(list) for k, v in s: d[k].append(v) d.items() [('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]  d = {} for k, v in s: d.setdefault(k, []).append(v) d.items() [('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]
  • 20. Python Performance Tips  built-in modules  string concatenation  lookups and local variables  dictionary initialization  dictionary lookups  import statements  loops
  • 21. Built-ins - Highly optimized - Sort a list of tuples by it’s n-th field  def sortby(somelist, n): nlist = [(x[n], x) for x in somelist] nlist.sort() return [val for (key, val) in nlist] n = 1 import operator nlist.sort(key=operator.itemgetter(n))
  • 22. String Concatenation  s = "" for substring in list: s += substring  s = "".join(list)  out = "<html>" + head + prologue + query + tail + "</html>"  out = "<html>%s%s%s%s</html>" % (head, prologue, query, tail)  out = "<html>%(head)s%(prologue)s%(query)s% (tail)s</html>" % locals()
  • 23. Searching:  using ‘in’  O(1) if RHS is set/dictionary  O(N) if RHS is string/list/tuple  using ‘hasattr’  if the searched value is an attribute  if the searched value is not an attribute
  • 24. Loops:  list comprehensions  map as for loop moved to c – if the body of the loop is a function call  newlist = [] for word in oldlist: newlist.append(word.upper())  newlist = [s.upper() for s in oldlist]  newlist = map(str.upper, oldlist)
  • 25. Lookups and Local variables:  evaluating function references in loops  accessing local variables vs global variables  upper = str.upper newlist = [] append = newlist.append for word in oldlist: append(upper(word))
  • 26. Dictionaries  Initialization -- try... Except  Lookups -- string.maketrans Regular expressions:  RE's better than writing a loop  Built-in string functions better than RE's  Compiled re's are significantly faster  re.search('^[A-Za-z]+$', source)  x = re.compile('^[A-Za-z]+$').search x(source)
  • 27. Imports  avoid import *  use only when required(inside functions)  lazy imports exec and eval  better to avoid  Compile and evaluate
  • 28. Summary on loop optimization - (extracted from an essay by Guido)  only optimize when there is a proven speed bottleneck  small is beautiful  use intrinsic operations  avoid calling functions written in Python in your inner loop  local variables are faster than globals  try to use map(), filter() or reduce() to replace an explicit for loop(map with built-in, for loop with inline)  check your algorithms for quadratic behaviour  and last but not least: collect data. Python's excellent profile module can quickly show the bottleneck in your code
  • 29. Might be unintentional, better not to be intuitive! The right answer to improve performance - Use PROFILERS
  • 30. Spot it Right!  Hotspots  Fact and fake( - Profiler Vs Programmers intuition!) Threads  IO operations  Logging  Encoding and Decoding  Lookups  Rewrite just the hotspots!  Psyco/Pyrex  C extensions
  • 31. Profilers  timeit/time.clock  profile/cprofile  Visualization  RunSnakeRun  Gprof2Dot  PycallGraph
  • 32. timeit  precise performance of small code snippets.  the two convenience functions - timeit and repeat  timeit.repeat(stmt[, setup[, timer[, repeat=3[, number=1000000]]]])  timeit.timeit(stmt[, setup[, timer[, number=1000000]]])  can also be used from command line  python -m timeit [-n N] [-r N] [-s S] [-t] [-c] [-h] [statement ...]
  • 33. timeit import timeit  timeit.timeit('for i in xrange(10): oct(i)', gc.enable()') 1.7195474706909972  timeit.timeit('for i in range(10): oct(i)', 'gc.enable()') 2.1380978155005295  python -m timeit -n1000 -s'x=0' 'x+=1' 1000 loops, best of 3: 0.0166 usec per loop  python -m timeit -n1000 -s'x=0' 'x=x+1' 1000 loops, best of 3: 0.0169 usec per loop
  • 34. timeit import timeit  python -mtimeit "try:" " str.__nonzero__" "except AttributeError:" " pass" 1000000 loops, best of 3: 1.53 usec per loop  python -mtimeit "try:" " int.__nonzero__" "except AttributeError:" " pass" 10000000 loops, best of 3: 0.102 usec per loop
  • 35. timeit test_timeit.py  def f(): try: str.__nonzero__ except AttributeError: pass if __name__ == '__main__': f()  python -mtimeit -s "from test_timeit import f" "f()" 100000 loops, best of 3: 2.5 usec per loop
  • 36. cProfile/profile  Deterministic profiling  The run time performance  With statistics  Small snippets bring big changes!  import cProfile cProfile.run(command[, filename])  python -m cProfile myscript.py [-o output_file] [-s sort_order]
  • 37. cProfile statistics E:pycon12>profile_example.py 100004 function calls in 0.306 CPU seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.014 0.014 0.014 0.014 :0(setprofile) 1 0.000 0.000 0.292 0.292 <string>:1(<module>) 1 0.000 0.000 0.306 0.306 profile:0(example()) 0 0.000 0.000 profile:0(profiler) 1 0.162 0.162 0.292 0.292 profile_example.py:10(example) 100000 0.130 0.000 0.130 0.000 profile_example.py:2(check)
  • 38. Using the stats  The pstats module  View and compare stats  import cProfile cProfile.run('foo()', 'fooprof') import pstats p = pstats.Stats('fooprof')  p.strip_dirs().sort_stats(-1).print_stats()  p.sort_stats('cumulative').print_stats(10)  p.sort_stats('file').print_stats('__init__')
  • 39. Visualization  A picture is worth a thousand words!  Other tools to visualize profiles  kcachegrind  RunSnakeRun  GProf2Dot  PyCallGraph  PyProf2CallTree
  • 40. RunSnakeRun  E:pycon12>runsnake D:simulation_gui.profile
  • 41. Don't be too clever. Don't sweat it too much.  Develop an instinct for the sort of code that Python runs well.
  • 42. References  http://docs.python.org  http://wiki.python.org/moin/PythonSpeed/PerformanceTips/  http://sschwarzer.com/download/optimization_europython2006.pdf  http://oreilly.com/python/excerpts/python-in-a-nutshell/testing- debugging.html