Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Python+gradle

6.043 Aufrufe

Veröffentlicht am

A presentation I gave at GradleSummit 2015 that discusses building Python code bases with Gradle.

Veröffentlicht in: Software

Python+gradle

  1. 1. Python+Gradle Stephen  Holsapple  
  2. 2. Python  
  3. 3. What  is  setuptools?   #!/usr/bin/env python from setuptools import setup setup( description=README, author='Stephen Holsapple', author_email='sholsapp@linkedin.com', packages=['foo'], install_requires=[], )
  4. 4. What  is  setuptools?  Everything.   •  To  Python,  setuptools  is  everything.   – Is  the  core  API  that  most  tools  nowadays  use.   – Is  the  brains  behind:   •  Build   •  Packaging   •  Metadata  discovery   •  Dependency  management  
  5. 5. Building  w/  setuptools •  Tools  like  virtualenv  and  pip  make  life   preHy  good.  
  6. 6. More  about  virtualenv   $ virtualenv my-env $ source my-env/bin/activate (my-env)$ python Python 2.7.5 >>> import sys >>> sys.path ['./my-env/lib/python2.7/site- packages'] (my-env) $ deactivate $
  7. 7. More  about  virtualenv •  A  command  line  tool  useful  for:   – CreaJng  isolated  Python  environments.   •  Really,  a  plaLorm  to  solve  dependency  and   version  problems.  
  8. 8. More  about  pip •  A  command  line  tool  useful  for:   – Finding  dependencies   – Installing  dependencies   – ArJfact  management   – Caching   •  Really,  an  implementaJon  of  the   setuptools  programming  interface.  
  9. 9. More  about  pip (my-env)$ pip install requests Downloading/unpacking requests Downloading requests-2.7.0-py2.py3-none- any.whl (470kB): 470kB downloaded Installing collected packages: requests Successfully installed requests Cleaning up...
  10. 10. Building  w/  setuptools   (my-env)$ pip install requests (my-env)$ python >>> import requests >>> ...
  11. 11. Packaging  w/  setuptools •  Tools  like  setuptools,  pip, pex  make  life   preHy  good.   – Source  distribuJons   – Binary  distribuJons   – Deployable  distribuJon  
  12. 12. Metadata  w/  setuptools •  Doesn't  provide  great  ways  to:   – Reason  about  builds  before/aOer   – Reason  about  arJfacts  programaJcally   •  Does  expose  programming  interfaces  to   integrate  with  exisJng  metadata  systems.   •  Requires  us  to  evaluate  Python  to  query  about   metadata.  
  13. 13. Metadata  w/  setuptools (my-env)$ pip show requests --- Name: requests Version: 2.7.0 Location: my-env/lib/python2.7/site-packages Requires: (my-env)$ pip freeze requests==2.7.0 wsgiref==0.1.2
  14. 14. Dependencies  w/  setuptools #!/usr/bin/env python import setuptools setuptools.setup(, # ... install_requires=[ "requests>=2.6", "pyOpenSSL==0.15", "urllib3", ], # ... )
  15. 15. Dependencies  w/  pip $ cat requirements.txt requests>=2.6 pyOpenSSL==0.15 urllib3
  16. 16. [pip/issues/988] “Requirements  files  are   used  to  force  pip  to   properly  resolve   dependencies.  As  it  is  now,   pip  doesn't  have  true   dependency  resolu=on,   but  instead  simply  uses   the  first  specifica=on  it   finds  for  a  project.”  
  17. 17. pkg_resources  is  greedy   •  Simply  use  the  first  specificaJon  found  for  a   dependency.   •  Ask  developers  to  pin  all  of  their   dependencies.   – pip freeze > requirements.txt
  18. 18. [VersionConflict] pkg_resources.VersionConflict: (requests 2.6.2, Requirement.parse('requests<=2.6.0')
  19. 19. [DistributionNotFound] pip.exceptions.DistributionNotFound: No distributions at all found for requests
  20. 20. Future  Work   •  Sincere  thanks  to   Donald  StuS  et  al.  on   PyPA  working  to   improve  state  of   Python  packaging.   •  Python  is  rapidly   changing  for  the   beHer.  
  21. 21. Python   •  Report  card   – Ecosystem  (great)   – Build  System  (good)   – Packaging  (good)   – Metadata  (okay)   – Dependency  Management  (bad)  
  22. 22. Scaling  Products   •  Builds   •  ArJfacts   – Downloading   – Caching   – Metadata   •  Dependencies   – Dependency  ResoluJon   – Conflict  ResoluJon   •  TesJng  
  23. 23. Building  BeHer  Building  Systems   •  We  have  a  core  set  of  great  tools  to  work   with:   – virtualenv – setuptools – pip – pex
  24. 24. A  polite  tool   •  The  pip  tool  is  polite  because:   – Allows  for  customizaJon  or  disabling  of   setuptools  and  pip  features:   •  --index-url •  --extra-index-url •  --no-deps – Allows  us  to  use  local  repositories.   – Allows  us  to  solve  our  problems  our  way.  
  25. 25. #!/bin/build.sh   vinit && source activate python setup.py install py.test python setup.py sdist
  26. 26. #!/bin/build.sh   vinit && source activate python setup.py install py.test python setup.py sdist pip wheel .
  27. 27. #!/bin/build.sh vinit && source activate for dep in $( cache/ ); do pip install --no-deps dep done python setup.py install py.test python setup.py sdist
  28. 28. #!/bin/build.sh vinit && source activate for dep in $( ???/ ); do pip install --no-deps dep done python setup.py install py.test python setup.py sdist
  29. 29. Gradle  
  30. 30. #!/bin/build.sh vinit && source activate for dep in $( gradle/ ); do pip install --no-deps dep done python setup.py install py.test python setup.py sdist
  31. 31. Philosophy   Let’s  let  Python  do  what  Python  is   good  at  and  let  Gradle  do  what  Gradle   is  good  at.  
  32. 32. Gradle  as  a  Build  Orchestrator   •  First,  resolve  my  dependencies •  Second,  run  my  build – pip install --no-deps dep •  Third,  run  my  package – python setup.py sdist •  Last,  run  my  publish
  33. 33. build.gradle apply plugin: 'python-venv' apply plugin: 'python-sdist' sdist { sourceDirectory 'src' }
  34. 34. build.gradle apply plugin: 'python-venv' apply plugin: 'python-pex' sdist { sourceDirectory 'src' }
  35. 35. Enhancing  Python   •  Represent  arJfacts  and  dependencies  in  a  way   Gradle  can  understand.   •  Integrate  exisJng  metadata  with  setuptools   using  custom  distribuJon  class.  
  36. 36. RepresenJng  Dependencies   <ivy-module version="1.0"> <info module="cryptography" revision="0.8.2" /> <publications> <artifact name="cryptography" conf="default"/> </publications> <dependencies> <dependency name="pyasn1" rev="0.1.7" /> <dependency name="six" rev="1.9.0" /> <dependency name="setuptools" rev="3.4.4" /> <dependency name="enum34" rev="1.0.4" /> <dependency name="cffi" rev="0.9.2" /> </dependencies> </ivy-module>
  37. 37. The  Cheese  Shop   •  Python  Package  Index  (PyPI)   – Community  hosts  packages  here   – Cannot  modify  packages  here   •  Internal  Python  Package  Index   – ArJfactory   – Add  Ivy  metadata  for  packages  
  38. 38. IntegraJng  Metadata   •  How  to  marry  setuptools  with  custom   metadata  systems.   – Product  names   – Product  versions   – SpecificaJon  files   – Dependencies  
  39. 39. IntegraJng  Metadata   #!/usr/bin/env python2.6 import setuptools from distgradle import GradleDistribution setuptools.setup( distclass=GradleDistribution, package_dir={'': 'src'}, packages=setuptools.find_packages('src'), include_package_data=True, namespace_packages=['linkedin'], )
  40. 40. Gradle  Infrastructure   •  python-product – Your  enterprise  build  logic,  covering  everything   from  your  resoluJon  strategies,  your  repositories,   your  metadata,  and  more.   – You  could  write  a  simple  plugin  that  simply  defers   to  setuptools,  if  you  want.  
  41. 41. Gradle  Plugins   •  python-venv – Builds  a  local  environment   •  python-sdist – Build  a  source  distribuJon  using  setuptools.   •  python-wheel   – Build  a  wheel  distribuJon  using  setuptools.   •  python-pex – Build  a  pex  distribuJon  using  pex.  
  42. 42. python-venv   •  Under  the  hood:   – CreaJng  a  virtual  environment.   – Installing  build  requirements.   – Installing  your  resolved  dependencies.   – Installing  your  project.   – Running  tests   – Packaging  arJfacts  
  43. 43. python-sdist   •  Under  the  hood:   – Applying  python-venv – Invoking  ./setup.py sdist
  44. 44. python-wheel   •  Under  the  hood:   – Applying  python-venv – Invoking  pip wheel •  Making  Python's  wheel  file  compaJble  with   Ivy  is  hard.  
  45. 45. python-pex   •  Under  the  hood:   – Applying  python-venv – Invoking  pex •  Prepare  to  baHle  with  pex's  dependency   resolver.  
  46. 46. python-???   •  ExisJng  plugins  easy  to  add  now.   •  AddiJonal  plugins  easy  to  add  later.   •  Mix  and  match  any  way  we  like.   – We  might  want  a  source  and  a  wheel  distribuJon.  
  47. 47. Python+Gradle   •  Report  card   – Ecosystem  (great)   – Build  System  (good)   – Packaging  (good)   – Metadata  (good)   – Dependency  Management  (great)  
  48. 48. What  is  Gradle  doing?   •  Reading  specificaJon  files   •  Dependency  and  conflict  resoluJon   •  Downloading  and  caching   •  OrchestraJng  creaJon  of  Python  arJfacts   •  CreaJng  build  metadata   •  Uploading  arJfacts   •  PyPI  
  49. 49. What  is  Gradle  not  doing?   •  Changing  how  you  develop  with  Python   •  Changing  how  Python  packages  arJfacts   •  Changing  Python  arJfacts  themselves   •  Pushing  Gradle-­‐isms  into  Python  
  50. 50. Major  Wins   •  Dependency  Management   •  Plugin  Architecture   •  Reusable  Logic   •  MulJ-­‐language  Builds  
  51. 51. MulJ-­‐language  Builds   •  One  build  system  to  rule  them  all.   – Java   – Scala   – C/C++   – Javascript   – Python   •  Products  can  contain  different  languages  
  52. 52. Why  Python?   •  Why  was  Python  so  easy  to  build  with  Gradle?   – Can  decouple  dep.  mgmt.  from  setuptools – Can  decouple  dep.  mgmt.  from  PyPI   – Can  represent  dependencies  with  Ivy   •  Other  languages  like  Python  can  also  easily   use  Gradle  
  53. 53. QuesJons?  

×