A presentation I gave at GradleSummit 2015 that discusses building Python code bases with Gradle.

1. Python+Gradle
Stephen
Holsapple

2. Python

3. What
is
setuptools?
#!/usr/bin/env python
from setuptools import setup
setup(
description=README,
author='Stephen Holsapple',
author_email='sholsapp@linkedin.com',
packages=['foo'],
install_requires=[],
)

4. What
is
setuptools?
Everything.
• To
Python,
setuptools
is
everything.
– Is
the
core
API
that
most
tools
nowadays
use.
– Is
the
brains
behind:
• Build
• Packaging
• Metadata
discovery
• Dependency
management

5. Building
w/
setuptools
• Tools
like
virtualenv
and
pip
make
life
preHy
good.

6. More
about
virtualenv
$ virtualenv my-env
$ source my-env/bin/activate
(my-env)$ python
Python 2.7.5
>>> import sys
>>> sys.path
['./my-env/lib/python2.7/site-
packages']
(my-env) $ deactivate
$

7. More
about
virtualenv
• A
command
line
tool
useful
for:
– CreaJng
isolated
Python
environments.
• Really,
a
plaLorm
to
solve
dependency
and
version
problems.

8. More
about
pip
• A
command
line
tool
useful
for:
– Finding
dependencies
– Installing
dependencies
– ArJfact
management
– Caching
• Really,
an
implementaJon
of
the
setuptools
programming
interface.

9. More
about
pip
(my-env)$ pip install requests
Downloading/unpacking requests
Downloading requests-2.7.0-py2.py3-none-
any.whl (470kB): 470kB downloaded
Installing collected packages: requests
Successfully installed requests
Cleaning up...

10. Building
w/
setuptools
(my-env)$ pip install requests
(my-env)$ python
>>> import requests
>>> ...

11. Packaging
w/
setuptools
• Tools
like
setuptools,
pip, pex
make
life
preHy
good.
– Source
distribuJons
– Binary
distribuJons
– Deployable
distribuJon

12. Metadata
w/
setuptools
• Doesn't
provide
great
ways
to:
– Reason
about
builds
before/aOer
– Reason
about
arJfacts
programaJcally
• Does
expose
programming
interfaces
to
integrate
with
exisJng
metadata
systems.
• Requires
us
to
evaluate
Python
to
query
about
metadata.

13. Metadata
w/
setuptools
(my-env)$ pip show requests
---
Name: requests
Version: 2.7.0
Location: my-env/lib/python2.7/site-packages
Requires:
(my-env)$ pip freeze
requests==2.7.0
wsgiref==0.1.2

14. Dependencies
w/
setuptools
#!/usr/bin/env python
import setuptools
setuptools.setup(,
# ...
install_requires=[
"requests>=2.6",
"pyOpenSSL==0.15",
"urllib3",
],
# ...
)

15. Dependencies
w/
pip
$ cat requirements.txt
requests>=2.6
pyOpenSSL==0.15
urllib3

16. [pip/issues/988]
“Requirements
files
are
used
to
force
pip
to
properly
resolve
dependencies.
As
it
is
now,
pip
doesn't
have
true
dependency
resolu=on,
but
instead
simply
uses
the
first
specifica=on
it
finds
for
a
project.”

17. pkg_resources
is
greedy
• Simply
use
the
first
specificaJon
found
for
a
dependency.
• Ask
developers
to
pin
all
of
their
dependencies.
– pip freeze > requirements.txt

18. [VersionConflict]
pkg_resources.VersionConflict:
(requests 2.6.2,
Requirement.parse('requests<=2.6.0')

19. [DistributionNotFound]
pip.exceptions.DistributionNotFound:
No distributions at all found for
requests

23. Future
Work
• Sincere
thanks
to
Donald
StuS
et
al.
on
PyPA
working
to
improve
state
of
Python
packaging.
• Python
is
rapidly
changing
for
the
beHer.

24. Python
• Report
card
– Ecosystem
(great)
– Build
System
(good)
– Packaging
(good)
– Metadata
(okay)
– Dependency
Management
(bad)

25. Scaling
Products
• Builds
• ArJfacts
– Downloading
– Caching
– Metadata
• Dependencies
– Dependency
ResoluJon
– Conflict
ResoluJon
• TesJng

26. Building
BeHer
Building
Systems
• We
have
a
core
set
of
great
tools
to
work
with:
– virtualenv
– setuptools
– pip
– pex

27. A
polite
tool
• The
pip
tool
is
polite
because:
– Allows
for
customizaJon
or
disabling
of
setuptools
and
pip
features:
• --index-url
• --extra-index-url
• --no-deps
– Allows
us
to
use
local
repositories.
– Allows
us
to
solve
our
problems
our
way.

28. #!/bin/build.sh
vinit && source activate
python setup.py install
py.test
python setup.py sdist

29. #!/bin/build.sh
vinit && source activate
python setup.py install
py.test
python setup.py sdist
pip wheel .

30. #!/bin/build.sh
vinit && source activate
for dep in $( cache/ ); do
pip install --no-deps dep
done
python setup.py install
py.test
python setup.py sdist

31. #!/bin/build.sh
vinit && source activate
for dep in $( ???/ ); do
pip install --no-deps dep
done
python setup.py install
py.test
python setup.py sdist

32. Gradle

33. #!/bin/build.sh
vinit && source activate
for dep in $( gradle/ ); do
pip install --no-deps dep
done
python setup.py install
py.test
python setup.py sdist

34. Philosophy
Let’s
let
Python
do
what
Python
is
good
at
and
let
Gradle
do
what
Gradle
is
good
at.

36. Gradle
as
a
Build
Orchestrator
• First,
resolve
my
dependencies
• Second,
run
my
build
– pip install --no-deps dep
• Third,
run
my
package
– python setup.py sdist
• Last,
run
my
publish

37. build.gradle
apply plugin: 'python-venv'
apply plugin: 'python-sdist'
sdist {
sourceDirectory 'src'
}

38. build.gradle
apply plugin: 'python-venv'
apply plugin: 'python-pex'
sdist {
sourceDirectory 'src'
}

39. Enhancing
Python
• Represent
arJfacts
and
dependencies
in
a
way
Gradle
can
understand.
• Integrate
exisJng
metadata
with
setuptools
using
custom
distribuJon
class.

40. RepresenJng
Dependencies
<ivy-module version="1.0">
<info module="cryptography" revision="0.8.2" />
<publications>
<artifact name="cryptography" conf="default"/>
</publications>
<dependencies>
<dependency name="pyasn1" rev="0.1.7" />
<dependency name="six" rev="1.9.0" />
<dependency name="setuptools" rev="3.4.4" />
<dependency name="enum34" rev="1.0.4" />
<dependency name="cffi" rev="0.9.2" />
</dependencies>
</ivy-module>

41. The
Cheese
Shop
• Python
Package
Index
(PyPI)
– Community
hosts
packages
here
– Cannot
modify
packages
here
• Internal
Python
Package
Index
– ArJfactory
– Add
Ivy
metadata
for
packages

42. IntegraJng
Metadata
• How
to
marry
setuptools
with
custom
metadata
systems.
– Product
names
– Product
versions
– SpecificaJon
files
– Dependencies

43. IntegraJng
Metadata
#!/usr/bin/env python2.6
import setuptools
from distgradle import GradleDistribution
setuptools.setup(
distclass=GradleDistribution,
package_dir={'': 'src'},
packages=setuptools.find_packages('src'),
include_package_data=True,
namespace_packages=['linkedin'],
)

44. Gradle
Infrastructure
• python-product
– Your
enterprise
build
logic,
covering
everything
from
your
resoluJon
strategies,
your
repositories,
your
metadata,
and
more.
– You
could
write
a
simple
plugin
that
simply
defers
to
setuptools,
if
you
want.

45. Gradle
Plugins
• python-venv
– Builds
a
local
environment
• python-sdist
– Build
a
source
distribuJon
using
setuptools.
• python-wheel
– Build
a
wheel
distribuJon
using
setuptools.
• python-pex
– Build
a
pex
distribuJon
using
pex.

46. python-venv
• Under
the
hood:
– CreaJng
a
virtual
environment.
– Installing
build
requirements.
– Installing
your
resolved
dependencies.
– Installing
your
project.
– Running
tests
– Packaging
arJfacts

47. python-sdist
• Under
the
hood:
– Applying
python-venv
– Invoking
./setup.py sdist

48. python-wheel
• Under
the
hood:
– Applying
python-venv
– Invoking
pip wheel
• Making
Python's
wheel
file
compaJble
with
Ivy
is
hard.

49. python-pex
• Under
the
hood:
– Applying
python-venv
– Invoking
pex
• Prepare
to
baHle
with
pex's
dependency
resolver.

50. python-???
• ExisJng
plugins
easy
to
add
now.
• AddiJonal
plugins
easy
to
add
later.
• Mix
and
match
any
way
we
like.
– We
might
want
a
source
and
a
wheel
distribuJon.

51. Python+Gradle
• Report
card
– Ecosystem
(great)
– Build
System
(good)
– Packaging
(good)
– Metadata
(good)
– Dependency
Management
(great)

52. What
is
Gradle
doing?
• Reading
specificaJon
files
• Dependency
and
conflict
resoluJon
• Downloading
and
caching
• OrchestraJng
creaJon
of
Python
arJfacts
• CreaJng
build
metadata
• Uploading
arJfacts
• PyPI

53. What
is
Gradle
not
doing?
• Changing
how
you
develop
with
Python
• Changing
how
Python
packages
arJfacts
• Changing
Python
arJfacts
themselves
• Pushing
Gradle-­‐isms
into
Python

54. Major
Wins
• Dependency
Management
• Plugin
Architecture
• Reusable
Logic
• MulJ-­‐language
Builds

55. MulJ-­‐language
Builds
• One
build
system
to
rule
them
all.
– Java
– Scala
– C/C++
– Javascript
– Python
• Products
can
contain
different
languages

56. Why
Python?
• Why
was
Python
so
easy
to
build
with
Gradle?
– Can
decouple
dep.
mgmt.
from
setuptools
– Can
decouple
dep.
mgmt.
from
PyPI
– Can
represent
dependencies
with
Ivy
• Other
languages
like
Python
can
also
easily
use
Gradle

57. QuesJons?