This presentation describes different strategies for installing Python software packages. This includes standard techniques like easy_install and pip, as well as newer techniques like virtualenv that are well-suited for users who do not have administrative privileges.
Apidays New York 2024 - The value of a flexible API Management solution for O...
Â
10 11-hart installing pythonsoftware
1. Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,
for the United States Department of Energyâs National Nuclear Security Administration
under contract DE-AC04-94AL85000.
Installing Python Software Packages:
The Good, The Bad and the Ugly
William E. Hart
Sandia National Laboratories
wehart@sandia.gov
2. Overview
This talks focuses on installing on Python packages
â Installing Python itself is usually a no-brainer
The Good
1. Installing on Windows with an installer executable
2. Installing with Linux application utility
3. Installing a Python package from the PyPI repository
4. Installing a Python package from source
The Bad
5. Using a virtual environment to isolate package installations
6. Using an installer executable on Windows with a virtual environment
The Ugly
7. Installing a Python extension package from source
8. PyCoinInstall â Managing builds for Python extension packages
Slide 2
3. About Python
Recent releases:
â 2.5 Very stable
â 2.6 Very stable
â 2.7 Recently released. Supports transition to Python 3.x
â 3.0 Includes changes that are not backwards compatible.
â 3.1 Stable 3.x release. Resolves 3.0 performance issues.
Package management for 3.x is still a bit snarky. Weâll focus on examples for
2.x
Slide 3
4. Python Packages
Package distributions
â Include one or more *.py files
â May include compiled source files (e.g. C++)
â Include a top-level setup.py file to coordinate installation
Three categories:
â Pure-Python packages
⢠Only include Python *.py files
â Independent extension packages
⢠Include compiled source files only on standard system libraries
â Dependent extension packages
⢠Include compiled source files that rely on third-party libraries
Slide 4
5. The Good
Assumption: you have administrative privileges
â Windows administrator group
â Linux super-user
Impact: you can install Python in system directories!
Idea:
â Install packages in the system Python directories
⢠Specifically, the site-packages directory is used
â Can easily install pure-Python packages or pre-compiled extension
packages
Slide 5
6. Example 1
Installing on Windows with an installer executable
Steps:
1.Download installer executable from package website
2.Execute installer
Notes:
â This uses the registry to find the Python installation
â The installer can contain pre-compiled extension packages
⢠These may differ for 32- and 64-bit installations
â The installer will likely be specific to the Python version
â This may work OK for dependent extension packages if the dependent
libraries are stored in a standard location
Slide 6
7. Example 2
Installing with Linux application utility
Idea: use a standard Linux package manager
â Ubuntu (and other Debian-based Linux variants)
⢠apt-get install packagename
â Fedora
⢠yum install packagename
Notes:
â The package manager can install pre-compiled extension packages
â The package manager handles package dependencies automatically
â Not all Python packages are available within these managers
â Itâs unclear whether these packages are updated frequently
Slide 7
8. Example 3
Installing a Python package from the PyPI repository
PyPI:
â A web-based service for hosting Python packages
â Includes some pre-compiled extension packages
â Several Python packages are used to install from PyPI
⢠setuptools, distribute, pip
Installing with pip:
â pip install packagename
Notes:
â The package manager handles package dependencies automatically
â Most Python packages are available from PyPI, including most releases
â PyPI does not include many pre-compiled extension packages
â The PyPI service has become a bit overloaded in the past year
â You need to specify the HTTP_PROXY environment to talk to proxy servers
Slide 8
9. Example 3 (cont)
Problem: pip is not a standard Python package!
Installing pip:
â wget http://peak.telecommunity.com/dist/ez_setup.py
â python ez_setup.py
â easy_install pip
Notes:
â pip is now generally recommended over setuptools and distribute
â The wget command is not a standard MS Windows utility
⢠Download from http://users.ugent.be/~bpuype/wget/
Slide 9
10. Example 4
Installing a Python package from source
Steps:
â Download package source from PyPI or project website
â Extract package from tarball or zipfile
â python setup.py install
Notes:
â This process avoids the need to install pip
â This is useful for packages that are not available on PyPI
â This installation process can build and install some extension packages
⢠Configuration of the compiler may be a problem
⢠Configuration of extension package dependencies is a problem
â This installation does not handle package dependencies
⢠Package dependencies may cause an install to fail
Slide 10
11. The Bad
Problem: users do not have administrative privileges
â This is not uncommon in industry
Problem: different users on a computer need different package versions
Idea:
â A user can create a virtual Python environment
â The user installs packages in this virtual Python directories
Impact:
â Users have complete control over their Python environment
⢠Can even have multiple environments (e.g. for developers)
â Can install both pure-Python packages or pre-compiled extension
packages
Slide 11
12. Virtual Python
Idea: create an isolated Python environment
virtualenv:
â Creates a directory structure that mimics the system Python installation
â Works on MS Windows, Linux and Mac OS
Installing virtualenv (with administrative privileges)
â Debian Linux
⢠apt-get install python-virtualenv
â Other
⢠wget http://peak.telecommunity.com/dist/ez_setup.py
⢠python ez_setup.py
⢠easy_install virtualenv
Slide 12
13. Virtual Python (cont)
Creating a virtual Python environment:
â virtualenv directory
Virtual Python executable:
â MS Windows: directory/Scripts/python.exe
â Linux: directory/bin/python
Notes:
â On MS Windows the executables are installed in the Scripts directory
⢠For simplicity, all subsequent examples will use the bin directory
â The setuptools package is installed in a virtual environment
⢠Can use the easy_install command to install packages
Example:
â directory/bin/easy_install pip
Slide 13
14. Virtual Python (cont)
Problem: setup a virtual environment without administrative privileges
vpy_install:
â Bootstrap a virtualenv installation from a script
Steps:
â wget http://doiop.com/pyutilib/vpy_install
â python vpy_install directory
Notes:
â Standard tools like setuptools and pip are installed by vpy_install
â The specified directory is a virtualenv Python environment
â On Linux, you can use a simpler command-line:
⢠vpy_install directory
â The â-site-packages option can be used to expose packages that are
installed in the system Python installation
⢠This is particularly useful when using precompiled extension packages,
which are installed in system Python directories
Slide 14
15. Example 5
Using a virtual environment to isolate package installations
Idea:
â Create a virtual python installation with vpy_install
â Install packages with pip
Example (Linux):
â wget http://doiop.com/pyutilib/vpy_install
â vpy_install mypy
â mypy/bin/pip install packagename
Notes:
â Within the virtual environment, pip is installed as an executable
⢠You do not need to execute it with the Python executable
Slide 15
16. Example 5 (cont)
Using a virtual environment to isolate package installations
Note: source package installations using a virtual Python executable are
installed in the virtual environment
Example (Linux):
â wget http://doiop.com/pyutilib/vpy_install
â vpy_install mypy
â cd packagename
â ../mypy/bin/python setup.py install
Slide 16
17. Example 6
Using an installer executable on Windows with a virtual environment
Problem:
â MS Windows installers use the system Python installation
â If a user does not have administrative privileges, then an installer cannot
be used directly!
Idea:
â âInstallâ the executable with the easy_install command
Example:
â easy_install.exe matplotlib-1.0.0.win32-py2.6.exe
Note: this only works if the executable was created by the Python distutils
package!
Slide 17
18. Example 6 (cont)
Using an installer executable on Windows with a virtual environment
Idea:
â Modify the MS Windows registry to register the virtual python executable
Steps:
1.Download the the_python.py script that is described at
http://doiop.com/nedbatchelder/register_python.html
2.Register the virtual python executable (as administrator)
⢠mypyScriptspython.exe the_python.py
1.Execute the MS Windows installer
2.Reregister the original python executable (as administrator)
Slide 18
19. The Ugly
Problem:
â you need to use an extension package that has dependencies
â prebuilt packages are not available
Observation:
â There is very little consistency between installation setup for different
packages
â Few packages use configuration management tools like autotools or
cmake
â Build instructions can be very different for different platforms
There are few options beyond diving deep into the package build process!
Slide 19
20. Example 7
Installing a Python extension package from source
Note: there is no standard way for doing this
Example: SciPy documentation
â General build instructions
â Separate instructions for Linux, Mac OS, Windows, and Solaris
Recommendations:
â Make sure that this package is going to be worth your time
â Donât hesitate to ask the developer team for help
â Have patience ...
Slide 20
21. Example 8
PyCoinInstall â Managing builds for Python extension packages
Note: this script is a relatively new project within CoinBazaar
â https://projects.coin-or.org/CoinBazaar/wiki/Projects/PyCoinInstall
Idea:
â Create a virtual environment with virtualenv
â Define package installation with a simple plugin system
⢠Packages define download, build and/or install operations
â A user creates a simple configuration file to specify the packages that will be
installed
â The pci command is used to manage the installation
Notes:
â Current development is focused on COIN-OR related packages
â Current plugins are tailored for building under Linux
Slide 21
22. Final Thoughts
Installing and distributing pure-Python packages is quite easy.
Installing extension packages is much more difficult.
â Try to leverage pre-compiled extension packages!
Virtual Python environments offer a flexible paradigm for managing different
Python configurations
â These are worth using even if you have administrative privileges!
Slide 22