Creating and Using a Virtual Python Environment
Table of Contents
What We Are Trying To Do
Your computer's system Python is used by your computer's operating system for many things including installing your OS itself and installing additional software on your OS. It is not good to pollute that Python with your own packages, and Python packages which your OS uses might interfere with your own packages.
Also, your computer's Python might not even be the right version for packages you want to use.
You may need many different versions of Python on your computer for doing different things.
You may also need different versions of Python packages for doing different things.
You may also find you have installed a Python package you don't like for some reason. You need a way to get rid of Python packages you don't want.
You want to do two things:
- Use the right version of Python for your job.
- Isolate the Python packages used by your applications from other applications.
A virtual Python environment will give you a separate Python for each application. That Python will be exactly the right version for your application. And that Python will have exactly the right version of packages installed for your application. You will be able to change the versions of packages used by your application without disturbing any of your other applications.
Some Terminology If You Get Lost
- module
- A file containing Python code. It may be executed directly by the Python interpreter, or imported into another module if found in a directory listed in
sys.path
. - package
- A hierarchy of directories containing Python modules. It may be imported by a Python module if found in a directory listed in
sys.path
. - site-packages
- A subdirectory in your Python's
lib
directory which contains your installed third party Python packages. Something is seriously wrong if this subdirectory is not listed insys.path
. - ez_setup.py
- A Python module which installs the
setuptools
package and theeasy_install
executable. - easy_install
- An executable in your Python's
bin
directory which can automatically install Python packages. - setuptools
- A Python package which facilitates creating and distributing third party Python packages.
setuptools
is a third party package that is not yet a part of the Python Standard Library of packages. - distutils
- A Python package which is the predecessor of
setuptools
still in wide use.distutils
is part of the Python Standard Library of packages. - setup.py
- A Python module which, when placed at the top level of a Python package, tells
setuptools
ordistutils
how to create a distributable form of the package. It normally consists of one function call (setup()
) with numerous arguments. Adistutils
distributed package can also be installed by unzipping it and running thesetup.py
module in the unzipped package directory.setup.py
is defined by the package author. - Pypi
- The Python Package Index (aka "Cheeseshop"). A master repository of third party Python packages. easy_install downloads packages from Pypi by default. Pypi contains packages in both "egg" form created by
setuptools
as well as packages created by
distutils
. - egg
- A zipped file created by setuptools which contains a distributable form of a Python package.
- virtualenv
- An executable in your Python's
bin
directory which can create virtual Python environment for an application. This environment will have its own isolatedbin
directory with virtualpython
andeasy_install
executables, as well as its ownlib/site-packages
subdirectory where the virtualeasy_install
will install private instances of third party Python packages.
There are three places to start:
- If you don't yet have a Python compiled separately from your system Python,
- If you have a Python instance separate from your system Python, but it doesn't yet have the setuptools package installed, or
- If you already have a separated compiled Python with the setuptools package installed.
If you don't yet have a Python compiled separately from your system Python
- XXX
- Proceed to:
If you have a Python instance separate from your system Python, but it doesn't yet have the setuptools module installed
- Get the latest version of ez_setup.py (Right click to save as python file)
- Run ez_setup.py with your separately compiled Python:
/usr/local/python2.6/bin/python ez_setup.py
- Your separately compiled Python's bin directory will now contain a new executable called easy_install
- Proceed to:
If you already have a separately compiled Python with the setuptools module installed
- Install virtualenv with easy_install:
/usr/local/python2.6/bin/easy_install virtualenv
- Your separately compiled Python's bin directory will now contain a new executable called virtualenv
Creating a Virtual Python Environment
- Use the virtualenv executable in your separately compiled Python to create a new virtual python environment:
/usr/local/python2.6/bin/virtualenv /opt/env/my-virt-py-env
- Your new virtual python environment at /opt/env/my-virt-py-env now contains its own bin/python executable, bin/easy_install executable, and lib/site-packages subdirectory
Using a Virtual Python Environment
- Install the packages required for your application in your virtual Python environment with easy_install:
/opt/env/my-virt-py-env/bin/easy_install Paste 2>&1 | tee /opt/env/my-virt-env/install-paste.stdout}}}
- You may need to manually install distutil-created (non-egg) packages the old way with the package's setup.py:
/opt/env/my-virt-py-env/bin/python /opt/env/my-virt-py-env/numpy/setup.py install 2>&1 | tee /opt/env/my-virt-py-env/numpy/install.stdout
- Run your application using your virtual Python environment's python executable:
/opt/env/my-virt-py-env/bin/python /opt/env/my-virt-py-env/my-application/my-script.py
Best Practices
- Download and untar distutils-created (non-egg) package into the virtual Python environment (see numpy example above)
- Run applications using fully qualified paths to the executable and all arguments (see my-script.py example above)
- Develop your application in your virtual Python environment directory
- While developing, you could change directories (cd) into your virtual Python environment directory to avoid having to fully qualify the paths to executables and arguments (i.e., just use bin/python and bin/easy_install
- Ignore the activate script supplied in your virtual Python environment's bin directory. No good comes of it. Use fully qualified paths.
- Do not set the environment variable PYTHONPATH. Only evil comes of it. Use fully qualified paths.
Resources
- virtualenv documentation on Pypi
- A short presentation on distutils
- A short presentation on setuptools
- A really good book about software engineering with Python written by the developer who is adding Pypi mirroring capability to setuptools. Thoroughly Covers eggs, setuptools, and virtualenv.
To Do
- How to compile Python
- How to cron an application in a virtual Python environment
- And other best practices
- Anatomy of a path file
- Anatomy of an egg
- Using easy_install to install from somewhere other than Pypi
- Better Resources
- Intro to Paste
- Intro to Buildout
- Creating a Python package
- Creating a Python egg
- Submitting a Python package to Pypi
- Creating a Pypi mirror