Creating and Using a Virtual Python Environment

What We Are Trying To Do

Your computer's system Python is used by your computer's operating system for many things including installing your OS itself and installing additional software on your OS. It is not good to pollute that Python with your own packages, and Python packages which your OS uses might interfere with your own packages.

Also, your computer's Python might not even be the right version for packages you want to use.

You may need many different versions of Python on your computer for doing different things.

You may also need different versions of Python packages for doing different things.

You may also find you have installed a Python package you don't like for some reason. You need a way to get rid of Python packages you don't want.

You want to do two things:

1. Use the right version of Python for your job.
2. Isolate the Python packages used by your applications from other applications.

A virtual Python environment will give you a separate Python for each application. That Python will be exactly the right version for your application. And that Python will have exactly the right version of packages installed for your application. You will be able to change the versions of packages used by your application without disturbing any of your other applications.

Some Terminology If You Get Lost

module

A file containing Python code. It may be executed directly by the Python interpreter, or imported into another module if found in a directory listed in sys.path.

package

A hierarchy of directories containing Python modules. It may be imported by a Python module if found in a directory listed in sys.path.

site-packages

A subdirectory in your Python's lib directory which contains your installed third party Python packages. Something is seriously wrong if this subdirectory is not listed in sys.path.

ez_setup.py

A Python module which installs the setuptools package and the easy_install executable.

easy_install

An executable in your Python's bin directory which can automatically install Python packages.

setuptools

A Python package which facilitates creating and distributing third party Python packages. setuptools is a third party package that is not yet a part of the Python Standard Library of packages.

distutils

A Python package which is the predecessor of setuptools still in wide use. distutils is part of the Python Standard Library of packages.

setup.py

A Python module which, when placed at the top level of a Python package, tells setuptools or distutils how to create a distributable form of the package. It normally consists of one function call (setup()) with numerous arguments. A distutils distributed package can also be installed by unzipping it and running the setup.py module in the unzipped package directory. setup.py is defined by the package author.

Pypi

The Python Package Index (aka "Cheeseshop"). A master repository of third party Python packages. easy_install downloads packages from Pypi by default. Pypi contains packages in both "egg" form created by setuptools as well as packages created by distutils.

egg

A zipped file created by setuptools which contains a distributable form of a Python package.

virtualenv

An executable in your Python's bin directory which can create virtual Python environment for an application. This environment will have its own isolated bin directory with virtual python and easy_install executables, as well as its own lib/site-packages subdirectory where the virtual easy_install will install private instances of third party Python packages.

There are three places to start:

• If you don't yet have a Python compiled separately from your system Python,
• If you have a Python instance separate from your system Python, but it doesn't yet have the setuptools package installed, or
• If you already have a separated compiled Python with the setuptools package installed.

If you don't yet have a Python compiled separately from your system Python

• XXX
• Proceed to:

If you have a Python instance separate from your system Python, but it doesn't yet have the setuptools module installed

• Get the latest version of ez_setup.py (Right click to save as python file)
• Run ez_setup.py with your separately compiled Python:

/usr/local/python2.6/bin/python ez_setup.py

• Your separately compiled Python's bin directory will now contain a new executable called easy_install
• Proceed to:

If you already have a separately compiled Python with the setuptools module installed

• Install virtualenv with easy_install:

/usr/local/python2.6/bin/easy_install virtualenv

• Your separately compiled Python's bin directory will now contain a new executable called virtualenv

Creating a Virtual Python Environment

• Use the virtualenv executable in your separately compiled Python to create a new virtual Python environment:

/usr/local/python2.6/bin/virtualenv /opt/env/my-virt-py-env

• Your new virtual Python environment at /opt/env/my-virt-py-env now contains its own bin/python executable, bin/easy_install executable, and lib/site-packages subdirectory

Using a Virtual Python Environment

• Install the packages required for your application in your virtual Python environment with easy_install:

/opt/env/my-virt-py-env/bin/easy_install Paste 2>&1 | tee /opt/env/my-virt-env/install-paste.stdout

• You may need to manually install distutil-created (non-egg) packages the old way with the package's setup.py:

/opt/env/my-virt-py-env/bin/python /opt/env/my-virt-py-env/numpy/setup.py install 2>&1 | tee /opt/env/my-virt-py-env/numpy/install.stdout

• Run your application using your virtual Python environment's python executable:

/opt/env/my-virt-py-env/bin/python /opt/env/my-virt-py-env/my-application/my-script.py

Best Practices

• Download and untar distutils-created (non-egg) package into the virtual Python environment (see numpy example above)
• Run applications using fully qualified paths to the executable and all arguments (see my-script.py example above)
• Develop your application in your virtual Python environment directory
• While developing, you could change directories (cd) into your virtual Python environment directory to avoid having to fully qualify the paths to executables and arguments (i.e., just use bin/python and bin/easy_install
• Ignore the activate script supplied in your virtual Python environment's bin directory. No good comes of it. Use fully qualified paths.
• Do not set the environment variable PYTHONPATH. Only evil comes of it. Use fully qualified paths.

Resources

• virtualenv documentation on Pypi
• A short presentation on distutils
• A short presentation on setuptools
• A really good book about software engineering with Python written by the developer who is adding Pypi mirroring capability to setuptools. Thoroughly Covers eggs, setuptools, and virtualenv.

To Do

• How to compile Python
• Fix location of site-packages in lib path
• Explore the effects of OS user and permissions when running virtualenv
• How to cron an application in a virtual Python environment
• And other best practices
• Anatomy of a path file
• Anatomy of an egg
• Using easy_install to install from somewhere other than Pypi
• Better Resources
• Intro to Paste
• Intro to Buildout
• Creating a Python package
• Creating a Python egg
• Submitting a Python package to Pypi
• Creating a Pypi mirror