imports and PYTHONPATH

Posted under » Python on 24 February 2010

PYTHONPATH is conceptually the same as Java's CLASSPATH.

Python has two forms of import statement. They look something like this:

import z3c.form.form
from z3c.form import form

Python is all about binding (or assigning) names to values, and the primary purpose of the import statement is to bind names to modules. The key difference between the two forms above is what names are made available. The first form lets you reference 'z3c.form.form' in your code; the latter lets you reference 'form' directly. Let's examine the first case:

>>> import z3c.form.form
>>> z3c.form.form

>>> form
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'form' is not defined

Contrast this with the second case:

>>> from z3c.form import form
>>> z3c.form.form
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'z3c' is not defined
>>> form

The rule of thumb is that you will always refer to the bit after the 'import' in following code; so, when you say 'import z3c.form.form' you'll be able to refer to 'z3c.form.form', and when you say 'from z3c.form import form' you'll simply be able to refer to 'form'. In both cases, what you are actually dealing with when you actually use that name (be that 'z3c.form.form' or just 'form') is exactly the same - in this case, the 'form.pyc' module from the z3c.form package.

PYTHONPATH is an environment variable, much like PATH. You can get a list of environment variables on UNIX-like operating systems by running the 'env' command. It's available in the Properties of My Computer in Windows. PYTHONPATH is similar to PATH in another way, in that it defines a search path. However, unlike PATH (which tells the operating system which directories to look for executable files in), PYTHONPATH is used by the Python interpreter to find out where to look for modules to import.

Let's create a file called hello.py in a directory ~/pymodules:

Now, as my current directory is the pymodules directory, I can fire up Python, import my hello module, and run print_hello():

lebah:pymodules nafi$ python
Python 2.4.6 (#2, Mar 19 2009, 10:00:53)
[GCC 4.3.3] on linux2
>>> import hello
>>> hello.print_hello()
hello!

However, this doesn't work if I'm not in that pymodules directory. To fix this, we need to tell Python to look in our new pymodules directory for libraries. We do this by setting the PYTHONPATH variable:

lebah:~ nafi$ export PYTHONPATH=$PYTHONPATH:/Users/nafi/pymodules 
lebah:~ nafi$ python
Python 2.4.6 (#2, Mar 19 2009, 10:00:53)
[GCC 4.3.3] on linux2
>>> import hello
>>> hello.print_hello()
hello!

The magic is in that 'export' line, which is appending the path '/Users/nafi/pymodules' to the environment variable PYTHONPATH. Note how we append, to avoid completely overwriting any existing values.

There are a few of 'special cases' to be aware of when thinking about where Python modules may be imported from. The first is that the Python installation's site-packages directory will always be placed on the search path automatically. Secondly, as we saw in the first 'hello' example, the current module's directory is placed on the search path, allowing relative imports (more on this shortly). Finally, the current directory is also placed on the search path.

How then can you definitively find out where Python is looking for modules? Well, like this:

lebah:~ nafi$ python
Python 2.4.6 (#2, Mar 19 2009, 10:00:53)
[GCC 4.3.3] on linux2
>>> import sys
>>> from pprint import pprint as pp
>>> pp(sys.path)
['',
 '/usr/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg',
 '/usr/lib/python2.4/site-packages/Mako-0.1.8-py2.4.egg',
 '/usr/lib/python2.4/site-packages/Beaker-1.5.1-py2.4.egg',
 '/usr/lib/python2.4/site-packages/SQLAlchemy-0.3.8-py2.4.egg',
 '/usr/lib/python2.4/site-packages/MySQL_python-1.2.3c1-py2.4-linux-i686.egg',
 '/usr/lib/python2.4',
 '/usr/lib/python2.4/plat-linux2',
 '/usr/lib/python2.4/lib-tk',
 '/usr/lib/python2.4/lib-dynload',
 '/usr/local/lib/python2.4/site-packages',
 '/usr/lib/python2.4/site-packages',
 '/usr/lib/python2.4/site-packages/Numeric',
 '/usr/lib/python2.4/site-packages/PIL']

As you can see, there's quite a lot of items in there! (Note also the use of pprint, the 'pretty printer' library to format the output nicely). Note that the first item is the empty list (the current directory) - this is what made our very first 'import hello' work, as noted before. Also notice that the site-packages directory has been placed on the search path. Finally, there's a bunch of items on there that Apple set up. You can also see that I have the virtualenv package installed in my system Python.

Contrast this with the list after we've set our PYTHONPATH manually as before:

lebah:~ nafi$ export PYTHONPATH=$PYTHONPATH:/Users/nafi/pymodules
lebah:~ nafi$ cd /tmp
lebah:tmp nafi$ python
Python 2.4.6 (#2, Mar 19 2009, 10:00:53)
[GCC 4.3.3] on linux2
>>> import sys
>>> from pprint import pprint as pp
>>> pp(sys.path)
['',
 '/Library/Python/2.5/site-packages/virtualenv-1.0-py2.5.egg',
 '/private/tmp',
 '/Users/nafi/pymodules',
 '/usr/lib/python2.4/site-packages/setuptools-0.6c11-py2.4.egg',
 '/usr/lib/python2.4/site-packages/Mako-0.1.8-py2.4.egg',
 '/usr/lib/python2.4/site-packages/Beaker-1.5.1-py2.4.egg',
 '/usr/lib/python2.4/site-packages/SQLAlchemy-0.3.8-py2.4.egg',
 '/usr/lib/python2.4/site-packages/MySQL_python-1.2.3c1-py2.4-linux-i686.egg',
 '/usr/lib/python2.4',
 '/usr/lib/python2.4/plat-linux2',
 '/usr/lib/python2.4/lib-tk',
 '/usr/lib/python2.4/lib-dynload',
 '/usr/local/lib/python2.4/site-packages',
 '/usr/lib/python2.4/site-packages',
 '/usr/lib/python2.4/site-packages/Numeric',
 '/usr/lib/python2.4/site-packages/PIL']

The two important things to note are:

My changed current working directory (/tmp) has been added to the search path
/Users/nafi/pymodules has been added to the path, having been read from the PYTHONPATH environment variable.