Cyclic module dependencies and relative imports in Python
Solution 1:
First let's start with how from import
work in python:
Well first let's look at the byte code:
>>> def foo():
... from foo import bar
>>> dis.dis(foo)
2 0 LOAD_CONST 1 (-1)
3 LOAD_CONST 2 (('bar',))
6 IMPORT_NAME 0 (foo)
9 IMPORT_FROM 1 (bar)
12 STORE_FAST 0 (bar)
15 POP_TOP
16 LOAD_CONST 0 (None)
19 RETURN_VALUE
hmm interesting :), so from foo import bar
is translated to first IMPORT_NAME foo
which equivalent to import foo
and then IMPORT_FROM bar
.
Now what IMPORT_FROM
do ?
let's see what python do when he found IMPORT_FROM
:
TARGET(IMPORT_FROM)
w = GETITEM(names, oparg);
v = TOP();
READ_TIMESTAMP(intr0);
x = import_from(v, w);
READ_TIMESTAMP(intr1);
PUSH(x);
if (x != NULL) DISPATCH();
break;
Well basically he get the the names to import from, which is in our foo()
function will be bar
, then he pop from the frame stack the value v
which is the return of the the last opcode executed which is IMPORT_NAME
, then call the function import_from()
with this two arguments :
static PyObject *
import_from(PyObject *v, PyObject *name)
{
PyObject *x;
x = PyObject_GetAttr(v, name);
if (x == NULL && PyErr_ExceptionMatches(PyExc_AttributeError)) {
PyErr_Format(PyExc_ImportError, "cannot import name %S", name);
}
return x;
}
As you can see the import_from()
function is quiet easy, it try first to get the attribute name
from the module v
, if it don't exist it raise ImportError
else return this attribute.
Now what this have to do with relative import ?
Well relative import like from . import b
are equivalent for example in the case that is in the OP question to from pkg import b
.
But how this happen ? To understand this we should take a look to the import.c
module of python specially to the function get_parent(). As you see the function is quiet long to list here but in general what it does when it see a relative import is to try to replace the dot .
with the parent package depending on the __main__
module, which is again from the OP question is the package pkg
.
Now let's put all this together and try to figure out why we end up with the behavior in the OP question.
For this it will help us if we can see what python do when doing imports, well it's our lucky day python come already with this feature which can be enabled by running it in extra verbose mode -vv
.
So using the command line: python -vv -c 'import pkg.b'
:
Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import pkg # directory pkg
# trying pkg/__init__.so
# trying pkg/__init__module.so
# trying pkg/__init__.py
# pkg/__init__.pyc matches pkg/__init__.py
import pkg # precompiled from pkg/__init__.pyc
# trying pkg/b.so
# trying pkg/bmodule.so
# trying pkg/b.py
# pkg/b.pyc matches pkg/b.py
import pkg.b # precompiled from pkg/b.pyc
# trying pkg/a.so
# trying pkg/amodule.so
# trying pkg/a.py
# pkg/a.pyc matches pkg/a.py
import pkg.a # precompiled from pkg/a.pyc
# clear[2] __name__
# clear[2] __file__
# clear[2] __package__
# clear[2] __name__
# clear[2] __file__
# clear[2] __package__
...
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "pkg/b.py", line 1, in <module>
from . import a
File "pkg/a.py", line 2, in <module>
from . import a
ImportError: cannot import name a
# clear __builtin__._
hmm what just happen before the ImportError
?
First) from . import a
in pkg/b.py
is called, which is translated as explained above to from pkg import a
, which is again in bytecode is equivalent to import pkg; getattr(pkg, 'a')
. But wait a minute a
is a module too ?!
Well here came the fun part if we have something like from module|package import module
in this case a second import will happen which is the import of the module in the import clause. So again in the OP example we need now to import pkg/a.py
, and as you know first of all we set in our sys.modules
a key for our new module which will be pkg.a
and then we continue our interpretation of the module pkg/a.py
, but before the module pkg/a.py
finish importing it call from . import b
.
Now come the Second) part, pkg/b.py
will be imported and in it turn it will first attempt to import pkg
which because pkg
is already imported so there is a key pkg
in our sys.modules
it will just return the value of that key. Then it will import b
set the pkg.b
key in sys.modules
and start the interpretation. And we arrive to this line from . import a
!
But remember pkg/a.py
is already imported which mean ('pkg.a' in sys.modules) == True
so the import will be skipped, and only the getattr(pkg, 'a')
will be called , but what will happen ? python didn't finish importing pkg/a.py
!? So only getattr(pkg, 'a')
will be called , and this will raise an AttributeError
in the import_from()
function, which will be translated to ImportError(cannot import name a)
.
DISCLAIM : This is my own effort to understand what is happening inside the interpreter, i'm far away of being an expert.
EDIt: This answer was rephrased because when i tried to read it again i remarked how my answer was bad formulated, hope now it will be more useful :)
Solution 2:
(Incedentally, the relative import doesn't matter. Using from pkg import
... reveals the same exception.)
I think what's going on here is that the difference between from foo import bar
and import foo.bar
is that in the first one, the value bar
could be module in pkg foo
or it could be a variable in a module foo
. In the second case, it's invalid for bar
to be anything but a module/package.
This would matter because if bar is known to be a module, then the contents of sys.modules
is sufficient to populate it. If it might be a variable in the foo
module, then the interpreter must actually look into the contents of foo
, but while importing foo
, that would be invalid; the actual module has not been populated yet.
In the case of a relative import, we understand from . import bar
to mean import the bar module from the package that contains the current module, but this is really just syntactic sugar, the .
name is translated to a fully qualified name and passed to __import__()
, and thus it looks for all the world like the ambigious from foo import bar
Solution 3:
As an additional Note:
I had the following module Structure:
base
+guiStuff
-gui
+databaseStuff
-db
-basescript
I wanted to be able to run my script by import base.basescript
, however this failed with an error since the gui
file had an import base.databaseStuff.db
which caused an import of base
. Since base
was only registered as __main__
it caused a second execution of the whole imports and the error above unless I used an external script above base
thus importing base
/ basescript
only once. To prevent this I put the following in my base script:
if __name__ == '__main__' or \
not '__main__' in sys.modules or \
sys.modules['__main__'].__file__ != __file__:
#imports here