Behavior of exec function in Python 2 and Python 3
Following code gives different output in Python2
and in Python3
:
from sys import version
print(version)
def execute(a, st):
b = 42
exec("b = {}\nprint('b:', b)".format(st))
print(b)
a = 1.
execute(a, "1.E6*a")
Python2
prints:
2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)]
('b:', 1000000.0)
1000000.0
Python3
prints:
3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit (Intel)]
b: 1000000.0
42
Why does Python2
bind the variable b
inside the execute
function to the values in the string of the exec
function, while Python3
doesn't do this? How can I achieve the behavior of Python2
in Python3
? I already tried to pass dictionaries for globals and locals to exec
function in Python3
, but nothing worked so far.
--- EDIT ---
After reading Martijns answer I further analyzed this with Python3
. In following example I give the locals()
dictionay as d
to exec
, but d['b']
prints something else than just printing b
.
from sys import version
print(version)
def execute(a, st):
b = 42
d = locals()
exec("b = {}\nprint('b:', b)".format(st), globals(), d)
print(b) # This prints 42
print(d['b']) # This prints 1000000.0
print(id(d) == id(locals())) # This prints True
a = 1.
execute(a, "1.E6*a")
3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit (Intel)]
b: 1000000.0
42
1000000.0
True
The comparison of the ids of d
and locals()
shows that they are the same object. But under these conditions b
should be the same as d['b']
. What is wrong in my example?
There is a big difference between exec
in Python 2 and exec()
in Python 3. You are treating exec
as a function, but it really is a statement in Python 2.
Because of this difference, you cannot change local variables in function scope in Python 3 using exec
, even though it was possible in Python 2. Not even previously declared variables.
locals()
only reflects local variables in one direction. The following never worked in either 2 or 3:
def foo():
a = 'spam'
locals()['a'] = 'ham'
print(a) # prints 'spam'
In Python 2, using the exec
statement meant the compiler knew to switch off the local scope optimizations (switching from LOAD_FAST
to LOAD_NAME
for example, to look up variables in both the local and global scopes). With exec()
being a function, that option is no longer available and function scopes are now always optimized.
Moreover, in Python 2, the exec
statement explicitly copies all variables found in locals()
back to the function locals using PyFrame_LocalsToFast
, but only if no globals and locals parameters were supplied.
The proper work-around is to use a new namespace (a dictionary) for your exec()
call:
def execute(a, st):
namespace = {}
exec("b = {}\nprint('b:', b)".format(st), namespace)
print(namespace['b'])
The exec()
documentation is very explicit about this limitation:
Note: The default locals act as described for function
locals()
below: modifications to the default locals dictionary should not be attempted. Pass an explicit locals dictionary if you need to see effects of the code on locals after functionexec()
returns.
I'd say it's a bug of python3.
def u():
exec("a=2")
print(locals()['a'])
u()
prints "2".
def u():
exec("a=2")
a=2
print(a)
u()
prints "2".
But
def u():
exec("a=2")
print(locals()['a'])
a=2
u()
fails with
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in u
KeyError: 'a'
--- EDIT --- Another interesting behaviour:
def u():
a=1
l=locals()
exec("a=2")
print(l)
u()
def u():
a=1
l=locals()
exec("a=2")
locals()
print(l)
u()
outputs
{'l': {...}, 'a': 2}
{'l': {...}, 'a': 1}
And also
def u():
l=locals()
exec("a=2")
print(l)
print(locals())
u()
def u():
l=locals()
exec("a=2")
print(l)
print(locals())
a=1
u()
outputs
{'l': {...}, 'a': 2}
{'l': {...}, 'a': 2}
{'l': {...}, 'a': 2}
{'l': {...}}
Apparently, the action of exec
on locals is the following:
- If a variable is set within
exec
and this variable was a local variable, thenexec
modifies the internal dictionary (the one returned bylocals()
) and does not return it to its original state. A call tolocals()
updates the dictionary (as documented in section 2 of python documentation), and the value set withinexec
is forgotten. The need of callinglocals()
to update the dictionary is not a bug of python3, because it is documented, but it is not intuitive. Moreover, the fact that modifications of locals withinexec
don't change the locals of the function is a documented difference with python2 (the documentation says "Pass an explicit locals dictionary if you need to see effects of the code on locals after function exec() returns"), and I prefer the behaviour of python2. - If a variable is set within
exec
and this variable did not exist before, thenexec
modifies the internal dictionary unless the variable is set afterwards. It seems that there is a bug in the waylocals()
updates the dictionary ; this bug gives access to the value set withinexec
by callinglocals()
afterexec
.
To sum it up:
- There is no bug in Python 2 nor in Python 3
- The different behavior of
exec
stems fromexec
being a statement in Python 2, while it became a function in Python 3.
Please note:
I do not tell anything new here. This is just an assembly of the truth out there found in all the other answers and comments. All I try here is to bring light to some of the more obscure details.
The only difference between Python 2 and Python 3 is, that, indeed, exec
is able to change the local scope of the enclosing function in Python 2 (because it is a statement and can access the current local scope) and cannot do this anymore in Python 3 (because it now is a function, so runs in it's own local scope).
The irritation, however, has nothing to do with the exec
statement, it only stems from one special behavior detail:
locals()
returns something, which I want to call "a scope-wise mutable singleton which, after the call to locals()
, always only references all variables in the local scope".
Please note that the behavior of locals()
did not change between Python 2 and 3. So, this behavior together with change of how exec
works looks like being erratic, but isn't, as it just exposes some detail, which always was there.
What does "a scope-wise mutable singleton which references variables in local scope" mean?
- It is a
scope-wise singleton
, as regardless how often you calllocals()
in the same scope, the object returned is always the same.- Hence the observation, that
id(d) == id(locals())
, becaused
andlocals()
refer to the same object, the same singleton, as there can only be one (in a different scope you get a different object, but in the same scope you only see this single one).
- Hence the observation, that
- It is
mutable
, as it is a normal object, so you can alter it.-
locals()
forces all entries in the object to reference the variables in the local scope again. - If you change something in the object (via
d
), this alters the object, as it is a normal mutable object.
-
-
These changes of the singleton do not propagate back into the local scope, because all entries in the object are
references to the variables in the local scope
. So if you alter entries, these changes the singleton object, and not the contents of where "the references pointed to before you change the reference" (hence you do not alter the local variable).-
In Python, Strings and Numbers are not mutable. This means, if you assign something to an entry, you do not change the object where the entry points to, you introduce a new object and assign a reference to that to the entry. Example:
a = 1 d = locals() d['a'] = 300 # d['a']==300 locals() # d['a']==1
Besides optimization this does:
- Create new object Number(1) - which is some other singleton, BTW.
- store pointer to this Number(1) into
LOCALS['a']
(whereLOCALS
shall be the internal local scope) - If not already exist, create
SINGLETON
object - update
SINGLETON
, so it references all entries inLOCALS
- store pointer of the
SINGLETON
intoLOCALS['d']
- Create Number(300), which is not a singleton, BTW.
- store pointer to these Number(300) into
d['a']
- hence the
SINGLETON
is updated, too. - but
LOCALS
is not updated, so the local variablea
orLOCALS['a']
still is Number(1) - Now,
locals()
is called again, theSINGLETON
is updated. - As
d
refers toSINGLETON
, notLOCALS
,d
changes, too!
-
For more on this surprising detail, why
1
is a singleton while300
is not, see https://stackoverflow.com/a/306353But please do not forget: Numbers are immutable, so if you try to change a number to another value, you effectively create another object.
Conclusion:
You cannot bring back the exec
behavior of Python 2 to Python 3 (except by changing your code), as there is no way to alter the local variables outside of the program flow anymore.
However, you can bring the behavior of Python 3 to Python 2, such that you, today, can write programs, which run the same, regardless if they run with Python 3 or Python 2. This is because in (newer) Python 2 you can use exec
with function like arguments as well (in fact, those is a 2- or 3-tuple), with allows to use the same syntax with the same semantics known from Python 3:
exec "code"
(which only works in Python 2) becomes (which works for Python 2 and 3):
exec("code", globals(), locals())
But beware, that "code"
can no more alter the local enclosing scope this way. See also https://docs.python.org/2/reference/simple_stmts.html#exec
Some very last words:
The change of exec
in Python 3 is good. Because of optimization.
In Python 2 you were not able to optimize across exec
, because the state of all local variables which contained immutable contents could change unpredictably. This cannot happen anymore. Now the usual rules of function invocations apply to exec()
like to all other functions, too.
I'm afraid I can't explain it exactly, but it basically comes from the fact that b inside the function is local, and exec()
appears to assign to the global b. You'll have to declare b to be global inside the function, and inside the exec statement.
Try this:
from sys import version
print(version)
def execute1(a, st):
b = 42
exec("b = {}\nprint('b:', b)".format(st))
print(b)
def execute2(a, st):
global b
b = 42
exec("global b; b = {}\nprint('b:', b)".format(st))
print(b)
a = 1.
execute1(a, "1.E6*a")
print()
execute2(a, "1.E6*a")
print()
b = 42
exec("b = {}\nprint('b:', b)".format('1.E6*a'))
print(b)
Which gives me
3.3.0 (default, Oct 5 2012, 11:34:49)
[GCC 4.4.5]
b: 1000000.0
42
b: 1000000.0
1000000.0
b: 1000000.0
1000000.0
You can see that outside the function, the global b is automatically picked up. Inside the function, you're printing the local b.
Note that I would have thought that exec()
always uses the global b first, so that in execute2()
, you don't need to declare it inside the exec()
function. But I find that doesn't work (which is the part I can't explain exactly).