How are python closures implemented?

I am interested in how python implements closures?

For the sake of example, consider this

def closure_test():
    x = 1
    def closure():
        nonlocal x
        x = 2
        print(x)
    return closure

closure_test()()

Here, the function closure_test has a local x which the nested function closure captures.

When I run the program, I get the following output

2

When I see the disassembly of the function closure_test,

  2           0 LOAD_CONST               1 (1)
              2 STORE_DEREF              0 (x)

  3           4 LOAD_CLOSURE             0 (x)
              6 BUILD_TUPLE              1
              8 LOAD_CONST               2 (<code object closure at 0x7f14ac3b9500, file "<string>", line 3>)
             10 LOAD_CONST               3 ('closure_test.<locals>.closure')
             12 MAKE_FUNCTION            8 (closure)
             14 STORE_FAST               0 (closure)

  7          16 LOAD_FAST                0 (closure)
             18 RETURN_VALUE

Disassembly of <code object closure at 0x7f14ac3b9500, file "<string>", line 3>:
  5           0 LOAD_CONST               1 (2)
              2 STORE_DEREF              0 (x)

  6           4 LOAD_GLOBAL              0 (print)
              6 LOAD_DEREF               0 (x)
              8 CALL_FUNCTION            1
             10 POP_TOP
             12 LOAD_CONST               0 (None)
             14 RETURN_VALUE

I see the instructions STORE_DEREF, LOAD_DEREF, MAKE_FUNCTION and LOAD_CLOSURE which I won't get if I write the whole program without functions and closures.

I think those are the instructions which are needed to use closures.

But how does Python manages this? How does it captures the variable off from the local variable table of the enclosing function? And after capturing the variable where does it live? How does the function get the access of the captured variable?

I want a complete low level understanding of how it works.

Thanks in advance.

Overview

Python doesn't directly use variables the same way one might expect coming from a statically-typed language like C or Java, rather it uses names and tags instances of objects with them

In your example, closure is simply an instance of a function with that name

It's really nonlocal here which causes LOAD_CLOSURE and BUILD_TUPLE to be used as described in When is the existence of nonlocal variables checked? and further in How to define free-variable in python? and refers to x, not the inner function named literally closure

  3           4 LOAD_CLOSURE             0 (x)

About nonlocal

For your case, nonlocal is asserting that x exists in the outer scope excluding globals at compile time, but is practically redundant because it's not used elsewhere docs
Originally I'd written that this was redundant due to redeclaration, but that's not true - nonlocal prevents re-using the name, but x simply isn't shown anywhere else so the effect isn't obvious
I've added a 3rd example with a very ugly generator to illustrate the effect

Example of use with a global (note the SyntaxError is at compile time, not at runtime!)

>>> x = 3
>>> def closure_test():
...     def closure():
...         nonlocal x
...         print(x)
...     return closure
...
  File "<stdin>", line 3
SyntaxError: no binding for nonlocal 'x' found
>>> def closure_test():
...     def closure():
...         print(x)
...     return closure
...
>>> closure_test()()
3

Examples of SyntaxErrors related to invalid locals use

>>> def closure_test():
...     def closure():
...         nonlocal x
...         x = 2
...         print(x)
...     return closure
...
  File "<stdin>", line 3
SyntaxError: no binding for nonlocal 'x' found
>>> def closure_test():
...     x = 1
...     def closure():
...         x = 2
...         nonlocal x
...         print(x)
...     return closure
...
  File "<stdin>", line 5
SyntaxError: name 'x' is assigned to before nonlocal declaration

Example which makes use of nonlocal to set the outer value
(Note this is badly-behaved because a more normal approach wrapping yield with try:finally displays before closure is actually called)

>>> def closure_test():
...     x = 1
...     print(f"x outer A: {x}")
...     def closure():
...         nonlocal x
...         x = 2
...         print(f"x inner: {x}")
...     yield closure
...     print(f"x outer B: {x}")
...
>>> list(x() for x in closure_test())
x outer A: 1
x inner: 2
x outer B: 2
[None]

Original Example without nonlocal (note absence of BUILD_TUPLE and LOAD_CLOSURE!)

>>> def closure_test():
...     x = 1
...     def closure():
...         x = 2
...         print(x)
...     return closure
...
>>>
>>> import dis
>>> dis.dis(closure_test)
  2           0 LOAD_CONST               1 (1)
              2 STORE_FAST               0 (x)

  3           4 LOAD_CONST               2 (<code object closure at 0x10d8132f0, file "<stdin>", line 3>)
              6 LOAD_CONST               3 ('closure_test.<locals>.closure')
              8 MAKE_FUNCTION            0
             10 STORE_FAST               1 (closure)

  6          12 LOAD_FAST                1 (closure)
             14 RETURN_VALUE

Disassembly of <code object closure at 0x10d8132f0, file "<stdin>", line 3>:
  4           0 LOAD_CONST               1 (2)
              2 STORE_FAST               0 (x)

  5           4 LOAD_GLOBAL              0 (print)
              6 LOAD_FAST                0 (x)
              8 CALL_FUNCTION            1
             10 POP_TOP
             12 LOAD_CONST               0 (None)
             14 RETURN_VALUE

About the ByteCode and a Simple Comparison

Reducing your example to remove all the names, it's simply

>>> import dis
>>> dis.dis(lambda: print(2))
 ​1           0 LOAD_GLOBAL              0 (print)
             ​2 LOAD_CONST               1 (2)
             ​4 CALL_FUNCTION            1
             ​6 RETURN_VALUE

The rest of the bytecode just moves the names around

• x for 1 and 2
• closure and closure_test.<locals>.closure for inner function (located at some memory address)
• print literally the print function
• None literally the None singleton

Specific DIS opcodes

• STORE_DEREF puts a value in slot i
• LOAD_DEREF retrieves a value from slot i
• MAKE_FUNCTION creates a new function on the stack and puts it in slot i
• LOAD_CLOSURE does just that, putting it on the stack at i

You can see the constants, names, and free variables with dis.show_code()

>>> dis.show_code(closure_test)
Name:              closure_test
Filename:          <stdin>
Argument count:    0
Positional-only arguments: 0
Kw-only arguments: 0
Number of locals:  1
Stack size:        3
Flags:             OPTIMIZED, NEWLOCALS
Constants:
  ​0: None
  ​1: 1
  ​2: <code object closure at 0x10db282f0, file "<stdin>", line 3>
  ​3: 'closure_test.<locals>.closure'
Variable names:
  ​0: closure
Cell variables:
  ​0: x

Digging at the closure itself

>>> dis.show_code(closure_test())  # call outer
Name:              closure
Filename:          <stdin>
Argument count:    0
Positional-only arguments: 0
Kw-only arguments: 0
Number of locals:  0
Stack size:        2
Flags:             OPTIMIZED, NEWLOCALS, NESTED
Constants:
  ​0: None
  ​1: 2
Names:
  ​0: print
Free variables:
  ​0: x
>>> dis.show_code(lambda: print(2))
Name:              <lambda>
Filename:          <stdin>
Argument count:    0
Positional-only arguments: 0
Kw-only arguments: 0
Number of locals:  0
Stack size:        2
Flags:             OPTIMIZED, NEWLOCALS, NOFREE
Constants:
  ​0: None
  ​1: 2
Names:
  ​0: print

Using Python 3.9.10

Other related questions

• Python nonlocal statement
• How does exec work with locals?