_ctypes.cpython-39-x86_64-linux-gnu.so: undefined symbol: PyFloat_Type in embedded Python loaded with dlopen

When a C-extension is imported in CPython on Linux, dlopen is used under the hood (and per default with RTLD_LOCAL-flag).

A C-extension usually needs functionality from the Python-library (libpythonX.Y.so), like for example PyFloat_Type. However, on Linux the C-extension isn't linked against the libpythonX.Y.so (the situation is different on Windows, see this or this for more details) - the missing function-definition/functionality will be provided by the python-executable.

In order to be able to do so, the executable must be linked with -Xlinker -export-dynamic, otherwise loader will not be able to use the symbols from the executable for shared objects loaded with dlopen.

Now, if the embedded python is not the executable, but a shared object, which loaded with dlopen itself, we need to ensure that its symbols are added to the dynamic table. Building this shared object with -Xlinker -export-dynamic doesn't make much sense (it is not an executable after all) but doesn't break anything - the important part, how dlopen is used.

In order to make symbols from text.python.so visible for shared objects loaded later with dlopen, it should be opened with flag RTLD_GLOBAL:

RTLD_GLOBAL The symbols defined by this shared object will be made available for symbol resolution of subsequently loaded shared objects.

i.e.

shared_lib = dlopen(path_to_so_file, RTLD_GLOBAL | RTLD_NOW);

Warning: RTLD_LAZY should not be used.

The issue with RTLD_LAZY is that C-extensions do not have dependency on the libpython (as can be seen with help of ldd), so once they are loaded and a symbol (e.g. PyFloat_Type) from libpython which is not yet resolved must be looked up, dynamic linker doesn't know that it has to look into the libpython.

On the other hand with RTLD_NOW, all symbols are resolved and are visible when a C-extension is loaded (it is the same situation as in the "usual" case when libpython is linked in during the linkage step with -Xlinker -export-dynamic) and thus there is no issue finding e.g. PyFloat_Type-symbol.

As long as the embedded python is loaded with dlopen, the main executable doesn't need to be built/linked with -Xlinker -export-dynamic.

However, if the main executable is linked against the embedded-python-shared-object, -Xlinker -export-dynamic is necessary, otherwise the python-symbols want be visible when dlopen is used during the import of c-extension.

One might ask, why aren't C-extension linked against libpython in the first place?

Due to used RTLD_LOCAL, every C-extension would have its own (uninitialized) version of Python-interpreter (as the symbols from the libpython would not be interposed) and crash as soon as used.

To make it work, dlopen should be open with RTLD_GLOBAL-flag - but this is not a sane default option.