Set LD_LIBRARY_PATH before importing in python

Python uses the PYTHONPATH environment-variable to determine in which folders it should look for modules. You can play around with it by modifying sys.path, which works nicely for pure Python-Modules. But when a module uses shared object files or static libraries, it looks for those in LD_LIBRARY_PATH (on linux), but this can't be changed as easily and is platform dependent as far as I know.

The quick-fix for this problem is of course to set the environment-variable or invoke the script like LD_LIBRARY_PATH=. ./script.py, but then you'll have to set it again for every new shell you open. Also, the .so files in my case will always be in the same directory as the .py file, but may very well be moved to another absolute path, so I'd like to set them automatically every time I invoke the script.

How can I edit the path in which the Python interpreter looks for libraries platform-independently on runtime?

EDIT:

I already tried os.environ['LD_LIBRARY_PATH'] = os.getcwd(), but to no avail.

Solution 1:

UPDATE: see the EDIT below.

I would use:

import os

os.environ['LD_LIBRARY_PATH'] = os.getcwd()  # or whatever path you want

This sets the LD_LIBRARY_PATH environment variable for the duration/lifetime of the execution of the current process only.

EDIT: it looks like this needs to be set before starting Python: Changing LD_LIBRARY_PATH at runtime for ctypes

So I'd suggest going with a wrapper .sh (or .py if you insist) script. Also, as @chepner pointed out, you might want to consider installing your .so files in a standard location (within the virtualenv).

See also Setting LD_LIBRARY_PATH from inside Python

Solution 2:

My solution to this problem is to put this as the first line of a Python script (instead of the usual shebang):

exec env LD_LIBRARY_PATH=/some/path/to/lib /path/to/specific/python -x "$0" "$@"

And here is how this works:

with no shebang the current shell treats the file as a shell script,
"exec" ensures that this first line is also the last command from this file executed by the shell,
"env" is used here to set any environment variables, e.g. LD_LIBRARY_PATH,
an exact path to Python's interpreter can specified or "env" can find one in PATH,
"-x" is a Python's option which causes the first line to be ignored by the Python interpreter,
"$0" is the script name, "$@" is substituted by positional parameters.

Solution 3:

Python, when gets the values of environment variables as in os.environ[‘LD_LIBRARY_PATH’] or os.environ[‘PATH’], it copies the values, into a dictionary, from it's parent process's environment, generally bash (bash process's environment get's carried to the child process, the python running instance).

you can see this environment variable section with env command output from bash.

you can also see/read this env data from /proc/<pid>/environ, by introducing an infinite loop(while 1: pass) after modifying any environment variable.

If you see/read this variable value/data from /proc/<pid>/environ after modifying it inside the python script, you would get to see that the real variable's data doesn't get modified, though the python script shows a modified dictionary key value, updated.

What actually happens when you modify an env variable inside python script, as in os.environ['LD_LIBRARY_PATH']='/<new_location>', is that it just updates the value in local dictionary, which is not mapped to process's env variable section. Hence it won't propagate all the way back to reflect in current process's environment, because ONLY a local dictionary was modified/updated/populated.

Hence if we want to have the new environment variable to be reflected, we should overwrite the memory image of the process with new environment variable data, using execv.

Example:

new_lib = '/<new_location>'
if not new_lib in os.environ['LD_LIBRARY_PATH']:
    os.environ['LD_LIBRARY_PATH'] += ':'+new_lib
    try:
        os.execv(sys.argv[0], sys.argv)
    except Exception as e:
        sys.exit('EXCEPTION: Failed to Execute under modified environment, '+e)

import xyz
#do something else

Limitation: Ideally, python should not allow such modification of os.environ variables. But because there is no constant dictionary data type, it allows modification of the data variable. There is absolutely no use of modifying the values, as it does nothing useful to reflect in running process's real environment, unless execv is used.

Solution 4:

The solution works fine if the env is reinit

import os

os.environ['LD_LIBRARY_PATH'] = os.getcwd()  # or whatever path you want

The code need to be taken in place....

os.execv(sys.argv[0], sys.argv)

Solution 5:

Since coreutils 8.30 it is possible to use env -S to split shebang line to separate arguments:

#!/usr/bin/env -S LD_LIBRARY_PATH=/path/to/lib python options

For compatibility with older systems, you can use the fact that the shell allows all commands to be enclosed in quotes, whereas in python it will be just strings:

#!/bin/sh
"export" "LD_LIBRARY_PATH=/path/to/lib:$LD_LIBRARY_PATH"
"exec" "python3" "$0" "$@"
# Further python program
import somemodule