uwsgi vassal does not use virtualenv

I am running a python application started by an uwsgi emperor. The application is supposed to run in a specific venv, but I cannot get that to work.

The ini file for the application specifies a venv created with python3.8. Still, the application is running with python3.6 and does not have access to the python modules installed in the venv.

The emperor is running in a virtualenv with python3.6. However, the python modules installed in that virtualenv are not available in my application either.

I may be missing some configuration, but I was not able to figure out which one. I have looked at the posts listed below as well as the official uwsgi documentation.

• uWSGI and python virtual env

• uWSGI cannot load virtualenv

• Setting UWSGI to run a module in a venv

Here is my ini file:

[uwsgi]
socket = /tmp/new.embeetle.uwsgi.sock
chdir = /srv/new.embeetle/app
wsgi-file = app.py
logto = /tmp/new.embeetle.uwsgi.log
enable-threads
virtualenv = /srv/new.embeetle/app/env

I have tried adding the two lines below (without really understanding what they mean), but these had no effect:

plugin = python3.8
home = /srv/new.embeetle/app/env

Here is what I get in the log file /tmp/new.embeetle.uwsgi.log after touching the ini file:

Fri Oct 16 23:56:51 2020 - received message 1 from emperor
...gracefully killing workers...
Gracefully killing worker 1 (pid: 23275)...
worker 1 buried after 1 seconds
binary reloading uWSGI...
chdir() to /srv/uwsgi/vassals
closing all non-uwsgi socket fds > 2 (max_fd = 1024)...
found fd 3 mapped to socket 0 (/tmp/new.embeetle.uwsgi.sock)
running /srv/uwsgi/uwsgi/bin/uwsgi
*** has_emperor mode detected (fd: 6) ***
[uWSGI] getting INI configuration from new.embeetle.ini
*** Starting uWSGI 2.0.19.1 (64bit) on [Fri Oct 16 23:56:52 2020] ***
compiled with version: 7.5.0 on 16 October 2020 20:28:25
os: Linux-4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020
nodename: foechoer.sikando.com
machine: x86_64
clock source: unix
detected number of CPU cores: 6
current working directory: /srv/uwsgi/vassals
detected binary path: /srv/uwsgi/uwsgi/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
chdir() to /srv/new.embeetle/app
your processes number limit is 80112
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 inherited UNIX address /tmp/new.embeetle.uwsgi.sock fd 3
Python version: 3.6.9 (default, Jul 17 2020, 12:50:27)  [GCC 8.4.0]
PEP 405 virtualenv detected: /srv/new.embeetle/app/env
Set PythonHome to /srv/new.embeetle/app/env
Python main interpreter initialized at 0x55af44ec7410
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145840 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***

followed by the output of my application, which fails due to the wrong version of python and missing modules.

Any suggestions on how to get this to work? Do I need to provide more information?

Solution 1:

Since nobody answered my question, I did some further research and found an answer. I did not find a clear answer in any other posts, so I will post it here in case anyone else has similar problems. I had to read some of the uwsgi source code to really understand this issue. Did I miss some obvious source of documentation for uwsgi, or is the documentation lacking?

First, the virtualenv option of uwsgi has three synonyms: home, venv and pyhome have exactly the same effect. This is good to know when looking for information in other posts.

The effect of using these options is not the same as the effect of activating a virtual environment (on the command line). When you activate a virtual environment, you also change PATH so that you get the python version of that virtual environment. In uwsgi however, python code is invoked directly using the python .so/.dll which is linked into the uwsgi executable, so the python version executing the code is always the version with which uwsgi was linked, regardless of any virtualenv option given.

There is however a solution. If you install uwsgi in the target virtual environment, say at /my/virtual/env, you can add to your .ini file:

unprivileged-binary-patch = /my/virtual/env/bin/uwsgi

(or --unprivileged-binary-patch /my/virtual/env/bin/uwsgi on the command line) to replace the current uwsgi by the target environment's version, which will be linked with the desired version of python. It will also activate the virtual environment without the need for an addition virtualenv option.

Note that unprivileged-binary-patch cannot be combined with emperor-tyrant mode (for reasons unclear to me). To use unprivileged-binary-patch, disable emperor-tyrant mode and make sure that the vassal's ini file cannot be edited by untrusted users.

When trying unprivileged-binary-patch, make sure that the user running your application (usually specified using the uid and gid options) has read permission on the target environment. For example, if you use uid = www-data and gid = www-data, run

chmod -R www-data:www-data /my/virtual/env

If you don't do this, you may get this error:

Fatal Python error: Py_Initialize: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

which is not very helpful to pinpoint the problem, to say the least.