Different Python versions under the same uwsgi Emperor?

I'm running an uwsgi Emperor with various Vassals that each serve a specific Python app from a different virtualenv. Since uwsgi was compiled with it's own Python 2.7 interpreter, trying to use a virtualenv with Python 3 in it produces the following error in vassal.log:

ImportError: No module named site

I believe the origin of this error is that uwsgi is using its built-in Python 2.7 interpreter, while the virtualenv directory it's running in only supports Python 3 interpreters. Indeed, when I use another uwsgi (simply by installing it with pip install uwsgi in the same virtualenv), the error dissappears. However, I'd like one Emperor to rule over several different virtualenvs, so installing a separate uwsgi in each is not an option.

According to this answer on Stackoverflow, The Right Way to solve this is to compile uwsgi with different Python interpreters as loadable modules. Before I commit to this approach, I'd like to know how I can configure my Vassals to each use another interpreter plugin.

Right now I have one Emperor that is started from my /etc/rc.local with the following settings:

[uwsgi]
uid = www-data
gid = www-data
master = true
emperor = /etc/uwsgi/vassals
daemonize = /var/log/uwsgi/emperor.log

Then I have a bunch of Vassals with ini files like this:

[uwsgi]
master = false
single-interpreter = true
socket = /tmp/%n.sock
virtualenv = /home/user/.virtualenvs/djangoproject
chdir = /home/user/djangoproject
wsgi-file = project/wsgi.py
logto = /var/log/uwsgi/%n.log

I have no problem compiling a tweaked version of uwsgi with several interpreter plugins, but I'd like to know what I have to change in my configuration to actually use these separate interpreters. Can I just say one vassal.ini:

plugin = python3.4

and in another:

plugin = python2.7

?

Please help me figure out how to combine Python 2.7 and Python 3 virtualenvs under the same uwsgi Emperor.

Solution 1:

Well, since I wasn't exactly overwhelmed by responses, here is the solution I came up with myself:

First, I created a new virtualenv with a Python 3 interpreter:

mkvirtualenv -p /usr/bin/python3 python3env

Then I installed the stock uwsgi from Pypi, which gets compiled automatically with a Python 3 interpreter:

pip install uwsgi

I created a configuration directory /etc/uwsgi-python3 that contains the emperor.ini and a subdirectory vassals, containing vassal.ini. Finally, I added the following line to /etc/rc.local

/home/user/.virtualenvs/python3env/bin/uwsgi --ini /etc/uwsgi-python3/emperor.ini

Now there's an uwsgi Emperor running that uses the Python 3 interpreter for its vassals. It doesn't interfere with another uwsgi Emperor that was already running and uses the Python 2.7 interpreter.

I know it's not optimal, because I'm not using the pluggable interpreter architecture that's explained in the documentation (thanks roberto! I don't know how I could've overlooked that). However, it runs flawlessly and I didn't have to touch my existing uwsgi installation that's serving a bunch of production apps.

Solution 2:

Under osx i made like this. I unistalled all uwsgi on my system (from brew from pip etc).

After that i downloaded under /usr/local the source code

wget https://projects.unbit.it/downloads/uwsgi-latest.tar.gz
tar zxvf uwsgi-latest.tar.gz

after

cd uwsgi-2.0.17
make PROFILE=nolang

In this way i created an executable without plugins for python.

After that i made each plugin for each version on my system:

PYTHON=python3.6 ./uwsgi --build-plugin "plugins/python python36"
PYTHON=python2.7 ./uwsgi --build-plugin "plugins/python python27"
PYTHON=python2.6 ./uwsgi --build-plugin "plugins/python python26"

Now i have 3 plugins.

In my ini files for the emperor i specified the plugins dir and the plugin version for each file

[uwsgi]
plugins-dir = /usr/local/uwsgi-2.0.17
plugin = python36

[uwsgi]
plugins-dir = /usr/local/uwsgi-2.0.17
plugin = python27

[uwsgi]
plugins-dir = /usr/local/uwsgi-2.0.17
plugin = python26

...

I symlinked the uwsgi binary in my /usr/local folder

ln -s /usr/local/uwsgi-2.0.17/uwsgi /usr/local/bin/uwsgi

And after run the emperor

uwsgi --emperor /PATH/TO/INI/FILES/FOLDER/

And voila now i can run python26, python27 and python36 project simultaneously

Solution 3:

Another possible solution is to reuse the system-wide "emperor", and only substitute the vassal with the new version. This way you don't need to invent any new folders under /etc nor launch new services to rc.local.

1. Install uwsgi via pip into a virtualenv.

2. Edit the /etc/uwsgi/apps-enabled/your-app.ini as follows:

   • Remove the plugins=... line (because pip-compiled uwsgi does not support plugins).

   • Add the line:

     unprivileged-binary-patch = /path/to/your/venv/bin/uwsgi
     

     This will force the uWSGI emperor launch your own uwsgi binary as the vassal.

3. Reload your app in the emperor service uwsgi restart your-app.

The last step somewhy reports a failure to restart the server:

 * Starting app server(s) uwsgi
   ...fail!

However, in reality, the new vassal starts fine as well as all the other apps. I did not find the time to debug this.