Database connection to postgresql refused for flask app under mod_wsgi when started by systemd
I'm having a weird problem where a flask application I am trying to run (https://github.com/pyfarm/pyfarm-master) under mod_wsgi under Apache under CentOS 7 cannot connect to a local postgresql server – but only if said apache has been started from systemd using systemctl. If I start apache manually using /usr/sbin/httpd -DFOREGROUND on the command line (the same command systemd says its using), everything works fine.
Apache, Postgresql and mod_wsgi are barely changed default installations from the CentOS repository or EPEL. Pyfarm-core and pyfarm-master have been installed with the help of pip.
The postgresql configuration is untouched from the default, apart from setting host based authentication to "trust" for everything that is localhost:
# "local" is for Unix domain socket connections only
local all all trust
# IPv4 local connections:
host all all 127.0.0.1/32 trust
# IPv6 local connections:
host all all ::1/128 trust
After that, I created a user and a database with
CREATE USER pyfarm WITH PASSWORD 'pyfarmpw';
CREATE DATABASE pyfarm WITH OWNER pyfarm;
I could log in to the database pyfarm as user pyfarm with psql afterwards.
Apache, too, is almost unchanged from the default configuration. The only thing I changed was to add a new file /etc/httpd/conf.d/pyfarm.conf with the following content:
<VirtualHost *>
ServerName pyfarm.produktion.local
ServerAlias pyfarm
SetEnv PYFARM_APP_INSTANCE True
SetEnv PYFARM_SECRET_KEY 2qgsd4hHaalvAn7
SetEnv PYFARM_CONFIG Prod
SetEnv PYFARM_DB_PREFIX ""
SetEnv PYFARM_DATABASE_URI postgresql+psycopg2://pyfarm:[email protected]/pyfarm
SetEnv TDB_DRIVER psycopg2
WSGIScriptAlias / /var/www/pyfarm/pyfarm.wsgi
DocumentRoot /var/www/pyfarm
<Directory /var/www/pyfarm>
Order allow,deny
Allow from all
</Directory>
</VirtualHost>
(The passwords and secret keys in these configuration files are not very important and will be changed anyway...)
This file references /var/www/pyfarm/pyfarm.wsgi, which looks like this:
import os
def application(environ, start_response):
for key in environ:
if key.startswith("PYFARM_") or key.startswith("TDB_"):
os.environ[key] = environ[key]
from pyfarm.master.entrypoints import app as application_
return application_(environ, start_response)
(It needs to be so complex because otherwise the app won't see the environment variables set in apache.)
Accessing any page from the application that does not actually need the database works fine after these steps, but once I try to access one that does, I get a 500 return code and the following errors in Apache's error_log:
[Thu Jul 03 18:01:20.099007 2014] [:info] [pid 18094] mod_wsgi (pid=18094): Initializing Python.
[Thu Jul 03 18:01:20.106601 2014] [:info] [pid 18094] mod_wsgi (pid=18094): Attach interpreter ''.
[Thu Jul 03 18:01:20.116951 2014] [:error] [pid 18080] \x1b[31m2014-07-03 18:01:20 ERROR - pyfarm.master - Exception on /api/v1/agents/ [GET]
[Thu Jul 03 18:01:20.116964 2014] [:error] [pid 18080] Traceback (most recent call last):
[Thu Jul 03 18:01:20.116968 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
[Thu Jul 03 18:01:20.116971 2014] [:error] [pid 18080] response = self.full_dispatch_request()
[Thu Jul 03 18:01:20.116974 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
[Thu Jul 03 18:01:20.116977 2014] [:error] [pid 18080] rv = self.handle_user_exception(e)
[Thu Jul 03 18:01:20.116989 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
[Thu Jul 03 18:01:20.116993 2014] [:error] [pid 18080] reraise(exc_type, exc_value, tb)
[Thu Jul 03 18:01:20.116996 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
[Thu Jul 03 18:01:20.116999 2014] [:error] [pid 18080] rv = self.dispatch_request()
[Thu Jul 03 18:01:20.117001 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
[Thu Jul 03 18:01:20.117005 2014] [:error] [pid 18080] return self.view_functions[rule.endpoint](**req.view_args)
[Thu Jul 03 18:01:20.117007 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/flask/views.py", line 84, in view
[Thu Jul 03 18:01:20.117010 2014] [:error] [pid 18080] return self.dispatch_request(*args, **kwargs)
[Thu Jul 03 18:01:20.117013 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/flask/views.py", line 149, in dispatch_request
[Thu Jul 03 18:01:20.117016 2014] [:error] [pid 18080] return meth(*args, **kwargs)
[Thu Jul 03 18:01:20.117019 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/pyfarm/master/api/agents.py", line 412, in get
[Thu Jul 03 18:01:20.117021 2014] [:error] [pid 18080] for host in query:
[Thu Jul 03 18:01:20.117024 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2405, in __iter__
[Thu Jul 03 18:01:20.117027 2014] [:error] [pid 18080] return self._execute_and_instances(context)
[Thu Jul 03 18:01:20.117030 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2418, in _execute_and_instances
[Thu Jul 03 18:01:20.117033 2014] [:error] [pid 18080] close_with_result=True)
[Thu Jul 03 18:01:20.117037 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2409, in _connection_from_session
[Thu Jul 03 18:01:20.117040 2014] [:error] [pid 18080] **kw)
[Thu Jul 03 18:01:20.117043 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 846, in connection
[Thu Jul 03 18:01:20.117046 2014] [:error] [pid 18080] close_with_result=close_with_result)
[Thu Jul 03 18:01:20.117048 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 850, in _connection_for_bind
[Thu Jul 03 18:01:20.117051 2014] [:error] [pid 18080] return self.transaction._connection_for_bind(engine)
[Thu Jul 03 18:01:20.117054 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 315, in _connection_for_bind
[Thu Jul 03 18:01:20.117057 2014] [:error] [pid 18080] conn = bind.contextual_connect()
[Thu Jul 03 18:01:20.117060 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1737, in contextual_connect
[Thu Jul 03 18:01:20.117063 2014] [:error] [pid 18080] self.pool.connect(),
[Thu Jul 03 18:01:20.117065 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/sqlalchemy/pool.py", line 332, in connect
[Thu Jul 03 18:01:20.117068 2014] [:error] [pid 18080] return _ConnectionFairy._checkout(self)
[Thu Jul 03 18:01:20.117071 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/sqlalchemy/pool.py", line 630, in _checkout
[Thu Jul 03 18:01:20.117074 2014] [:error] [pid 18080] fairy = _ConnectionRecord.checkout(pool)
[Thu Jul 03 18:01:20.117076 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/sqlalchemy/pool.py", line 433, in checkout
[Thu Jul 03 18:01:20.117079 2014] [:error] [pid 18080] rec = pool._do_get()
[Thu Jul 03 18:01:20.117082 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/sqlalchemy/pool.py", line 949, in _do_get
[Thu Jul 03 18:01:20.117085 2014] [:error] [pid 18080] return self._create_connection()
[Thu Jul 03 18:01:20.117091 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/sqlalchemy/pool.py", line 278, in _create_connection
[Thu Jul 03 18:01:20.117094 2014] [:error] [pid 18080] return _ConnectionRecord(self)
[Thu Jul 03 18:01:20.117097 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/sqlalchemy/pool.py", line 404, in __init__
[Thu Jul 03 18:01:20.117099 2014] [:error] [pid 18080] self.connection = self.__connect()
[Thu Jul 03 18:01:20.117102 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/sqlalchemy/pool.py", line 530, in __connect
[Thu Jul 03 18:01:20.117105 2014] [:error] [pid 18080] connection = self.__pool._creator()
[Thu Jul 03 18:01:20.117108 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 95, in connect
[Thu Jul 03 18:01:20.117111 2014] [:error] [pid 18080] connection_invalidated=invalidated
[Thu Jul 03 18:01:20.117113 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 189, in raise_from_cause
[Thu Jul 03 18:01:20.117116 2014] [:error] [pid 18080] reraise(type(exception), exception, tb=exc_tb)
[Thu Jul 03 18:01:20.117119 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 89, in connect
[Thu Jul 03 18:01:20.117122 2014] [:error] [pid 18080] return dialect.connect(*cargs, **cparams)
[Thu Jul 03 18:01:20.117125 2014] [:error] [pid 18080] File "/usr/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 376, in connect
[Thu Jul 03 18:01:20.117127 2014] [:error] [pid 18080] return self.dbapi.connect(*cargs, **cparams)
[Thu Jul 03 18:01:20.117130 2014] [:error] [pid 18080] File "/usr/lib64/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect
[Thu Jul 03 18:01:20.117133 2014] [:error] [pid 18080] conn = _connect(dsn, connection_factory=connection_factory, async=async)
[Thu Jul 03 18:01:20.117136 2014] [:error] [pid 18080] OperationalError: (OperationalError) could not connect to server: Permission denied
[Thu Jul 03 18:01:20.117138 2014] [:error] [pid 18080] \tIs the server running on host "127.0.0.1" and accepting
[Thu Jul 03 18:01:20.117141 2014] [:error] [pid 18080] \tTCP/IP connections on port 5432?
[Thu Jul 03 18:01:20.117144 2014] [:error] [pid 18080] None None\x1b[39m
However, when I stop apache with systemctl stop httpd.service and restart it again with /usr/sbin/httpd -DFOREGROUND, everything works fine, including the pages that do require database access.
When Apache is running like this, I can see the app creating and using a database connection in tcpdump -i lo. When Apache is started from systemd, no such connections are visible in tcpdump, not even a SYN-packet.
Obviously, starting Apache by hand every time is not a long term solution. Does anybody have an idea what is going wrong here? Maybe some kind of systemd-specific capability restriction that keeps some service from making outbound TCP connections?
Solution 1:
When you have SELinux enabled, the web server (and processes that run within it, such as mod_wsgi, and on EL 7, on similar server processes even run separately from the web server) are not allowed to make outgoing network connections, unless you explicitly allow them to do so, e.g. using an SELinux boolean.
To resolve the problem, set the boolean to allow network connections to a database:
setsebool -P httpd_can_network_connect_db 1