Stopping a Python Multiprocessing BaseManager 'serve_forever' server?
I have the following setup in Python 3.6 for interprocess communication:
from multiprocessing.managers import BaseManager
class MyManager(BaseManager): pass
MyManager.register('get_instance', callable=lambda:my_instance)
m = MyManager(address=('', 50000), authkey=b'key')
s = m.get_server()
s.serve_forever()
To avoid blocking my message loop of a larger application, I used a thread to contain this setup. So s.serve_forever()
actually runs inside a run
function of a thread.
This is done according to the documentation. And the setup itself worked fine with client managers calling into the shared instance.
However, I found no way to stop this 'serve_forever' server. Since the documentation didn't mention this, I checked the source code. There's a stop_event
which supposedly I can set
to quit the loop. But it didn't work, as the accepter
daemon/thread is still running. And I can't call shutdown
on the server object because I don't have the socket object c
.
So how do I shut down this server?
ps. Using BaseManager.start()
isn't really an option for me because the server in my case shares an async message loop which can only be accessed by the starting process. BaseManager.start()
spawns a new process which no longer has access to the message loop. get_server().serve_forever()
on the other hand, runs within the calling process.
Try this in the server:
import threading
s = m.get_server()
stop_timer = threading.Timer(1, lambda:s.stop_event.set())
MyManager.register('stop', callable=lambda:stop_timer.start())
s.serve_forever()
And in the client:
MyManager.register('stop')
m.stop()
UPDATE:
I solved the timeout issue by delaying the stop_event.set()
with a threading.Timer
OLD:
However, across machines, you then run into long timeouts as the stop()
method fails to receive data, this answer talks about that - but I could not get it work:
properly disconnect multiprocessing remote manager
I'm new to the multiprocessing managers, I will update this answer if I find a better solution. (UPDATE: Found, updated)
Here is a hack after reading the multiprocessing.managers
source, stripped from _finalize_manager()
, basically creating a connection to the server and send a shutdown msg
from multiprocessing.managers import dispatch,listener_client
_Client = listener_client['pickle'][1]
# address and authkey same as when started the manager
conn = _Client(address=('127.0.0.1', 50000), authkey=b'key')
dispatch(conn, None, 'shutdown')
conn.close()