Why does python multiprocessing Pool.map() RAM consumption increase over time?
Setting the maxtasksperchild parameter in Pool() solves the issue.
From Python documentation:
maxtasksperchild is the number of tasks a worker process can complete before it will exit and be replaced with a fresh worker process, to enable unused resources to be freed. The default maxtasksperchild is None, which means worker processes will live as long as the pool.
The fact that settings maxtasksperchild solves the issue seems to indicate that memory is leaking (See https://stackoverflow.com/a/54975030/7097579).
But I still don't understand where or why this is happening.