"chunksize" parameter in multiprocessing.Pool.map

If I have a pool object with 2 processors for example:

p=multiprocessing.Pool(2)

and I want to iterate over a list of files on directory and use the map function

could someone explain what is the chunksize of this function:

p.map(func, iterable[, chunksize])

If I set the chunksize for example to 10 does that means every 10 files should be processed with one processor?

Solution 1:

Looking at the documentation for Pool.map it seems you're almost correct: the chunksize parameter will cause the iterable to be split into pieces of approximately that size, and each piece is submitted as a separate task.

So in your example, yes, map will take the first 10 (approximately), submit it as a task for a single processor... then the next 10 will be submitted as another task, and so on. Note that it doesn't mean that this will make the processors alternate every 10 files, it's quite possible that processor #1 ends up getting 1-10 AND 11-20, and processor #2 gets 21-30 and 31-40.