How can I profile a multithread program in Python?

I'm developing an inherently multithreaded module in Python, and I'd like to find out where it's spending its time. cProfile only seems to profile the main thread. Is there any way of profiling all threads involved in the calculation?

Solution 1:

Please see yappi (Yet Another Python Profiler).

Solution 2:

Instead of running one cProfile, you could run separate cProfile instance in each thread, then combine the stats. Stats.add() does this automatically.

Solution 3:

If you're okay with doing a bit of extra work, you can write your own profiling class that implements profile(self, frame, event, arg). That gets called whenever a function is called, and you can fairly easily set up a structure to gather statistics from that.

You can then use threading.setprofile to register that function on every thread. When the function is called you can use threading.currentThread() to see which it's running on. More information (and ready-to-run recipe) here:

http://code.activestate.com/recipes/465831/

http://docs.python.org/library/threading.html#threading.setprofile

Solution 4:

Given that your different threads' main functions differ, you can use the very helpful profile_func() decorator from here.

Solution 5:

Check out mtprof from the Dask project:

https://github.com/dask/mtprof

It's a drop-in replacement for cProfile that, if your threads are launched in the usual way and complete before your main thread, will roll-up their stats into the same reporting stats. Worked like a charm for me.