How to profile multi-threaded C++ application on Linux?

I used to do all my Linux profiling with gprof.

However, with my multi-threaded application, it's output appears to be inconsistent.

Now, I dug this up:

http://sam.zoy.org/writings/programming/gprof.html

However, it's from a long time ago and in my gprof output, it appears my gprof is listing functions used by non-main threads.

So, my questions are:

1. In 2010, can I easily use gprof to profile multi-threaded Linux C++ applications? (Ubuntu 9.10)
2. What other tools should I look into for profiling?

Solution 1:

Edit: added another answer on poor man's profiler, which IMHO is better for multithreaded apps.

Have a look at oprofile. The profiling overhead of this tool is negligible and it supports multithreaded applications---as long as you don't want to profile mutex contention (which is a very important part of profiling multithreaded applications)

Solution 2:

Have a look at poor man's profiler. Surprisingly there are few other tools that for multithreaded applications do both CPU profiling and mutex contention profiling, and PMP does both, while not even requiring to install anything (as long as you have gdb).

Solution 3:

Have a look at Valgrind.