How can I identify "most thread hungry" applications/services on my system?

In htop, I see over 1500 threads. How do I know their distribution related to what is running on the machine?

It is remarkable that only 2 are running; so maybe a lot of CPU time is wasted just to visit sleeping threads?

Solution 1:

Press T to see threads in a tree view, under their parent processes. You can also add the NLWP column showing the number of threads for each process through the F2 htop settings screen. (Despite its name, the column shows you the count of kernel threads and has nothing to do with userspace LWPs.)

In general, these numbers are nowhere near "unusual" or "excessively large" (even more so for a modern 12-core system). I'd only bother investigating if the total reached 6–7k.

Processes or threads which are not running but only waiting to be woken up by some specific event (e.g. waiting for the poll(2) syscall to return) do not occupy any CPU time. The scheduler does not keep checking what each task is waiting for and whether needs to be woken up – instead, the event itself knows which tasks to wake up when it completes.

For example, if you have Apache httpd with its 256 threads waiting for incoming connections on a socket, the scheduler doesn't even look at them unless there is activity on the socket.

In other words, the only thing that actually needs to spend CPU time per process or thread is htop itself (as it does indeed rescan each task's status every second).