How to determine the cause of stuck `D` status processes?

Solution 1:

You can see stack of the process:

cat /proc/<process pid>/stack

which will give you information on what it was doing when it ended up in D-state.

echo w > /proc/sysrq-trigger; dmesg

will tell kernel to report all stack traces for D-state processes in dmesg buffer.

Processes in D-state cannot be killed. There are situations where process stays in D-state for long time but occasionally finishes I/O and is interruptible for short period of time and then goes back to the same I/O activity and ends up in D-state again. Then with

while (true); do kill -9 PID; done

there is a little chance of delivering KILL signal while process is interruptible.