How to kill ostensibly immortal process?

Solution 1:

OK, time for something even more drastic!

echo 1 >/proc/sys/kernel/panic

This instructs the kernel that when you kernel panic, to reboot the host in 1 second.

echo c >/proc/sysrq-trigger

This forces the kernel to panic. So hopefully you'll end up rebooting the host.

Solution 2:

You've started the shutdown process, so your best bet is to get that thing rebooted. If its a physical machine, then can you physically power off the server or via a service processor?

If not, and you think its these specific processes hanging up, then try and Kill all processes named mv and gzip with this:

killall mv

killall gzip

As a general disclaimer, Be careful with that killall command, as you don't want to accidentally kill some system process, so just be aware of what you're killing.

Also, see whats mounted with df -h and try umounting them. I've seen my Linux systems hang on shutdown when they won't let go of an nfs mount. I usually have to "lazy" unmount them with

umount -l /path/of/mount/point

EDIT 1

Other ways to gracefully reboot:

Does your system respond to Ctrl+Alt+Del?

If not, try the magic SysRQ key combo: Alt+SysRq+R+E+I+S+U+B. Sysrq=PrintScreen button. While holding down alt and sysrq keys, you type the REISUB key combo one after the other in order. It basically kills off all processes first, does some other cleanup, and reboots. This only works if magic sysrq is enabled in your kernel. FYI:

R: Switch the keyboard from raw mode to XLATE mode
E: Send the SIGTERM signal to all processes except init
I: Send the SIGKILL signal to all processes except init
S: Sync all mounted filesystems
U: Remount all mounted filesystems in read-only mode
B: Immediately reboot the system, without unmounting partitions or syncing