Why is rm -rf ~/.Trash/* much faster than clicking "empty trash"?
When you use rm -rf ~/.Trash/*
you're directly using a system call that unlinks these files from the filesystem, freeing the space allocated by them. If you use a GUI tool instead (I suppose you are referring to it when you say "click empty trash") it is probably saving or moving these files somewhere else so you can possibly undelete them, which is not directly possible using the shell command line. That should be the reason it is faster, but it wouldn't explain the GUI tool taking "forever".
How many times faster do you perceive the command line erase comparing to the GUI tool?