Absolutely! At work we have multiple Mac Pro's that have 1 monitor between them hot-swapped and while there's up to 5 seconds of delay for the video to come through, it works great.
You can hot-swap displays, but not video cards - if that makes sense.