Is there a way to quick compare a weapon with the one in your off hand?

Yes, you can press the Alt key to compare the weapon with the off-hand.