RDP Connection to Windows 7 stays really slow
I had the same problem, and downloading and installing the latest drivers for the Network Interface Card from the manufacturer's site solved it.
Turning off the offload is apparently not a "solution," but covering up a problem because your remote connection is supposed to work fine with offloading on.
But just in case your problem cannot be solved with latest drivers, I am linking all the pages I found on the topic.
http://www.windowsreference.com/windows-vista/slow-network-copy-and-connection-drops-in-windows-7/
https://ticketing.nforce.com/index.php?/Knowledgebase/Article/View/19/0/fix-slow-remote-desktop-connection
I have the same issue. Mine sometimes locks up altogether and it is definately the same problem with complex window contents drawing slowly. Opening IE sometimes hangs it completely.
I promise to return and post an answer. All threads on this on the web go dead!
NOT Fix
For me, this was a case of a bad (default MS) graphics driver for the IBM eServer 326m I'm running. Since going to Win 2008 it needed repointing to the driver.
In the unlikely event your server also has an ATI ES1000:
http://www-947.ibm.com/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-5074558&brandind=5000008
*UPDATE 1
This actually didn't solve it. Scrolling in IE is not a good test. It seems that it doesn't like drawing the Windows 2008 MMC snap-ins and the entire connection hangs when there are too many open.
FIX (PROPER)
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
DWORD, DisableTaskOffload. Set its value to 1.
Oh yes!!!!!