What is ideal internet speed for remote desktop connection?

In Remote Desktop Connection (RDP), The Server does not actually sends the screen to the client screen. That process is quite expensive as you have to constantly poll the screen for any changes which can be easily +7MB of data. RDP uses special protocol. It basically uses virtual screen. It kind of interacts with the graphic card. In stead of sending screen to the actual one, it send screen content to a virtual screen.

This article and this by by Nadim Abdo of Microsoft explains it very well. It talks about all the myths about RDP. The following is an bandwidth graph from the article

Typical high bandwidth of RDP is 130 Kbps (Kilo bits per second) Typical low bandwith of RDP such as scrolling etc is about 5kbps.

Note that my problem is mostly with scrolling and switching tabs in firefox. Since I have 4Mbps speed (3.6Mbps effective), this should not be a problem. I am assuming the slow speed that I am getting is because of my server computer which at times does get slow and I know is not the fastest machine on the planet.