Command line is horribly slow in Ubuntu Server
Your framebuffer is enabled. Certain framebuffer configurations are not accelerated and will use the CPU to redraw the screen - in effect turning your CPU in to a bitblit engine when it decides to "scroll" the text. Disable it using the instructions found here, which will "tell" VirtualBox that you want "text" and not "graphics".