Why can't I connect to my VirtualBox using Remote Desktop?

I was having the exact same problem until I installed the Extension Pack. Turns out the VRDP extension isn't installed by default even though the docs make it sound like it is.

Reading the log file for the VM showed

00:00:00.230 VRDE: VirtualBox Remote Desktop Extension is not available.


Don't connect to the IP assigned to Windows running inside VirtualBox, but to the host, or simply localhost if connecting from the same machine. This way it works like a charm.