What are differences between VBoxVGA, VMSVGA and VBoxSVGA in VirtualBox?

Based on what I've found in the source code:

VBoxVGA

This emulates a graphics adapter specific to VirtualBox, the same as in previous versions (<6.0.0).

• This is the default for images created for previous versions of VirtualBox (<6.0.0) and for Windows guests before Windows 7.
• It has some form of 3D passthrough, but – if I remember correctly – uses an insecure approach that just lets the guest dump any and all commands to the host GPU.
• Using it on a Linux guest requires installing the guest additions because this adapter is not (yet) supported by the mainline Linux kernel.
• Only supports OpenGL 1.1 on 64bit Windows 10 and all Linux guests.

This option likely exists just to provide continuity – after upgrading to 6.0, all old VMs have this mode selected automatically so there's no unexpected change in behavior; you don't lose whatever acceleration you previously had.

VMSVGA

This emulates the VMware Workstation graphics adapter with the "VMware SVGA 3D" acceleration method.

• Contrary to what the manual says, this is currently the default for Linux guests.
• It is supposed to provide better performance and security than the old method.
• This is supported by the mainline Linux kernel using the SVGA driver.
• Supports OpenGL 2.1 on all Windows and Linux guests.
• It might also have the advantage of supporting old operating systems which had VMware guest additions available but not VirtualBox guest additions. (I plan to test this with Windows 9x, which is otherwise a massive pain to get even VESA graphics working with VirtualBox)

VBoxSVGA

This provides a hybrid device that works like VMSVGA (including its new 3D acceleration capabilities), but reports the same old PCI VID:PID as VBoxVGA.

• This is the default for Windows guests.
• The advantage of this mode is that you can upgrade existing VMs (which previously used VBoxVGA and had the VirtualBox Video driver installed) and they don't lose their graphics in the process – they still see the same device, until you upgrade the "guest additions" at any later time to enable 3D acceleration.
• Also, because it's still VMware SVGA emulated by VirtualBox, choosing this option and using the VirtualBox driver may still have advantages over the VMware one, e.g. allow to make use of VirtualBox-specific additional features.

"None"

This is, obviously, no emulated graphics at all. Only use it if you provide a real GPU via PCI passthrough, or if your system absolutely doesn't need a GPU.

Sources

• VirtualBox Manual chapter 3.6. Display Settings
• VirtualBox VGA Source Code

But they do tell you quite explicitly when you should prefer them:

– VBoxSVGA: The default graphics controller for new VMs that use Linux or Windows 7 or later. This graphics controller improves performance and 3D support when compared to the legacy VBoxVGA option.

• Linux or Windows >7
• improves performance and 3D support

– VBoxVGA: Use this graphics controller for legacy guest OSes. This is the default graphics controller for Windows versions before Windows 7. 

• (by inference) slightly lower performance but more compatibility with older OSes
• Old OS or Windows < 7

– VMSVGA: Use this graphics controller to emulate a VMware SVGA graphics device. 

• Emulates a VMWare device
• probably best when your VM was originally set up on VMWare and has their tools installed
• may not be as fast as VBox cards because it's trying to be compatible with something else.

– None: Does not emulate a graphics adapter type.

• you don't want a graphics adapter
  i.e. you want to run the machine headless or over SSH and don't need the overhead of it emulating graphics.
• could improve performance as it removes a layer or emulation

Performance is one difference. Testing with glxgears, I get:

• VBoxVGA: 60fps +/- 2 (very low, but moving a window around is not smooth so it seems plausible)
• VMSVGA: 1570fps +/- 50 (after a reboot, I get 1300fps +/- 200?!)
• VBoxSVGA: 970fps +/- 30 (in this mode, Cinnamon warns me that the desktop environment's graphics are running on CPU)

The guest additions were already updated from the Debian repositories before it spawned the VBoxVGA deprecation warning the first time, so I could just switch it without needing to update first. Check your version using dpkg -l | grep virtualbox.

Guest additions are version 6.0.4, mesa-utils (containing glxgears) 8.4.0-1+b1, base OS is Debian 10 (Buster) with Cinnamon.

VM has PAE/NX enabled, 3D acceleration but no 2D acceleration, 2 CPUs assigned (the host has 2 cores with 2 threads each), PV default, VT-x/AMD-V enabled, and nested paging.

I was able to auto set the resolution of the guest with wayland compositor only using VBoxSVGA. virtualbox-guest-additions couldn't auto-resize guest display with wayland when using VMSVGA.