Why is my Windows 8 virtual machine not using my NVIDIA card?

A virtual machine does not get direct access to your hardware. The hypervisor emulates hardware, what hardware that is emulated depends on which hypervisor you are using.

The Vmware Compatibility Guide includes details about exactly what Guest Operating systems are supported. Windows 8 is not on the list.

For supported guest operating systems VMware does provide a set of drivers for officially support Operating systems that greatly improve the performance of this emulation. With Windows 7 the graphics performance is good enough for Aero to work. Windows 8 is not supported, and I would bet is not likely to be supported for a while.

To my knowledge, most VMs use a special guest driver to emulate graphics; in some cases, passing the host GPU's capabilities via the aforementioned guest driver.

VirtualBox's user manual describes how VirtualBox (in particular) emulates 2D/3D acceleration.

https://www.virtualbox.org/manual/ch04.html#guestadd-video