Why can't a 64 bit OS run a 16 bit application?

From my understanding, it's because when running in Long Mode (x64 native), the CPU itself doesn't support going into 16 bit mode. See Wikipedia. So, in order to support 16 bit mode, the NTVDM (the 16 bit layer in Windows) would have to fully emulate a 16 bit processor.

I suppose they weighed re-implementing an emulation layer vs using already extant virtualization software (VirtualPC, VirtualBox) to handle this, and it was decided to cut the VDM.

Because 64-bit handles have 32 significant bits:

Note that 64-bit Windows does not support running 16-bit Windows-based applications.
The primary reason is that handles have 32 significant bits on 64-bit Windows.
Therefore, handles cannot be truncated and passed to 16-bit applications without loss of data.

In Windows, programs pass around "handles" to the OS and vice-versa (which are numbers that the OS uses to uniquely identify a particular resource, such as a window).

To support 16-bit programs, 32-bit Windows only generates a handles that have 16 significant bits -- the 16 upper bits are ignored by the OS (even though programs are not to be taking advantage of this fact). So no program can interact with more than 2¹⁶ objects, which is actually rather low.

However, in order to improve this, 64-bit Windows increased the number of significant bits in a handle to 32. But now that means that handles cannot be passed to 16-bit programs without loss of information. So 16-bit programs cannot run on 64-bit Windows.

For Windows, it's because the x86 versions of the OS includes 16-bit emulation that allows them to run those older DOS processes. In the x64 versions, they already have to emulate x86 execution (they call it WoW64) to allow 32-bit processes to run, and I guess using Wow64 to further emulate the 16-bit emulator caused too many problems.

A handful of recognized 16-bit processes will run because the emulation is hard-coded to handle them, but the rest don't work because emulation isn't included in x64.

See "No 16-bit code" at the MSKB article: http://support.microsoft.com/kb/282423

The situation is different for Dos applications and 16 bit windows applications.

For Dos applications the problem is that virtual 8086 mode is not available under long mode. This is a CPU architecture limitation.

For 16 bit Windows appliations (which run in 16 bit protected mode) the reason is that MS wasn't prepared to do the work to implement a suitable compatibility layer. Amusingly Wine is perfectly capable of running 16 bit windows apps on 64-bit linux.