How to resolve "No 3D support is available from the host" error in VMware Fusion 11

• Revert to VMware Fusion 10.x, and open that app. It may prompt you to allow something in System Preferences and such; do so.
• Start up every VM that is not in a fully shut-down state (anything you can "Resume"). Shut each down entirely.
• Upgrade again to VMware Fusion 11.
• For each VM in which you need 3D stuff, go into its Settings and pick the Compatibility tab, then Advanced Options drop-down, then select Use Hardware Version: 16.
• Then in the Display tab, turn on Accelerate 3D Graphics (if it's not on already), and set the Shared Graphics Memory you need.
• Shut down VMware Fusion.

• Edit your main VMware Fusion preferences file. This should be "~/Library/Preferences/VMware Fusion/preferences". Add the following lines (or edit any that are present to make them conform), then save the file:

  mks.enableMTLRenderer = "0"
  mks.enableGLRenderer = "1"
  mks.gl.allowBlacklistedDrivers = "TRUE"
  

• If you have a 2013 or later Mac, and a recent version of macOS, and are running the Metal 2 3D graphics library, you can try setting the first of those (the MTL one) to "1", and could try setting the one below it (GL) to "0", but I have not tested this personally.

• If it still doesn't work, shut down the VM, and VMware Fusion, then go find your VM in the macOS filesystem, right-click or ctrl-click it, and pick "Show package contents". Open its [whatever].vmx file, and add the mks.gl.allowBlacklistedDrivers = "TRUE" line to that file as well. Save the file.
• Under no circumstances should you fire up the VM entirely (i.e., past the point where you get a login screen and can pick the option to shut the VM down) until this "No 3D support" problem is fixed. That is apt to really booger something up.
• If you just cannot get it to work at all, set the VM's Hardware Level back to 14, and its VRAM (Display > Shared Graphics Memory) back to no more than 2048 (the max in VMware 10), then downgrade the VMware app back to VMware Fusion 10.x. In that, see if there's an option to go to Hardware Level 15; if not, leave it at 14. Your VMs should now work properly again. If not, undo the two changes made to the config files (preferences and *.vmx).
• If it did work in VMware Fusion 11, then after restarting your VM fully, re-install VMware Tools from within the VM. This will upgrade the video drivers, etc., to VMware Fusion 11 versions.

What seems to have happened was that either Hardware Version 14 did support DirectX and this support was removed, or there was a Version 15 that did and that entire version was removed. Either way, any VMs under VMware Fusion 10.x that supported DirectX, under the highest Hardware Version then available, suddenly lost that ability in VMware Fusion 11.x, until upgraded to Hardware Version 16. And on top of this, VMware silently assumed everyone was using Metal 2 and had abandoned OpenGL, which of course isn't true.

On the up side, you now get up to 3GB shared VRAM instead of 2GB, so you'll be able to play some less-ancient games and use slightly more modern 3D-rendering apps. However, VMware is still lagging badly on 3D support (DirectX 10 has been obsolete since 2008, so VMware's a decade behind user needs – it's still stuck at DirectX 10.1).

Credits:

• Charles Green and Amandeep Dhindsa in this Ubuntu VMware Workstation thread at Stack Exchange: https://askubuntu.com/questions/537787/enable-3d-hw-acceleration-on-vmware-workstation-10-on-ubuntu-14-04/1095171#1095171
• Mikero in this Macs and VMware Fusion thread at VMTN: https://communities.vmware.com/thread/597366

By experiment, I found out with Prefer External GPU checked on Catalina:

If you enable on Fusion 11.5 High Performance Graphic support on Windows 10 guest with above lines added to preferences, it will use Open GL framework from dGPU. If you, however, disable High Performance Graphic support for the guest entirely, it will use eGPU, probably via Metal. Open GL with subjectively similar performance to host, perhaps bogged down by number of vCores assigned.

I ran a few WebGL demos and I can see eGPU (Vega 64 FE) utilized to 30% before hitting limits on CPU allocation (2C in my case). Performance with eGPU is significantly better than performance with 5500M dGPU where I see GPU maxed out before CPU.