looking for something more modern than X11 "streaming" with local Vagrant Linux VMs

Solution 1:

X11 in general is not really "obsolete".

You compare it to Wayland, but Wayland has only very recently become something that can be used in a standard install of Linux for example. It still has rough edges and use cases that are not implemented at all - so many Linux distributions has X11 as the default, or if they default to Wayland they have an option to use X11 instead.

For connecting to Linux VMs with desktop environments like XFCE and LXDE, I would still recommend X11 as the best option.

You can use XQuartz, or you could use for example the xorg-server package from MacPorts, which has a more recent version of basically the same software as XQuartz).

Perhaps a better alternative would be use the X2Go (free) or NoMachine NX (commercial) software for accessing your X11 system from macOS.

Another option would be to use VNC. Many VNC clients exists for Mac, and Linux has software like x11vnc that exposes X11 software (such as XFCE and LXDE environments) over VNC. The VNC protocol works substantially different than the X protocol, and I wouldn't recommend it if you're using this on a more permanent basis, as you probably will see some degradation of graphics quality, and you would loose the ability to have individual X11 apps appear as individual windows in macOS.

Considering Wayland on macOS. You would actually not need a client as you write, it is the server you're after. However, as you're trying to access a Linux VM over a network (even though it is locally on the same machine) - this is not natively supported by Wayland. The standard Wayland setup requires the clients (individual applications) and the server (what actually handles the composition of all the applications onto the displays) to be on the same computer. There are some, very new, attempts at accessing Wayland remotely over the network - but currently they actually just work like a VNC server would. I.e. there's no advantage to using Wayland here.