Cannot connect to x11vnc server from other client

Solution 1:

x11vnc is a server for X displays - i.e. you can only connect to the X display running on your Mac - not the ordinary Mac desktop. This means that if you're running for example an X desktop in a window with emacs, xeyes, or whatever odd program - you can connect to that.

You cannot connect to a desktop running Finder, Safari, and similar. Those are not X programs and are not displayed on an X display. In order to connect to them, you'll need a macOS native VNC server. Such a server is bundled with macOS. You'll need to open System Preferences, then Sharing and then enable Screen Sharing.