How to run X-Wing Alliance on modern hardware?

You'll probably need to play around with XWA Hacker, from a guy named Reimar. You can obtain it at the XWA Upgrade forums. The thread I linked to has to do with graphical glitches, which sounds like what you want.

I believe I had this working a few years ago on my machine, so I think it's possible.

Another path you could take is to try emulating it through DOSBox or virtualizing an older operating system. One of those should work well for you.

This may be a little off topic, but yesterday I successfully started X-WING in a MS-DOS 6.22 virtual machine, set up with VMWare Player. At first I tried VirtualBox but EMM386 didn't work properly. With VMWare I had no problem to configure the OS to have enough memory to start the game. I assume X-Wing Alliance would run with VMWare too. Of course you may stumble across other issues than me. Let us know how you did it in the end.