What's this visualization artifact and how can I avoid it?
This looks a lot like a common screen tearing artifact. Usually this is fixed by installing proper graphics drivers and then enabling VSYNC on the output. Your video player is likely to have an option for VSYNC and so does MAME.
If the problem persists for the video player, try using mpv or, if you're on Windows, use MPC.