Switching to another TTY after running startx and returning breaks Xorg
Solution 1:
Ah, the solution was relatively simple, instead of running
$ startx /usr/bin/xterm -- :1
Running
$ startx /usr/bin/xterm -- :1 vt$(tty | sed -e "s:/dev/tty::")
Will prevent X from breaking when I switch TTYs.