Xorg High (100%) CPU on Xubuntu 17.04

I installed Xubuntu 17.04 a couple days ago on my XPS 13 9360. Since then every once in a while xorg uses 100% CPU for a while. Primarily this occurs right after boot or resuming from suspend. I've read through https://wiki.ubuntu.com/X/Troubleshooting/HighCPU but none of those situations seem to match mine.

I've confirmed that it's not using the software rendering instead of the intel video card. The system is still usable but a little sluggish with fans constantly spinning. I did not have this issue with Ubuntu 16.10.

Nothing jumps out at me in Xorg.0.log. How I can figure out what Xorg is trying to do?

Xorg.0.log: https://gist.github.com/anonymous/826812e3808aa22f531cbfe19521f8b3

After noticing that xfsettingsd was also constantly using cpu along with xorg I stumbled across https://bugs.launchpad.net/ubuntu/+source/linux/+bug/998310 which indicated that ~/.Xmodmap is the culprit.

I removed Xmodmap, rebooted and am happy to report no excessive CPU usage.