Very high Xorg memory usage with Ubuntu 18.04
Solution 1:
UPDATE:
I managed to figure out that the issue was a custom wallpaper systemd service that I was running to update the wallpaper every 30 seconds using pscirlce. Apparently I misconfigured the service and it wasn't removing something from memory properly. I don't know the exact cause of the issue, but removing this fixed my issue.
Unfortunately I lost the .service files so I don't have any more information for further research.