Tons of general protection faults
Solution 1:
It seems it was indeed a memory issue. Memtest gave some errors on one of the four modules, and after removing this module it hasn't crashed since. Thanks for all the suggestions.
Solution 2:
I would use the older kernel (2.6.x) first before jumping onto the hardware. If Ubuntu 10 kernel is still installed, reboot the machine and run the server with the old kernel. Ubuntu 12 is 3.x? while Ubuntu 10 is 2.6.x.
If 2.6 linux-image is not available, you can add the lucid repository to /etc/apt/sources.list.d, do "apt-get update", and "aptitude versions linux-image" and install 2.6 kernel.
If the situation does not change with the older kernel, you can conclude that it's not the kernel but hardware. If the situation improves, it's probably either buggy driver or kernel. AFAIK, using older kernel does not affect your system. I've installed 2.6 kernel from Lucid and used it with Precise in order to avoid the Intel graphics problem and my machine worked just fine.
Only annoying part of it was that I had to hack the "grub" (/etc/grub.d/) in order for the 2.6 kernel to show up in the grub menu so that I can edit /etc/default/grub to choose the 2.6 kernel as default, and I had to edit the /etc/grub.d/ multiple times since every time I take the linux-image update, it reverts the hacked files in /etc/grub.d/*. (Maybe someone else knows how to use the older version of kernel as default in grub.)