What is /dev/mem?

Presumably it's somehow related to memory? What would

sudo cat /dev/urandom > /dev/mem

do? Trash all RAM? All non-kernel virtual memory? None of the above?

It provides access to the system's physical memory.

The mem(4) man page explains more about what /dev/mem is.

Yes -- it could cause all sorts of problems. A reboot should fix you, but bad things can happen very easily. Be careful! :-)

/dev/mem provides access to the system's physical memory, not the virtual memory. The kernels virtual address space can be accessed using /dev/kmem.

It's primarily used to access IO memory addresses related to peripheral hardware, like video adapters.

sudo cat /dev/urandom > /dev/mem won't do anything, since sudo will elevate the privilege of cat but not of the redirect. You can either do sudo su and then work in the root shell, or use
sudo dd if=/dev/urandom of=/dev/mem

/dev/mem provides access to physical memory, i.e. all of the RAM in the system, however this doesn't mean that it gives you full read/write access to RAM (see CONFIG_STRICT_DEVMEM option in this document). Also note that some regions of physical memory will have other devices like video card memory, etc. mapped onto it.

Writing blindly to /dev/mem will result in an uncertain behaviour, here is a youtube video doing the same.