Should I upgrade to the "mainline" kernels?

Upgrading to a mainline kernel is usually not a good idea

Most of the basic information in this answer is from the Mainline Builds wiki

1. They are provided only for testing and are unsupported

The mainline kernels are built from the latest unmodified "mainline" Linux kernel sources.
The Ubuntu kernel team provides these only for testing and debugging purposes, to see whether issues have been fixed "upstream", i.e. by the Linux kernel developers.
They are therefore not supported and must be used at your own risk; you can report possible bugs to kernel.org via kernel-oops, or if you want a faster solution, try posting to the Linux Kernel Mailing List

2. They will often break drivers, especially Nvidia/AMD and wireless (Broadcom)

The mainline kernels do not include any Ubuntu-provided drivers or patches
This means no binary drivers for graphics, wireless, etc. are provided
If you try installing binary drivers downloaded directly from the manufacturers, there is a very good chance they will not work because the mainline headers may be incompatible.
- This will be especially true for non-LTS versions after 12.04 (12.10, 13.04, ...), because the mainline kernels are built using the last LTS toolchain (compilers, etc.), which is generally older than the toolchain on the latest non-LTS release.

3. You should only install these if you believe they may fix a critical problem you are having with the current kernel

Newer kernels sometimes contain fixes for a hardware or filesystem problem you may have. For example, the internal HD4000 graphics on the latest Intel Ivy Bridge CPUs occasionally froze, a bug which was fixed in kernels 3.3.6 and newer.
You can try installing a mainline kernel in these circumstances, and see if it helps your problem.
If it does, you should consider upgrading to the latest Ubuntu+1 kernel instead, which does have binary drivers available for it.

4. If you install a mainline or other newer kernel, you can still choose to use your old (stable) kernel by selecting it at boot-time:

Keep Shift pressed after turning on your computer.
You will see the Grub menu as shown below. User the down arrow to go to "Previous Linux Versions" and press Enter
Select the option which has a version of x.x.x-generic, NOT one that has x.x.x-yyyyyy-generic and press Enter. This is usually the first option unless you have installed multiple mainline/custom kernels.

Instead of mainline, you should consider LTS enablement stack

https://wiki.ubuntu.com/Kernel/LTSEnablementStack

Where, a newer kernel / graphics stack is backported officially, only for LTS Ubuntu releases.

In this case, Precise 12.04.5 has Trusty's kernel (3.13)

If you know of a specific fix or improvement in how the newer kernel handles your hardware, I say go for it.

I would not have upgraded to a mainline kernel due to lack of support, except that I was having some trouble with N64 game emulation running very sluggishly with lots of bugs. After trying every configuration I could find for the emulator as well as xorg, I read Here's Why Radeon Graphics Are Faster On Linux 3.12. This was enough to motivate me to test it and the results in appearance and performance were amazing with no new problems arising as of yet.

One thing to note is whether you have external modules (aka out-of-tree) installed (see Indentify out of tree modules to get an idea where these are). If you need these modules, consider whether they will build against the new kernel version. Do some research and test them yourself on the new kernel version. Worse case scenario, you can boot into the previous kernel version and uninstall the new one.

It is recommended to remove external or proprietary modules before attempting to upgrade to the mainline kernel. I did have to remove virtualbox to avoid the "Error! Bad return status for module build" during the kernel install.