What happens 'internally' on a unstable/overclocked/undervolted CPU?

All logic in digital electronics is basically made up of transistors. The zero and one states have certain thresholds that define what the are. For example if your transistors are run at 1.8V, there may be a definition that 0-0.75V is logical zero and 1.05-1.8V is logical one. There may also be different values for input and output, but let's simplify.

Any value that is between these is unknown, since we don't want there to be a case when the values can be either one with unknown probability.

Now, when you undervolt the transistors you are reducing the value area, especially in the logical one part. The transistors might give you 0.9V for logical one and that is not accepted, therefore resulting in an error state. Depends on the design and circuit how this manifests. It could be in a memory chip and might cause a zero to be stored. It could be in a calculation instruction inside the CPU and cause false result. Or it could be a bit that is never used, or happily understood as logical one and you never even notice it was there.