Why are modern CPUs "underclocked"?

I keep reading since years about magical overclockability of modern CPUs. It is said that CPUs (with open multiplier) which are shipped to customers with (let's say) preset 3GHz, can be overclocked "without problems" to (let's say) 4GHz.

But I can't believe that CPU manufacturers are fooling themselves. I am thinking: should a CPU always support 1GHz higher clock rate without problems, why not sell such CPU with 1GHz more preset?

Would you say, that my assumption is correct, that in fact there are lot of problems with overclocking, and for that reason, modern CPUs are preset to lower frequencies (compared to those they could handle)?

I quote from OverClocking Risks:

Usually processors in the lower range are produced with the same manufacturing process as the CPUs sold in the mid to high range. The higher rated CPUs are factory overclocked and tested, then sold for a premium. Many users will buy the cheaper processor and over clock it to nearly the same speeds of the premium CPUs. This is a great deal if and only if you are lucky enough to get a CPU that just missed the premium cut during initial factory testing. Sometimes, you will be stuck with a CPU that only over clocks slightly above its rating.

In other words, the processor that you have was factory-tested and found to be incapable of sustaining in the long-term much higher clock-rates than what the specs say. If it could, it would have been sold as another model and with a different price.

If you are really lucky, your CPU was one that just missed by a hair a higher rating, and therefore can be overclocked with safety. However, the only way to find out is to overclock in increments until you start getting strange crashes. Of course, while exploring the limits of your CPU this way, you take the risk of burning it out or of destroying your disk etc. It is for you to decide.

As harrymc pointed out, the manufacturing process for a range of processors is same. Sometimes, it is difficult for manufacturers to calculate how many processors will fall into each category in a range and that can cause problems. I remember that several years ago (Athlon XP era, if I remember correctly) AMD did some miscalculations and got large amount of top of the line processors. That was a problem, since such processors are much more expensive than mid level processors. So they took those processors, underclocked them and sold them as cheaper processors, in order to cover manufacturing costs.

Another interesting point somewhat related to this are AMD processors with strange number of cores. For example some 3 core processors were actually 4 core processors with one core faulty. Some people managed to unlock the extra core and get some performance.

Another point is processor testing. Usually manufacturers put processors under much higher stress than expected when the processor is in normal use. This way they can be sure that processors will produce correct output. While it is true that processor bugs usually affect only a small number of people, they can be very embarrassing for manufacturers. Another problem is that most users assume that processors will provide correct output when they are working and there's very little protection against processor bugs available, so it is very important that processors work correctly.

shipped to customers with (let's say) preset 3GHz, can be overclocked "without problems" to (let's say) 4GHz.

The key phrase here is "without problems". This is the part that just isn't true. While many will handle this just fine, a certain significant (large) percentage of those processors certainly will have problems, even if its not until a year or so down the road when the cooling fan starts to wear down and there's dust clogging airflow through the heat sink. It doesn't take a large percentage of returns from this issue to wipe out a manufacturers profit margins and damage it's reputation.

Another fallacy here is the assumption that performance is the primary virtue of a cpu. For large business, which makes up a huge portion of their customer base, stability, endurance, and electrical consumption all factor in as part of their TCO (total cost of ownership) equation. We're now to the point where most processors on the market will choose to deliberately underclock themselves dynamically depending on the current workload, to save on energy use and heat output. We also have chips like the Atom and Neo that are designed to underperform other processors on purpose to gain the advantage of low cost and higher battery life.

. I am thinking: should a CPU always support 1GHz higher clock rate without problems,

That's not correct.

CPUs. It is said that CPUs (with open multiplier) which are shipped to customers with (let's say) preset 3GHz, can be overclocked "without problems" to (let's say) 4GHz.

Gross simplification. Can they be overclocked ? Sure. Will be they be shipped factory overclocked ? Unlikely.

For the CPUs to run faster than their rated clock speed ( hence "overclocking") - you'll have to do certain changes. Changes such as replacing stock cooling with aftermarket cooling for better heat dissipation. CPU's are "locked" to run at frequencies which are determined after running through a set of tests without causing any problems.

that my assumption is correct, that in fact there are lot of problems with overclocking

Maybe, may be not, depends - depends on a lot of factors - ambient temparture, cooling systems, yield of the silicon, motherboard support, stable power supply being amongst the factors

Overclocking generates more heat which will cause the CPU to fail if it is excessive. Underclocking makes the CPU more resiliant to cooling problems. Most modern CPUs also have adaptive speed setting so that they can reduce the heat they generate.

I have and HP laptop that died due to poor heatsinking of the graphics chip. The excessive heat appearently causes the soldering to break.

My server runs almost constantly at its lowest CPU setting. This still leaves most of the CPU cylcles for BOINC processes.