What exactly is the cause of RAM incompatibility?

The question of RAM compatibility is extremely complicated, so fitting RAM to motherboard is chancy if the RAM is not certified. To understand the complexity, here are some of the factors you should take under consideration when fitting RAM to motherboard.

Memory technology
DDR, DDR2, DDR2, DDR3, DDR4 (they won't even fit in the same slots).

CAS latency
How many clock cycles the memory module will delay in returning data requested by the CPU.

Voltage
Higher voltage is usually faster, but low voltage generally means less stress to the CPU memory controller.

Timings
Given through a series of numbers, for example 4-4-4-8, indicate the number of clock cycles that it takes the memory to perform a certain operation. In the BIOS timings are usually detected automatically, but some BIOS allow for timings to be preset as part of over-clocking. Denoted usually by CL-tRCD-tRP-tRAS-CMD, where:

• CL: CAS Latency. The time it takes between a command having been sent to the memory and when it begins to reply to it. It is the time it takes between the processor asking for some data from the memory and then returning it.
• tRCD: RAS to CAS Delay. The time it takes between the activation of the line (RAS) and the column (CAS) where the data are stored in the matrix.
• tRP: RAS Precharge. The time it takes between disabling the access to a line of data and the beginning of the access to another line of data.
• tRAS: Active to Precharge Delay. How long the memory has to wait until the next access to the memory can be initiated.
• CMD: Command Rate. The time it takes between the memory chip having been activated and when the first command may be sent to the memory. Sometimes this value is not announced. It usually is T1 (1 clock cycle) or T2 (2 clock cycles).

RAM frequency
The signalling rate what words of data can be transferred into or out of the memory. This is part of the RAM label, for example DDR3-2400.

RAM Density
Denotes the on-chip organization. Low density modules have 100% compatibility with all systems and chipsets. High density modules only have 10% compatibility and are very slow.

RAM controller
Controllers are limited in the amount of RAM they can address, thus limiting the motherboard size of the RAM stick.

Chipset technology
The motherboard chipset affects using memory, an example is regarding multi-channel configurations.

I think we can already conclude that for a RAM stick to be compatible with a motherboard, both must agree on a very large number of parameters. Any incompatibility on any one parameter will make a given RAM unusable on the motherboard.

Some motherboards will accept RAM whose settings are in a certain range, rather than one specified amount, but others will give error or even fail to detect RAM that does not suit.

RAM bought from the manufacturer of the computer is guaranteed compatible, but may be costly. When looking for third-party RAM, it is usually hit-and-miss, since the motherboard specs are never fully published.

I have several times used the Crucial Advisor tool, or even better, the Crucial System Scanner, found on this page. In my experience their results are always correct, and the RAM that is proposed by Crucial has a very competitive price.

References :

• Wikipedia Random-access memory
• Wikipedia DDR SDRAM
• Understanding RAM Timings
• Myth: Low Density vs High Density memory modules

As in, why can't motherboard manufacturers just list the specifications of the RAM they will accommodate, and support every memory chip of that category instead of listing specific SKUs from specific brands?

Because that would give people who wanted guaranteed, tested compatibility no way to get it.

What happens electrically/physically that causes boards to fail to POST / operate with RAM that is otherwise 'compatible' with all of the specifications of RAM that is on the QVL?

It can be all kinds of things. The most obvious is voltage. Most motherboards only have a single voltage generator, which means all the RAM sticks must run at the same voltage. But it can also be details of trace length, load levels, and all kinds of details like that. We're talking about very high speed links with very precise timing requirements. All kinds of things can go wrong, though most of the time it just works.

The current setup is the best of both worlds. Those who want guaranteed compatibility can stick with the list of tested and guaranteed combinations. Those who want freedom to choose can compare specifications and draw their own conclusions.

In the old days, weird incompatibilities were not that unusual. But since the DDR2 standard, cases of incompatibility not involving one of the explanations mentioned are fairly rare. Most of those cases are likely to be components that are not quite meeting their specifications.