Where does "X-factor" meaning: "noteworthy, special talent or quality" come from?

Solution 1:

I reckon that the idea in widespread public consciousness that X means something not only mysterious but extraordinary dates from at least the discovery and naming of X-rays in the late 19th Century.

The letter was chosen as "signifying an unknown quantity" according to Wikipedia. See here for more information, including the original adoption of X and Y as unknown variables by Rene Descartes in the 17th Century.

Once their use in medicine became widespread, their "magical" nature captured the imagination of the general public.

See here for examples of the term being used to market various products unconnected with radiation, from prophylactics to batteries. It was a fin-de-siecle buzz-word.

EDIT -

It doesn't seem a huge leap to apply the letter in other ways. Sorry for the indirect reference:

Tolstoy suggests in War and Peace that "in warfare the strength of an army is the product of its mass and of something else, some unknown factor x."

(from Systems Intelligence in Leadership and Everyday Life , 2007 p.25)

Solution 2:

Note that "X factor" has only really meant a "noteworthy talent" since the talent show. In earlier versions of Idol, they had spoken about contestants "having an X-factor", which was basically used equivalent to "having a certain je-ne-sais-quoi".

It's an unknown factor, not a particular talent.

It's origin is related to algebra, where in a simple formula, if there is a single unknown, it usually gets designated as an x. "The X factor" therefore becomes "the one thing we do not conclusively know", but has lately been used to mean "an indescribable quality. We know it's good, we just can't put it into words."

As to the source of why the X factor as opposed to another letter; that's based on algebra.

Edit minor update to answer.