History of notation: "!"
Earliest Uses of Various Mathematical Symbols will help you for the origin of math symbols.
Factorial is in the category "probability and statistics" and we can read:
The notation $n!$ was introduced by Christian Kramp (1760-1826) in 1808. In his Élémens d'arithmétique universelle (1808), Kramp wrote [in old French]:
Je me sers de la notation trés simple $n!$ pour désigner le produit de nombres décroissans depuis n jusqu'à l'unité, savoir $n(n - 1)(n - 2) ... 3\cdot 2\cdot 1$. L'emploi continuel de l'analyse combinatoire que je fais dans la plupart de mes démonstrations, a rendu cette notation indispensable.
My translation:
I used the very simple notation $n!$ to refer to the product of decreasing integers from n to 1, ie $n(n - 1)(n - 2) ... 3\cdot 2\cdot 1$. I had to do it since I've nominated this product a large number of times in my demonstrations.
As noted in Fabien's answer, the first stop for questions about notation is Cajori's A History of Mathematical Notations. Section 713 there contains an excerpt with Augustus de Morgan's observations on notation, including his opinion of the use of "!" for the factorial. He, for one, was not a fan. Here's an excerpt of the excerpt:
"Mathematical notation, like language, has grown up without much looking to, at the dictates of convenience and with the sanction of the majority. Resemblance, real or fancied, has been the first guide, and analogy has succeeded....
Among the worst of barbarisms is that of introducing symbols which are quite new in mathematical, but perfectly understood in common language. Writers have borrowed from the Germans the abbreviation $n!$ to signify $1\,.\,2\,.\,3\,.\,.\,.\,.\,(n-1)\,.\,n$, which gives their pages the appearance of expressing surprise and admiration that $2$, $3$, $4$, etc., should be found in mathematical results."
According to Ian Stewart, the symbol "!" was introduced because of printability. Before 1808
$\underline{n\big|} = n \cdot (n-1) \cdots 3 \cdot 2$
was [widely?] used to denote the factorial. Because it was hard to print [in non-computer ages], the French mathematician Christian Kramp chose "!".
Source: Professor Stewart's Hoard of Mathematical Treasures