Why can we add "a number of" before nouns?

You cannot generalize it to nouns.

You can do it with countable nouns.

You cannot put a number of before mass nouns.

A number of knowledge is wrong, even though it is a noun.

CGEL (The Cambridge Grammar of the English Language) has (at 3.3, p.349) a charmingly cumbrous term for terms like a lot, lots, a great deal, plenty, oodles and a number when these are used with of NOUN: it calls them “number-transparent quantificational nouns”. Let’s call them ntqns.

That is, although these terms are by syntactic form the heads of the noun phrases (NPs) ntqn of NOUN, the semantic head of the NP is taken to be the ‘oblique’, NOUN, the term which is the object of of, and it is this semantic head which governs agreement. An ntqn like “lot is number-transparent in that it allows the number of the oblique to percolate up to determine the number of the whole NP.”

In other words, semantics trumps syntax^†: the verb agrees with the sense rather than the form.

(Note, by the way, that ntqns are not always singular forms; the same phenomenon is observable with plural ntqns anticipating singular semantic heads:

Oodles of chocolate was provided. )

That of course merely names the phenomenon; it does not provide any sort of ‘rule’ governing what nouns can become ntqns. I'm afraid that the only ‘rule’ that's operative here is the ‘rule’ of common sense: the speaker’s commonsense notion of what she means, and the sense common to all her hearers that that’s how English actually works.

^† Do not conclude from this that ntqn of is somehow ‘transformed’ into a syntactic quantifier like many or some. CGEL presents (on pp.351-2) “compelling arguments” against understanding ntqn of as a constituent.