Use of the pronoun "which"
Solution 1:
Although you didn't ask specifically about which as opposed to that, your question about the ambiguity of a clause's noun phrase referent also relates to the clause's restrictiveness, if I'm reading it correctly.
(Side note on that vs. which: The most commonly followed distinction between the use of that and which relates to whether the clause they precede is restrictive or non-restrictive. The Perdue Online Writing Lab defines a restrictive clause as one in which "the information in the clause is necessary to understand the preceding noun [phrase]." A non-restrictive clause doesn't change, or isn't necessary to understand, the preceding noun phrase. Note almost all guides suggest non-restrictive phrases should be set off using commas.)
If your question (#2) is really how to make the phrase "which is smoothly parametrized by a finite number of real parameters" refer to "statistical model," that seems to be a non-restrictive phrase. In that case you would need something to the effect of:
A statistical model, which is smoothly parametrized by a finite number of real parameters, is a family of probability distributions of a random variable.
If, on the other hand, you're trying to distinguish between the referents "family of probability distributions" and "random variable" in order to define a particular type of probability distribution (which seems more likely given your comment), your noun phrase is restrictive and you'll want to clarify which antecedent noun phrase is being restricted. But restriction doesn't use which in quite the way you do.
For example, if it's the random and smoothly parameterized variable that distinguishes a particular type of probability distribution from others, you'd want something like:
A statistical model is a family of probability distributions in which [or where, or wherein] a random variable is smoothly parametrized by a finite number of real parameters.
If it's the smooth parameterization that distinguishes a type of probability distribution, you definitely need a that. For example:
A statistical model is a family of probability distributions of a random variable that is smoothly parametrized by a finite number of real parameters.
In this last case, it's still ambiguous whether the smooth parameterization refers to the unit "probability distribution of a random variable" or just to "random variable". However, if there's no such thing as a probability distribution that's not of a random variable, there's no ambiguity—but if the randomness of a variable is what makes a probability distribution a thing in the first place, it seems to me "of a random variable" may be redundant.
Caveat: I'm parsing this sentence as someone who remembers basically nothing about statistics. But unless (and really, even if) you're writing for an audience solely made up of experts, you'd want to craft the sentence in the least ambiguous (and, of course, accurate) way, so that your meaning comes across clearly to the widest possible audience.
And finally: which and that are some of the trickiest words in English to use unambiguously, so your question isn't silly at all. I'm constantly second-guessing myself on which one to use (and have been corrected by editors more than once). I just hope my lack of familiarity with statistics hasn't resulted in several completely absurd rephrasings of what's actually a straightforward sentence to statisticians.
Solution 2:
You commented that the sentence is intended to define "statistical model"; in that case, it's inadvisable to attach a clause like "which is smoothly ..." to it. That would be rather like trying to define what a king is by a statement that's only about the king of Sweden.
In the sentence as written, the "which" clause would be understood as grammatically modifying "family" (or, as others have said, "family" with the attached phrase "of probability distributions ..."). But since the sentence says that the statistical model and the family are one and the same thing, it implies that the "which" clause also applies to the model. In other words, by saying that a statistical model is a suitably parametrized family, you've automatically said that a statistical model is suitably parametrized.
So I think your sentence is fine as is, provided (as Edwin Ashworth pointed out in a comment on another answer) your readers know enough about statistics to know what makes sense. A reader who doesn't know that much might take the "which" clause to refer to "variable". (It can't refer to "distributions" because of the singular verb in "which is".) In the (unlikely) event that you need to address such readers, you might consider omitting "of a random variable".