Is Edwin Jaynes correct about statistics?
I've recently been reading Edwin Jaynes's book, Probability Theory: The Logic of Science, and I was struck by Jaynes's hostile view of what he dubs "orthodox statistics." He repeatedly claims that much of statistics is a giant mess, and argues that many commonplace statistical techniques are ad hoc devices which lack a strong theoretical footing. He blames historical giants like Karl Pearson and Ronald Fisher for the field's sorry state, and champions Bayesian methods as a healthier alternative.
From my personal perspective, his arguments make a lot of sense. However, there are a number of factors that prevent me from taking his criticism of statistics at face value. Despite the book's being published in 2003, the majority of its contents were written in the mid 20th century, making it a little dated. The field of statistics is reasonably young, and I'm willing to bet it's changed significantly since he levied his critiques.
Furthermore, I'm skeptical of how he paints statistics as having these giant methodological rifts between "frequentists" and "Bayesians." From my experience with other fields, serious methodological disagreement between professionals is almost nonexistent, and where it does exist it is often exaggerated. I'm also always skeptical of anyone who asserts that an entire field is corrupt--scientists and mathematicians are pretty intelligent people, and it's hard to believe that they were as clueless during Jaynes's lifetime as he claims.
Questions:
Can anyone tell me if Jaynes's criticisms of statistics were valid in the mid 20th century, and furthermore whether they are applicable to statistics in the present day? For example, do serious statisticians still refuse to assign probabilities to different hypotheses, merely because those probabilities don't correspond to any actual "frequencies?"
Are "frequentists" and "Bayesians" actual factions with strong disagreements about statistics, or is the conflict exaggerated?
Solution 1:
I can't speak for all statisticians, but here's my view of these two questions:
First, I think that (1) and (2) are mostly the same question, each is asking about the relevance of the Bayesian-Frequentist debate.
In general, I'd say the debate has shifted significantly, notably via input from Deborah Mayo and Andrew Gelman (among many others...but I read these two the most), away from philosophizing about the nature of probability as applied to statistics to the idea of how we judge/check our statistical models. For example, a Bayesian claiming a 95% probability (posterior) of their model being correct is really not saying much, since we can't place that 95% in an objective context. Why should 95% be reassuring? What does 95% mean in terms of observable outcomes?
If you say, "Well, I accept my model as valid when the posterior probability is over 95%", then does that imply 5% of your accepted models are wrong (or something to that effect)? If so, you are implicitly using a frequentist calibration of your model. If you deny the validity of the above statement, then what can you offer to support confidence in your 95%? Why can't we say its 60%? How would you show that I am wrong in a non-circular way?
However, I digress a bit. My main point is not to criticize subjective probability, but to say that it's become largely irrelevant to modern statistical practice. Here's why:
Regardless of your philosophy, you need to validate your models!
No one, no matter how good their reputation or beautifully coherent their logic should get a free pass. You need to show your model is a reasonably good approximation of reality. To do this, you're going to need to compare your output to data, and if your outputs are probabilities, you can't get away from calculating frequency distributions/histograms/density estimates.
As far as I'm concerned, the philosophical part of Bayesian/Frequentist debate is largely over, with frequentism winning out as the preferred method of calibration or checking of models. There simply aren't many alternatives to showing "hey, my model is pretty good!" without appealing to some sense of frequency estimates. Logical coherence alone is not a valid way to empirical truth.
However, I'd also say that Bayesian models are strongly on the ascendance. Modeling your parameters as "random effects" from some meta model basically means you are entertaining a richer space of models, with the added benefit that your estimates may have less variance and lower overall error (if your priors are not completely off the mark).
So, yes, I'd say Jayenes was a product of (a) his ego and (b) his times. Both Bayesian and non-Bayesian (I'm not going to say Frequentist) approaches have advanced markedly, with quite a bit of convergence at the philosophical level, but proliferation at the model/methodological level. For example we have the bootstrap, empirical likelihood, MCMC, EM, AIC/BIC, diffuse priors/informative prior/non-informative priors, etc...so many ways to skin the same cat (but at least we all seem to agree on what it means to have skinned it!).
For (2), I actually think I partially answered this in my answer to (1). If you read modern Bayesians like Gelman, you'll see a strong push towards model checking and verification by comparing observed frequencies vs modeled frequencies or similar measures (e.g., he proposes a version of the P-value in his book for predictive model checking). The massive decrease in computing cost means you can also check a lot more measures of "misfit" than when Fisher and Pearson came up with their P-values and significance tests. Gelman and others seem to support the idea of holistic checks, not blind adherence to a single statistics. I agree with this, and I'm happy to see how objective verification has returned to Bayesian modeling.
Anyway, my 2 cents.