I’ve always had a bit of a mixed reaction to the reports published by the Journal of Wine Economics. On the one hand, I love the fact that serious statistical attention is being given to topics like wine awards, in the hopes that scientific examination will help reveal more about how wine and consumers interact. BUT… I’ve also had to deconstruct their lead articles to highlight what I felt to be conclusions that they draw from their analysis that I felt weren’t adequately supported by their data.
Well, now it seems that the American Association of Wine Economists has gone off the deep end.
The latest issue of the JWE (Volume 4, Issue 1, Spring 2009) contains a lead-off article by Robert T. Hodgson titled An Analysis of the Concordance Among 13 U.S. Wine Competitions. After reading the nine-page analysis, I’d go so far as to say that the AAWE’s release is bordering on being totally irresponsible. In my opinion, the science of how the statistics are applied is, at best, specious, and at worst might be downright deceitful.
Heady criticism, right? Let’s get deconstructin’!
The report examines data from 13 U.S. wine competitions in 2003. Here’s a bit of excerpt from the article abstract (emphasis is mine):
“An analysis of the number of Gold medals received in multiple competitions indicates that the probability of winning a Gold medal at one competition is stochastically independent of the probability of receiving a Gold at another competition, indicating that winning a Gold medal is greatly influenced by chance alone.”
Stochastic independence is simply another way of saying that the events are not related. For example, if you roll a 5 on a die, the event of rolling a 5 on your second role are independent. In other words, a wine winning a medal in one competition doesn’t impact what it will or won’t win in another competition. Which is exactly what you’d expect from a different competition, with different judges, and competing against different wines. The problem is that none of those other conditions are detailed in the JWE report.
Ignoring the fact that 13 competitions might not be a statistically relevant sample, not detailing the other factors that would certainly impact the outcome of the wine competitions is a seriously glaring omission.
Things get worse…