blogger web statistics/a>
Wine Scores On Trial (The Wine Trials 2011 Robin Goldstein Interview) | 1 Wine Dude

Wine Scores On Trial (The Wine Trials 2011 Robin Goldstein Interview)

Vinted on November 2, 2010 binned in book reviews, interviews
WP Greet Box icon
HiYa! If you're new here, you may want to Sign Up to get all the latest wine coolness delivered to your virtual doorstep. I've also got short, easily-digestible mini wine reviews and some educational, entertaining wine vids. If you're looking to up your wine tasting IQ, check out my book How to Taste Like a Wine Geek: A practical guide to tasting, enjoying, and learning about the world's greatest beverage. Cheers!

He’s baaaaaaaaaaaack.

Robin Goldstein, who shook the wine world’s foundations in 2008 when he won Wine Spectator’s restaurant Award of Excellence after creating a fictitious restaurant whose wine list included some of their lowest-scoring Italian wines in the past two decades (triggering one of the most heated public debates of the year in the wine world), is back.

With a vengeance.

Not that Robin’s disappeared since my last interview with him (which long-time 1WD readers will recall generated some very compelling debate – some of which, you will come to learn, influenced his latest project): he blogs regularly at BlindTaste.com, helped follow up the 2010 edition of The Wine Trials with The Beer Trials (a similar take on blind tasting ratings, applied to commercial beers), and has co-authored the new release The Wine Trials 2011.

Once again, I greedily devoured the results in my review copy of The Wine Trials, and just as in the 2010 versions, I found the them nothing short of compelling.

For starters, the consumers’ choices (for the most part) are very good bargain wines: take Dona Paula, Aveleda, Hugel, Nobilo, and Sebeka for examples.

Additionally, the blind tasting regimen for the trials (which once again pitted inexpensive wines against similar but much pricier brands) was enhanced with a bit more of the science behind them explained, and the results were similar to those in 2010: non-experts prefer less expensive wines, by a significant statistical margin.

Finally, Robin and his co-authors seem to take an even harder line in The 2011 Wine Trials against the use of point scores by leading wine publications, including taking Wine Spectator to task for how they handled the Award of Excellence kerfuffle in 2008. Whether or not you agree with their stance and their findings, the Wine Trials team at Fearless Critic Media are clearly not interested in backing down anytime soon.

Robin (once again) kindly agreed to talk to me about his controversial new release, and (once again) he has a lot to say about Wine Spectator, the 100 point wine scoring system, and how wine consumers can enhance their own perceptions (and use their own preferences to rally against snobbery in the wine world). Oh, yeah, and he talks RUSH!

Enjoy!…

1WineDude: Our interview back in January created quite a stir and some great discussion both in support and in criticism of the 2010 Wine Trials. Did the discussions and splash coming from the previous releases influence the approach to the 2011 edition?

Robin Goldstein: Well, for starters, the title of your post was certainly provocative. It’s definitely the first time I’ve been referred to as “Satan” (or “Savior”, for that matter) in a headline. Both in terms of the number of comments and the content, it was one of the most spirited debates I’ve seen since the flurry that followed release of our first edition in spring 2008. I love listening to (and participating in) debates about the purpose, value, and importance of blind tasting, as one of the central purposes of the first half of The Wine Trials was to encourage that kind of discussion.

One thing that surprised me was how some people seemed to think that the importance of context in our experience was an argument *against* blind tasting as a way of evaluating wine. If you taste blind, some people argued, you’re missing a crucial element of wine’s pleasure. (This point had also been made in the past by Eric Asimov–who almost always tastes blind these days, by the way–and others.) In fact, that’s precisely my point: context matters a lot, and you’ll get *much* more pleasure–not just a little bit–when you’re given positive information about the wine. That’s why we need to taste blind when we’re rating: so that context doesn’t bias critical quality assessments that have real impact in the marketplace.

The fact that people’s experiences of wine are so individual and subjective isn’t an argument *against* the blind tasting method; it’s an argument *for* it. Consumers are already subject to the forces of bias that the label, appellation, etc. create. The critic’s rating is supposed to act as a foil for those biases by providing an independent variable that evaluates the wine’s physical properties — an additional piece of information that we’re to use when making a buying decision. If that data point is subject to the same forces of bias that consumers are, then it’s no longer an independent variable. It loses its value in our buying equation. We should evaluate wine blind and drink it non-blind.

That discussion was part of what encouraged me to incorporate more specific scientific evidence for this point into the 2011 version of the “blind tasting manifesto” in The Wine Trials. It also helped me to better articulate my views on Robert Parker’s guide to wines under $25 in the book review that I published in the latest Journal of Wine Economics. All of this said, I think that among my friends who read the article, more people asked me about my comments on RUSH than any of my long-winded answers about perceptual bias and sensory experience. That’s either another testament to the fact that you are much better than I am at knowing what will attract readers’ immediate attention, or it’s evidence that, contrary to your argument, it’s actually you that’s Satan, and the members of RUSH once sold you their souls in exchange for the opportunity, while on Earth, to be superior to all other entities in the known universe, whether physical or intellectual. That’s also possible.

[ Editor’s note: at the advice of counsel, I decline to comment! ]

“The critic’s rating is supposed to act as a foil for those biases by providing an independent variable that evaluates the wine’s physical properties — an additional piece of information that we’re to use when making a buying decision. If that data point is subject to the same forces of bias that consumers are, then it’s no longer an independent variable. It loses its value in our buying equation. We should evaluate wine blind and drink it non-blind.

1WD: What’s different between the 2011 and 2010 book releases? Which I guess is another way of saying “sell me on buying the new edition” :-)

RG: Wine gets a new vintage every year, so a wine guide needs one, too. In a way, it’s misleading for us just to call it a “new edition”–as far as the wine reviews go, it’s an entirely new book. The 175 full-page reviews in The Wine Trials 2011 (up from 150 in 2010) are based entirely on a blind tasting process that starts over every year. Although certain facts about the winery or bottle sometimes remain constant, every single review in this book is based on the tasting panel’s assessment of a new, different vintage of each wine in consideration. Wines vary vastly from one vintage to the next, and it’s a myth that inexpensive wines are immune to this phenomenon–on the contrary, they vary considerably, sometimes for reasons that are less related to climate than to a producer’s decision to change the style of winemaking in response to consumer demand trends.

As such, and also because of stiffer competition in the ever-more-crowded under-$15 market, less than half of the winners of the 2010 Wine Trials re-qualified for the 2011 edition, and even for those that did re-qualify, the 2010 book doesn’t describe the current release. It might help you do better than chance at the wine store now, but only The Wine Trials 2011 actually reviews the wines that are now available in the marketplace.

1WD: Have you received any death threats written using cut-out letters from magazine and newspaper articles but inadvertently photocopied onto
Wine Spectator letterhead?

RG: No death threats…but I still get a steady stream of thank yous for the role played by the Osteria L’Intrepido experiment in exposing the Wine Spectator Award of Excellence program’s misrepresentation of an advertising scheme as an exercise of expert judgment. I find it fascinating that two years later, Wine Spectator has still never apologized to its readers for this fraudulent awards program. They have never explained why the magazine would define as “excellent” a wine program whose high-priced “reserve” list was a catalog of the very worst Italian wines, as measured by Wine Spectator’s own ratings.

As I wrote in my response to Thomas Matthews’ post in the comments on your Wine Trials 2010 article: “Every wine on my reserve list was priced above US$100, every one of them was rated by *Wine Spectator*, and the average score of the wines on that list, in your magazine’s own judgment, was 70.6 points. Of the 35,498 Italian wines rated in WS’s database, only 239 of them–less than 1%–scored 71 or below. That would put the average wine on Osteria L’Intrepido’s “reserve list” safely in the bottom 1% of all Italian wines rated by *Wine Spectator*. Is that your magazine’s definition of ‘excellence’?”

Perhaps unsurprisingly, Matthews did not respond to that comment. Neither he nor anyone else at Wine Spectator has ever announced any changes to the judgment process, and the program seems to be continuing just as before. I think that, in retrospect, the editorial staff’s decision to try to smear me personally instead of apologizing to its readers and fixing the awards program was a poor one. At the time Wine Spectator people calling me a “mugger” guilty of “malicious duplicity,” restaurateurs and wineries everywhere were thanking me for exposing what they viewed as an extortion scheme. It was clear from the blogosphere and other coverage of the exposé that a large swath of wine writers, industry people, and academics found my results to be salient and the Awards of Excellence shown to be fraudulent, which made these charges of “malicious duplicity” seem a bit ironic. I think that many readers might have responded better to an apology by the editors and a pledge to change the program.

All of this said, I harbor no personal grudge against Thomas Matthews or anyone else at Wine Spectator. I know that Matthews was a former academic, and, from all accounts I’ve heard, an intelligent, mild-mannered guy. And it’s clear from his writing that he’s thoughtful and articulate. But he works for an institution whose journalistic ethics are seriously suspect. When large media organizations become too concerned with profitability, the people who work there can tend to forget the financial consequences that their actions have on the “little guys”–real financial consequences. Every time Wine Spectator rewards a big, well-capitalized restaurant for its ability to afford the $250 entry fee, or agrees to review a wine that’s been submitted by an advertiser when another similar wine from a non-advertiser goes unnoticed, the little guy loses just because he’s little–or because he’s principled and doesn’t believe in this form of media racketeering. I don’t have any reason to believe that the Wine Spectator are bad people, but it seems evident that there’s a certain peripheral blindness to the magazine’s editorial vision. Through the haze of their particular corporate culture, they seem not to see what is so obvious to so many: when a well-known critic accepts money from the entities he or she is purporting to criticize, it does a disservice to readers, consumers, and businesses.

“[Wine Spectator] seem not to see what is so obvious to so many: when a well-known critic accepts money from the entities he or she is purporting to criticize, it does a disservice to readers, consumers, and businesses.”

1WD: The new release’s Appendix (with details on the statistics behind The Wine Trials) isn’t easy reading. If you had to dumb it down to one or two sentences, what does the science behind The Wine Trails tell us?

RG: The appendix describes how we conducted, and analyzed the results of, our 2008 blind-tasting experiment that tested the connection between the price of wine and how much people like the wine. We used two different methods for the analysis (known as “ordered probit” and “OLS”), both of which revealed the same small but significant effect: on the whole, amongst the population at large, the more expensive the wine, the less people liked it.

This claim was counter-intuitive enough that we felt it would be helpful for those econ or stats geeks out there who were skeptical to look at our math a bit more closely. But we’re not under the delusion that most people read the appendix. Anyone who doesn’t like looking at regression models with Greek letters should definitely feel free to skip it; the methodology of the experiment is explained in plain English (in more detail than here) in the text of the book’s first couple of chapters.

1WD: Regarding luxury wine brands, The Wine Trials 2011 states that “The Central problem is that wine pricing is almost completely arbitrary… To accept the premises of this book and pay real attention to every glass you drink is to shift the burden back onto luxury producers to make a produce that actually differentiates itself.”  Are there any wines that you feel are too damn expensive but that you drink often anyway because you just dig them?

RG: Yes, and I’d do it even more if I could afford to: old red Burgundy. There’s something magical about it. Its cost of production is vastly higher than most wine, but that factor can explain only a small fraction of the retail price. What can I say? It’s overpriced. I save it for special occasions, and I drink it less than old Italian wine, for one (very lucky) reason: in the late 1990s, when I was living in Italy as a teacher and travel writer, I bought and cellared a lot of wine from the late-’80s-to-mid-’90s that it would be impossible or prohibitively expensive for me to buy now. Every time I came back to the US to visit friends or family, I’d bring 15 or even 20 bottles back with me. (These were the days of carry-on liquids and 2,400 lire–84 euro cents–to the dollar.)

I was less confident in my wine preferences at the time than I am now, and I made some (in retrospect) poor buying decisions based more on “tre bicchieri” ratings from the Gambero Rosso wine guide than on a real understanding of my palate. But even the over-concentrated New-World-ish wines, made in a style that I’ve come to dislike in the years since, have softened enough over time that they now have some of those same beautiful, mulchy, flowery, tea-like aromatics that I love in old Burgundy, yet still retain a bit more power at the back end. Some of them are also totally oxidized, or cooked, or washed out with acetic acid, or just DOA. I haven’t been going through them fast enough, partly because I travel too much, and partly because I am possessed by a certain sense of loss every time I open one, like a reminder that I’m growing old.

But the guessing game of old wine is the fun of it, too. I think I have more of a taste for past-its-prime wine than most people I know. I have a lot of old stuff from Sicily that probably peaked 10 years ago, but I still like it — it’s dusty, barnyardy, soft, weird, and kind of awesome. I bought a lot of Sangiovese di Romagna in those days, because Emilia-Romagna was one of my Fodor’s regions for many years, and most of it is aging really well. Remembering where and when you bought such-and-such a bottle 15 years ago can be an exquisite pleasure–a temporary bout with the insane metaphysics of escapist wine geekdom. Imagine drinking blind the bottle of 1988 Gaja Barbaresco that my roommate Nikia bought for me as a going-away present! What a monumental waste that would be.

1WD: The Wine Trials doesn’t treat extravagant Bordeaux prices, or Robert Parker’s influence over Bordeaux wine prices, very fondly. In my interview with Parker, he seemed to agree at least on the Bordeaux prices (“With respect to Bordeaux, future prices have certainly gotten out of control. There was a real incentive to buy futures in a top vintage when I started 32 years ago because the prices were low, and if you bought the right wines, they obviously appreciated in value… The Bordelais have increasingly raised prices for their most prestigious estates. Except for the top wines of the great vintages, they really don’t appreciate much in price, so there is no point in buying them as futures and tying up your money two years prior to delivery”).

Why do you think Parker and WS have such price influence in the first place? Is it due to lazy marketing based on points, a lack of wine publications to challenge widespread use of their point reviews? Both? Something else entirely?

RG: It’s something else entirely–and this is one of the central issues that I tackle in the Parker book review that I mentioned above. (It’s a book review in the academic-paper sense only–that is, using a review of someone else’s book as a platform for making much larger points yourself.) I think the above quote reflects well on Parker, to whom I think I’m very fair in the review. The quote reinforces one of the central points that I make in that review–that it’s unfair to suggest that Parker has any designs on driving prices up.  Here’s a clip:

“Many of the people within the wine world that have become increasingly disgusted with so-called ‘Parkerization’—the tinkering with a style of winemaking to bring out more fruit, more oak, and more alcohol in hopes of improving a Parker score—would paint the celebrated critic as a power-hungry dictator with designs on reshaping the wine world just to please his palate and fortify his wealth. But to adopt that view is to misunderstand the fundamental human mechanics of Parker’s vast appeal. Winemakers may feel obliged to please him, but consumers are under no obligation to follow him. If you want to understand Parker, look in the mirror.

Robert Parker is no dictator. He is a storyteller. The magnetism of his prose is that of J.K. Rowling’s, too: you’re first presented with a set of familiar facts and situations, and then, slowly, you’re seduced into suspending reason and believing in the perfectly impossible. Escape into a Parker review, and for a few sentences, there you are, back in junior high, the great critic’s palate—and yours, too—cured of its nagging mortality. In this counter-factual place, there is no perceptual bias, just perception. There is no confidence interval, just confidence. Parker’s 100-point wine is Gatsby’s green light, the orgiastic ghost of taste’s future, the tongue a sudden lattice of infinite resolution, the nose a sudden instrument of preternatural whiff.”

“Parker’s 100-point wine is Gatsby’s green light, the orgiastic ghost of taste’s future, the tongue a sudden lattice of infinite resolution, the nose a sudden instrument of preternatural whiff.”

1WD: Still no mention of RUSH in the Wine Trials? Honestly, what the hell are you waiting for already? I mean, they get passed up for the Rock & Roll Hall of Fame again and now *this*?!??

RG: While I will acknowledge that the Rock & Roll Hall of Fame snub is evidence against the Joe Roberts as Satan theory, I don’t think it’s *conclusive* evidence. It’s possible that LL Cool J and/or Neil Diamond were also working side deals with you.

[ Editor’s note: to the best of my knowledge, no evidence exists linking me to LL or Neil D… ]

1WD: Do you think The Wine Trials shows that wine scores are a dying breed, “Q.E.D.” style?

RG: Tough question. I think that one change that’s already happened is that wine scores are becoming more and more associated with nouveau-riche conspicuous consumption, and that segment is becoming more and more marginalized by people who are more traditionally sophisticated about wine–MWs, sommeliers, wine writers, wine bloggers, and such. In those circles, the rejection of 100-point-scale scores has become fashionable, and new status symbols, like knowledge about obscure regions and appellations, have emerged. (Going back to Eric Asimov, I think he’s representative of this new fashion, and I don’t mean that in a bad way–it’s a much more interesting fashion!)

At the same time, neuroscience research points ever more clearly toward the problems with sensory inter-subjectivity. I’ve spent the last three days at the fascinating Society of Sensory Professionals conference in Napa, where I’ve spent some time with Tim Hanni, whose work pushes us away from the outright dismissal of people who don’t like, say, wine with very high acidity and toward the segmentation of tasters according to their palate “phenotypes.”  Tim’s work should be marginalized less and appreciated more as our understanding of interpersonal neurophysiological diversity advances, particularly as concerns sensory pathways that work with ion channels, like olfactory receptors.

[Editor’s note: read more about Tim’s work in his recent 1WD interview ].

All sounds great, right? The problem is that at the same time, wine collectors and drinkers in emerging markets, especially China, are loving Parker and WS scores more than ever–they’ve become mainstream conspicuous consumption icons. In that sense, the wine world is becoming more fragmented–between the new American-style “experts” who understand our sensory systems better than ever before, and new conspicuous-consumption markets popping up everywhere. Unfortunately, it may turn out that wine producers find that catering to the latter is far more lucrative than catering to the former, and thus will still continue to care about scores.

“Wine collectors and drinkers in emerging markets, especially China, are loving Parker and WS scores more than ever–they’ve become mainstream conspicuous consumption icons. In that sense, the wine world is becoming more fragmented–between the new American-style “experts” who understand our sensory systems better than ever before, and new conspicuous-consumption markets popping up everywhere.”

1WD: The Wine Trials has a great central message, I think, which at its heart is providing measurable data to support what many people know instinctively: they should be confident in asserting their own preferences and buying wines to suit, and not be afraid to try new things and expand their wine horizons inexpensively. However, it cautions that wine is not somehow above any other luxury good in terms of how it’s marketed, and that a lot of wine media may be a bit out of touch with the average wine consumer’s preferences. How would you recommend that those new to wine get started, and what resources they should use to get their wine knowledge?

RG: I couldn’t agree more. I’m a big advocate of blind tasting as a way of learning about wine — it trains your palate to discriminate between different styles of wine, and it makes your own preferences much clearer. Brown-bag six bottles of red, and test yourself on whether you can identify which wine is which. Throw two identical wines into the mix, and see if you can separate them out. Once you’re used to *identifying* wine, you’ll be naturally led to a clearer picture of your preferences, and you’ll start to feel less insecure about what you like and don’t like. Beyond The Wine Trials, I think Mark Oldman’s new book, Brave New World of Wine, is a good resource for beginners. I also like Jancis Robinson’s entry-level books. Beginners should also consider amateur wine courses. Tim Hanni does seminars and educational programs, and Karen MacNeil (author of “The Wine Bible,” another Workman title) is a great wine educator at the CIA Greystone in Napa. WSET courses (I’d skip the overly elementary Level 1 and start with Level 3), although they don’t focus enough on blind tasting, are another option. But there’s no substitute for tasting — a lot — at home.

[ Editor’s note: for more on Mark Oldman’s new book, see my recent interview with Mark; for more info. on the WSET and other wine certs, see the Learning Wine category ]


1WD: What’s next for Fearless Media and the Wine / Beer Trials?

RG: I’m working on an as-yet-untitled book on taste perception and the tricks our brains play on us when we eat and drink, and Fearless Critic has just released its Seattle restaurant guide, which will be followed by a San Antonio guide in November and Dallas in the spring. The Beer Trials will get an update next year, and next summer we’ll release a Liquor Trials book.

Cheers!

(images: amazon.com, blindtaste.com)

Don't miss the good vino! Sign-up now for non-SPAMmy delivery of 1WineDude updates to your Inbox.

Email address:

The Fine Print

This site is licensed under Creative Commons. Content may be used for non-commercial use only; no modifications allowed; attribution required in the form of a statement "originally published by 1WineDude" with a link back to the original posting.

Play nice! Code of Ethics and Privacy.

Contact: joe (at) 1winedude (dot) com

Google+

Labels

Vintage

Find