Or should I say "The Emperor's new score." Poking a hole in wine judging If you sub in three identical wines, will any judge notice? Carlos Avila Gonzalez/The Chronicle, 2001 If you sub in three identical wines, will any judge notice? The latest study to cause a fluster among winemakers has nothing to do with price points, labels or anything involving less household spending. It's about wine judging, specifically judges at wine competitions. A paper released last month in the Journal of Wine Economics found that judges in the California State Fair wine competition were able to score the same wine with the same rating only about 10 percent of the time, and that less than half the judging panels provided consistent judging results. Nor were judges who judged consistently one year necessarily consistent the next. The study was conducted by Robert Hodgson, a retired professor at Humboldt State University who also runs Fieldbrook Winery in Humboldt County and has served as a wine judge at the state fair. He performed the study with the cooperation of the competition's organizers. (You can read the full paper here.) Hodgson has said he wanted to delve into the inconsistencies that seem to plague wine competitions. A wine scores a gold in one; comes home empty-handed in another. Based on his own experiences, he suspected that judging dozens of wines per day might be too much to ask. As he told the Los Angeles Times: "Wine judges in the setting of a competition must make about a hundred decisions a day. It is in this environment where I think their ability is taxed beyond a reasonable level." The study makes for interesting reading, and though I haven't had a chance to discuss the results with Hodgson directly, I'd love to query him in particular about this last notion -- that it's a matter of fatigue, presumably both mental and physical. There's inevitably a portion of wine folk out there who are gazing at these findings with a big "So what?" on their faces, since it's no secret that judging results are often wildly inconsistent from competition to competition. But there is plenty of interesting fodder in the results. Let's just separate two different issues at work here. The first is whether judges are able to score the same wine with similar scores when presented with it. That goes to consistency, and though wine can show itself differently depending on when you're tasting it, how it has evolved and whether it's from the same bottle, Hodgson arranged for triplicate samples of the same wines to be served to 16 judging panels, typically all in the same flight and typically during the second flight of the day, before any palate fatigue set in. The second is to understand whether a wine is being judged on its own merits or on the subjective palate preferences of a judge. This latter one is more difficult, and more interesting. My own experience judging tells me that both factors are in play; my panel at this year's Chronicle wine competition (which uses a similar judging method) spent a lot of time discussing technical execution of the wines we tasted, in part because I once again got the pleasure to taste with Dick Peterson, and to benefit from his decades of technical expertise in winemaking. Even in panel, it was clear that one person's technical fault in a wine may be what someone else considers an extra bit of character. (That's true even for the world's top wines, though a topic for another day. Sometime we'll discuss the volatile acidity in the 1947 Cheval Blanc.) But beyond that, there's an inherent tension between the notion of wine judging as an extension of agricultural tradition -- the state fair connection very much a salient one -- versus being an aesthetic consideration of talent and quality. In the first, judging wine becomes just like judging ewes or pumpkins (though I suspect some squash and sheep connoisseurs out there take similar issue). In the second, wine becomes a matter of taste -- considered blind, perhaps, but ultimately not so different than the Academy Awards or, dare I say, the Grammies, or any award program that uses a series of expert judges, including those for journalism. Does this Merlot suck? Was "Wall-E" robbed? Discuss. There are other factors to chew on, too -- ones that are somewhat harder to study. The claim that competitions are set up to hand out as many medals as possible, for instance. Or what the real value of a gold medal might be. Hodgson gives a nod to this last by noting that wineries spend over $1 million in entry fees at four California competitions, yet many consumers barely care whether the wine they're buying won a medal. Which brings us to the ultimate question: Are wine competitions still relevant in a world flooded with 100-point scores, blogs (natch) and endless sources of opinion? What do you think? Posted By: Jon BonnéEmail) | February 04 2009 at 09:30 AM Listed Under: Events, News, Tastings, Winemakers | Comments (11) : Post Comment http://www.sfgate.com/cgi-bin/blogs/wine/detail?&entry_id=35402 :wq -- ------------------------------ * Dr. James Ellingson, jellings(a)me.umn.edu * * University of Minnesota, mobile : 651/645-0753 * * Great Lakes Brewing News, 1569 Laurel Ave., St. Paul, MN 55104 *