Go For The Gold - I
After a stay in the hospital with bad food and no wine, I'm Back and Okay. Does a Gold medal equal a 100 point rating? 95 points? 90 points? Does a Gold medal mean anything? Does a 100 point rating mean anything? Does a wine blogger's opinion mean anything? Okay! Enough! Basta! Bob Tovey recently sent along a piece in the L.A.Times by Jerry Hirsch about Robert Hodgson, winemaker and economist who did "An Analysis of the Concordance Among 13 U.S. Wine Competitions," published in the Journal of Wine Economics.
"An analysis of over 4000 wines entered in 13 Wine Competitions shows little concordance among venues in awarding Gold Medals. Of the 2440 wines entered in more than three competitions, 47% received Gold medals, but 84% of these same wines also received no award in another competition. Thus, many wines that are viewed as extraordinarily good in some competitions are viewed as below average at others. An analysis of the number of Gold Medals received in multiple competitions indicates that the probability of winning a Gold Medal at one competition is stochastically independent of the probability of winning a Gold Medal at another competition, indicating that winning a Gold Medal is greatly influenced by chance. (J. Wine Econ., Vol4. Issue 1. Spring 2009, p. 1)"
What does this mean? There is no correlation among scores at different competitions? A Gold Medal winner at one competition doesn't mean much and and is unlikely to win another Gold. Is this a case of " garbage in, garbage out"? Winemakers and wineries seem to believe in and invest in Gold Medals, but apparently wine consumers don't ( Thach, Wine Business Monthly, 2008) . It is understandable that winemakers would seek validation for their efforts. Everybody seeks recognition, but it appears that the consumer may be smarter than the CEOs and marketing types at wineries and wine conglomerates. Of course, most consumers choose a wine based on a pretty label or maybe a wine rating. I used to hope that a medal at least meant that the wine was not bad, because at least a small group of wine tasters liked it but now I wonder.
How does a wine get a Gold Medal and what does it mean? The methodology varies tremendously among wine competitions? At least most competitions are single blind, that is, the tasters are told the varietal they are tasting or the region. Sometimes the name of the specific wine is revealed before the final rating. Usual a flight is tasted by a group of tasters, and while the identity of the wine is unknown they discuss the wine and a certain consensus among a certain number of tasters is required to give an award. Thus, the process becomes politicized and pressure can be brought to bear to award a medal. In fact, it is rumored that judges, who don't award enough medals don't get asked back. Now, it is difficult to taste dozens or hundreds of wines in a day and there is a lot of palate fatigue. So typically, tasters are encouraged to refresh their palates with red meat, olives, cheese and bread. This is all very well, except that most of these victuals enhance the taste of wine. The French have a saying, "Buy on bread, sell on cheese." Why do you think so many wineries offer cheese, salami, etc., in their tasting rooms? Altruism? State law?
Then, there is the question of statistics? Ratings are what mathematicians call ordinal numbers which are not really amenable to the usual calculations done with normal or nominal numbers. Are the ratings reliable? Do raters rate the wines the same way or are they all over the place? How are tasters chosen? By qualifications? By reputation? By who's available? I doubt that most tasters are reliable in the statistical sense. That is, are their rating reproducible and is there any relationship among their ratings? Then there is validity. Even if the tasters are consistent among themselves and each and everyone of them, do their ratings validly relate to anything else like whether you like the wine?
So much for competitions. What about ratings and points? Is 90 better than 89? Is Parker reliable and valid, that is consistent and accurate? Wine Spectator? Others? Tune in!
"An analysis of over 4000 wines entered in 13 Wine Competitions shows little concordance among venues in awarding Gold Medals. Of the 2440 wines entered in more than three competitions, 47% received Gold medals, but 84% of these same wines also received no award in another competition. Thus, many wines that are viewed as extraordinarily good in some competitions are viewed as below average at others. An analysis of the number of Gold Medals received in multiple competitions indicates that the probability of winning a Gold Medal at one competition is stochastically independent of the probability of winning a Gold Medal at another competition, indicating that winning a Gold Medal is greatly influenced by chance. (J. Wine Econ., Vol4. Issue 1. Spring 2009, p. 1)"
What does this mean? There is no correlation among scores at different competitions? A Gold Medal winner at one competition doesn't mean much and and is unlikely to win another Gold. Is this a case of " garbage in, garbage out"? Winemakers and wineries seem to believe in and invest in Gold Medals, but apparently wine consumers don't ( Thach, Wine Business Monthly, 2008) . It is understandable that winemakers would seek validation for their efforts. Everybody seeks recognition, but it appears that the consumer may be smarter than the CEOs and marketing types at wineries and wine conglomerates. Of course, most consumers choose a wine based on a pretty label or maybe a wine rating. I used to hope that a medal at least meant that the wine was not bad, because at least a small group of wine tasters liked it but now I wonder.
How does a wine get a Gold Medal and what does it mean? The methodology varies tremendously among wine competitions? At least most competitions are single blind, that is, the tasters are told the varietal they are tasting or the region. Sometimes the name of the specific wine is revealed before the final rating. Usual a flight is tasted by a group of tasters, and while the identity of the wine is unknown they discuss the wine and a certain consensus among a certain number of tasters is required to give an award. Thus, the process becomes politicized and pressure can be brought to bear to award a medal. In fact, it is rumored that judges, who don't award enough medals don't get asked back. Now, it is difficult to taste dozens or hundreds of wines in a day and there is a lot of palate fatigue. So typically, tasters are encouraged to refresh their palates with red meat, olives, cheese and bread. This is all very well, except that most of these victuals enhance the taste of wine. The French have a saying, "Buy on bread, sell on cheese." Why do you think so many wineries offer cheese, salami, etc., in their tasting rooms? Altruism? State law?
Then, there is the question of statistics? Ratings are what mathematicians call ordinal numbers which are not really amenable to the usual calculations done with normal or nominal numbers. Are the ratings reliable? Do raters rate the wines the same way or are they all over the place? How are tasters chosen? By qualifications? By reputation? By who's available? I doubt that most tasters are reliable in the statistical sense. That is, are their rating reproducible and is there any relationship among their ratings? Then there is validity. Even if the tasters are consistent among themselves and each and everyone of them, do their ratings validly relate to anything else like whether you like the wine?
So much for competitions. What about ratings and points? Is 90 better than 89? Is Parker reliable and valid, that is consistent and accurate? Wine Spectator? Others? Tune in!
posted by Gene Stein, Ph.D. @ 5:18 AM
0 comments
