There are hundreds of different large-scale wine scandals that suggest the institutions responsible for reviewing, rating, and certifying wine don’t deserve the reputations they pride themselves on. The authorities responsible for training sommeliers and informing the interested public about oenology are also bankrupt. Many wine scandals indicate credulity rather than expertise; others suggest that the skills sommeliers and their like have claimed to hone are limited at best and in so far as those skills inform the refinement of their palates, those skills and that refinement may be as good as nonexistent and perhaps little more than prejudice.
judges for the years 2005 to 2008 were systematically assessed. In each panel, the wines that judges were blindly issued contained triplicates (i.e., three identical pours) poured from the same bottles so we could see how consistent their ratings were for the exact same wines from the exact same bottles.
How consistent were they? Not very. Only 10% of judges consistently placed the exact same bottles in the same medal range (i.e., within four points in a twenty-point scale). Another 10% awarded scores that were twelve points apart or worse. The typical level of consistency was an eight point gap, or the difference between a high bronze and a low gold medal placement. Perfect consistency was only achieved 18% of the time, but almost always for wines that judges entirely rejected—obviously bad wine!
(1) there is almost no consensus among the 13 wine competitions regarding wine quality, (2) for wines receiving a Gold medal in one or more competitions, it is very likely that the same wine received no award at another, (3) the likelihood of receiving a Gold medal can be statistically explained by chance alone.
When almost 600 people were asked to blind comment on red and white wines priced between £3.49 to £29.99 per bottle, they classified ones costing £5 and less differently from the ones costing £10 and more about as well as you’d expect from a coin flip. More to the point, people do not tend to rate higher-priced wines as more enjoyable when they don’t know the prices.
In 2003, California Grapevine tracked 4,000 wines across fourteen competitions. Of those 4,000, some 1,000 received a gold medal in at least one competition and then went on to fail to place in any of the others. Why?
Is it because panels of judges vary that much? If reviewers and sommeliers are to be believed, that should not be possible—there should be standards!
Is it because different bottles vary that much? If producers are to be believed, that should also not be possible, and there’s a good case they’re right because there are production standards that should keep ‘the same’ wines similar across bottles.
Is it because of luck that other wines would outcompete them just-so? Highly unlikely!
Or to make this simpler, given 4,000 bottles and fourteen contests with 25% ever winning a gold, the implied per-contest gold rate under independence is about 2%. That predicts about 870 bottles winning exactly once and about 130 winning multiple times, and that roughly a quarter of any contest’s winners should show up with a gold somewhere else. The observed pattern—basically just 1,000 “one-off” winners with very little overlap across the other thirteen contests—is many standard deviations away from those expectations. And if judges are even moderately reliable (consistent in their judgments of different wines across contests), then repeats should be much more common than under independence. So, the data are inconsistent with any reasonable level of judge reliability and strongly suggest unreliable criteria or noise rather than robust, repeatable quality judgments