"Experiment" #4 - Lies, Damned Lies, and Wine Judgings

I recently had the distinct pleasure of reading about someone doing the same thing I'm doing here, except much better, in the Wall Street Journal, of all places. An illuminating essay about how wine ratings are largely bullshit, A Hint of Hype, A Taste of Illusion, focuses on a series of studies about the fallibility of so-called wine experts.

Below are some of the more salient and startling points from this essay, for your reading pleasure:

• Researchers secretly added red food coloring to a dry white wine to simulate a sauterne, sherry, rosé, Bordeaux and burgundy and asked experts to rate the sweetness of the various wines. The ratings corresponded to the type of wine the tasters thought they were drinking.
  
• Two identical Bordeaux wines were served to a panel of experts, one in an expensive Grand Cru bottle, the other in what looked like a cheap table wine bottle. The tasters showed a significant preference for the Grand Cru bottle, describing it as "excellent", while calling the contents of the table wine bottle "unbalanced" and "flat".
• An experiment where the same wines were poured 3 separate times for a panel of wine judges showed that the judges' ratings of the same wine varied wildly between tastings. Only about one in 10 regularly rated the same wine within a range of 2 points.
• After some data manipulation of the results of wine competitions over several years, it was determined that the odds of one wine winning a gold medal in a competition were at about 9% for any given wine. In other words, a wine that won the gold one year was just as likely to win a gold the next year as any other wine in the competition.

I didn't find these results too surprising. If you're an established wine critic, you have a reputation to uphold, and thus your rating heuristic is going to be a little bit different from that of an average wine drinker. You cannot be too extreme or drastic in your reviews because you will probably lose credibility when your peers rate the same wine highly. Chances are, most of the wines you will be trying will be at a high enough level where saying that they have colorful overtones of dog shit or a mild hint of battery acid on the nose will only paint you as an ignorant newb. And, given that most of the tastings you do will not turn out to be experiments in disguise, your heuristic probably works well more often that not.

But what about the average wine drinker who does not have a reputation to protect? Can we tell a cheap table wine from a more expensive label? And so, inspired by and riding on the coattails of the WSJ essay, I bring you "experiment" #4, the wine battle.

In this experiment, I procured a bottle of Charles Shaw, aka Two Buck Chuck, Cabernet (2008, $2) and pitted it against a Cabernet from the Napa Valley winery of St. Supéry (2003, $20). The vintner describes the St. Supéry Cab as:
"Luxurious, rich and concentrated with aromas of cherry, currant and blueberry these are just the start of this 2003 Cabernet Sauvignon. The harmonious flavors of black cherry, anise and plush cassis explode upon tasting. This is a classic Napa Valley Cabernet which is wonderful young but has everything it needs to age beautifully. Enjoy now through 2011."

Not surprisingly, I couldn't find a vintner's description of Chuck online, and I regrettably threw away the bottle, which I think had some writing on the back, during a drunken stupor brought on by chugging everything that was left after the conclusion of the experiment.

PROCEDURE:
The experimental design was very simple. I marked the base of a bunch of paper cups with the letters P or Q. P was the St. Supéry. Q was the Chuck... because Q stands for Quality. Then I gave each subject a P cup (ha ha) and a Q cup and asked which letter of wine they preferred.

I also tasted both wines, having someone else pour them, of course.

RESULTS:
Out of a total of 14 subjects, the results came up dead even. Seven people preferred Supéry, and seven preferred Chuck. I asked the subjects if they were frequent wine drinkers, and though I didn't do any formal statistical analysis, it seemed like both groups consisted of both connoisseurs and casual drinkers.

WHAT I THOUGHT THE WINES TASTED LIKE:
I drink wine a lot. I don't know too much about it, but I think I know enough to wow people who don't know jack shit. (The same goes for how well I speak any number of foreign languages.) I found the Supéry to be almost undrinkable, ie I would drink it but only to get wasted. It was very tannic and had a definite battery acid/vodka nose. The Chuck, on the contrary, was quite pleasant. It didn't have too much going on, smell or taste-wise, but it was delightfully inoffensive.

WHAT OTHER PEOPLE HAD TO SAY:
One of my favorite comments came from a guy who was a member of the St. Supéry wine club. After the tasting, but before being told which wines he had tasted, he said that he preferred P (the Supéry). I asked if it was by a huge margin. He said that it wasn't and that, frankly, he didn't really care for either one. When, later on, I told him which wines he had tasted, he started laughing and, after telling me he was in the wine club, he admitted that, had he known he was tasting Supéry, he would have been much more enthusiastic about it.

Another guy, after tasting the wine, correctly placed the P wine (Supéry) in Napa. I was suitably impressed... until he informed me that the Q wine (Chuck) must have come from Sonoma.

CONCLUSION:
The fact that a $2 wine and a $20 wine tied is compelling enough. It is interesting to note, as well, that of the people who preferred the more expensive wine, only ONE person preferred it by any kind of margin. Most people, regardless of which wine they preferred, said that neither was that great.

So, where do we go from here? By no stretch of the imagination does this study mean that you should always buy Two Buck Chuck while self-righteously snickering at the morons buying a $12 bottle of wine one aisle down. However, I do think that you shouldn't buy a bottle of wine just because it's highly rated or expensive... unless you're trying to impress someone because then all logic and reason goes out the window anyway. I guess, if there's any overarching take-home message to be had, it's this. Don't trust anyone but yourself when it comes to wine. And don't trust yourself either, unless you're blindfolded. And it's probably a great idea to buy one expensive-looking bottle of wine, pour out what is probably shit-swill, and then buy a case of Chuck and funnel it into the bottle whenever you have company.