Great post. Now, please do the same to beer, coffee, and bourbon.
A steakhouse in College Station did/does a March madness bourbon elimination. In the years I went there, the legendary Van Pappy Bourbon never got out of the first round.
Similar posts could definitely be done for other foodstuffs. I only selected wine for this because it was personally relevant and it has such an esteemed reputation for discernment.
I think the situation with beer is different, since, from my experience, there is a greater difference in flavour between beers than there is between wines. I believe it is easier to detect a significant difference in flavour between a porter or stout and a pale ale than it is to tell the difference between red wine and white wine. Where I would agree with there being problems with "beer experts" is more that they are often following fashions and can be influenced by successful PR campaigns.
I have heard a story, possibly apocryphal, about how the American pale ale trend started with one brewery failing to create a proper Indian pale ale. Rather than admit that they had made a mistake and created something that had a flavour one would more expect from foraging in a florist than visiting a pub, they instead embarked on a massive PR campaign that managed to convince some pretentious "experts" that their over-hopped failure was in fact good. This then resulted in the past 15 or so years of mediocre pale ales selling for around £10 a pint and snobbish idiots praising garbage that tastes just as awful as some of the mass produced lagers that are, rightfully, regarded as tasting like tramp's urine.
That is fair enough, but part of the reason I can detect more flavour differences between beers is because there are more varied ingredients that go into different types of beers. Chocolate stouts often feature chocolate malt or cacao nibs whilst milk stouts are brewed with lactose. This is in stark contrast to wine, which can only be made from fermented grape juice. This isn't to say that a connoisseur of wine or of a particular type of beer may be able to better detect flavour differences within their preferred tipple to a greater degree than a novice, but beer in general does have a greater number of potential ingredients.
A relative used to tell me this all the time growing up - about how with blind testing the cheapest wines can taste as good as the worst.
On retirement, the same fellow dragged out every bottle of wine he owned, googled it to see the ratings, and classified each wine he owned 1, 2, 3 depending on how 'good' it was, and that determines which guests get which.
If you give me a random beer and a random wine, I'm going to prefer the wine in every single contest, even if I might err in judging their triplicate pours.
But the difference between a California Chardonnay and a white burgundy is much less that the difference between red wine and white wine dyed red. So it does seem inconsistent that judges have repeatedly (if unknowingly) detected the first difference but not the second.
All it proves — and as I noted in a prior comment this study is often overstated, including by you — is that visual stimuli impact perception of flavor. Which we already knew.
A real study would blindfold the judges and ask them point blank if it was red or white. Even then you have temperature issues.
Fun article. Doesn't the very fact that people rely on tasters and "experts" to tell them what's good reek of subjectivity? If people were just relying on what tastes good to them, popularity would dictate which wines sold best, and presumably weed out the expensive frauds. That seems to be pretty close to what happens with beer, although there is an element of snob appeal in the niche craft brew industry.
There are good reasons for consumers to rely on advice: they very rarely get to taste a wine before buying. Even if you live in a wine region and visit wineries you only taste wine from those producers. There are wine fairs where you can taste and buy a large variety of wines, but only a tiny proportion of wine drinkers go there.
What you don't mention, but which seems relevant, is that even though wine tasters claim to not get drunk at these tastings they almost certainly end up drinking some wine. There's no doubt that drunk people are bad at actual taste issues, a fact known at least 2000 years ago since it's mentioned in the bible. Chances are that in the larger wine events where testers have to rate dozens of wines they simply lose their ability to discriminate properly, assuming it was there in the first place.
Regarding wine scandals, there have been many wine scandals detected by wine sellers selling more wine than they could possibly make from the vineyards the grapes allegedly come from. This included the monks on the island just off Cannes who have a tiny vineyard but sold more wine than several wineries on the mainland...
In some cases in the EU the prime sucker for the scam seems to be the EU's agricultural subsidies rather than consumers, and in many of those cases it seems likely that the wine simply did not exist, but there were others I recall where the same wine was counted as two different wines through careful use of unlabeled bottles.
Being a sommelier, wine critic, and judge I have some first hand experience in this. The TL;DR is that yes, you are right, there is a certain level of BS when of comes to the entirety of your article. i.e. the industry not what you wrote. But there are valid reasons for the these variations and anomalies.
And now......the rest of the story:
Judging anything (food, beverage, art, sports, etc.) at its core is a personal and oftentimes unique experience. When looking at reviews or scores or evaluations of these things we can see large variations of opinion. Especially in the first three categories listed.
With sports, athletes tend to hit the required criteria for that sport. Did they nail the landing, did they do enough turns, did they do all the required movements, etc. And at competitions such as the Olympics, the differences between skill levels for most of the athletes is pretty minimal. So the scores may be pretty close to each other unless the athlete made a major mistake. But even then you'll have some judges give scores much higher or lower than the rest because of politics or preferences.
We've seen plenty of competitions across many categories where experts will have differences of opinion. We also see where they will agree. In most competitions with a panel of judges the top and bottom scores will be eliminated. This is done to prevent any particular judge from skewing a result too much.
With things like F&B and art, so-called experts will view these things differently than the general public. And not every expert is created equally. Some grew up in that world and have intimate knowledge whereas others had to start from scratch. Some are trained either academically or through work experience in that field. Phrases like "beauty is in the eye of the beholder," and "the best wine is the one you like," and "opinions are like...., everyone has one" demonstrate that we each evaluate things from our perspective.
Bringing it back to wine and competitions. A typical competition receives hundreds or thousands of entries. Those entries are then categorized. It's up to the competition to decide those categories based on the entries. Sometimes not enough entries come in to be very specific and sometimes they are. For instance, one flight could be United States Pinot Noir $15-18 whereas another could be Other White $10-30. For each flight the judges are given some kind of information to go on to evaluate the wines.
When it comes to the actual scoring, there will be 4-6 or maybe more criteria to use to evaluate. But basically it will be Nose, Taste, Balance, Overall. Sometimes Color, Structure, or some other criteria. You'll assess a numerical value to each (usually 0-5) and then add them up. That total will correspond to a specific medal, or no medal.
Once the flight is finished the head judge will then compile the rest of the table's scores and then come up with an average. As mentioned before, a low score and/or a high score will usually be thrown out. So with a table of 5 judges, including the head judge, you could have only 3 scores determine the final score.
When scoring these wines, judges are supposed to judge them within their class. So if you're judging those Cali Pinot Noirs $15-18, then you're judging them on if they meet certain quality levels for wines in that class, NOT compared to the best Pinot Noirs in the world. So, yes, a $15 or even $10 Pinot could get a Double Gold....in its category. Also, the wines are not really competing. They are not supposed to be evaluated against each other. They are evaluated on their own. So calling these things competitions is really incorrect. With that said, it's hard not to compare wines within a flight, but the goal isn't to have a winner among the flight. They could all be Double Golds or all suck.
FTR, this is basically the same thing that happens at publications. Wines are (supposedly) evaluated blind as far as knowing producer, but are put into appropriate categories. The publication's panel will evaluate them based on that category and not versus all wines of that overall category.
During this phase, we will typically have a general discussion of the flight. So this might be a time where each of the judges may make a case for a wine to be score higher or lower than the rest. Also the head judge may ask the table if any judges would like to give a wine more points if it's on the verge of a higher medal. No one has to, but sometimes the head judge's opinion can sway others to change their score. Now this won't change a Bronze to a Gold or a Silver to a Double Gold, but if 1 or 2 points from the judges will increase the average to the next higher medal, then that might happen.
Head judges are usually people in the industry that are considered the most experienced or knowledgeable. Maybe it's a winemaker, or a Master Sommelier, or a top critic. Or sometimes it's someone no more experienced than the rest of table and they're picked because they are good are keeping things organized or moving. Or maybe "just because."
As far as the rest of the judges. Typically you have to have some kind of certification or experience equivalency. But that doesn't guarantee that your panel is perfect. A panel will usually be a mixture of industries. So out of 5 you might have a journalist/media person, someone in restaurants, someone in retail, a winemaker or winery employee, and someone in distribution. This helps spread out specific experiences and perspectives. Each judge will use their own experience in evaluating each wine.
A retailer may approach a wine differently than a winemaker. A winemaker may look more at how "sound" a wine is while the retailer may say "can I sell this wine?" A critic may be closer to the winemaker while a distributor or restauranteur may be closer to the retailer. In addition to this, not every judge has experience in every grape variety, wine region, and style of wine. I've been the "wine critic" and I've been the "seller" in the past. For most of us, we know what we make or know what we sell. If you don't make Sangiovese, then you may not know what a good Chianti is. Or at least what it's supposed to taste like. You should be able to detect faults or flaws. If you work in an Italian restaurant, then you may not have much experience in French wines.
Add to this, not everyone's ability to detect aromas, flavors, structure, and faults is equal. We each have a different sensory threshold we can detect something. Or you don't have the experience in detecting something. Or genetically you are unable to detect something. There are winemakers who cannot detect cork taint (TCA). Many other professionals can't or it has to be much higher than the average person's minimum threshold.
I know professionals who struggle to smell or taste various things as well. Sometimes that's critical in identifying a specific wine in an exam situation. So they have to rely on "tricks" to get them to identify it. For example I know one MS that can't detect the distinctive aroma of petrol (TDN) in Riesling. Now, it's not that they can't detect TDN at all, just as petrol. If they smell shower curtain, then they know that compound is there. Both are petroleum products.
I smell "accordion case" in many Northern Italian reds. That comes from a unique personal experience growing up. I can't use that phrase in an exam however as the evaluators have no reference. So I have to translate that to leather, felt, and dust. The same for someone saying "it smells like my grandmother's closet."
The point is that we all have variations in how we perceive flavors and aromas. If you've never had a mango, then how can you detect it? So in wine, if you are evaluating a wine you don't have a lot of experience with, then how can you reliably rate the taste or aroma if you don't really know how to distinguish what's an appropriate level for that style? You may fall back on whether you like the wine or not. Which is also valid.
It's not that you're not qualified overall, just that you're not as experienced. And the reality with wine competitions is that they're only as good as the judges that they can get. And most of the time judges are drawn from the local population. Out of town judges travel on their own dime. And everyone needs to use vacation time or just not get paid for their time off. In some cases, people who need to travel long distances may get their travel expenses taken care of. But that's going to be for someone like a Master Sommelier, Master of Wine, a well-know wine critic, or someone of similar stature. For the vast majority of judges this is not the case.
Add to all of this. If you're struggling with allergies, or you're stressed out about your job, or having marital problems, or behind on you're bills, or a myriad of other things that might distract you, then you may have a hard time focusing on the task at hand. Also, you may hate Cali Pinots so every Pinot Noir in that flight gets a Bronze at best. Maybe you're being hypercritical of each wine. Or maybe you're putting yourself into the shoes of the average consumer and deciding what they like regardless of quality or your preference.
Finally, wine can taste differently from day to day to even the same person for much of the above. Bottle variation may happen (it's a lot less common now). Some even believe in the voodoo of the Biodynamic Calendar where there are better days to taste than others.
To address someone not being able to detect the same wine poured multiple times next to each other. What was the scenario? Was the taster told all the wines were different? If so, then by doing that you've broken the trust the taster has with you. If I'm told I have 6 different wines in front of me, and 2 or 3 of them are the same exact wine, then I may detect that those wines are at least the same type of wine, but I might try really hard to come up with slight differences between those wines. In that example, did the tasters fail to identify that all the wines were, say, California Cabernet Sauvignon or did they say they were completely different wines from different grapes and different regions?
Also, when it comes to tasting exams and practicing for them, we are given a list of wines that are fair game. At least for the Court of Master Sommeliers. As you go higher in certifications, that list grows. At the MS level they officially can give you any wine, but the reality is that it's based on the same list as the Advanced with maybe a handful of additional reds and whites that are from very classic areas. i.e. you might get Châteauneuf-du-Pape Blanc, but you're not going to get Cabernet Sauvignon from Galilee. So if you're giving me a "non-testable" wine and I'm trying to fit it into one of the approved categories, I'm probably going to fail. Why? Because I'm trusting you to play by the rules.
This doesn't mean that as an experienced taster I haven't questioned something that was put in front of me. Especially in a wine competition. It's happened to me more than once that a flight was given to my table and we all immediately detected it was the wrong flight based on grape variety. I've had blind tasting where I've confirmed that the person poured the flight in the wrong order. Or that the wine may not be on the allowed list. So many of us can and do correctly identify mistakes. The real test is to use black glasses. I've only done it once but I correctly identified red vs. white.
However, we take everything into account so if I'm seeing a red wine, then I'm expecting to smell and taste red wine. This can be done with food. With the right preparation, seasoning, sauces, etc. you can make something taste like another. Or at least close enough. And if you add in some kind of implication that you have been served a specific kind of food, then your brain will try to fit that square peg into a round hole. Plant-based "meats" come to mind. Without knowing ahead of time, it's not hard to trick the average person, maybe even a "foodie."
Without knowing that red food coloring study, did the people at least say that the wine seemed unusual for being a red wine? Some white wines have what we call "phenolic bitterness" aka tannin so with a little bit of food coloring you could fool someone into thinking it's a wine with a low amount of tannin. These whites have some kind of extended skin contact (not to the extent of orange wine). It's a regional style for specific grapes. Also, depending on the experience level of those tasters, just adding food coloring may have yielded red colors not typical of many classic red wines.
Another personal example. Nebbiolo is the grape used for Barolo and Barbaresco. Visually it can look like Pinot Noir. Even on the nose and palate it can mimic Pinot. Structurally both can be the same. Except tannin. Nebbiolo has a lot more tannin than Pinot Noir. However, the type of tannin in Nebbiolo takes longer for my gums to react to than for other people. So while they may immediately detect tannin as being high, I may initially say low, medium minus, or medium.
Even so, some wines do look and act like each other. The Bermuda Triangle of whites - Pinot Grigio, Gruner Veltliner, and Albariño trick many a sommelier. Including MSes. Sometimes Albariño is replaced with another white, but the first two are always in the trio. All three are fairly neutral. But each have certain characteristics that differentiate each from the others. I've already mentioned how someone can struggle to taste these differences.
As far as the scandals, yep they happen. This is why we have controls and ways to detect them. Most of the time it was because there was a scandal in the first place. For instance, because of the "anti-freeze" scandal in Austria, they have to strictest testing of any country to prevent any kind of harmful adulteration of a wine. Other countries or governing bodies will do the same. Sometimes it's just to make sure the wine meets specific requirements, not necessarily looking for anti-freeze. Many appellations have tasting panels to ensure a wine meets the legal requirements (as far as look, smell, taste, etc.). Even so, wines may slip through. Or wines may get denied certification due to internal politics. And, yes, fraud exists. But that's no different than in many other industries. A skilled counterfeiter can fool most people most of the time.
The thing is, human perception of things is very difficult to quantify into an equation. No matter how complicated it looks. Sometimes the standard deviation will make sense, and other times it won't. Yes, some things are almost universally agreed to be great or awful. But many things fall into the middle based on personal experience and bias. Last example. Most people think U2 is a great band. I think they suck. I'm not a fan of Country Music even though it's one of the most popular styles. And I have a B.A. in Music. So what does that mean? Nothing.
FTR, I stopped scoring wines I review more than 10 years ago.......
Thanks for unmasking the BS. Hey I though I might as well make wish list for your next destruction - is there any substance to the sulphite issue? I do feel like they´re no good for me the next day, but, you know... placebos dominate any real signal. So we should exploer the power of placebos instead of real stuff, perhaps.
I enjoyed this essay but was slightly confused by the target: is it that sommeliers can judge good wine from bad? or is it whether sommeliers can identify wines in their particulars? It seems that they cannot do the former, but can do the latter. But then the results of the latter don't necessarily bear on the former.
There's also the confounding fact that as a rule, wine producers now make good wines. Don't you think it was possible that the French were some of the only *consistent* winemakers in the olden days? Especially in the pre-industrial era, or worse, the early industrial era? Good wine exists, as your essay suggests, but it might have once been a gamble whether your wine would be good. French provenance might've been a guarantee of quality for various reasons.
A sommelier in the old days, you could suppose, had the job of just enforcing a certain meagre standard, like a health and safety official. (c.f. the etymology: 'Earlier it meant the person in a large household, convent, hospital, etc. who was in charge of wines, (1680s) in French contexts, and was regarded as akin to an English butler (q.v.). The original French sense was of an officer who had charge of provisions (13c.)')
Setting aside the historical part of this point: the conclusion is something like the 'the best wine is the one you like' and yet the sommeliers aren't very good judges... of something. I'm assuming that these days, the competition is fierce: every wine they encounter is good, rather than bad (think: backyard rocket-fuel.) So you'd think that discerning the particulars of what makes a good wine good is much harder than discerning good wine from bad. You'd think that the winemakers today also overdetermine sommelier judgements: the vintner culls bad wines before they even reach the judges. Yet the judges must still judge 'good' from 'bad' within a selection of good wines: a doomed task.
Another way of putting it: Are you getting at whether sommeliers can enforce 'standards of taste' or just whether they have delicacy of taste?
I'd take the opportunity to refer to that wine bit of Hume's The Standard of Taste if I wasn't already going approaching tl;dr.
For some reason, when this topic comes up, its always isolated studies like this. There are organisations that certify sommeliers, and they do blind tasting in their exams. Assessing the "difficulty" and methodological validity of those exams seems much more reliable than compiling heterogenous studies with much less context.
For example, if you have two studies on the influence of different factors in the same setup, that would be less isolated.
>And that certification is evidently not worth very much.
Right, so you should look at the most prestigious ones and show how little ability is required to pass them. I think there is much less chance of unexpected context messing up the result in that investigation, because the exam is designed as an adversarial situation, by people who supposedly understand the context.
>Idea covered in the article you're replying to.
The idea of "experts" blind tasting things is covered. What I mean is that, if they really are so bad at it, then those blind tests in the exams must be a sham or trivial somehow, and to find out how exactly.
If the test is hard but doesnt require actually high skill in taste identification, then that is presumably something you can find out by looking at the test.
I know that your results here imply what these certs/test must approximately be like. I think that looking at them directly would be better evidence about the topic as a whole than the things you picked.
Basically, all sorts of objections pro-sommelier people could make to the experimental setup automatically fall flat by using a test they designed.
A few things I want to add from a somewhat informed perspective:
1) the wine contests are a very bad proxy for quality. Most are pure marketing, with very few wines presented thus all but guaranteeing a medal in one category or other. The judges tend to be amateurs, not equiped (see below) to offer deep analysis and thus overly dependent on scoring rulebooks (cahier des charges) that are often shared beforhand with favoured candidates...
2) Most really good wines - in france at least - never go to contests. that is a low-status way of getting renown. so there is much greater variability, in what is effectively the high-middle class of wines.
3) It is very well attested that training (most likely from a formal education in oenology, or perfumery) leads to much better performance of the nose as a discriminating engine, leading to a depth of note perception and analysis that seems like either a superpower or a grift to untrained people. (disclaimer: trained in perfumery for a bit, my sister is a full blown oenologist). Like all things, natural capacity plays a big role here, the top oenologists will have both talent and systematic training and practice, but will make a vanishingly small proportion of even pro wine tasters.
4) add to that that good taste is also unevenly distributed, and you end up in a place where gifted, well trained wine tasters with good taste in wine (not just learned taste in wine, e.g: price, area recognition and pattern matching) will be very, very rare. And tastes change with time! There are not many Anna Winetours.
Take all the above plus the ungodly amount of money sloshing around, and you can explain a lot of the inconsistencies. If you restrict your analysis to large contests (many contestants in each categories) with professional wine tasters (maybe even a celebrity one or two) as judges and a publicly disclosed scoring rule you'll get much better repeatability.
“But more importantly for us, they showed that cultured, haughty oenophile prejudices were bereft of substance; they laid bare the reality that the refined judgments of the top sommeliers were lacking in predictive power.”
This conclusion is logically inconsistent. “Refined judgment of the top sommeliers” are what placed the California wines above the French ones. These are the same judgments that hold the French wines in high esteem in the first place.
Therefore, if these judgements are the basis of your validation of the quality of California wines, then you must also conclude that the French wines are of high (but lesser) quality. And therefore, the prejudices — while perhaps not always accurate — are not totally bereft of substance.
"'Refined judgment of the top sommeliers' are what placed the California wines above the French ones."
No, quite the opposite. The judgment that the California wines were better defied their refined expectations.
"These are the same judgments that hold the French wines in high esteem in the first place."
No, they are not. The French wines were preferred for prejudicial reasons, not because they were observed to be better.
"Therefore, if these judgements are the basis of your validation of the quality of California wines"
If you read the whole article, you'll notice that I actually say that the best wine is the one you like, not that California wines are better than French wines.
“No, quite the opposite. The judgment that the California wines were better defied their refined expectations.”
— defied their expectations, but not their judgement. They judged the California wines to be better (which we take as a measure of the Cali wines’ quality).
“No, they are not. The French wines were preferred for prejudicial reasons, not because they were observed to be better.”
— The French wines were expected to be better, due to prejudice yes, but this was based on them having been judged to be excellent in the past. This judgement can be no less valid than the judgement which rated the Cali wines highly.
“If you read the whole article, you'll notice that I actually say that the best wine is the one you like, not that California wines are better than French wines.”
—- Yes, i did read it, and I concur. My point was that the judgements are not totally bereft of substance (which your article does later conclude).
Anyway, thank you for the compelling read. I think we are more in agreement than disagreement.
"defied their expectations, but not their judgement"
Prejudice is a prejudgment and it is what I was referring to. I thought this was clear, but I had someone else read it just now and they also understood what I meant.
"which we take as a measure of the Cali wines’ quality"
You*
"but this was based on them having been judged to be excellent in the past."
Evidence?
"My point was that the judgements are not totally bereft of substance"
I said that the *prejudices* were bereft of substance.
“Prejudice is a prejudgment and it is what I was referring to. I thought this was clear, but I had someone else read it just now and they also understood what I meant.”
—- I can agree that the pre-judgements were bereft of substance.
“"but this was based on them having been judged to be excellent in the past."
New suggested name for your substack: “Cremieux Ruins Everything.” Like the old show but better, and with less snark.
I always said there was only two types of wine: good and bad
No, I think you missed the point, my friend. There are only two types of wine: those you like and those you don't.
Great post. Now, please do the same to beer, coffee, and bourbon.
A steakhouse in College Station did/does a March madness bourbon elimination. In the years I went there, the legendary Van Pappy Bourbon never got out of the first round.
Similar posts could definitely be done for other foodstuffs. I only selected wine for this because it was personally relevant and it has such an esteemed reputation for discernment.
Agreed. Olive oil was especially interesting. Apparently the world doesn't make close to as much virgin as is sold
I think the situation with beer is different, since, from my experience, there is a greater difference in flavour between beers than there is between wines. I believe it is easier to detect a significant difference in flavour between a porter or stout and a pale ale than it is to tell the difference between red wine and white wine. Where I would agree with there being problems with "beer experts" is more that they are often following fashions and can be influenced by successful PR campaigns.
I have heard a story, possibly apocryphal, about how the American pale ale trend started with one brewery failing to create a proper Indian pale ale. Rather than admit that they had made a mistake and created something that had a flavour one would more expect from foraging in a florist than visiting a pub, they instead embarked on a massive PR campaign that managed to convince some pretentious "experts" that their over-hopped failure was in fact good. This then resulted in the past 15 or so years of mediocre pale ales selling for around £10 a pint and snobbish idiots praising garbage that tastes just as awful as some of the mass produced lagers that are, rightfully, regarded as tasting like tramp's urine.
I could not possibly argue with anyone who also realizes over-hopped beers are a sign of poor craftsmanship.
You may detect more flavor differences between beers, but I detect more between wine. I suspect this is because I like wine more than beer.
And I can’t detect differences between whiskeys at all — to me they all taste like gasoline.
That is fair enough, but part of the reason I can detect more flavour differences between beers is because there are more varied ingredients that go into different types of beers. Chocolate stouts often feature chocolate malt or cacao nibs whilst milk stouts are brewed with lactose. This is in stark contrast to wine, which can only be made from fermented grape juice. This isn't to say that a connoisseur of wine or of a particular type of beer may be able to better detect flavour differences within their preferred tipple to a greater degree than a novice, but beer in general does have a greater number of potential ingredients.
A relative used to tell me this all the time growing up - about how with blind testing the cheapest wines can taste as good as the worst.
On retirement, the same fellow dragged out every bottle of wine he owned, googled it to see the ratings, and classified each wine he owned 1, 2, 3 depending on how 'good' it was, and that determines which guests get which.
If wine judges are radically unreliable, how was the Judgment of Paris able to be replicated? One would expect regression to the mean, no?
I don't see any contradiction.
If you give me a random beer and a random wine, I'm going to prefer the wine in every single contest, even if I might err in judging their triplicate pours.
But the difference between a California Chardonnay and a white burgundy is much less that the difference between red wine and white wine dyed red. So it does seem inconsistent that judges have repeatedly (if unknowingly) detected the first difference but not the second.
Fooling judges with dye seems like it introduces novel variance.
All it proves — and as I noted in a prior comment this study is often overstated, including by you — is that visual stimuli impact perception of flavor. Which we already knew.
A real study would blindfold the judges and ask them point blank if it was red or white. Even then you have temperature issues.
Fun article. Doesn't the very fact that people rely on tasters and "experts" to tell them what's good reek of subjectivity? If people were just relying on what tastes good to them, popularity would dictate which wines sold best, and presumably weed out the expensive frauds. That seems to be pretty close to what happens with beer, although there is an element of snob appeal in the niche craft brew industry.
There are good reasons for consumers to rely on advice: they very rarely get to taste a wine before buying. Even if you live in a wine region and visit wineries you only taste wine from those producers. There are wine fairs where you can taste and buy a large variety of wines, but only a tiny proportion of wine drinkers go there.
And even if you could taste them, some stores have hundreds of options.
What you don't mention, but which seems relevant, is that even though wine tasters claim to not get drunk at these tastings they almost certainly end up drinking some wine. There's no doubt that drunk people are bad at actual taste issues, a fact known at least 2000 years ago since it's mentioned in the bible. Chances are that in the larger wine events where testers have to rate dozens of wines they simply lose their ability to discriminate properly, assuming it was there in the first place.
Regarding wine scandals, there have been many wine scandals detected by wine sellers selling more wine than they could possibly make from the vineyards the grapes allegedly come from. This included the monks on the island just off Cannes who have a tiny vineyard but sold more wine than several wineries on the mainland...
In some cases in the EU the prime sucker for the scam seems to be the EU's agricultural subsidies rather than consumers, and in many of those cases it seems likely that the wine simply did not exist, but there were others I recall where the same wine was counted as two different wines through careful use of unlabeled bottles.
The scandals are alas common in food in general. Look up virgin olive oil.
Indeed!
True
Being a sommelier, wine critic, and judge I have some first hand experience in this. The TL;DR is that yes, you are right, there is a certain level of BS when of comes to the entirety of your article. i.e. the industry not what you wrote. But there are valid reasons for the these variations and anomalies.
And now......the rest of the story:
Judging anything (food, beverage, art, sports, etc.) at its core is a personal and oftentimes unique experience. When looking at reviews or scores or evaluations of these things we can see large variations of opinion. Especially in the first three categories listed.
With sports, athletes tend to hit the required criteria for that sport. Did they nail the landing, did they do enough turns, did they do all the required movements, etc. And at competitions such as the Olympics, the differences between skill levels for most of the athletes is pretty minimal. So the scores may be pretty close to each other unless the athlete made a major mistake. But even then you'll have some judges give scores much higher or lower than the rest because of politics or preferences.
We've seen plenty of competitions across many categories where experts will have differences of opinion. We also see where they will agree. In most competitions with a panel of judges the top and bottom scores will be eliminated. This is done to prevent any particular judge from skewing a result too much.
With things like F&B and art, so-called experts will view these things differently than the general public. And not every expert is created equally. Some grew up in that world and have intimate knowledge whereas others had to start from scratch. Some are trained either academically or through work experience in that field. Phrases like "beauty is in the eye of the beholder," and "the best wine is the one you like," and "opinions are like...., everyone has one" demonstrate that we each evaluate things from our perspective.
Bringing it back to wine and competitions. A typical competition receives hundreds or thousands of entries. Those entries are then categorized. It's up to the competition to decide those categories based on the entries. Sometimes not enough entries come in to be very specific and sometimes they are. For instance, one flight could be United States Pinot Noir $15-18 whereas another could be Other White $10-30. For each flight the judges are given some kind of information to go on to evaluate the wines.
When it comes to the actual scoring, there will be 4-6 or maybe more criteria to use to evaluate. But basically it will be Nose, Taste, Balance, Overall. Sometimes Color, Structure, or some other criteria. You'll assess a numerical value to each (usually 0-5) and then add them up. That total will correspond to a specific medal, or no medal.
Once the flight is finished the head judge will then compile the rest of the table's scores and then come up with an average. As mentioned before, a low score and/or a high score will usually be thrown out. So with a table of 5 judges, including the head judge, you could have only 3 scores determine the final score.
When scoring these wines, judges are supposed to judge them within their class. So if you're judging those Cali Pinot Noirs $15-18, then you're judging them on if they meet certain quality levels for wines in that class, NOT compared to the best Pinot Noirs in the world. So, yes, a $15 or even $10 Pinot could get a Double Gold....in its category. Also, the wines are not really competing. They are not supposed to be evaluated against each other. They are evaluated on their own. So calling these things competitions is really incorrect. With that said, it's hard not to compare wines within a flight, but the goal isn't to have a winner among the flight. They could all be Double Golds or all suck.
FTR, this is basically the same thing that happens at publications. Wines are (supposedly) evaluated blind as far as knowing producer, but are put into appropriate categories. The publication's panel will evaluate them based on that category and not versus all wines of that overall category.
Continued in my reply to this comment.....
During this phase, we will typically have a general discussion of the flight. So this might be a time where each of the judges may make a case for a wine to be score higher or lower than the rest. Also the head judge may ask the table if any judges would like to give a wine more points if it's on the verge of a higher medal. No one has to, but sometimes the head judge's opinion can sway others to change their score. Now this won't change a Bronze to a Gold or a Silver to a Double Gold, but if 1 or 2 points from the judges will increase the average to the next higher medal, then that might happen.
Head judges are usually people in the industry that are considered the most experienced or knowledgeable. Maybe it's a winemaker, or a Master Sommelier, or a top critic. Or sometimes it's someone no more experienced than the rest of table and they're picked because they are good are keeping things organized or moving. Or maybe "just because."
As far as the rest of the judges. Typically you have to have some kind of certification or experience equivalency. But that doesn't guarantee that your panel is perfect. A panel will usually be a mixture of industries. So out of 5 you might have a journalist/media person, someone in restaurants, someone in retail, a winemaker or winery employee, and someone in distribution. This helps spread out specific experiences and perspectives. Each judge will use their own experience in evaluating each wine.
A retailer may approach a wine differently than a winemaker. A winemaker may look more at how "sound" a wine is while the retailer may say "can I sell this wine?" A critic may be closer to the winemaker while a distributor or restauranteur may be closer to the retailer. In addition to this, not every judge has experience in every grape variety, wine region, and style of wine. I've been the "wine critic" and I've been the "seller" in the past. For most of us, we know what we make or know what we sell. If you don't make Sangiovese, then you may not know what a good Chianti is. Or at least what it's supposed to taste like. You should be able to detect faults or flaws. If you work in an Italian restaurant, then you may not have much experience in French wines.
Add to this, not everyone's ability to detect aromas, flavors, structure, and faults is equal. We each have a different sensory threshold we can detect something. Or you don't have the experience in detecting something. Or genetically you are unable to detect something. There are winemakers who cannot detect cork taint (TCA). Many other professionals can't or it has to be much higher than the average person's minimum threshold.
I know professionals who struggle to smell or taste various things as well. Sometimes that's critical in identifying a specific wine in an exam situation. So they have to rely on "tricks" to get them to identify it. For example I know one MS that can't detect the distinctive aroma of petrol (TDN) in Riesling. Now, it's not that they can't detect TDN at all, just as petrol. If they smell shower curtain, then they know that compound is there. Both are petroleum products.
I smell "accordion case" in many Northern Italian reds. That comes from a unique personal experience growing up. I can't use that phrase in an exam however as the evaluators have no reference. So I have to translate that to leather, felt, and dust. The same for someone saying "it smells like my grandmother's closet."
The point is that we all have variations in how we perceive flavors and aromas. If you've never had a mango, then how can you detect it? So in wine, if you are evaluating a wine you don't have a lot of experience with, then how can you reliably rate the taste or aroma if you don't really know how to distinguish what's an appropriate level for that style? You may fall back on whether you like the wine or not. Which is also valid.
It's not that you're not qualified overall, just that you're not as experienced. And the reality with wine competitions is that they're only as good as the judges that they can get. And most of the time judges are drawn from the local population. Out of town judges travel on their own dime. And everyone needs to use vacation time or just not get paid for their time off. In some cases, people who need to travel long distances may get their travel expenses taken care of. But that's going to be for someone like a Master Sommelier, Master of Wine, a well-know wine critic, or someone of similar stature. For the vast majority of judges this is not the case.
Add to all of this. If you're struggling with allergies, or you're stressed out about your job, or having marital problems, or behind on you're bills, or a myriad of other things that might distract you, then you may have a hard time focusing on the task at hand. Also, you may hate Cali Pinots so every Pinot Noir in that flight gets a Bronze at best. Maybe you're being hypercritical of each wine. Or maybe you're putting yourself into the shoes of the average consumer and deciding what they like regardless of quality or your preference.
Finally, wine can taste differently from day to day to even the same person for much of the above. Bottle variation may happen (it's a lot less common now). Some even believe in the voodoo of the Biodynamic Calendar where there are better days to taste than others.
To address someone not being able to detect the same wine poured multiple times next to each other. What was the scenario? Was the taster told all the wines were different? If so, then by doing that you've broken the trust the taster has with you. If I'm told I have 6 different wines in front of me, and 2 or 3 of them are the same exact wine, then I may detect that those wines are at least the same type of wine, but I might try really hard to come up with slight differences between those wines. In that example, did the tasters fail to identify that all the wines were, say, California Cabernet Sauvignon or did they say they were completely different wines from different grapes and different regions?
Also, when it comes to tasting exams and practicing for them, we are given a list of wines that are fair game. At least for the Court of Master Sommeliers. As you go higher in certifications, that list grows. At the MS level they officially can give you any wine, but the reality is that it's based on the same list as the Advanced with maybe a handful of additional reds and whites that are from very classic areas. i.e. you might get Châteauneuf-du-Pape Blanc, but you're not going to get Cabernet Sauvignon from Galilee. So if you're giving me a "non-testable" wine and I'm trying to fit it into one of the approved categories, I'm probably going to fail. Why? Because I'm trusting you to play by the rules.
This doesn't mean that as an experienced taster I haven't questioned something that was put in front of me. Especially in a wine competition. It's happened to me more than once that a flight was given to my table and we all immediately detected it was the wrong flight based on grape variety. I've had blind tasting where I've confirmed that the person poured the flight in the wrong order. Or that the wine may not be on the allowed list. So many of us can and do correctly identify mistakes. The real test is to use black glasses. I've only done it once but I correctly identified red vs. white.
However, we take everything into account so if I'm seeing a red wine, then I'm expecting to smell and taste red wine. This can be done with food. With the right preparation, seasoning, sauces, etc. you can make something taste like another. Or at least close enough. And if you add in some kind of implication that you have been served a specific kind of food, then your brain will try to fit that square peg into a round hole. Plant-based "meats" come to mind. Without knowing ahead of time, it's not hard to trick the average person, maybe even a "foodie."
Continued in my reply to this comment.....
Without knowing that red food coloring study, did the people at least say that the wine seemed unusual for being a red wine? Some white wines have what we call "phenolic bitterness" aka tannin so with a little bit of food coloring you could fool someone into thinking it's a wine with a low amount of tannin. These whites have some kind of extended skin contact (not to the extent of orange wine). It's a regional style for specific grapes. Also, depending on the experience level of those tasters, just adding food coloring may have yielded red colors not typical of many classic red wines.
Another personal example. Nebbiolo is the grape used for Barolo and Barbaresco. Visually it can look like Pinot Noir. Even on the nose and palate it can mimic Pinot. Structurally both can be the same. Except tannin. Nebbiolo has a lot more tannin than Pinot Noir. However, the type of tannin in Nebbiolo takes longer for my gums to react to than for other people. So while they may immediately detect tannin as being high, I may initially say low, medium minus, or medium.
Even so, some wines do look and act like each other. The Bermuda Triangle of whites - Pinot Grigio, Gruner Veltliner, and Albariño trick many a sommelier. Including MSes. Sometimes Albariño is replaced with another white, but the first two are always in the trio. All three are fairly neutral. But each have certain characteristics that differentiate each from the others. I've already mentioned how someone can struggle to taste these differences.
As far as the scandals, yep they happen. This is why we have controls and ways to detect them. Most of the time it was because there was a scandal in the first place. For instance, because of the "anti-freeze" scandal in Austria, they have to strictest testing of any country to prevent any kind of harmful adulteration of a wine. Other countries or governing bodies will do the same. Sometimes it's just to make sure the wine meets specific requirements, not necessarily looking for anti-freeze. Many appellations have tasting panels to ensure a wine meets the legal requirements (as far as look, smell, taste, etc.). Even so, wines may slip through. Or wines may get denied certification due to internal politics. And, yes, fraud exists. But that's no different than in many other industries. A skilled counterfeiter can fool most people most of the time.
The thing is, human perception of things is very difficult to quantify into an equation. No matter how complicated it looks. Sometimes the standard deviation will make sense, and other times it won't. Yes, some things are almost universally agreed to be great or awful. But many things fall into the middle based on personal experience and bias. Last example. Most people think U2 is a great band. I think they suck. I'm not a fan of Country Music even though it's one of the most popular styles. And I have a B.A. in Music. So what does that mean? Nothing.
FTR, I stopped scoring wines I review more than 10 years ago.......
Straight into my veins! The fact that so many wine "experts" have been tricked is under-discussed.
Also, I live in France, and Americans in France are obsessed with pretending French wine is the only good wine. That is dumb.
There are genuinely excellent wines from Virginia, CA, Spain, Israel-- all for easily under $20/bottle.
That movie Somm from 2012 was still kinda fun though.
https://theonion.com/connoisseur-only-one-not-enjoying-wine-1837999823/
Thanks for unmasking the BS. Hey I though I might as well make wish list for your next destruction - is there any substance to the sulphite issue? I do feel like they´re no good for me the next day, but, you know... placebos dominate any real signal. So we should exploer the power of placebos instead of real stuff, perhaps.
I enjoyed this essay but was slightly confused by the target: is it that sommeliers can judge good wine from bad? or is it whether sommeliers can identify wines in their particulars? It seems that they cannot do the former, but can do the latter. But then the results of the latter don't necessarily bear on the former.
There's also the confounding fact that as a rule, wine producers now make good wines. Don't you think it was possible that the French were some of the only *consistent* winemakers in the olden days? Especially in the pre-industrial era, or worse, the early industrial era? Good wine exists, as your essay suggests, but it might have once been a gamble whether your wine would be good. French provenance might've been a guarantee of quality for various reasons.
A sommelier in the old days, you could suppose, had the job of just enforcing a certain meagre standard, like a health and safety official. (c.f. the etymology: 'Earlier it meant the person in a large household, convent, hospital, etc. who was in charge of wines, (1680s) in French contexts, and was regarded as akin to an English butler (q.v.). The original French sense was of an officer who had charge of provisions (13c.)')
Setting aside the historical part of this point: the conclusion is something like the 'the best wine is the one you like' and yet the sommeliers aren't very good judges... of something. I'm assuming that these days, the competition is fierce: every wine they encounter is good, rather than bad (think: backyard rocket-fuel.) So you'd think that discerning the particulars of what makes a good wine good is much harder than discerning good wine from bad. You'd think that the winemakers today also overdetermine sommelier judgements: the vintner culls bad wines before they even reach the judges. Yet the judges must still judge 'good' from 'bad' within a selection of good wines: a doomed task.
Another way of putting it: Are you getting at whether sommeliers can enforce 'standards of taste' or just whether they have delicacy of taste?
I'd take the opportunity to refer to that wine bit of Hume's The Standard of Taste if I wasn't already going approaching tl;dr.
For some reason, when this topic comes up, its always isolated studies like this. There are organisations that certify sommeliers, and they do blind tasting in their exams. Assessing the "difficulty" and methodological validity of those exams seems much more reliable than compiling heterogenous studies with much less context.
"isolated studies"
What is a non-isolated study?
"There are organisations that certify sommeliers"
And that certification is evidently not worth very much.
"and they do blind tasting in their exams"
Idea covered in the article you're replying to.
>What is a non-isolated study?
For example, if you have two studies on the influence of different factors in the same setup, that would be less isolated.
>And that certification is evidently not worth very much.
Right, so you should look at the most prestigious ones and show how little ability is required to pass them. I think there is much less chance of unexpected context messing up the result in that investigation, because the exam is designed as an adversarial situation, by people who supposedly understand the context.
>Idea covered in the article you're replying to.
The idea of "experts" blind tasting things is covered. What I mean is that, if they really are so bad at it, then those blind tests in the exams must be a sham or trivial somehow, and to find out how exactly.
So regarding isolated studies, there's no such thing as a non-isolated study and it's just personal judgment. Got it!
Nothing you said about certification makes sense. The Titan Test is a hard test even if it's not a valid IQ test.
The nature of what tasters pick up is what I was referring to being covered.
If the test is hard but doesnt require actually high skill in taste identification, then that is presumably something you can find out by looking at the test.
I know that your results here imply what these certs/test must approximately be like. I think that looking at them directly would be better evidence about the topic as a whole than the things you picked.
Basically, all sorts of objections pro-sommelier people could make to the experimental setup automatically fall flat by using a test they designed.
You have greatly misunderstood what I wrote. Because others were able to understand it correctly, I'll just leave this conversation.
A few things I want to add from a somewhat informed perspective:
1) the wine contests are a very bad proxy for quality. Most are pure marketing, with very few wines presented thus all but guaranteeing a medal in one category or other. The judges tend to be amateurs, not equiped (see below) to offer deep analysis and thus overly dependent on scoring rulebooks (cahier des charges) that are often shared beforhand with favoured candidates...
2) Most really good wines - in france at least - never go to contests. that is a low-status way of getting renown. so there is much greater variability, in what is effectively the high-middle class of wines.
3) It is very well attested that training (most likely from a formal education in oenology, or perfumery) leads to much better performance of the nose as a discriminating engine, leading to a depth of note perception and analysis that seems like either a superpower or a grift to untrained people. (disclaimer: trained in perfumery for a bit, my sister is a full blown oenologist). Like all things, natural capacity plays a big role here, the top oenologists will have both talent and systematic training and practice, but will make a vanishingly small proportion of even pro wine tasters.
4) add to that that good taste is also unevenly distributed, and you end up in a place where gifted, well trained wine tasters with good taste in wine (not just learned taste in wine, e.g: price, area recognition and pattern matching) will be very, very rare. And tastes change with time! There are not many Anna Winetours.
Take all the above plus the ungodly amount of money sloshing around, and you can explain a lot of the inconsistencies. If you restrict your analysis to large contests (many contestants in each categories) with professional wine tasters (maybe even a celebrity one or two) as judges and a publicly disclosed scoring rule you'll get much better repeatability.
“But more importantly for us, they showed that cultured, haughty oenophile prejudices were bereft of substance; they laid bare the reality that the refined judgments of the top sommeliers were lacking in predictive power.”
This conclusion is logically inconsistent. “Refined judgment of the top sommeliers” are what placed the California wines above the French ones. These are the same judgments that hold the French wines in high esteem in the first place.
Therefore, if these judgements are the basis of your validation of the quality of California wines, then you must also conclude that the French wines are of high (but lesser) quality. And therefore, the prejudices — while perhaps not always accurate — are not totally bereft of substance.
"This conclusion is logically inconsistent."
No, quite the opposite.
"'Refined judgment of the top sommeliers' are what placed the California wines above the French ones."
No, quite the opposite. The judgment that the California wines were better defied their refined expectations.
"These are the same judgments that hold the French wines in high esteem in the first place."
No, they are not. The French wines were preferred for prejudicial reasons, not because they were observed to be better.
"Therefore, if these judgements are the basis of your validation of the quality of California wines"
If you read the whole article, you'll notice that I actually say that the best wine is the one you like, not that California wines are better than French wines.
“No, quite the opposite. The judgment that the California wines were better defied their refined expectations.”
— defied their expectations, but not their judgement. They judged the California wines to be better (which we take as a measure of the Cali wines’ quality).
“No, they are not. The French wines were preferred for prejudicial reasons, not because they were observed to be better.”
— The French wines were expected to be better, due to prejudice yes, but this was based on them having been judged to be excellent in the past. This judgement can be no less valid than the judgement which rated the Cali wines highly.
“If you read the whole article, you'll notice that I actually say that the best wine is the one you like, not that California wines are better than French wines.”
—- Yes, i did read it, and I concur. My point was that the judgements are not totally bereft of substance (which your article does later conclude).
Anyway, thank you for the compelling read. I think we are more in agreement than disagreement.
"defied their expectations, but not their judgement"
Prejudice is a prejudgment and it is what I was referring to. I thought this was clear, but I had someone else read it just now and they also understood what I meant.
"which we take as a measure of the Cali wines’ quality"
You*
"but this was based on them having been judged to be excellent in the past."
Evidence?
"My point was that the judgements are not totally bereft of substance"
I said that the *prejudices* were bereft of substance.
“Prejudice is a prejudgment and it is what I was referring to. I thought this was clear, but I had someone else read it just now and they also understood what I meant.”
—- I can agree that the pre-judgements were bereft of substance.
“"but this was based on them having been judged to be excellent in the past."
Evidence?”
—- some evidence (non exhaustive):
- The Château Mouton Rothschild 1945 was evaluated by famous critics such ad Michael Broadbent and Michel Dovaz (https://www.decanter.com/learn/wine-legend-chateau-mouton-rothschild-1945-374280/)
- The Château Haute-Brion was taste-tested and evaluated (at least) the year prior (https://hogsheadwine.wordpress.com/2016/08/23/a-history-of-the-california-barrel-tastings-part-4-fine-american-wines-can-now-hold-their-own-with-fine-wines-from-france-the-wines-served-at-the-early-tastings/)
- The Château Montrose was tested in the same comparative tasting as the above, as was the Château Léoville Las Cases
- The Beaune Clos des Mouches Joseph Drouhin was evaluated at the New York Wine Tasting of 1973 (https://en.wikipedia.org/wiki/New_York_Wine_Tasting_of_1973)
Its hard to find sources for all of them, but generally, the standing of these wines was based on actual evaluation and not solely prejudice.
“”My point was that the judgements are not totally bereft of substance"
I said that the *prejudices* were bereft of substance.”
—— that’s clear now; thanks. I misread, initially.