The Whiskey Reviewer A World of Whiskey, Poured Every Weekday
Don’t Believe Everything You Read, Especially About Statistics
By Richard Thomas
Is this…
(Credit: Randall H. Borkus)
Although the business press was abuzz last week about Brown-Forman’s surprise acquisition of BenRiach and bourbon forums boiled that former Four Roses Master Distiller Jim Rutledge might have been pushed out of his job, in the wider (and softer) mainstream media the major whiskey story was how people can’t tell the difference between bourbon and rye whiskey. In an article for the Journal of Food Science this month, Dr. Jake Lahne of Drexel University published the results of a study delving into that topic. Yet did it actually answer the question of whether whiskey drinkers are able to tell the difference between bourbon and rye?
Foodies with good news alerts might already be familiar with Dr. Lahne, whose bailiwick is how and why people perceive food and drink in particular ways. For my part, I both follow the whiskey beat and read plenty of British news sources, and those Brits seem to take plenty of pleasure from quirky science stories, judging from how many they publish.
Because of that last point, I have learned to take the expansive media claims made about statistical studies with, not a grain, but a spoonful of salt. Sometimes even the study itself has serious flaws, obvious to a critical mind with even a modicum of statistical training.
In summary, Lahne selected a group of 21 people, gave them an unlabeled set of five bourbons and five ryes, and instructed them to nose the whiskeys and then organize them according to what was largely their own critera. He then recalled this group a few days later, gave them the same 10 whiskeys, now with random labels, and asked them to organize them again. His conclusion, based on how his subjects organized the whiskeys in all manner of ways that had nothing to do with mashbills, was that his subjects couldn’t tell the difference between the two whiskey styles.
I must confess I haven’t read the original article in the Journal Of Food Science, but the first problem with the study that jumps out at me is that his subjects were allowed to organize their whiskey samples as they pleased. I’m better versed in whiskey than most, but it might or might not occur to me to organize a group of whiskey samples into bourbon and rye if I was free to do something else. Lahne’s study indicated not whether bourbon can be told from rye, but about how people choose organize samples with and without information about them.
Another problem with the study was the small sample size of subjects, a mere 21. With so few people in the study, any flaws with subject selection would be greatly magnified in the results. What is more (and this is a frequent problem with studies like this), the results cannot even begin to qualify as conclusive or even as science unless they are verified by repetition, a very basic requirement. They have not.
With those points out of the way, if I could read the article, I would ask these two questions:
- How rigorous was the subject selection process? What were the criteria? I find many casual whiskey drinkers have these sorts of identification issues, and if that was the case here the results are unsurprising, especially with the aforementioned problems in play.
- What whiskeys were used in the study? It’s important to recall that the Kentucky rye style uses a 51% rye mashbill, just barely over the rye line.
… really the same as this?
(Credit: Richard Thomas)
Frankly, I don’t find it hard to believe that some people can’t tell bourbon from rye. Barrel aging accounts for between 40 and 80% of what an American whiskey tastes like, so the mashbill could very well be a minority factor in a given whiskey. What I find completely wrong headed is that the suggestion that the difference cannot be told. If you collected up 21 members of quality control teams from different distilleries, gave them the same test but also told them what it was they were testing for, I guarantee you the results would be night and day from Lahne’s study.
I have a number of concerns with your commentary:
– Throwing out a peer-reviewed, published study with vague questions about study design is intellectually lazy.
– 21 participants may be statistically valid with proper design and evaluation. To stipulate that the study size is too small without access to the data really doesn’t further the concept of healthy skepticism. Instead it seems to further the populist opinion that the mainstream media and academia are not to be trusted.
– I too did not read the study. However, the abstract for this study is readily available. The authors did not suggest that participants couldn’t differentiate between rye and bourbon, instead they suggest that participants did not utilize mashbill as a deciding factor in grouping the products. The authors suggest that other factors, such as age, are better grouping determinants. Interviews with the primary author seem to just question if mashbill, which is perceived by the authors to be the determining factor in product difference, is actually the appropriate way to differentiate the products.
Thank you for comment.
First, skepticism and asking questions, whether you consider them vague or not, is rarely, if ever, intellectually lazy. Skepticism is only lazy if it is not connected to raising questions, but is instead an exercise in closed-mindedness. On the contrary, overruling your own critical faculties and accepting an opinion merely because an authority tells you to is what is actually (and supremely) lazy. Calling it “populist” is akin to labeling an axiom of critical thinking as conspiracy theory.
Second, my article was as critical of the way in which these studies are reported as it is of the study itself. There is a whole paragraph stating that only sometimes is it the study that is part of the problem.
Third, you restated one of my concerns — that flaws in subject selection and study design might skew the results in a small study size — in a different way emphasizing the positives rather than the potential problems.
Fourth, while peer review is certainly better than no review at all, it isn’t a foolproof process and should not be seen as raising anything above suspicion. Plenty of flawed science passed peer review. Acts of fraud have passed peer review. Science as a whole wouldn’t be confronted with a reproducibility crisis if peer review was doing what it is supposed to do.
Coming from my PR perspective, your faith in the media is very, very misplaced. A lot of the stories you read are “canned,” and based on a press release. This stuff about the bourbon and rye study is a case in point. I saw a dozen or so stories in my news alerts. They were virtually identical, with overlapping quotes. None of them presented a contrary point of view. That isn’t real reporting. It’s republishing a press release, either from Drexel, Lahne or the journal.
I have known guys like Matt. If it i called “science,” it is above criticism. Criticizing it in any way makes you National Enquirer reading, tin foil hat wearing quack to them.
That was a BS study. Period.
I’ve just posted a detailed review of the methods and results of this paper on my whiskyanalysis.com site. There, I go to some length to explain the actual analysis results and their selection of whiskies.
http://whiskyanalysis.com/index.php/2016/05/03/can-you-tell-the-difference-between-bourbon-and-rye/
To summarize, while the actual statistical analysis method is sound, there is great cause for concern of a selection bias in their choice of whiskies – based on previous scientific studies. And their actual results show that a slight tweaking of the selection process would have produced the opposite finding (namely, that ryes and bourbons were discriminated by the panel).
I also cover the points about the study design and participant selection. Short version, there are greater limitations to these than they acknowledge. But the real issue is the whisky selection. It is hard to accept that such a small specific sampling could be balanced, based on their own previously published work.
If I were at a logical debate, I’d accuse Matt of a fallacious appeal to authority. In the real world, I’d accuse him of being an arrogant, pretentious douche.
I’m coming late to this, but anyone who doesn’t think you can’t or shouldn’t challenge the mainstream media didn’t see you lead the charge against CNN for that bogus Scotch shortage story, based solely on the BS opinions of Scotch hoarders in Hong Kong!
I think Matt’s objections are spot-on. Criticism & dismissal of any study one didn’t read is a major concern that smells of confirmation bias. Questioning statistical significance w/o reviewing the results is pure rubbish. Yes, the study did not make the conclusion suggested by this headline.
Note that in Lahne’s 2016 study subject smelled but did not taste! My opinion is that an experienced whiskey drinker, familiar with rye’s characteristic flavor, can readily detect rye, BUT bourbons generally contain rye (some prominently) and the rye aroma is detectable, but not nearly as prominent as rye flavor.
That untrained judges don’t substantially distinguish variable amounts of rye aroma among the sea of unfamiliar and complex scents it’s hardly surprising.
“Confirmation bias” is a pretty sophisticated term for a Kruger-Dunning University graduate. You should feel good about yourself.