You may have seen headlines in the past week or so along the lines of “coffee drinkers live longer,” the result of a study that appeared in the latest New England Journal of Medicine. “Great!” the average reader might think, “I’ll brew another pot and add another five years to my life!” However, this is one of those cases where it is important to look beyond the headline and at the actual methodology of the study—which is what Discover did. The primary issue—and, as the NEJM editor pointed out in a subsequent interview with the Boston Globe, publishing the study was fairly controversial editorially—is that the study was an observational study. Essentially, researchers questioned people about their various lifestyle habits. As such, it’s easy to jump to the fallacious conclusion that correlation implies causation, that drinking coffee directly impacts longevity. But in these kinds of studies, there are always “confounding variables” that need to be controlled for. To wit:

In this coffee study, for example, they initially found that coffee drinkers died younger, but coffee drinkers are also more likely to be smokers. When they controlled for smoking as a confounding variable though, the result flipped: coffee drinkers lived longer.

So whenever I see headlines about how some food or beverage has some kind of impact on health, I immediately want to know what the mechanism of that impact would be. It’s not enough to say, “Yeah, coffee drinkers lived longer than non-coffee drinkers.” I want to know why. What substance in coffee impacts human biology to impart greater longevity? (For example, we can specifically identify substances in cigarettes that have been demonstrated to very negatively impact human health.) In the absence of any kind of physical mechanism, it’s just anecdotal, and opens the door to questions about confounding variables. So, as the editor-in-chief of the NEJM pointed out, “the study did not control for health insurance, high blood pressure, or high cholesterol levels” all of which could play a role in the results. This is not to deem observational studies completely useless, or even to denigrate this study in particular; it is studies like this that can identify phenomena that may warrant further, more rigorous study. Rather, the point is to criticize the press, which often repeats these topline findings without context or caveats, which misrepresents the point or conclusions of a study. Gary Taubes has an excellent post on some of the fallacies of nutrition research that relies on observational studies, and why stories about such studies should be taken with a grain of salt (but not too much salt; high blood pressure, you know...or maybe not). Another example of how to interpret scientific research, closer to Going Green’s proper topic, can be found at the Real Climate blog, in which the authors of a study about ice sheets and glaciers put their findings into perspective, revealing nuances that are often absent from the “blunt instruments” that mainstream media stories often come off as. Here is an exchange from the comments section that sums up quite well the cautious approach we should take when looking at the conclusions of studies:

The concluding statements are nicely put. While saying 10X is implausible, and 2X might occur, the reader is still left with the idea that a crisis of ocean rise is possible or probable. The number these researchers are speaking of is considerably less than “crisis”, but by not giving a number, they cannot be criticized – by either side. [Response: Which is precisely right of course, because the reality is that this new paper provides no real guide here. It simply says "here are the data" and if we MUST say something about sea level, here's what we can say. It may be boring, but it's right. By the way, a crisis of ocean rise is actually CERTAIN already if you live in particular places. The question is how many places that will wind up being, and over what timescale.--eric]

Not every scientific study has groundbreaking, life-changing conclusions; more often than not, any given study provides some clarifying data that helps build up a body of knowledge and that, in combination with other and further research, may well lead to paradigm-changing conclusions some day. But that is not every working scientist’s daily goal. Although it would be nice.