(Mis)Understanding Studies

Nate Silver and his merry band of data journalists recently re-launched fivethirtyeight.com, a fantastic site that tries to communicate original analyses of data relating to science, politics, health, lifestyle, the Oscars, sports, and pretty much everything else. It's unsurprising that articles on the site receive a fair amount of criticism. In reading the comments on the articles, I was heartened to see people debate the proper way to explain the purpose of a t-test (we're a long way from the typical YouTube comments section), but a bit saddened that the tone of the comments made them seem more like carping and less like constructive criticism. Instead of saying someone is "dead wrong", why not make a suggestion as to how their work might be improved?

One article on the site got me thinking about a topic I've already been thinking about as I begin teaching classes on news literacy and information literacy: how news articles about research misrepresent findings and what to do about this phenomenon. The 538 piece is wonderfully specific and constructive about what to do. It provides a checklist that readers can quickly apply to the abstract of a scientific article, and advises readers to take into account this information, along with their initial gut reaction to the claims, when deciding whether or not to believe the claims, act on them, or share the information. It applies to health news articles in the popular press, but I think it could be applied to articles about media effects.

Now, the list might not be exhaustive, and there might be totally valid findings that don't possess any of the criteria on the list, but I think this is a good start. And really, that's what I love about 538. I recognize it has flaws, but it is a much needed step away from groundless speculations based on anecdotes that are geared toward confirming the biases of their niche audience (i.e., lots of news articles and commentary). And they appear to be open to criticism. Through that, I hope, they will refine their pieces to develop something that will really help improve the information literacy of the public.

The piece got me thinking about the systematic nature of the ways in which the popular press misleads the public about scientific findings. They tend to follow a particular script: The researchers account for most likely contributors to an outcome in their studies and test these hypotheses in a more-or-less rigorous fashion. The popular press does not mention the fact that they accounted for certain possible contributing factors because of limited space and the need to attract a large, general audience. When people read the news article about the research study, they think "well, there's clearly another explanation for the finding!" But in most (not all, but most) cases, researchers have already accounted for whatever variable you imagine is affecting the outcome.

In other cases, the popular press simply overstates either the certainty that we should have about a finding or the magnitude of the effect of one thing on another thing. Again, if we look at a few things from the original research article (like the abstract and the discussion section), we should be able to know whether or not the popular press article was being misleading, and we wouldn't even have to know any stats to do this. 

The popular press benefits from articles and headlines that catch our eyes and confirm our biases. That's just the nature of the beast. Instead of just throwing out the abundant information around us, it's worth developing a system for quickly vetting it, and taking what we can from it. 

The Two Webs

There are two dialogs on human behavior (which includes political, economic, and social behavior) taking place on the web. In effect, there are two webs.

One web consists of data on human behavior and commentary about this data. This one connects some folks in academia to folks in policy circles and the private sector around the world. This is Big Data.

The advantages of this mode of inquiry is that it harnesses the power of new media technologies to provide more information to help improve our predictive power when trying to understand something as enormously complex as individual and collective human behavior. Many of the critiques of quantitative study of human behavior were grounded in the fact that studies simply didn't have enough information to predict and explain the variance in behavior. Whereas other sciences (physics, chemistry) had enough information about a system to predict outcomes within that system, social sciences did not. But if we were to assume, for a moment, that a team of researchers had access to every single bit of information about every human thought, feeling, or behavior for thousands of years, then that team's ability to predict human behavior would be comparable, I think, to those in other sciences. With better predictions come better answers to questions: how best to minimize suffering, or the spread of disease, or human's impact on the environment, or whatever.

Of course, this mode of inquiry is not without its flaws (or at least perceived flaws). The collection and analysis of so much data on human behavior is viewed as being exploitative in some way: those collecting the information benefit and those who are the subjects do not. There are privacy issues: privacy is seen as a prerequisite to mental and emotional health as well as a means of maintaining some power over determining the course of your life (the actual value of privacy would be difficult to determine within this purely quantified conversation about human behavior). There is also the fear that someone with enough information about human behavior will be able to manipulate people to suit their ends (but if those collecting and analyzing data discuss findings freely and don't hoard secrets, this critique doesn't make much sense to me). It can easily be abused by people thinking there's a causal relationship when then there is only a correlational one. Statistics, when misused, create the illusion of certainty. Statistics could always be misused, but the more powerful and widespread they become, the more likely misuse might be and the damage that could be done.

You might call this the rational web. It views behaviors and events as probablistic (or it should, anyway) and it takes into account the degree to which outcomes are affected. If, say, it was determined that people's political party identification determined the amount they were paid when controlling for occupation, abilities, etc., you could find an answer to the question of how much difference it made in terms of pay (maybe Republicans make $4000 more on average when controlling for other relevant variables). In fact, there is an expectation that you answer the "how much" question.

The other web consists of rhetoric: emotional appeals to pre-existing, deeply held beliefs about human behavior. The most commonly used technique here is to select a few emotionally charged stories and try to get the audience to empathize with them. As access to the web has increased, it has become easier to find a subject (that is, the individual at the center of the story) whose situation exemplifies the pre-existing beliefs about human behavior held by the author of the story and the intended audience. Its easier to find the emotionally charged stories, the one or 3 or 100 personal stories that, when you look at parts of them the right way, support your pre-existing belief that, say, capitalism or socialism is harmful or that a certain policy does more harm than good. This technique connects some other folks in academia with the public at large, particularly disenfranchise members of the public worldwide.

Here, I think one possible danger of the wide-spread use of this technique might be that the echo chamber effect (where certain factions become less able to take the perspective of others and become more hostile towards others) gets stronger. Confirmation bias runs amok, and fewer people take into account new information in order to make better decisions. Any holder of an opinion, no matter how wacky, can find others supporting their opinion. This social support, this sense that one is not alone in one's beliefs, is essential to the persistence or propagation of an idea or ideology. Even if one is in the minority, all one has to do is draw an analogy to a group that was in the minority that eventually became the majority (the rebelling colonists in early America or civil rights crusaders in the later 1950's) in order to justify one's beliefs.

You might call this the emotional web. Rather than being probablistic, it is principled. It rarely asks the question "how racist is a statement?" or "how much privacy is being sacrificed?" or "how much freedom is the right amount of freedom"? In this way, it seems irreconcilable with the rational web.

You can't easily categorize certain websites as one or the other. Two of my favorite news sites - the New York Times and Slate - have some stories that appeal to statistics and analyses of statistics and other stories (usually editorials) that appeal to emotion by cherry-picking individual stories. In fact, a journalistic standard seems to be to combine the two: start with an individual's story and then zoom out to the larger trend. Hook the audience with emotion and convince their inner skeptic with data.

Still, I can see, at the very least, certain blogs that are more emotional or more rational, and it would be interesting to see if certain people gravitated toward either emotional/rhetoric arguments or rational/data-driven ones. Last month, I saw a great paper at the International Communication Association's annual conference by Brian Weeks titled "Partisan enclaves or diverse repertoires? A network approach to the political media environment" that suggested that the self-selection ideological bias (dems watch only MSNBC, repubs watch only Fox News) is a misconception and that personal media repertoires are more diverse, at least ideologically, than many believe. They may be diverse (or rather, balanced) in terms of their emotional or rational content as well. But maybe they are not, in which case we really are two different groups of people having two fundamentally different conversations about human behavior. Definitely an avenue worth exploring.