(Mis)Understanding Studies

Nate Silver and his merry band of data journalists recently re-launched fivethirtyeight.com, a fantastic site that tries to communicate original analyses of data relating to science, politics, health, lifestyle, the Oscars, sports, and pretty much everything else. It's unsurprising that articles on the site receive a fair amount of criticism. In reading the comments on the articles, I was heartened to see people debate the proper way to explain the purpose of a t-test (we're a long way from the typical YouTube comments section), but a bit saddened that the tone of the comments made them seem more like carping and less like constructive criticism. Instead of saying someone is "dead wrong", why not make a suggestion as to how their work might be improved?

One article on the site got me thinking about a topic I've already been thinking about as I begin teaching classes on news literacy and information literacy: how news articles about research misrepresent findings and what to do about this phenomenon. The 538 piece is wonderfully specific and constructive about what to do. It provides a checklist that readers can quickly apply to the abstract of a scientific article, and advises readers to take into account this information, along with their initial gut reaction to the claims, when deciding whether or not to believe the claims, act on them, or share the information. It applies to health news articles in the popular press, but I think it could be applied to articles about media effects.

Now, the list might not be exhaustive, and there might be totally valid findings that don't possess any of the criteria on the list, but I think this is a good start. And really, that's what I love about 538. I recognize it has flaws, but it is a much needed step away from groundless speculations based on anecdotes that are geared toward confirming the biases of their niche audience (i.e., lots of news articles and commentary). And they appear to be open to criticism. Through that, I hope, they will refine their pieces to develop something that will really help improve the information literacy of the public.

The piece got me thinking about the systematic nature of the ways in which the popular press misleads the public about scientific findings. They tend to follow a particular script: The researchers account for most likely contributors to an outcome in their studies and test these hypotheses in a more-or-less rigorous fashion. The popular press does not mention the fact that they accounted for certain possible contributing factors because of limited space and the need to attract a large, general audience. When people read the news article about the research study, they think "well, there's clearly another explanation for the finding!" But in most (not all, but most) cases, researchers have already accounted for whatever variable you imagine is affecting the outcome.

In other cases, the popular press simply overstates either the certainty that we should have about a finding or the magnitude of the effect of one thing on another thing. Again, if we look at a few things from the original research article (like the abstract and the discussion section), we should be able to know whether or not the popular press article was being misleading, and we wouldn't even have to know any stats to do this. 

The popular press benefits from articles and headlines that catch our eyes and confirm our biases. That's just the nature of the beast. Instead of just throwing out the abundant information around us, it's worth developing a system for quickly vetting it, and taking what we can from it. 

Wordling Through Public Opinion

I've been examining audience interpretations of movies, TV shows, and web video for a few years now. Recently, a friend turned me on to this Wordle application which makes word clouds of any block of text you give it. Word clouds got a lot of press recently when CNN used it to analyze one of Obama's speeches (sadly, people didn't "get" what word clouds can tell them). Anyway, like any good data visualization tool, it allows you to see, at a glance, some of the trends in your data. There are, of course, limitations (as will become apparent in the examples i give below), but it could be an exciting new way to look at public opinion quickly. Take blog posts or comments from thousands of different people, feed it into Wordle, and you'll get a general idea of what that portion of the public is thinking about (or what they think about a certain issue).

Here are the results of my wordle of people's evaluation of several movies on IMDB. I drew the comments from the headings of the first 300 posts for each movie. Check 'em out.
Shawshank Redemption (the highest rated film on IMDB). No big surprises. People think its the best. Its interesting to note that they think its the best ever. If you look closer, you can see adjectives like "brilliant" and "moving." These may not seem surprising, but when you start comparing the different word clouds to each other, even the films that everyone loved, you start to see that fans put emphases on certain elements of certain films.
Amelie (44 on the list of top 250 IMDB). Interesting to see the word "overrated" in there. People define this movie by the fact that its French, and they find it charming.
Love Guru (I had to pick a bad one for contrast). This represents the limits of assuming that word choice (taken out of context) will tell you what people think. The word "funny" might have been part of the phrase "not funny," but Wordle can't tell you that. Its interesting to see that "bad" is used far more than "worst," and that people think of this as a Mike Myers film more than anything (again, obvious, but worth noting). Also, the word "critics" indicates that the critics lambasting of this film carried some weight with people (you don't see "critics" in many other worldes for other films).Casablanca (11 on Top 250 list). Again, "best" and "greatest" are there. "Classic" is there. It would be interesting to see at what year the word "classic" becomes commonly used by people. You would expect it to grow in highly-rated movie word clouds the older the movie is, but maybe there are certain characteristics that make people call a movie "classic" (the centrality of a romance? PG content?). In any case, its far more interesting to me to look at the ways in which hundreds or thousands of viewers use words like "classic" than the ways in which a handful of critics use them. Definately something worth exploring further.

Movies and Data Visualization

Check out this amazing chart (or is it a graph? Maybe I'll just call it an interactive graphic) @ the New York Times. Box office earnings of every movie since 1986 represented graphically. At first glance, it just shows what one would suspect: summers and holidays are when the hit movies come out. But if you look closer, you can follow the paths of each film as its earnings rise and drop. You can quickly see which movies had "long tails." Its interesting to think about what those movies might have in common with one another.

Just looking at this, I start to see the flawed, delayed feedback system that movie creation and distribution is based on, all due to something that is becoming increasingly irrelevant: "shelf space" (or in this case, theater space). Studios release big movies at certain times b/c more people go to the movies then (summer/holidays), but people have started going to movies more at those times b/c that's when the big movies are out, not necessarily b/c that's the only time they want to go see movies. But b/c they're making huge, bloated-budget movies that have to compete with one another, studios have to make us starve for any half-way decent movies during the off-months (this April is pretty bad), not b/c its what we actually want, but b/c of finite shelf-space and bloated budgets, both of which could be (and perhaps are being) eliminated. Here's hoping everyone stays home this summer and watches hulu and YouTube.

Promotional space is still finite, though, so its still the movies with the biggest promo budgets that have the big numbers (they tend to peak faster and drop off faster than indy word-of-mouth hits). I really want to see how well films could do on their own merit. It would be easy enough to make a graph that corrected for promotional budgets, if only you could that information, if only studios weren't so protective of that data. I'm starting some work with a professor here at Michigan on the amount of information that flows into our households each day. Perhaps it would be easier to just track the amount of ads one can see instead of trying to get those stats from the people who put them out there. Advertising and promotion, by definition, is visible. Its trying to be seen, has nowhere to hide. You could just do a web-search for a movie, see how many hits you got, code explicit promos separately from mentions on blogs, etc. If one movie is more visible in its explicit and unofficial promotion than another movie that makes the same amount of money over the course of a similar time period, then you might conclude that the first movie actually succeeded based on its own merits and not on extensive promotion.

Long-term sales of DVDs might also get around the promotion issue, getting us closer to what people actually like.

Specific stats (things like "smallest drop from 5th to 6th week") are available in numerical form on sites like Box Office Mojo. In some cases, you're looking at the graphic representation to find specific things, in which case you'd be better off with a list of numbers. But still, images give you a bunch of patterns to notice that you can't notice right away by looking at numbers.

Mapping Narratives

I thought I'd post an abridged version of the paper I presented at this year's Society for Cinema and Media Studies conference here.

Here's what i had in mind: at any given moment in a film, information is being imparted to the viewer. At the same time, the viewer is aware (to varying degrees) of whether or not various characters are privy to this information. If you see a monster sneaking up behind the heroine, you are aware of the mortal threat to the heroine and you are aware of the fact that she is not aware of the monster. Thus, you feel that the heroine is quite vulnerable, more vulnerable than she would have been had she known that the monster was right behind her.

I'm positing that many (if not all) emotions a viewer experiences are dependent on fluctuations in these relative ranges of knowledge of events over the course of a narrative. For this project, I made some rather crude charts to illustrate my proposed mode of analysis. These charts show the course of movie narratives, and points during that narrative at which a key bit of information (an obvious threat to the protagonist or an obvious aid in helping he or she achieve his or her goal). 1=high level of knowledge ; 0=low level of knowledge. Click on slides for detail.



Flaws: as someone was kind enough to point out at my presentation, I have left out so many subtle aspects of narration (information revelation) in this crude model. The slightest bit of music or glance from a character could be considered a bit of information that would affect our emotions and our perception of goals and threats. To this I would say: I'm working on it. As I said, this model is very basic and crude. While watching movies for this paper, I became aware of the almost infinite amount of information that narratives dole out. And yet, it is not truly infinite. Also, it can be categorized, into threats and goals, by character awareness, perhaps by level of ambiguity. With a more sophisticated data visualization tool, I think I could create a chart that incorporates every bit of information in a narrative that could possibly affect viewer emotion. You could zoom in on various sections of the narratives, or highlight certain types of information.

Then there's the question of motivation. This model works well enough when the goals and threats are well defined, and when viewer identification with the protagonist is relatively straight forward. But what about when its not? When you look at them closely, many movies are bound to have moments of ambiguity concerning motivation, or the status of some bit of information vis a vis a given character. To that, I would say: this is one of the reasons why viewers do not have identical experiences of the same movie. Their experience and values will shape their identification and also their interpretation of some information as possibly helping or hindering a character. And yet, I do not think it is infinitely varied.

In any case, there is a way to set about proving the worth of this mode of analysis. I propose hooking a few viewers up to machines that measure their physiological states and comparing the read out yielded from those sessions with some narrative maps. We may not be able to say that a moment on our viewer physiological reaction chart represents "guilt" or "suspense," but I think we would be able to see fluctuations in the viewer's mental and physiological states. If those fluctuations corresponded to fluctuations on the more sophisticated narrative charts I hope to make in the future, in terms of the fluctuation's spacing, their duration, their degree (which I believe they would) then I think we could start to see how the machinery of narrative actually works on our emotions.

I think we already know the basics of this, intuitively. We're all familiar with the tropes of suspense, with dramatic irony, with farce. This is extending (and exacting) that intuition.