Experts Worry: Predictive News Headlines in the Age of COVID-19

Cataclysms, personal or shared, have a way of distorting your perception of time.

March 12th, 2020 was the day that I felt the Coronavirus crisis escalate. That day, I felt as though I was living simultaneously in three distinct realities. First and most vividly, there was the physical world around me. It was one of those blissful early spring days - bright and breezy. Some friends were meeting up at the local watering hole for a happy hour that was tinged with a different energy than previous ones. I think we knew it might be the last time we'd see each other in person for awhile, but this didn't make us glum. Instead, there was a kind of enhanced camaraderie, laughing at the craziness of it all, because what else was there to do?

Then, existing in what seemed like another universe, there was the reality that existed inside my computer and phone, on news websites and social media: a world quickly falling apart. There was an ever-escalating series of fear-evoking stories, every one of them true.

And then there was the thing itself: the unknowable reality of threat posed by the virus. Though the virus itself is knowable insofar as we are able to know viruses and what they do to various types of human bodies, the matter of immediate concern - the precise way the virus will spread through a given population and affect each individual person - could not be known. That reality is contingent upon too many things to be knowable, at least in mid-March, and perhaps now in early May and for the foreseeable future. The threat of the virus depends on future government policies at federal, state, and local levels, future workplace policies, the speed with which treatments will be developed and manufactured, and the future behavior of billions of individuals, as each individual's decision to, say, stay home and watch Netflix or say 'fuck it' and go out to a bar (when bars were/will be open) affects all downstream outcomes. Scientists of various stripes can imperfectly predict the spread of the virus and its deadliness by looking at various models based on prior behavior, but they cannot as yet know with absolute certainty (or really much certainty at all, it would seem) who the virus will infect in a particular social context, when it will infect them, how long it will stick around a population, who or how many it will kill.

As I moved into the gently mandated quarantine stage of the event, I started to think more about the disjuncture among these three worlds. A couple of months previous, I'd had time to think about the ways in which the social internet (news, commentary about current events on social media) presented its users with a distorted view of the world. The long and short of it is this: it enhances threats.

In the case of the virus, the threats are multiple. There is the virus itself, and then there is the effect of virus containment on the economy. Also, as always in the U.S. and perhaps elsewhere, there is the threat of the other political tribe: Would Trump invoke martial law? Would protesters spread the virus? Would liberals exaggerate the threat to make Trump look bad?

The key word in all of these questions is 'would.' I came to realize that many of the headlines I read contained words like 'would,' 'could,' 'may.' Some of the bolder ones contained the word 'will.'

They were about bad things that had not happened yet. 

On a podcast from mid-March, Malcolm Gladwell recounted something he had read that day that quoted experts from the University of California San Francisco, a leading medical school. The experts predicted that there would be more than 1 million Americans deaths from the virus.

From March 17, 2020 on the New York Times: 'There may be two to four more rounds of social distancing.'

And later, from CNN.com on April 14th, 2020: 'US may have to keep social distancing until 2022, scientists predict.'

From May 4th in the Washington Post: 'Draft report predicts covid-19 cases will reach 200,000 a day by June 1.' This article alluded to a leaked report from the CDC that also predicted that there will be 3,000 deaths per day in the second half of May.

These headlines were accompanied by opinion-piece headlines that would have seemed more at home on less prestigious news websites a few months ago. From the New York Times: 'One simple idea explains why the economy is in great danger'; 'Stop saying everything is under control. It isn't'.' 'More severe than the great recession.'

Most of the headlines and stories quoted experts who offered informed predictions about the virus or the economy. Right away, I thought of Phil Tetlock's work on expert political judgment. Tetlock found that when political experts of all ideological stripes were forced to make falsifiable predictions about a range of outcomes, they were not much better than chance or non-experts. The more famous the experts were, the worse their prediction records were. In his book The Signal and the Noise, Nate Silver reviews the incentives experts have to make predictions, why they are rewarded for more outrageous predictions with more coverage and fame, and why they are not punished for being wrong.

There are various lessons you might take away from Tetlock's ongoing project to assess the ability of experts to forecast a range of outcomes. The one I keep coming back to is that forecasting any outcomes that involve a large number of people's behavior is really difficult. The more people that may influence the outcome (that is, the more people who's individual behavior is part of the system you're trying to observe and extrapolate from), the harder the outcome is to predict.

Some of the predictions about the virus and the economy that dominated headlines were not falsifiable, as they did not provide a time range and thus could eventually be proven true even if they were not true on a particular date (they would never be false, but simply not true yet). But many were falsifiable: they made specific predictions about the duration or magnitude of an economic recession or depression; they made specific predictions about the number of infections or deaths resulting from the virus (the falsifiability of which assumes you accept data collected by authorities).

Sometimes, the experts would try to convey their levels of uncertainty in their predictions. Sometimes, they would not. Most times, this uncertainty level would not be conveyed in the news article; it was almost never conveyed in the headlines.

Aside from the inherent unpredictability of large groups of people, there is another reason why many forecasts relating to the virus or the economy will turn out to be wrong. Predictions like this can function as warnings that are then heeded by people who take action, which prevents the predicted outcome from occurring. Nate Silver calls this a 'self-cancelling prediction' or 'self-cancelling prophecy.' Similar problems plague predictions about environmental catastrophe: the more dire the predictions, the more likely they are to spur innovation or regulation that prevents the predictions from coming true.

I wonder if some folks are engaging in a kind of deliberately misleading, exaggerating framing of virus threats. Perhaps journalists and those who post on social media are aware of the shortcomings of the data they are working with, aware that they are focusing on the most dire scenarios and ignoring others. They do this because they believe, rightly, that the more dire the predictions, the more likely they will be to spur action that will prevent the more dire predictions from coming true.

I think that people often derive the wrong lesson from self-cancelling predictions. They do not prove the predictions to have been correct. It is possible that the prediction would have been wrong had no action been taken. In and of themselves, they do not provide much evidence of the accuracy of the prediction. I think the better lesson to draw from the possibility of self-cancelling predictions is that in order to have faith in our predictions in which those who learn of the prediction might plausibly affect its outcome, we must understand the mechanisms by which the predicted outcome will or won't occur. We must be able to account for the effects of particular behaviors in isolation (e.g., the effect of social distancing on viral transmission; the effect of carbon monoxide on sea levels) in order to really understand and predict complex phenomena.

The recent spate of predictive headlines brings to mind another domain examined in Nate Silver's book: weather predictions. Meteorologists' predictions of the weather on any given day were often wrong; no surprise there, as weather is another complex, hard-to-predict system. What's interesting is that the errors were systematic: meteorologists tended to predict rain on days that turned out to be sunny more often than they predicted sun on days that turned out to be rainy. As a reason for this, Silver noted that meteorologists were 'punished' for one type of wrong answer more severely than they were for the other. Most people saw sun on a supposedly rainy day as a pleasant surprise, while they tended to get angry at meteorologists who failed to warn them about the negative outcome: rain on a supposedly sunny day.

Many of the predictions dominating news headlines will inevitably turn out to be wrong, but will they be systematically wrong, wrong in a particular direction? I suspect that most journalists and news consumers see virus infections, deaths, and economic hardship in much the same way people see rain: they would rather the predictions turn out to have been too dire than not dire enough.

However, this creates a problem. If the predictions about virus infections and deaths are unnecessarily dire, this will cause consumers to spend less and investors to invest less, leading to worse economic outcomes. If the predictions about the economy are too dire, policy makers, business owners, voters, and consumers will push for re-opening too soon, resulting in worse health outcomes. All unnecessarily dire predictions will likely harm people's mental and emotional health, and any wrong prediction will harm subsequent trust in news sources.

I suspect that predictions in most mainstream news outlets will overestimate negative outcomes associated with the virus and underestimate negative outcomes associated with the economy. Of course, there's a political element to the predictions (those on the Left tend to be more concerned with the virus while those on the Right tend to be more concerned with the economy), but beyond that, I think the negative outcomes associated with the virus (mass death; dying alone) are more vivid, more viscerally repellent than those associated with the economy (lagged rises in social unrest, substance abuse, domestic abuse, and violent crime that typically accompany prolonged mass unemployment) which tend to be more diffuse and less easily depicted.

What to do about all this? Well, before going any further, it seems worthwhile to test all of my assumptions. In the spirit of putting my money where my mouth is, here are a few falsifiable hypotheses:

• The number of 'predictive headlines' (i.e., headlines that relay information about an event or state of the world that has yet to occur at the time of publication) has increased since the middle of March 2020. 

• Of the predictive headlines that are falsifiable at present, more headlines will have overestimated threats than will have correctly estimated or underestimated threats. 

• The more vivid the threat, the greater the magnitude of the error in prediction. 

• News consumers exposed to more dire predictions will be more likely to take action (or intend to take action) than those exposed to less dire predictions. 

• The inclusion of information about the confidence levels of experts (e.g., swapping out the word 'will' for the word 'could' or 'might') will have no effect on news consumers' behaviors or intentions. 

To motivate journalists and those who post on social media not to post speculative 'news,' perhaps we could shame the behavior with a catchy, albeit misleadingly reductive moniker: 'eventually fake news,' or something like that.

I can understand the desire to compulsively speculate at a time like this. Typically, there's a certain amount of uncertainty in the world. You might not know some of the details about what will happen over the next year, but you often have a rough idea of what it will be like, what you'll do, where you'll be. At a time when so much is uncertain, maybe we can't help ourselves from making predictions, even if we know most of them will turn out to be wrong. But even in times of great uncertainty, there must be something we can learn from our wrongness. Right?

Puppies & Iraq

I just saw Page One, a documentary about the New York Times, which raised some interesting (if oft repeated) questions about journalism that come along with the financial instability of the industry: is there something about a traditional media outlet like the NYTimes that is superior to the various information disseminating alternatives (news aggregation sites, twitter, Facebook, Huffpo, Daily Kos, Gawker, etc.) and, if so, what is it? What is it about the New York Times (or the medium of newspapers in general) that would be missed if it was gone?

Bernard Berelson asked a similar question in a study of newspaper readers who were deprived of their daily newspaper due to a workers' strike in 1945. The reasons people liked (or perhaps even needed) the paper back then - social prestige, as an escape or diversion, as a welcome routine or ritual, to gather information about public affairs - are all met by various other websites and applications, some of which seem to be "better" - that is, more satisfying to the user - at one or all of these things than any newspaper is.

I want to pick apart this idea of that which is "more satisfying" to the user, or what it means to say that they "want" something. The mantra of producers in the free market, no matter what they're selling, is that they must give the people what they want. Nick Denton of Gawker has a cameo in Page One in which he talks about his "big board", the one that provides Gawker writers with instant feedback about how many hits (and thus, how many dollars) their stories are generating. Sam Zell, owner of the Tribune media company, voiced a similar opinion: those in the information dissemination business should give people what they want. Ideally, you make enough money to do "puppies and Iraq" - something that people want and something that people should want. To do anything else is, to use Zell's phrase, "journalistic arrogance".

Certainly, a large number of people are "satisfied" with the information they get from people like Denton and Zell. But Denton and Zell, like any businessmen, can only measure satisfaction in certain ways: money, or eyeballs on ads. There are other, often long-term social, costs paid when people get what they supposedly want. When news is market driven, the public interest suffers. So goes the argument of many cultural theorists. But who are they to say what the public interest is? Why do we need ivory tower theorists to save the masses from themselves?

Maybe that elitist - the one who would rather read a story about Iraq than look at puppies - is not in an ivory tower but inside of all of us, along with an inner hedonist (that's the one that would rather look at puppies all day). There are many ways to measure what people like, want, need, or prefer. I'm not talking about measuring happiness as opposed to money spent/earned. I'm considering what happens when we're asked to pay for certain things (bundled vs. individually sold goods) at certain times (in advance of the moment of consumption vs. immediately before the moment of consumption). There is plenty of empirical evidence to suggest that those two variables, along with many others situational variables external to the individual, alter selection patterns of individuals. Want, or need, or preference does not merely emanate from individuals. When we take this into account, we recognize that a shift in the times at which individuals access options and the way those options are bundled together end up altering what we choose. We click on links to videos of adorable puppies instead of links to stories about Iraq because they're links (right in front of us, immediate) and because they've already been paid for (every internet site is bundled together, and usually bundled together with telephone and 200 channels of television). If it wasn't like that, if we had to make a decision at the beginning of the year about whether we "wanted" to spend all year watching puppy videos or reading about Iraq...well, I guess not that many people would want to spend all year reading about Iraq. But I reckon that many people would want, would choose some combination of puppies and Iraq if they had to choose ahead of time. The internet is a combination of what we want and what we should want, and so is the NYTimes, but they represent a different balance between those two things. The Times is 100 parts puppies, 400 parts Iraq. The internet is 10000000000 parts puppies, 100000000 Iraq (or something to that effect. When you change how things are sold, you may not change what people want, as many theorists claim, but you do change how we measure what people want.

Maybe we never have to defer to a theorist to tell us what we should be reading or watching in order to be a better citizen. Maybe we just need to tweak our media choice environment so that it gives the inner elitist a fighting chance against the inner hedonist.

Portable Technology: Size, Time, & Weight

My new computer - a netbook - has me thinking about how the physical characteristics of a device can influence how I feel about it and then what I do with it. I'm not talking about technological affordance - what the software/hardware allow me to do - but rather how the size and weight of the object influence my feelings about it.

My first working hypothesis: the smaller the device, the more "handy" it is, the more it will be suited for short bursts of use. Its hard to bust out anything bigger than my hand when I'm on the go, on a bus or walking around campus. Also, the use of these smaller devices for longer periods of time seems somehow fatiguing. Trying to block out all this other sensory information while concentrating on a smaller screen for a prolonged period of time is more difficult than concentrating on a larger laptop screen. For these reasons, smaller devices = shorter duration of use sessions.

Then I thought about whether weight has anything to do with use. I don't bring my laptop everywhere I bring my netbook b/c of the weight of my laptop. Its not so much that its literally too heavy for me to carry, but that it feels burdensome. I'm constantly reminded that its in my backpack. If I had a super light MacBook Air, I might feel better about bringing it more places b/c I wouldn't feel burdened by its presence.

So far, I'm finding that the netbook is making me more productive b/c I can "sneak up on myself" and start working on a project. This is inspired by a project I'm embarking on regarding study habits and affirmation (with Emily Falk and Elliot Berkman) related to my dissertation work on self control and virtue/vice media habits. Basically, if I think about going to a place (usually my office in my house), sitting down and doing work, I don't feel good about it, and I tend to avoid that place. But if I can take the "place" of work out of the equation, if I can get to work as impulsively as I can engage in time-wasting leisure activity, if work becomes as accessible as play, then I think I can get to work before I have a chance to dread it. At least that's the way it worked today.

How Election Coverage Can Decide Elections

I've been thinking about the press coverage of Reverend Wright's recent speeches, in particular the coverage of the major cable news networks and the NYTimes, though I'd suspect what holds true for these outlets holds true for most media. All agree that there are two priorities that, at times, conflict with one another: getting a certain candidate elected (Obama) and that candidate or other high-profile people linked (however vaguely) to the candidate being able to speak their minds. How much must one sacrifice in order to get elected? How many games does a candidate have to play?

To answer the question, you have to look at the poll numbers. Its a common complaint that the public is too focused on poll numbers, not focused enough on the issues, and this is b/c the news media frames elections as "horse races." I think that heavy use of the "horse race" frame (which emphasizes poll #s over all else) leads to more frequent/larger shifts in those numbers. That is, the more self-aware a public becomes of its opinion, the more likely it is to shift. The reasons for the shift are essentially arbitrary. It might be Reverend Wright, it might be "bitter-gate." There will always be something that either the competing candidate, the news media, or bloggers who support the competing candidate will exploit, either to get their candidate elected, to raise their own stature as "opinion leader," and/or to boost their ratings and make a profit. Elisabeth Noelle-Neumann's Spiral of Silence sums up this brilliantly. Its a must-read for anyone who is genuinely trying to understand why the primary is going the way it is going.

Fluctuations in polls are not due to the larger public's reaction to an event (like Wright's "controversial" remarks on race), nor are they the inevitable result of increasingly visible poll numbers per se (hating the pollsters and the NYTimes for posting poll #s gets us nowhere). The larger public reacts to professional interpretation of minor fluctuations in public opinion. First, the media (main stream or bloggers, doesn't matter) select an event which they can interpret as "controversial" enough to plausibly effect voter opinion. Then they limit their polling to one small but purportedly influential segment of the general populace (undecideds, superdelegates, other bloggers, white working class females between 25-40 since last Tuesday). How long this time period is and what the event happens to be are of no consequence. Both main stream media and bloggers will dig until they find an event that can be spun as controversial and a small enough sliver of the public to show that there is some movement in the polls that is plausibly correlated to that event. In doing this, they justify their own existence. They are the source of information about public opinion, and our conception of public opinion is, for better or worse, what we base our voting decisions on (if you don't believe this, read Noelle-Neumann's book).

This creates a cycle: larger and larger segments of the population accept the premise that public opinion is being altered by the event, making the connection between the event and public opinion ever more plausible.

It becomes acceptable (perhaps laudable) to change one's position on a candidate. This is the "change" election in the sense that voters are expected to change their opinion on candidates several times over the course of the year.

Why does all this work against Obama? Maybe b/c he was ahead, and favorite-toppled-by-underdog makes for a more compelling story than underdog-can't-come-back, which is why both candidates were trying so desperately to frame themselves as underdogs. It doesn't help that most people who publicly rush to Obama's defense are perceived by many as elite (the digerati, the NYTimes).

I think that the new technology and the ways it allows information to spread changes how public opinion fluctuates and so it changes how our leaders are elected. The first step is to understand how it works, to give up, for a moment, our dreams of perfect democracy or a perfect candidate as well as our nightmares of a totalitarian mainstream media cabal. Just take a step back and try to see how it all works. Then you can make your value judgment and think about how you might change the system. Personally, I think that the way out of this bind is...another technological innovation. I've seen innovations on assessing user demand work on small scales, on YouTube or within online communities that introduced wiki-ratings or similar widgets. Different tools, all under the heading of "new media" or "internet," can change the flow of public opinion and could get us to recognize how we're all shaped by public opinion and yet all have the power to resist it and decide things based on judgments of the candidates and the issues.

Terrorism and Contagious Media

There was a truly provocative blog post on the NYTimes' Freakonomics by Stephen Levitt. The blog entry solicited ideas on which terrorist attack would wreak the most havoc. Predictably, the article prompted both scathing rebukes and praise for its openness in roughly equal measure.

It got me to thinking about whether the scathing critiques had any merit. As I understand it, the worry of most of the naysayers was that either terrorists or unhinged people looking for ways to lash out at the world would get ideas from this blog and be more apt to try to carry those ideas out. It might be that they actually get a specific idea on how to cause the most fear, or it might be that just talking about terrorism in this manner puts it in the forefront of their minds and gets them to act out while not necessarily cribbing an actual idea from the website.

This is similar territory to that which I covered in my blog post-VA Tech. In both cases, we imagine an unhinged individual with nothing to live for who wants some sort of revenge on the rest of the world. He has this nebulous rage built up, but its unclear as to how it will be released. Maybe if he is presented with one set of stimuli (say, a lot of ultimate fighting videos and some death metal), he will train to become an ultimate fighter and beat the shit out of similarly frustrated young males. If he is presented with another set of stimuli (say, non-stop coverage of a mass murder or extensive, detailed speculation as to how to carry out a terrorist attack that would cause the most fear), then he might be more inclined to carry out such an act. A third set of stimuli might prompt him to merely kill himself, etc. With Virginia Tech, the worry seemed to be more emotional than logistic. The images of the gunman had a certain visceral power that offended people. In the case of today's NYTimes blog, its just words.

Many comments on the blog that fall into the pro-openness, pro-Levitt category take a "cat's out of the bag" approach to the potential harmfulness of information. This assumes that all nodes on the information network are equal. If a bit of information is on some obscure message board, then its liable to have the same effect on people's behavior as if it were on a higher-profile webpage. The linked nature of the internet means that if a bit of information is interesting, funny, or dangerous enough to warrant attention, it will get attention via digg, delicious, or the viral spread of blogs, vlogs, and emails.

Here's my problem with that reasoning as it applies here. What Levitt wrote isn't what might actually cause harm. He sketched out only one scenario. Its the aggregation of reader comments that could contain the terrorist scenarios that are superior to any that have been thought of before.

I've been waiting for the Wisdom of Crowds wiki-logic to hit the war on terror. By aggregating these scenarios, we seem to be doing the terrorists' work for them. It takes time, energy, and intellectual ability to think up plausible scenarios for terrorist attacks. One writer (e.g. Tom Clancy) could be pretty good at that, and a bunch of devoted terrorists could be just as good if not better, but a larger group of well-educated, creative people (if they worked collectively) would certainly be better at it than either Clancy or the terrorist. So I think you'd be mistaken to say that if we can come up with a bright idea for causing terror, it would've already been thought of. Even the most sophisticated think tank is probably no match for the collective wisdom of the NYTimes' readership (as I pat myself on the back).

Then there's this paradox: the people who think the information is harmful and comment accordingly are, in some sense, aiding and abetting the harmful information by making it more visible. In the inexorable logic of online community popularity, if a comment has many comments, it is more likely to be considered "important," to be forwarded, to be read. The virus spreads.

If indeed this discussion is followed by a large scale terrorist attack (or a few of them), we shouldn't assume that it caused it/them, nor should we fall back on the well worn truth that terrorism is extremely uncommon and therefore is nothing to worry about. Personally, I have never been hit by a car even once in my life. Does this mean that I shouldn't look both ways before crossing a street? We have lived in a world where Tom Clancy and other writers have dreamed up scenarios for terrorism, and one where groups of terrorists have spent a lot of time and energy thinking up ways to disrupt societies, but I don't think we've ever had an instance of a large number of creative, intelligent people brainstorming about ways to cause mass fear. In the sense that this is unprecedented, I think its impossible to definitively say whether or not this kind of openness is necessarily good.

But it could be good. The quicker we can think of potential problems, the quicker we can plan solutions. Dubner and Levitt have always been convinced that people are worried about the wrong things (handguns instead of swimming pools, for instance). There are those who believe that the threat of terrorism is way overblown and think that Levitt's exercise proves that by showing the disparity between possible scenarios (lots) and actual events (very, very few), we prove this. But I would say that we might learn what kinds of terrorism would be worth prepping for by discussing in this way. If we talk about it openly, we might discover that we should spend the money we're spending on airline safety on protecting the food and water supply or developing a well-known, well practiced quarantine protocol. Dubner and Levitt are all about correcting conventional wisdom when its out of whack, and this would seem to be an instance where they're needed.

You could also argue that by familiarizing us with possible doomsday scenarios, the article and the discussion makes eventual hysteria less likely. And really, that's what would cause a society to collapse: not the attack, but the ensuing hysteria. If we can convince potential attackers that we'll bounce right back from an attack (either b/c of our preparedness, our short attention span, or both), they'll be less likely to attack in the first place.

Hmm. Short attention spans...

So maybe its good that our attention spans have been whittled away by advertisements. This way, we can't stay scared for very long.