Experts Worry: Predictive News Headlines in the Age of COVID-19

Cataclysms, personal or shared, have a way of distorting your perception of time.

March 12th, 2020 was the day that I felt the Coronavirus crisis escalate. That day, I felt as though I was living simultaneously in three distinct realities. First and most vividly, there was the physical world around me. It was one of those blissful early spring days - bright and breezy. Some friends were meeting up at the local watering hole for a happy hour that was tinged with a different energy than previous ones. I think we knew it might be the last time we'd see each other in person for awhile, but this didn't make us glum. Instead, there was a kind of enhanced camaraderie, laughing at the craziness of it all, because what else was there to do?

Then, existing in what seemed like another universe, there was the reality that existed inside my computer and phone, on news websites and social media: a world quickly falling apart. There was an ever-escalating series of fear-evoking stories, every one of them true.

And then there was the thing itself: the unknowable reality of threat posed by the virus. Though the virus itself is knowable insofar as we are able to know viruses and what they do to various types of human bodies, the matter of immediate concern - the precise way the virus will spread through a given population and affect each individual person - could not be known. That reality is contingent upon too many things to be knowable, at least in mid-March, and perhaps now in early May and for the foreseeable future. The threat of the virus depends on future government policies at federal, state, and local levels, future workplace policies, the speed with which treatments will be developed and manufactured, and the future behavior of billions of individuals, as each individual's decision to, say, stay home and watch Netflix or say 'fuck it' and go out to a bar (when bars were/will be open) affects all downstream outcomes. Scientists of various stripes can imperfectly predict the spread of the virus and its deadliness by looking at various models based on prior behavior, but they cannot as yet know with absolute certainty (or really much certainty at all, it would seem) who the virus will infect in a particular social context, when it will infect them, how long it will stick around a population, who or how many it will kill.

As I moved into the gently mandated quarantine stage of the event, I started to think more about the disjuncture among these three worlds. A couple of months previous, I'd had time to think about the ways in which the social internet (news, commentary about current events on social media) presented its users with a distorted view of the world. The long and short of it is this: it enhances threats.

In the case of the virus, the threats are multiple. There is the virus itself, and then there is the effect of virus containment on the economy. Also, as always in the U.S. and perhaps elsewhere, there is the threat of the other political tribe: Would Trump invoke martial law? Would protesters spread the virus? Would liberals exaggerate the threat to make Trump look bad?

The key word in all of these questions is 'would.' I came to realize that many of the headlines I read contained words like 'would,' 'could,' 'may.' Some of the bolder ones contained the word 'will.'

They were about bad things that had not happened yet. 

On a podcast from mid-March, Malcolm Gladwell recounted something he had read that day that quoted experts from the University of California San Francisco, a leading medical school. The experts predicted that there would be more than 1 million Americans deaths from the virus.

From March 17, 2020 on the New York Times: 'There may be two to four more rounds of social distancing.'

And later, from CNN.com on April 14th, 2020: 'US may have to keep social distancing until 2022, scientists predict.'

From May 4th in the Washington Post: 'Draft report predicts covid-19 cases will reach 200,000 a day by June 1.' This article alluded to a leaked report from the CDC that also predicted that there will be 3,000 deaths per day in the second half of May.

These headlines were accompanied by opinion-piece headlines that would have seemed more at home on less prestigious news websites a few months ago. From the New York Times: 'One simple idea explains why the economy is in great danger'; 'Stop saying everything is under control. It isn't'.' 'More severe than the great recession.'

Most of the headlines and stories quoted experts who offered informed predictions about the virus or the economy. Right away, I thought of Phil Tetlock's work on expert political judgment. Tetlock found that when political experts of all ideological stripes were forced to make falsifiable predictions about a range of outcomes, they were not much better than chance or non-experts. The more famous the experts were, the worse their prediction records were. In his book The Signal and the Noise, Nate Silver reviews the incentives experts have to make predictions, why they are rewarded for more outrageous predictions with more coverage and fame, and why they are not punished for being wrong.

There are various lessons you might take away from Tetlock's ongoing project to assess the ability of experts to forecast a range of outcomes. The one I keep coming back to is that forecasting any outcomes that involve a large number of people's behavior is really difficult. The more people that may influence the outcome (that is, the more people who's individual behavior is part of the system you're trying to observe and extrapolate from), the harder the outcome is to predict.

Some of the predictions about the virus and the economy that dominated headlines were not falsifiable, as they did not provide a time range and thus could eventually be proven true even if they were not true on a particular date (they would never be false, but simply not true yet). But many were falsifiable: they made specific predictions about the duration or magnitude of an economic recession or depression; they made specific predictions about the number of infections or deaths resulting from the virus (the falsifiability of which assumes you accept data collected by authorities).

Sometimes, the experts would try to convey their levels of uncertainty in their predictions. Sometimes, they would not. Most times, this uncertainty level would not be conveyed in the news article; it was almost never conveyed in the headlines.

Aside from the inherent unpredictability of large groups of people, there is another reason why many forecasts relating to the virus or the economy will turn out to be wrong. Predictions like this can function as warnings that are then heeded by people who take action, which prevents the predicted outcome from occurring. Nate Silver calls this a 'self-cancelling prediction' or 'self-cancelling prophecy.' Similar problems plague predictions about environmental catastrophe: the more dire the predictions, the more likely they are to spur innovation or regulation that prevents the predictions from coming true.

I wonder if some folks are engaging in a kind of deliberately misleading, exaggerating framing of virus threats. Perhaps journalists and those who post on social media are aware of the shortcomings of the data they are working with, aware that they are focusing on the most dire scenarios and ignoring others. They do this because they believe, rightly, that the more dire the predictions, the more likely they will be to spur action that will prevent the more dire predictions from coming true.

I think that people often derive the wrong lesson from self-cancelling predictions. They do not prove the predictions to have been correct. It is possible that the prediction would have been wrong had no action been taken. In and of themselves, they do not provide much evidence of the accuracy of the prediction. I think the better lesson to draw from the possibility of self-cancelling predictions is that in order to have faith in our predictions in which those who learn of the prediction might plausibly affect its outcome, we must understand the mechanisms by which the predicted outcome will or won't occur. We must be able to account for the effects of particular behaviors in isolation (e.g., the effect of social distancing on viral transmission; the effect of carbon monoxide on sea levels) in order to really understand and predict complex phenomena.

The recent spate of predictive headlines brings to mind another domain examined in Nate Silver's book: weather predictions. Meteorologists' predictions of the weather on any given day were often wrong; no surprise there, as weather is another complex, hard-to-predict system. What's interesting is that the errors were systematic: meteorologists tended to predict rain on days that turned out to be sunny more often than they predicted sun on days that turned out to be rainy. As a reason for this, Silver noted that meteorologists were 'punished' for one type of wrong answer more severely than they were for the other. Most people saw sun on a supposedly rainy day as a pleasant surprise, while they tended to get angry at meteorologists who failed to warn them about the negative outcome: rain on a supposedly sunny day.

Many of the predictions dominating news headlines will inevitably turn out to be wrong, but will they be systematically wrong, wrong in a particular direction? I suspect that most journalists and news consumers see virus infections, deaths, and economic hardship in much the same way people see rain: they would rather the predictions turn out to have been too dire than not dire enough.

However, this creates a problem. If the predictions about virus infections and deaths are unnecessarily dire, this will cause consumers to spend less and investors to invest less, leading to worse economic outcomes. If the predictions about the economy are too dire, policy makers, business owners, voters, and consumers will push for re-opening too soon, resulting in worse health outcomes. All unnecessarily dire predictions will likely harm people's mental and emotional health, and any wrong prediction will harm subsequent trust in news sources.

I suspect that predictions in most mainstream news outlets will overestimate negative outcomes associated with the virus and underestimate negative outcomes associated with the economy. Of course, there's a political element to the predictions (those on the Left tend to be more concerned with the virus while those on the Right tend to be more concerned with the economy), but beyond that, I think the negative outcomes associated with the virus (mass death; dying alone) are more vivid, more viscerally repellent than those associated with the economy (lagged rises in social unrest, substance abuse, domestic abuse, and violent crime that typically accompany prolonged mass unemployment) which tend to be more diffuse and less easily depicted.

What to do about all this? Well, before going any further, it seems worthwhile to test all of my assumptions. In the spirit of putting my money where my mouth is, here are a few falsifiable hypotheses:

• The number of 'predictive headlines' (i.e., headlines that relay information about an event or state of the world that has yet to occur at the time of publication) has increased since the middle of March 2020. 

• Of the predictive headlines that are falsifiable at present, more headlines will have overestimated threats than will have correctly estimated or underestimated threats. 

• The more vivid the threat, the greater the magnitude of the error in prediction. 

• News consumers exposed to more dire predictions will be more likely to take action (or intend to take action) than those exposed to less dire predictions. 

• The inclusion of information about the confidence levels of experts (e.g., swapping out the word 'will' for the word 'could' or 'might') will have no effect on news consumers' behaviors or intentions. 

To motivate journalists and those who post on social media not to post speculative 'news,' perhaps we could shame the behavior with a catchy, albeit misleadingly reductive moniker: 'eventually fake news,' or something like that.

I can understand the desire to compulsively speculate at a time like this. Typically, there's a certain amount of uncertainty in the world. You might not know some of the details about what will happen over the next year, but you often have a rough idea of what it will be like, what you'll do, where you'll be. At a time when so much is uncertain, maybe we can't help ourselves from making predictions, even if we know most of them will turn out to be wrong. But even in times of great uncertainty, there must be something we can learn from our wrongness. Right?

‘The Curdling of the Internet’: The Open Public Sphere Internet as Threat Amplifier

A weekend retreat to a cabin in the woods provided me with time to finally get around to reading Jia Tolentino’s essay, ‘The I in the Internet.’ In a way, I’ve been thinking about what Tolentino describes as the ‘curdling’ of the internet for the past two years, since our research team started reviewing research on various forms of online hostility. More broadly, I’ve been thinking about the effects of social media on societies for the past decade.

I’m generally opposed to sweeping pronouncements about the negative effects of the internet, smartphones, or social media, mostly because it doesn’t seem to fit the evidence. Social media use, by itself, doesn’t seem to have much of an effect on well-being or depression; that is, if one person spends three hours a day on social media and the other spends one hour, or no hours, the person who uses more social media is no more likely to feel bad about themselves or about life in general. There’s also evidence that the internet exposes users to a wider range of opinions, rather than sorting users into filter bubbles or extreme echo chambers. Bad things may be happening in today’s world, and social media use might have increased at the same time, but that doesn’t mean that one caused the other to occur.

And yet, despite the apparent lack of evidence of a strong, negative effect of social media use on individuals, I can’t help but wonder if we, as researchers, are missing something, and how we might adjust so as to capture those things.

One way Tolentino’s feeling and the lack of individual, direct effects on social media users can both be true is if the internet/smartphones/social media are having a profoundly negative effect on society in general, but it occurs in some kind of indirect way. The most concise way of describing my hunch about this would be the ‘threat amplification’ effect. These technologies amplify certain voices, making certain kinds of people and behavior that were formerly invisible more visible. A sub-set of highly active social media users see the platforms as battlefields on which an ideological war is being fought, and so they post information that furthers their agendas, and/or a subset of especially passionate people simply express how strongly they feel about some issue. These expressions are perceived as threatening by a sub-set of other users, not necessarily because they contain explicit threats of violence, but maybe because they clash with some fundamental belief of theirs, or portend an escalation in the encroachment on their rights (e.g., the right to free speech; the right to exist; the right to defend themselves). These posts stick out to many people, draw attention to themselves in the way that any perceived threat in our environment sticks out to us. If future visibility of posts is determined through the amount of attention paid to them (via algorithms or 'most read' lists, etc.), then we tend to see more and more of these types of posts as time goes on.

Also, well-meaning users (journalists, re-tweeters, etc.) draw further attention to these posts in an effort to make sure that others see the threat because to ignore the threat would be hazardous. They might do so out of a sense of duty and compassion to others (i.e., they see others are in danger from the threat and must warn them) but might also do so out of self-interest, or in-group-interest. By drawing otherwise apathetic individuals’ attention to the threat, they may enlist them to join their fight. In such a way, much of the online discourse comes to resemble either a threat or a response to a threat.  

If we react to this by leaving the internet, by not participating, this may preserve our piece of mind, but it cedes the public forum of the internet to those holding more extreme views. So, perhaps we try to meet the threat with an equal and opposite assertion of our beliefs or values (which may, of course, be viewed by others as extreme and threatening). Or perhaps we just conform to the now-established norm of value-assertion-through-strong-takes in order to be heard (because in order to gain greater visibility in the form of likes, shares, and subscribers, it helps to conform to norms of what is popular), or simply because humans are prone to unconsciously conform to social norms of expression. All of these would lead to an internet that actually is becoming more threatening (again, defining 'threats' not in terms of what was intended, but how they are perceived by someone), and exposure to it would likely lead to increased polarization, depression, abuse, harassment, toxicity, etc. The internet may not have started out that way, but it could be the case that it is becoming that.

And that’s the tricky part. The internet that Robert Kraut analyzed 20 years ago in his landmark study of the effects of internet use had certain kinds of content and certain kinds of experiences and perspectives posted by users, and the internet that the research summed up in Hancock et al. has other content/experiences/perspectives, and today’s/tomorrows internet has/will have other content/experiences/perspectives. And if those three stages of the internet are sufficiently different from one another in terms of their contents, they may have very different effects. Thus, though we haven’t observed strong negative consequences of social media use, it’s impossible to rule out them occurring in the future, or right now, given the publishing lag and, perhaps, that we're looking at the wrong outcome variables.

The negative effects of internet use may not be depression, narcissism, or physical aggression. They may manifest themselves in voting behavior (e.g., voting for increasingly partisan, extreme candidates, so as to counter the perceived threat from the other side), or some other behavior that reflects a distrust in others (e.g., unwillingness to live in certain places or send one’s kids to schools with certain types of people, a kind of self-segregation). Eventually, this kind of mutual distrust may manifest itself in physical violence. Or perhaps we’ll view one another as very, very different and threatening, but ultimately leave one another alone, co-existing peacefully if not without mutual resentment (stranger things have happened).

But here’s an important limitation: everything I’ve called ‘the internet’ up until this point really refers to part of the internet: the open ‘public sphere’ internet: Twitter, Reddit, comments sections on news websites, the part of YouTube in which YouTubers engage in a kind of running commentary about the world; maybe certain users on Facebook and Instagram who use those platforms in this way. Tolentino does note that she's mostly talking about the 'social internet.' This is the part of the internet in which a single individual can post something that reaches many other individuals (one-to-many). But many of Tolentino's observations also seem to apply to news online, which is tightly linked to Twitter (the bubble that journalists live in is the bubble of the open public sphere internet). It’s worth remembering that the internet is a lot more than just this. It is Netflix and other streaming services: ‘top-down’ platforms in which content is created by a small group of professionals and consumed by a large audience. It is small-group communication via messaging apps (one-to-one, or several-to-several). Tolentino’s observations about online hostility probably don’t fit as well to small group communication online (which is 'social' and is online, though maybe it's pedantic to call it part of the 'social internet), and they only apply to Netflix and other mass media insofar as they try to reflect the zeitgeist of the open public sphere internet.   

The Open Public Sphere Internet as Difference Revealer

Now that there are so many people participating in public discourse online, the open public sphere internet has exposed groups to one another that are so unlike one another to begin with, in terms of their values, experiences, and perspectives, that independent of how aggressive or antagonistic they are, this difference is so shocking and threatening that people get freaked out. That is, it isn’t necessary to have people act in a hostile way toward another group, or in a strategic way so as to counter the messaging of others. Merely by expressing themselves, by showing who they are and what they believe, they may set others off.

Human difference was depicted and conceived in a certain way by mass media. It was often visual, concise, and fictional. So yes, traditional/legacy media depicted difference, but in a circumscribed way, made generic through the use of tropes and bracketed as fiction. While we weren’t consuming images of difference via mass media, we spent our time around people who are not terribly different in terms of beliefs, values, and experiences. We spent a lot of time around family, friends, schoolmates, and workmates. If we were out of sync in our beliefs or values, we often avoided those subjects in order to avoid conflict.

The differences that we see online are differences in values being expressed. Those differences always existed (there were always people who believed something that you would have found abhorrent), but we didn’t have to see them every day. We probably were never going to be particularly good at reacting to these differences. Many of us probably always would have perceived them as a threat to our own values. So, it could be argued that the internet is the first technology, the first moment in history, where we really have been confronted with values differences as they have always been. Of course, these online expressions are not pure reflections of any group's beliefs, are manipulated in various ways (though when we’re quickly scanning a feed and clicking on links, we may not recognize this). Nevertheless, despite the fact that they are not representative of the larger group, they are expressions of the beliefs and values of many real people. And maybe we feel this when we go online; we feel how real they are. They are not fictional characters dreamed up by a screenwriter. These are actual people who actually believe the exact opposite of what we believe.

It’s not surprising that people feel threatened by those differences. And so many folks react to the threat with a kind of siege mentality, and then many subsequent cultural expressions and public life become a reflection of that. And the only alternatives seem to be to carve out small clusters within online life via group messaging apps or forsake social media entirely.

What impact does the curdling of the open public sphere internet have on the world?

Why does any of this matter? Is it just a bunch of people bickering in cyberspace or does it have some clear connection to the rest of civic life? If some of us decide ‘to hell with it’ and only use social media for small-group communication and leave Twitter to the trolls, what would it matter?

I do wonder if the relationship between what is said on the open public sphere internet and power (political power, economic power) could change, or is changing. Perhaps when the open public sphere first became widely used in the United States, most people, including journalists and politicians, saw it as a kind of proxy for public opinion, overlooking the fact that only certain types of people expressed themselves. The apparent connection between what went on in the run-up to the 2016 election (expressions of extreme partisanship, particularly an insurgent group of Trump supporters engaging in a kind of online battle on behalf of their chosen candidate) and the outcome of that election (the election of that candidate) supports this view. 

But what if, in the coming months, the same open public sphere internet were to heap praise on Bernie Sanders and Elizabeth Warren while continuing to deride Donald Trump and deriding or ignoring Joe Biden, and Biden or Trump end up winning the election? What if that inconsistency between opinion on the open public sphere internet and a democratic election outcome were to happen a few more times in high-profile elections in different democracies? Many people might come to believe that opinion on, say, Twitter, neither reflects nor causes shifts in larger public opinion. Once this is acknowledged, I do wonder if at least some people, either in the general public or in positions of power (e.g., widely-read journalists), would be less likely to take their cues from opinion expressed on the open public sphere internet.

There’s a bit of a self-fulfilling prophecy aspect to it, I think. When people believe that Twitter is a bellweather, they look to Twitter and are subsequently influenced by opinion on Twitter. The evidence on which they initially base their belief that it is a bellweather might be suspect, but that doesn’t matter: once they believe it, they essentially bring it into being by assuming that it is public opinion writ large, are influenced by it as such, and, through their writing, influence others. That is, opinion leaders see a popular trend on Twitter and write about it as though it were popular more broadly. Others take cues from the opinion leaders and act on that information, turning it into a truly popular opinion. However, if opinion leaders stopped believing this (and again, it wouldn’t matter whether the reason they stopped believing it made much sense or was supported by evidence), they might ignore it and thus Twitter and the like would lose its power to influence public opinion writ large.

Maybe people in a certain social stratum are already leaning this way, avoiding social media (or at least the open public sphere internet) themselves, raising their kids to avoid it as well. These people are often quite powerful: upper class, working in tech, politically active. If they start ignoring opinion as expressed on the open public sphere internet, I can imagine all of it becoming like certain comment sections of certain news websites: it exists, it is ugly, but most readers simply ignore it, and it doesn’t really have much of an impact outside of itself.

In any case, it’s important to not simply assume that the open public sphere internet has a stable relationship with public opinion in general. I think it’s also worth thinking about the popularization of the open public sphere internet as the first moment in human history that we saw how different our views of the world really are, how directly they conflict with one another (there is also much agreement, though the current iteration of the open public sphere internet doesn’t seem suited to highlight that). It’s hard to imagine such a moment going smoothly. But perhaps it’s a matter of getting over the initial collective shock of it. Or perhaps we just need a little time to ourselves.