Public opinion tends to move in the opposite direction of policy. But how does the public learn that policy is changing, enabling their adjustment? And does a polarized and inattentive public still react together in response to policy? Stuart Soroka and Christopher Wlezien, the key developers and testers of the thermostatic model, find that television and newspaper coverage provides a good signal of which way national policy is heading. And the public as a whole, not just the most informed or a shrinking middle, receive that signal and respond by adjusting their preferences–favoring more spending when it declines and less when it increases. The result is that they tend to think each side goes too far moving policy in their preferred ideological direction. That’s frustrating to policymakers, but it may represent democracy in action.
Guests: Stuart Soroka, UCLA; Christopher Wlezien, University of Texas
Studies: Information and Democracy, Associated Website: http://mediaaccuracy.net/
Transcript
Matt Grossmann: How does the public move right when policy moves left? This week on the Science of Politics. From the Niskanen Center, I’m Matt Grossmann. Public opinion tends to move in the opposite direction of policy, a well established regularity across democracies known as thermostatic politics. When military, health, or welfare spending goes up, Americans want less spending. When spending goes down, they want more. That’s frustrating the policy makers, but may represent democracy in action. But how does the public learn that policy is changing, enabling their adjustment? And how does the polarized public still provide a clear signal in response to policy? This week, I talked to Stuart Soroka of the UCLA, and Christopher Wlezien of the University of Texas, the key developers and testers of the thermostatic model. Their new Cambridge book, Information and Democracy, points to the key role of the mass media in informing the public and driving opinion change.
They find that television and newspaper coverage does usually provide a good cumulative signal of which way national policy is heading. And the public as a whole, not just the most informed, receive that signal. In most issue areas, they respond by adjusting their preferences, saying enough when spending rises too much for their taste or too little when spending falls. The media, left, right, and center covers policy accurately when it changes despite all of our protestations and the public pays enough attention to notice and adjust. But the result is that they tend to think each side goes too far moving policy in their preferred ideological direction. It may not be the failures of democratic politics that produce an ungrateful, seemingly vindictive public, but actually its successes. The public learns what policy makers are doing, but that means less people demanding more of that and more demanding less. Here’s our conversation. Stuart, why don’t you start us off with a summary of the book? What were the big findings and takeaways?
Stuart Soroka: So the purpose of representative democracy is to produce public policy that matches public preferences. And that requires not just that governments are responsive to publics, but also that publics are responsive to government actions. Citizens in the aggregate at least need to have some basic sense for what governments are doing so they can hold governments accountable, and also so they can have informed preferences worth representing in the first place. So democracy depends on a minimally informed citizenry. And while it’s easy to point to specific instances of failure, our past work and work by others finds evidence of responsive publics. Governments increase spending in defense, for instance, and public preferences for more or less defense spending change. Not always, but on average, over decades in large salient policy domains like defense and welfare and health. And the question we’re going into this book with is how does this happen?
Most of us don’t have any direct experience with policy. It seems very likely then, indeed probable, that we learn this information from mass media. So accurate media are central to the functioning of modern representative democracy. And the purpose of this book is to try to capture what we refer to as the media policy signal to identify the information that media are conveying, accurate or otherwise, about policy. And then to examine the degree to which that mediated information, accurate and otherwise, fuels public responsiveness. In some, I think I’d say drawing on roughly four decades of public opinion, budgetary policy, and content analyses of media coverage, the book offers what we believe is the first, or at least one of the largest tests of the role of mass media in modern representative democracy.
Matt Grossmann: So this has been a long collaboration for you all. So Chris, talk about how this builds on your past work and how this book came to be.
Christopher Wlezien: Well yeah, goes back quite fire back to 2001. We were both at Nuffield College in Oxford in the UK, and I don’t know, it was about a month or two in after I arrived there, he was already there as post doc. We started a collaboration looking at opinion policy relationships, building at least in part on what I’d done in the United States. Stuart was intrigued, was wondering about comparative applications. So we took it to the UK. Later on, took it to Canada, which ultimately culminated in the book Degrees of Democracy. And then we kept going with that in a whole variety of directions. And it’s in the book, Degrees of Democracy, we featured the media as a mechanism. But we really didn’t do anything with it empirically.
And it always had to be part of the story, but we weren’t sure really how to do it. And we thought about it, we talked about it. We decided to start in with the economy because that’s something we figured we could get our heads around and our measures around and that worked pretty well. And then after a little while, we decided, “Let’s try it with policy.” And we were able to find a way to measure media coverage of defense spending. That went pretty well. And then next thing you know, we were doing it in a whole bunch of different spending domains. And that takes us up to the present in this book.
Matt Grossmann: So Stuart, you start with these two examples of defense spending and the Bush tax cuts. Maybe take us through those and maybe add one from a democratic president so we get a sense of what the dynamics look like?
Stuart Soroka: Yeah, in our first chapter we use those two examples to set up a mystery that requires solving, a mystery novel. There’s a large body of work focused on how uninformed the public is, and that work is partly correct. But there are also moments, and I think we would say many moments, in which aggregate public preferences respond sensibly to policy change. And that’s the mystery. So here are two examples, a mystery that is given how uninformed many people believe that the public is. So here are two examples. Going into 1980, the American public supports a lot more defense spending. Reagan makes large increases to defense spending. And over the next several years, public preferences adjust downwards, that is Reagan increases defense spending, the public responds thermostatically, adjusting our preferences for more spending downwards. We want a lot, he gives us a lot. We adjust.
Years later in the early 2000s, a large proportion of Americans believed taxes are too high. Bush implements tax cuts and Americans’ beliefs adjust downwards. Not all Americans, mind you. Bush’s tax cuts primarily helped the rich, and to a lesser extent the middle class. And it’s these citizens who show the largest subsequent decrease in the belief taxes are too high. So those are two examples of our mystery, but we can also look, for instance, at public preferences for health spending in response to Obamacare, where we see a similar kind of what we would call based on Chris’s prior work, a thermostatic dynamic. That’s the kind of mystery that we’re setting up at the beginning of the book. On the one hand, there’s lots of work suggesting that individual members of the public are ignorant about lots of different policies. But at the same time, in the aggregate, the public sometimes responds quite sensibly to policy change. How do they do this? The spoiler alert are partly media.
Matt Grossmann: So Chris, what was the assumed role of the media in the thermostatic model, and has your view of the media’s role in it changed at all from this book? And maybe give us a sense of if there’s anything out there, other potential mechanisms? Do people have direct experience with policy? Do they hear party messages and campaigns or direct from politicians or is the media still the central mechanism driving this?
Christopher Wlezien: Well, we assumed that media played a role, mass media played a role, and they could partly to these other alternative mechanisms that you mentioned, including campaign messages and the like which can find a way into news coverage. This direct experience with policy is a slightly different possibility, and I could circle back to that if you’d like, because that is the subject of a lot of research, as you know, on positive feedback, which is something different to thermostatic feedback, which is a form of negative feedback. I think the most interesting finding, and I’d love to hear what Stuart thinks about this too, we didn’t get a chance to talk about this in advance. But to me at least, but I also think to him is the form of media influence.
Christopher Wlezien: Media is informing the public, but it’s not as if the public is updating based on reporting solely of current changes in policy. It’s more along the lines of the public adjusting to this cumulation of signals they’re getting about policy changes over time. And so I think we were both expecting it to be more change focused. They have their preferences, it’s fully informed at the prior point in time. We get a change and that change is quickly incorporated into change in policy reflected in media coverage and that change in information is quickly incorporated. And that’s not always what we find and not can consistently what we find. And so there’s a bit of a lag to this process as it plays out over time.
Matt Grossmann: Stuart, without getting us completely lost in the details, give us a sense of how you measured media coverage, and in particular, your use of what we now call old media or legacy media, especially newspapers and televisions, and why that constitutes still the best measure of the signals that citizens would get.
Stuart Soroka: I’ll answer the second part first because I think it’s pretty straightforward. Our analysis go back to 1980. And in order to have an analysis that extends over that time period, we have to focus on legacy media. So that’s not to say that they’re the only media, of course, and for the latter, let’s say, third of our data set, especially there are other media that are going to matter. But for the bulk of the time that we’re looking at what we now call legacy media, television and newspapers, are just media. That’s partly why we focus on those media, although we have played with social media in a latter part of the book and another work as well. Now I’ll answer the part about where that all comes from. We begin by scraping principally from the LexisNexis full text archive, all relevant content from 17 major newspapers and six television stations. For newspapers, that means we capture any story that is even tangentially related to each of the five policy domains we examined, defense, welfare, health, education, and the environment.
TV transcripts are archived differently. They are not archived by story, but by the entire transcript. So we actually grab all television content. And then we identify in all of that content any sentence that refers to spending change in any of those policy domains. We do it initially using what we’re calling either a layered or hierarchical dictionary approach. Essentially, we have dictionaries of words that refer to budgets and spending, that refer to policy domains, and that refer basically to up or down keywords, up or down signals. And the combinations of policy words, budget words, and up or down words pretty reliably identify sentences in mass media, television and newspapers, that suggest policy change in one direction or the other. And then if you count up all those sentences, let’s say by fiscal year, and subtract the number of down sentences from the number of up sentences, you get a representation of what a citizen might learn from media coverage that is over a fiscal year, what was the aggregate signal the media was sending?
And we’re able to compare those annual up and down signals with the direction in magnitude of actual budgetary change. I’ll add that it’s important that we’re not talking about a couple sentences here and there in the New York Times and nowhere else. There actually are a lot of sentences on a near weekly basis in US media that signal up or downward change in one policy domain or the other. We’re talking about tens of thousands of sentences. So when Chris is talking about the impact of cumulative media coverage over years, what we’re talking about is a lot, a lot, a lot of sentences in media coverage that signal upward or downward spending change, enough so that any citizen probably paying even passing attention to mass media is getting at least a trickle of information about what governments are doing in terms of spending in these domains.
Matt Grossmann: So Chris, the nice thing about this is you have objective measures of what policy is doing, but you also try to look a little bit at whether the media might predict changes, maybe cover the policy debate before it happens. There’s been recent controversy about whether is it the media coverage of the policy changing or is it the media coverage of the debate whether or not it actually passes? So what can you tell us about that, and what did you learn from looking at each of these policy issue areas individually?
Christopher Wlezien: Well, it’s a good question. It’s something we take very seriously in the book. We might not go as far as you and listeners might like, but we’re on the case, and I’ll tell you a bit about that in a moment. In the book, we do pretty clearly demonstrate, I think, that media follow what policy makers do, so they’re not leading. But that analysis is based on measures of coverage over the entire fiscal year. And so it conceals what is happening during the year. It’s just not clear. But it is one thing we’re exploring, the timing. And it looks like based on the analyses that we’re actually literally in the middle of, when we have the time to do it, it looks like media is reacting later in the fiscal year to the decisions about spending for the following year. And so it’s not mostly to things like presidential requests and state of the union addresses or early debates, but it could still be reflective of later debates.
Christopher Wlezien: But these are basic results mostly motivated by a desire to unpack causal direction. And so there’s a lot more work to be done here, and it’s something that we’ve talked about doing moving forward. About domains, yeah, well most of what we’re seeing here we see it holds in some domains. We have to keep in mind that survey organizations only ask questions about domains that the people tend to care about. And we focus on a subset of those that we think are particularly salient, and that’s defense and welfare and health and the environment and education. And we find pretty nice effects of policy on media coverage and then in turn on public preferences in three of those domains, defense, welfare, and health. But not in the other domains. And got to keep in mind that what we’re focusing on here is a pretty high level. We’re talking about are you spending… pretty high level. We’re talking about our spending going up on defense. It’s not the details about the spending, and programs involved. And the same thing for welfare and health. And so it doesn’t hold everywhere. It’s not three cheers for American media, or public opinion, but it’s better than I think a lot of people expect.
Matt Grossmann: Stuart, you use both hand coding from research assistants and computerized text analysis, and you’ve let folks see the details on a website, I believe, media organization by media organization. So tell us what’s available, and what people will see in terms of how this content analysis was done.
Stuart Soroka: That’s right. I spoke about our dictionary coding earlier, but all the dictionaries and results are tested using human coders, and we use crowdsource coding to test the reliability of the dictionaries. We also use the crowdsource coding to fuel machine learning algorithms, to see if we get roughly the same results using that supervised machine learning.
In the book, we spend two whole chapters, one, first developing the dictionary approach, and the second, testing that dictionary approach in many different ways, including racing it against machine learning. And in past work with Lindsay Dunn, we’ve also explored the degree to which we get the same results using just what we’re calling this hierarchical dictionary approach versus machine learning.
And then on the website, mediaaccuracy.net, we provide more detailed results for each individual media outlet by domain, we give accuracy scores, and we have long pages there for those who are interested on the methodology behind those scores.
Short answer about the relationship between the machine learning versus human coding is that aggregated measures of the media policy signal, estimated using either approach, are highly correlated. The correlation coefficients in each domain are above 0.8, sometimes above 0.9. We’re getting essentially the same signal using these two different approaches, even though these two different approaches start from very different points. One is about capturing sentences that have all of these words in it. The other is about getting humans to code a bunch of sentences, and then teaching a machine to replicate that human coding.
Now, why is that the case? It’s the case, partly because we’re relying on highly aggregated measures of the media signal, both the dictionary and machine learning approaches make lots of errors at the level of individual sentences, that’s going to be true in almost any automated coding. There’s no systematic difference between one or the other method, like there’s no put place where clearly humans are doing better here, and the humans and the machine are doing better here, and the dictionary is doing better there. They’re both making different kinds of, essentially, from what we can tell, random errors.
What we can say is that both systems are making errors at the level of individual sentences. When we aggregate up sentences, and we’re talking about tens of thousands of sentences in each fiscal year, what we get are very, very similar measures using either approach.
Matt Grossmann: And real quick on the media differences, listeners can go look at them in detail, in the specific outlets, but it doesn’t seem like there was a broad pattern that this was reliant on conservative or liberal leaning media, or television versus newspapers, or any particular media framing decision, is that right?
Stuart Soroka: Yeah, I think that’s accurate. In the book, we don’t look at individual media outlets very much, we spend a very brief amount of time on that. And then on mediaaccuracy.net, what we do is, for every single media outlet, on every single domain, provide essentially an accuracy coefficient. We show both a volume and accuracy of coverage in each domain.
And there are, in some cases, let’s say in the defense domain, actually each individual outlet is doing a relatively good job, but there are variations in how well they’re doing. In environment, each outlet’s doing a pretty bad job, but there’s a little bit of variation in how badly they’re doing.
Matt Grossmann: All right, Chris, this seems to be a more redeeming portrait of the media than we usually hear, but probably critics of the media will think this is a pretty easy case, that they get the direction of spending right. So tell us how responsive you think this is to broader critiques of media framing of policy, or overall levels of coverage of policy with your analysis.
Christopher Wlezien: Well, I don’t think we disagree with those hypothetical critics. It’s not everything we want of the media, but it’s certainly good news. This information about policy is being conveyed pretty accurately, at least in certain domains that the public considers to be important, and the public gets that information and uses that information. Again, it’s not three cheers for mass media in America, but it seems better than we might expect.
Maybe most importantly, we’re all political scientists, right? We know that people were, and still continue to be, surprised by thermostatic response. Tends to be at a pretty highly aggregated level, on a very general level. And we’re finding that response is driven by media coverage, which also is [cast 00:21:35] at that level. And that’s why we studied it that way, we were trying to get at a public response of a particular type, not trying to assess media coverage generally in all of its ways.
Christopher Wlezien: Most importantly, I think, it’s potentially, and seemingly, enough to guide politicians. And then also hold them accountable, at least in a very general way.
Matt Grossmann: So since I’m regularly touting this to the media itself though, I do regularly get these responses that, “No, this is about the media is falsely covering what Biden is trying to do, or they’re overemphasizing the liberal aspects of it, or it’s about the debate and the inability of the democratic members of Congress to get across their messages.” Do you see this as responsive to those kinds of challenges, or are those just excuses?
Christopher Wlezien: Well, I think to some degree, right? Is it fully responsive to all of those? No, but I think there’s a little gain saying that things are happening in Washington, policies or decisions are happening, that are getting covered, and they’re getting covered pretty accurately. And that seems somewhat contrary to what some of those hypothetical critics you’ve raised at least are saying. And it’s good news, but again, it’s not perfect.
Matt Grossmann: So Stuart, you also find that the public is responsive, in broad terms, to this media coverage, but they’re also responsive to some basic economics, things like growth and inequality, and they’re also responsive to just the party of the president. So tell us, how does the media fit into that? And does the party of the president finding cut against a little bit the notion that the public is learning about policy itself from the media, rather than just reacting negatively overall?
Stuart Soroka: Right. We know from past work, our own and many others, that public preferences for policy react to economic indicators, and the partisanship of government, and changes in policy itself.
So one of the things we explore in the latter chapters of the book, is whether media coverage matters above and beyond those factors? Do media simply convey the state of the economy, or the partisanship of government, and policy change? If that’s the case, then including economic and spending variables in a model of preferences should remove any observed influence of media. That’s the context in which we’re exploring these other influences on public preferences.
But, of course, media don’t perfectly reflect the economy, or spending, or probably any other real world factor. Media can be, and most likely are, partly accurate, but also partly flawed, or noisy reflection of what’s going on around us. And media coverage includes information, as you’ve just been saying, information about other things as well, either completely irrelevant to policy, or relevant, but not necessarily captured by medias of the … variables rather of the economy or government.
So to what extent does that information matter? Is it possible for media, for this other information in media, to matter above and beyond the factors which we already know matter to public preferences, the economy and so on? And our analyses in those later chapters suggest that media do. That whatever it is that media are conveying, that information can have a significant impact on preferences above and beyond the party of the president, above and beyond information of budgetary policy itself, and above and beyond the economy.
Matt Grossmann: But just so we’re clear, the party of the president finding survives, and shows that the public is not just responding to either policy itself, or the media coverage of policy, but they’re are also just a turn whenever the party of the president changes, is that right?
Stuart Soroka: Yeah, that’s true. There are two ways to talk about that. One version is the party just responds to whatever the party of the president is, and that’s meaningless information, and because it’s a Democrat, the public swings Republican, and vice versa.
But, of course, the party of the president is not meaningless information where policy is concerned. Democratic presidents on average make democratic policies, Republican presidents on average make Republican policies, and so the party of the president is a meaningful cue about the likely direction of spending in a whole bunch of different domains. Not a perfectly accurate cue, just like the media are not a perfectly accurate cue, but it’s a meaningful cue.
It’s not completely irrational for someone to look at the party of president and think, “Given this president’s partisanship, I suspect that defense spending is going upwards, and welfare spending is going downwards, and I’m going to adjust my preferences accordingly.” That’s not a completely contentless cue, the party of the president is meaningful.
Christopher Wlezien: Yeah. Can I chime in on that too?
Matt Grossmann: Go ahead.
Christopher Wlezien: Yeah. One of the things that we … The way we look at that finding is, spending, and media coverage of spending, matter independently of the party of the president. What we’re trying to do in the book with that analysis, is to show that, “Hey, just in case you were wondering …” As a number of people have asked us over the years, is this all just about people responding to the partisanship of the president and/or the Congress? And it’s not that, it’s more than that.
Matt Grossmann: So, Chris, you also disaggregate the public, and show that this isn’t just the high attentive voters that are responsive. So talk about that finding a little bit, and maybe some other disaggregations. I know you have all been involved in debates over whether this is just about the highest class respondents, or about issue publics, or whether changes in the media mean that these signals are no longer getting through. This seems to be a pretty powerful signal against those kinds of ideas.
Christopher Wlezien: Yeah. Thank you for the question. There’s a lot of parallelism here, and this goes back to the page in Shapiro’s classic work on public opinion. There’s parallel publics, people respond in similar ways. We find that across education information levels. We find that across income levels.
Perhaps the most, I think, interesting and striking finding may be the parallelism across partisans. Preferences move together, we see thermostatic response that’s virtually identical, and all are responding to the media. Although, it looks like there’s some interesting differences across partisans as well, where the Republicans, and to a lesser degree independents, appear to be more responsive to the media signal.
Keeping in mind that the media signal doesn’t just reflect spending change, it picks up other stuff. And it looks like the Republicans are more responsive to that other stuff, and independents also to some degree. And so their preferences are affected, not just by spending, as is mediated, but by these other things. And this is something that Stuart and I are playing to explore, but haven’t started in on just yet.
Matt Grossmann: So Stuart, we’ve already done the hard work of redeeming the media, now we have to redeem the US public. That should be easy. On the one hand, this seems to be good evidence the public can learn from public policy, can accurately tell you what direction it’s moving, and can move against it, as we would expect.
On the other hand, we have some long time patterns that cut against the notion that we have an enlightened public here, even in this domain. So even in spending, we ask people about welfare versus spending for the poor, we get very different answers, and always have. We ask people about spending on individual items versus spending as a whole, and again we get pretty different answers, and always have. So place this evidence within that broader set of regular findings about American public opinion.
Stuart Soroka: First, regarding the difference in responses to, for instance, questions about welfare versus questions about spending on the poor, there are big level differences there, right? When we say welfare, preferences for more spending are much lower than when we say spending on the poor. But if we track those two questions over time, responses move almost perfectly in parallel. When preferences for welfare spending are moving upwards, so are questions for spending on the poor.
Chris has written a fair bit on the way in which we should be interpreting these questions. They’ll be good at showing us over time change. They’re kind of ambiguous on the levels front, that is, it’s very hard to learn from survey questions the level of policy that people would prefer. And when you ask about welfare versus spending on the poor, you get some different levels.
But there is meaningful information there. You track that over time, and over time change is meaningful. And all of what we are looking at here is about over time change. So given question wording, given for instance, asking about spending on welfare versus asking on spending on the poor, what does the over time change in those survey responses look like relative to policy? It’s meaningful.
So there are level differences that are a function of question wording, for instance, or we could say policy framing, but there also is meaningful over time change. That’s maybe the boring methodological answer to your question.
Maybe this one is better. Thinking more generally, I think there’s one reading of our book, and indeed of our work in general, that emphasizes a look at democracy working story, that Chris and I should be showing up to conferences with pompoms and cheering for functioning representative democracy. And to be clear, I think we both stand by that story. Maybe not the pompoms… clear. I think we both kind of stand by that story. Maybe not the pom-poms part of it, but the general story. There clearly are domains in which governments are held accountable, and publics respond sensibly to policy change, and policy makers represent public preferences. And all of that is facilitated in part by predominantly accurate media coverage. So there are instances in which media are working and democracy is working and the public is responding, but we tried in all our work to also make clear that this is not the case all the time. This is not a black versus white story. It’s a gray one, that there are domains in which things work and domains in which things don’t. And in our prior work, we’ve identified policy domains in which there’s no representation and no public responsiveness. And in this book we find domains in which media coverage barely reflects public policy at all.
So I’m not sure that our evidence redeems the public, in the sense that we’re not arguing that the public is always really perfectly informed and that democracy works as a result, but we are arguing that there are salient large policy domains in which there is enough media coverage and enough attention from the public for informed public preferences. And that justifies representative democracy, because there are preferences worth representing. And it also makes representative democracy work because governments are held accountable for their policy decisions. That can happen. It can be simultaneously true that there are huge failures in some domains and successes in others. And that’s what we find.
Matt Grossmann: And Chris, why don’t you give your take on American public opinion as redeeming as well, and maybe answer it in the context of alternative measures that we hear about from Jim Stimpson as to the overall kind of levels of liberalism and conservatism of the American public and their responsiveness to policy and part of the president. In this case, you all are more disaggregated. So you’re actually able to show the public is responding, not just to kind of one signal, but to these individual signals.
Christopher Wlezien: Right. I mean something we’ve looked into … this has a recurring theme in our work over a long time now, but it’s not by assumption entirely. I mean, some of the work I did back about 15, 16 years ago, almost 20 years ago now, it does demonstrate … it does not make an assumption that people are responding issue by issue and the policy makers are responding issue by issue, but allows for that possibility. So it doesn’t assume, as Stimpson and a lot of others have said, that it’s just one big dimension, that it may be that the public notices what’s going on in welfare and has preferences that are specific to welfare and policy makers respond to those welfare specific signals.
And empiric analysis suggests, again, there’s heterogeneity here. You see some on defense, you see specificity, you find domain specificity on welfare. You find it again on health, but you don’t find it elsewhere. So even to the extent you’re finding thermostatic response and representation in those other domains, it tends to be more collective. So it tends to be more like Stimpson and others have assumed, but that’s not the case in the three domains that not coincidentally are the ones where we find media effects in our book, information and democracy.
Matt Grossmann: Sure. I know you only have the data that you have, but place this kind of spending data in kind of the larger debates that we’re having in American politics. There are people saying that we’ve moved completely on from the redistributive dimension to emphasizing the cultural dimension. Stimpson actually finds public opinion moving more liberal on cultural issues, but then there are other people who say the public is only responding to cultural issue changes now and not economic policies. So we’ve chosen this area of spending. How should we place that in these broader debates?
Stuart Soroka: So it’s true that our data focus entirely on budgetary policy, that what we’re able to speak to are trends in budgetary policy, trends in media coverage of budgetary policy, and trends in public preferences about budgetary policy, essentially. So it’s possible for us to find a kind of sensible interaction, let’s say, between budgetary policy and public preferences for budgetary policy, fueled by media coverage of budgetary policy. It’s possible for all of that to be true, and also for there to be other areas in which media can be either misleading or can be focused on something entirely different from budgetary policy and changing public preferences in ways that are not captured by our data.
What we can say is this is the extent to which public preferences for budgetary policy respond to budgetary policy. In that sense, we find evidence that there’s a kind of media policy signal that is representative, but we don’t argue that it’s perfectly representative of policy change. There is noise in there. And part of the noise, even in our signal, is going to be all of the other debate and discussion and framing that comes with any kind of policy change in the US, and anywhere else for that matter.
Christopher Wlezien: One of the things to keep in mind, that relative preferences can change for two reasons. One, because policy changes, but also because our preferred level of policy changes. And that’s part of what you were asking about earlier with things like the economy and inequality. And in defense, we may have effects of national security threat and the like that impact our preferences. So P-Star can change over time. And interestingly, going back to my original work, actually, on the thermostatic model back in 1995 now, one of the things that we have found consistently is this tendency for people’s underlying preferences for spending to increase over time. So if spending does not increase by 2-3% in real dollars in the United States, for instance, over time, public preferences are going to go up.
Now, if they increase by too much, the preferences are going to go down. If they increase by too little, they’re going to go up, but if they increase by just the right amount and nothing else happens, the change preferences, then they’ll remain the same as long … again, depending on how much is being spent. So there is a trend here in P-Star for spending, not unlike trends we see in social domains. And a thermostatic model can encompass that, to the degree you can capture that kind of variation in what we call P-Star, the public’s underlying preferred level of policy. And that could be done, in theory, not only in spending domains, but also non-spending domains.
Matt Grossmann: So Chris, an objection I’m constantly hearing in trying to explain your work is that this is what Biden promised he would do in the campaign, or why did the public turn against health spending under Obama because he proposed Obamacare during the campaign? So I’m trying to constantly tell people that, like immigration policy now, it doesn’t mean that Biden is counteracting what he said he was going to do in the campaign. It just means he’s dramatically different than was Trump on immigration policy. So talk through that. What’s the logic, that it doesn’t really matter if the president or the party in power is kind of overshooting what they said they would do? Why doesn’t that matter that much?
Christopher Wlezien: Overshooting does matter. That’s kind of the point of the thermostatic model. I mean one way to … and maybe I’m misunderstanding the question, so please don’t hesitate following up if I’m misunderstanding, but in one way we see overshooting and undershooting for a whole variety of reasons, because politicians make mistakes, even to the extent they’re trying to represent, say the average voter. And when I’m thinking about overshooting and undershooting, I’m thinking about the average voter, not what they promised, because the promises are going to be predicated not just on the average voter. So party control matters. So Democrats come in and they push policy off to the left and the average voter is looking and saying, “Hey, that’s too far to the left.”
And the longer they’re in office, the further things go. And that could lead us to have time for a change instinct after two terms, which we see pretty frequently in American politics, and can more generally help us account for cost of ruling in other countries. And party is part of that story. If parties perfectly, if government’s perfectly represented the public, so no sort of partisan tendency and no mistakes, then change in public preferences would be entirely driven by these P-Star shocks that we just talked about, things that cause relative preferences to change that are unrelated to policy. So the economy may go up, there may be a shock to national security that may cause us to want more spending. And then if policy makers respond, and given the assumption that they do perfectly, if they do it quickly, then we would adjust our preferences back downward quickly. If they do it more slowly, deliberately, it would take us time to do that.
Matt Grossmann: Yeah. I think people are comparing what Biden is doing versus what they thought Biden would do, rather than comparing what Biden is doing versus what Trump is doing. So I think that that’s part of the issue, but also, I think people don’t sort of have a sense that there might still be a response to health spending going up, even if there were some people who wanted health spending to go up before. Now, those people are satisfied and now there’s going to be a few more people who want it to go down.
Christopher Wlezien: Yeah, some people may be thinking it went too far.
Matt Grossmann: So Stewart, the media … we talked about a little bit the media covering not just the policies that change, but the policy debates as they are occurring. So we said in the Time series, we are confident that they’re not kind of anticipating the change, necessarily, but is the media still responsible for this broader pattern of kind of undercutting policy when it occurs? The debate for Obamacare, Obamacare opinions go down as the public learns more about Obamacare or as the debate goes on further, and then when it’s threatened to be repealed, we see the exact same scenario in reverse, that support for repeal is high at the start, and then goes down as the debate occurs. So kind of fit that into the model that you all have. Is the media also just kind of informing people more about a policy, which somehow makes them more against it? Is that part of the spending signal? How does that fit in?
Stuart Soroka: I mean, that can be part of the story. I should say first that we know from work that just looks at the relationship between opinion and policy, like our past work that leaves media out of it, that can’t be the predominant part of the story, because we do see domains in which the public is responding to policy change as it happens. But at the same time, when we focus on media coverage, in our past work on the economy, the work that Chris and I have done with Dominik Stecuła, we see a media that is kind of forward looking, that media reporting on the economy is partly about what the economy is going to do. And then we see in analyses in this book that media reporting on policy change is partly about policy change in the previous fiscal year, partly about policy change in the current fiscal year, and partly about policy change in the upcoming fiscal year.
So there is a component, and it varies by outlet and it varies by policy domain, but there is a component of media coverage that is forward looking, that is talking about what people are saying will happen in policy. And maybe that’s about a debate about what the government might do next year, or maybe it’s a journalist pontificating about what governments might do next year. It could be any number of things, but there is a component of that media coverage that is about what’s going to happen next year. So to the extent that the public is following that media signal and to the extent that that signal is forward looking, it could be that what we have in some domains over time is a public that is responding to policy change that has yet to happen.
That’s not the predominant relationship, because we see a public that responds to change as it happens overall, but one constraint, one kind of limiting factor in that relationship, that is the relationship between the public preferences and current year fiscal change, one constraint may be that part of what’s going on with the public is they’re responding to change that hasn’t happened.
Matt Grossmann: But what about, I guess, the media coverage of politics, rather than policy as a factor? So there are some people who would say in current debates, the public is for childcare spending and health spending that the Democrats are trying to do, but the issue is that the public is not for political conflict and controversy. So if they see the enactment scenario, it’s going to look bad, even if they like those components. Do you that’s a part of the story?
Stuart Soroka: Well, there are ways in which that can be. I mean, kind of one version, actually to draw a little bit on one of your prior podcasts with Yana and John, I mean one version might be that people just stop consuming, or are consuming media less because there’s just so much argument in media. And as a consequence, we see constrained public responsiveness that is the result of getting less information about what policy is doing. But again, even as there can be lots of sources of noise, lots of sources of kind of error in public responsiveness to policy, there still is, in large domains, a signal getting through. Even within all of that other debate, there still clearly is, in defense, for instance, a pretty clear signal about what is going on in defense spending. That is a media signal, what is going on in defense spending, and public responsiveness accordingly. So both of those things can be going on. That is a kind of reasoned thermostatic responsiveness and all kinds of other sources of noise, those things can be going on at the same time.
Matt Grossmann: Chris, this seems somewhat inconsistent with the media model that we’re usually hearing around polarization that’s sort of based on this [inaudible 00:47:55] idea that the parties are giving off different signals and its partisans on each side who are receiving their party’s signals and going to different …who are receiving their party’s signals and going to different corners, but you find similar responsiveness across the side. So can we reconcile those perspectives or how do we still have this ongoing polarization over time with this responsiveness?
Christopher Wlezien: Well, something needs to be revised, but I mean, parallelism is in the data. It’s been there. There may some divergence across partisans, but it’s not the primary in preferences over time. [inaudible 00:48:40], as I’ve already discussed, right? There’s some differences in media effects across partisan groups, but thermostatic response is really striking to someone. And so they’re all getting and using the information. So it seems to me that, I mean, part of the story I think is if you look at… And this is I think clearer when we look at the by newspaper and particularly by network, television network analysis, which we get into in the book and also there’s a website, right, where we provide this data and media accuracy.net.
Interestingly, you find the signals that people are receiving across different media sources. You take, for instance, MSNBC, CNN, and FOX are strikingly some… At least as regards this kind of information. And so we really wouldn’t be all that surprised if people just watching one or the other of these networks to get their news. And that still tends to be the plurality winner, right? For news sources on television that is. They’re all going to be getting pretty pretty similar information. And so maybe that’s, I mean, that may be a really surprising finding for some. I don’t see how… What I see us providing is more of a challenge to that other work, but also an answer to responsiveness that we, thermostatic responsiveness that we’ve demonstrated in previous work, right? That these signals are really not all that different.
Stuart Soroka: If I can add to that briefly. I think that, I mean, Chris has highlighted what I think is a really interesting finding. And that is that when we look at accuracy scores, let’s say in different domains across television networks, we often find the highly accurate scores for both and equally accurate score for both MSNBC and FOX News. Now they clearly are not saying the same things about policy all the time, but they are sending a very similar signal about spending. So somehow through all of the editorializing in amongst all of that editorializing on those stations, there are sentences about whether spending is going upwards or downwards. And over the course of a fiscal year, those sentences are roughly in line with whether spending is going or downwards. It’s possible to watch FOX News or MSNBC and get roughly the same signal about spending. And that fuels the parallel thermostatic responsiveness that we see in this and another work.
But of course that’s not the whole, that’s not all of the story about FOX News and MSNBC, because around all of those other, all of those sentences about spending, there’s all kinds of other commentary that quite clearly is in very different directions. So we’re getting on each of those channels and on probably in any outlet, a combination of accurate information about what policy is doing, and then all of other information. Some of what we’re getting is an accurate signal that is producing reason thermostatic responsiveness, and some of what we’re getting is other stuff. And that other stuff could be useful or it could be nonsense.
Matt Grossmann: This isn’t just an American pattern, but of course the US system has some unique particularities, especially this very strong two party system. So what can out of comparative research on this add to our understanding of what’s going on in the US and how unique it is globally?
Stuart Soroka: I think in some ways, there isn’t anything unique about what we observe in the US. As in we expect that media are important to thermostatic responsiveness almost regardless of the system, so long as we have a sufficiently free media, like a media that is free to report on things. Then we should see media coverage being integral to public responsiveness to policy. And that should be true in other countries as well. So we’ve looked at thermostatic responsiveness in other countries, we haven’t looked at media coverage of policy in those other countries, but I think our expectation would be where there is strong public responsiveness to policy, we think that is probably partly fueled by accurate media coverage and capturing that media coverage and exploring accuracy is totally something that to be done across other countries. And we would expect roughly the same results.
To the extent there are differences in between the American system and other systems. I think on the first on one hand, we’ve already talked about the party of president as a useful cue about the direction that policy is going in. And of course, as we move to systems in which there are more parties or in which power is shared, the partisanship of government is a less useful cue to where policy is going. That may put more pressure on the public and on media to produce a signal that’s sufficiently representative of policy, so that there can be informed thermostatic responsiveness. That might be one difference. A source of complication in the US is that it is a federal system. And so there are some domains in which most domains aside from defense, in which part of the policy being made is at the state, or even at the local level. And in countries with more centralized policy making, public responsiveness should be easier. And actually media reporting of what’s going on in policy should be easier.
In the US, it may be particularly once we get to a domain like education, for instance, that part of the reason why we see limited, constrained public responsiveness and limited accuracy in media coverage is that we’re only focused on in our work that is, on federal spending on education, but part of what media might be doing, and part of what the public might be doing as well is responding to some combination of federal spending and state level spending and local spending. And that’s one regard in which the US system is relatively complex. So on the one hand, partisanship makes responsiveness in the US maybe a little easier. Federalism makes responsiveness in the US a little bit harder.
Matt Grossmann: So Chris, your story might be an answer to this question people have about why we have this competitive party system where we don’t seem to have either party being able to gain a permanent governing majority. On the other hand, you imply that if a governing party did not overshoot that maybe they would be able to maintain their support. So why are people so mad? Especially given that the overall pattern seem to be spending does go up a little it beyond inflation over time. It’s going more into things like health and less into things like defense that people seem to want. And yet people are mad even though the political system is giving them these compromises over spending that they desire.
Christopher Wlezien: Not everybody’s happy or to say not everybody’s mad, but not everybody’s happy. To me, it’s unclear the focus here. I mean, is it social media we’re seeing, is it reporting of more left wing papers and right wing papers that are picking up on audiences which are either really, really happy or really, really unhappy? It’s not entirely clear to me. And so I think that is part of the story and that may be heated by social media. And so it may not be so much that social media is causing the public to be different. Although there’s evidence that may be the case, but it certainly conveys a difference to us, right? The people in the middle, those voting voters, they’re not the ones showing up at Trump rallies, right? They’re not the ones taking to Twitter and Facebook.
Matt Grossmann: But I guess what about the other side? Because you have, there certainly are constant surveys where people say that the government isn’t doing what they want, they have low support for it. And yet of course there is the alternative direction that you all have also shown that the policy does respond to public preferences over time. So why don’t people notice that?
Stuart Soroka: I think people partly do notice that. I think there are these caricatures of the people who are entirely caught up in partisan arguing on Facebook or people who are completely aggravated that the government isn’t doing what they want the government to do. But oh, let me paint a picture that I think is much more accurate. It’s perfectly reasonable for someone to think, “I’m not going to go on Facebook anymore because there’s just too much craziness going on Facebook. And I just can’t withstand that partisan debate. And I’m pretty aggravated about Biden’s immigration policy because he hasn’t done as much as I wanted him to do on immigration, but what he did do, at least I support, I just wish he did more. So I’m aggravated at him about at how limited his response was on immigration, but I support what did do. And I wish that he did more.”
And that’s a perfectly reasonable position that is being simultaneously being aggravated about politics, being upset that the government didn’t do as much as you wanted them to do, and also recognizing what the government has done and that’s like part ways there. So imagine that that is the information that you’re getting over several years. Biden is getting a little bit closer to you on immigration and you’re adjusting how much more you want him to do over time accordingly. I think that’s maybe most people, right? That we would like the government to do something. They get part way there, we adjust. That’s thermostatic responsiveness. And simultaneously aggravation about the government not doing exactly what we want. Those two things can coexist.
Matt Grossmann: So what’s next for you all? What’s book number three going to look like? What are the big remaining holes in the thermostatic model or the places where it can be extended to explain new things?
Stuart Soroka: We leave a decade between books. So we need to let that percolate a little longer. I mean, one of the things we’re thinking about doing now, and actually trying, I should say, trying to do right now is to try to frame these results and make these results a little more public facing, a little more outward looking. Because we started in on this project trying to explain thermostatic responsiveness in a very pointy headed political sciencey way because that’s our jam, but along the way, we ended up with these measures of media accuracy that speak to the degree to which media are fulfilling their goal and representative of democracy. And are relevant for communities interested in misinformation and disinformation and media accuracy generally right now. And also, these are findings that are actionable for journalists and editors and also for media consumers.
Christopher Wlezien: Matt, to be honest, I mean, you anticipated a lot of what we’re already doing in your questions. And we’ve already touched on I think two of the mains that we’re working on now. One is the media reflex paper, entitled Media Reflect, which focuses on whether, you could tell from title, whether the media are actually affecting policy and also the public or reflecting policy on the public. And builds a little bit on what we do in later chapters in the book but I think much more explicitly and also using information available at different points of the fiscal year. And the conclusion, so far at least, is it’s mostly media reflect. Also the partisan differences. And this is something we really haven’t started in on yet, but it’s something that we’re really keen to and are hoping to later in the year, because it’s fascinating to us that you find much more media reliance among Republican voters and to a lesser degree in independents and [inaudible 01:01:45] Democrats. And also be useful to see how that connects up with different sources and different kinds of information.
Matt Grossmann: There’s a lot more to learn. The Science of Politic is available biweekly from Niskanen Center and part of the Democracy Group Network. I’m your host, Matt Grossmann. If you liked this discussion, you should check out our previous episodes. Do Americans implicitly trust government despite our public anger? Does the public respond to threats to democracy? Inflation hurts presidents, and it’s not the media’s fault. The future of the Biden agenda in Congress, and how the media economy drives local news. Thanks to Stuart Soroka and Christopher Wlezien for joining me. Please check out their new book, Information and Democracy, and then listen in next time.