Polls missed the 2016 election outcome and did even worse in 2020 on the margin, underestimating Donald Trump again. Should we believe the polls this time? What have pollsters changed? Have they overcorrected? In an era of one percent response rates for phone surveys and opt-in Internet panels, should we even talk about them in the same way? Michael Bailey finds that our theories about random sampling don’t really apply anymore. And weighting with larger samples does not solve our non-response biases. Brian Schaffner finds that weighting on several factors has increased, likely helping pollsters avoid undercounting Trump supporters. They both say survey research is important to get right but that the solutions are not obvious.

Guests: Michael Bailey, Georgetown University; Brian Schaffner, Tufts University
Study: Polling at a CrossroadsCooperative Election Study

Transcript

Matt Grossmann: Can we believe the polls? This week on the Science of Politics. For the Niskanen Center, I’m Matt Grossman. Polls missed the 2016 election outcome, expecting a Hillary Clinton victory. Then they did even worse in 2020 on the margin, underestimating Donald Trump again. Should we believe the polls this time? What have pollsters changed and have they overcorrected? In an era of 1% response rates for phone surveys and opt-in internet panels, should we even talk about them in the same way? This week, I talked to Michael Bailey of Georgetown University about his new Cambridge book, Polling at the Crossroads.

He finds that our theories and evidence about random sampling don’t really apply anymore, and waiting with larger samples does not solve our problems of numerous important non-response biases. But I also talked to Brian Schaffner of Tufts University about his research and work managing the Cooperative Election Study, a giant commonly used academic election survey. He finds that waiting on education party past vote and urbanicity have increased likely helping pollsters avoid undercounting Trump supporters. They both say survey research is important to get right, but that the solutions are not obvious. They join me together in anticipation of the hotly contested election. I think you’ll enjoy our conversation even if it makes you a bit more nervous about the polls. Michael, you have a new book out, Polling at a Crossroads. What are the main findings and takeaways?

Michael Bailey: Yeah, well, thanks for having me and yeah, I’m excited about the books. And so the basic idea and motivation behind the book was just to point out that our polling.. The theory we use in polling and survey research has become really, really disconnected from the practice. And so at some point that just has to break and we have to move to a new paradigm. And so right now the state of the art, it seems to be, is to look back at random sampling, which is amazing. And I was teaching this in class recently and this is science. Random sampling has such amazing properties and anything you want in expectation, you’ll get the right proportions and expectation with random sampling. We’re not doing random sampling. We’re not, but we still use margin of error and we still use sample size and we still nationally representative, we’re still using the language either of random sampling or kind of adjacent to random sampling, which my term for it is poll washing.

We kind of take the like, wow, we have a margin of error and we have the sample size and it’s naturally representative, which are all things that are relevant in random sampling. But when you have a 1% response rate or often less than 1%. And for subgroups, a lot less than 1% response rate or when you don’t even try to have a response rate. And if you start looking around at some of these firms, it is literally clickbait and you enter in and they just take a warm body and they’ll just get a response. Honestly, that doesn’t mean we’re wrong when we do those things at all. It just means we’re not doing what the theory says we’re doing. We’re doing something differently. And so the point of the book is to look it in the eye and say, we’re not… Just random sampling is dead, right?

As a theory, it’s amazing, but as practice is dead and so let’s appreciate what that means. And it means there’s some intuitions that are very, very different in the real world versus the random sampling world. And then where we are going to put our effort, especially on the academic side, is let’s look at the things that need to be true for this thing to work and let’s… Make sure is a bit optimistic, but let’s try as best we can to see if the things that have to be true are true. And in the random sampling world, we just needed to know if we had a random sample and we had a really high response rate. That’s it, we’re done. And then we go crazy in a really scientifically robust way. We are not in that world and we need… I really think that the whole field needs to move to a new paradigm.

Matt Grossmann: So presumably this affects both the phone, base polls, and the online surveys that may still actually get people to respond, but that initially were based on recruitment that’s subject to some similar findings. But what are the big sources of non-response bias that we should be concerned about?

Michael Bailey: Well, and one of the things that I think is really fascinating and really important, both just to point out, but then important as we do what I want a shift to new ways of thinking about things is I believe that the polls in practice are vulnerable to non-response. That’s very, very, very different than saying they always have non-response bias. And in fact, I think there’s a ton of polls and there’s good examples where pretty crazy stuff going on, less than 1%, or we just had ads on the internet and we got people in and they do fine. So I’m not saying there’s always non-response, I’m saying we’re always vulnerable to non-response. And in fact, I think that’s one of the things that slowed down progress in the field is because people can say, well, look, they were kind of okay in 2022 and 2018. They’re pretty good.

And then so why should we change? And then I keep 2016, 2020, but it’s not crazy to say these things kind of work sometimes. So I mean, we’ll talk more about it, I’m sure about the exact nature of non-response. And there are some patterns. I mean, people are more politically engaged or more likely to respond even when you control for covariates and you get these kind of patterns, but only sometimes does it lead to trouble. But when it leads to trouble, it leads to trouble quickly. You start going off the rails and you go into the ditch pretty fast with the current environment in a way that was not at all true with random sampling.

Matt Grossmann: So Brian, you run the cooperative election study, a big survey that’s used by academics but obviously is vulnerable to some of these other same problems. So what are the similarities and distinctions in your survey in how it’s dealing with these issues?

Brian Schaffner: So the Cooperative Election Study is carried out through a vendor called YouGov. They are maybe one of the oldest firms doing online polling. We’re certainly not going out for each unique poll and taking a random samples. So what YouGov has been doing for I guess maybe two decades now is recruiting people to take surveys. They recruit a lot of people to take surveys. So I think on their panel they have hundreds of thousands if not millions of people who have signed up to take surveys. So what you don’t want to do is just go in and say, let’s take a random sample of a thousand of those people because they are by definition a biased group of people. People who want to take surveys are not always going to look like and are online and whatever, are not going to look like the general population.

So the solution that Doug Rivers who founded YouGov came up with was what he calls sort of a matched random sample approach. So essentially the idea is to sort of acknowledge what Michael is saying about that we’re not dealing in a world of random samples, but still try to use some of the logic of a random sample and then add modeling to that. So the logic of YouGov’s approach is that essentially what they do is they essentially are taking census data and they’re taking a… So if you say, I want a sample of nationally represented of a thousand people from YouGov, what they’re going to do is they’re going to take a sample from census data essentially, and they obviously can’t contact those people. We don’t know who those people are, but we know their demographics and then they’re going to go in and find the closest matching person in their panel to each of those people.

So the idea is we have taken a random sample, we don’t know… We can’t talk to these people, but we can talk to people who look as much like them as possible on a whole host of demographic and political factors. So it’s trying to use some of the logic of random sampling, but not exactly random sampling our sample, but we’re trying to use the logic at the front end to get as representative a sample as possible. Then we recruit those people off the panel who look as much as possible like the random sample we took, interview them, and then make any post stratification adjustments, essentially any adjustments after the fact weighting the data, which all polls are doing essentially to make sure that if there’s any slippage between the sample we have from our panel and what the national figures should look like on education, race, age, et cetera, et cetera. That we’re basically, again adjusting that sample to look as much as possible like that.

So like most polls now there’s modeling involved to try to fix these issues that we’re not doing random sampling. And I think in the current state of the world, all polls are doing some sort of modeling. Some of the modeling is way more, I don’t know if I want to say sophisticated, but just way more involved than other forms. And I think my take would be that to the extent that polls are able to produce estimates that are fairly reliable, a lot of that is really weighing on the modeling side of things, that that’s where basically pollsters are earning their keep essentially. And so I think that makes things harder. I mean, Michael obviously knows this because if you’re just like, I have a random sample sample, therefore all these properties are true, that’s easy. Now the difference between one poll and another is what are they waiting on?

How are they sampling? What kind of quality control are they doing to remove respondents who are not being faithful respondents, et cetera. There’s all these options and a lot of that’s under the hood, not really… Either it’s I think not out there for people to look at or if it is, it’s like most people can’t really even understand it anyway. So I think there are all these challenges that make it much harder in the current climate because we can’t easily compare one sample to another because all the modeling is going… It is just so different.

Matt Grossmann: So your survey is obviously used primarily by academics. So on the one hand you have the advantage that it’s not the end of the world if you’re two or three points off on the margin and that you get to interview people afterward and you get to know whether they vote in the end. So that’s a lot. On the other hand, you might be seen as a disadvantage because you have to get not just the presidential result right, you have to get all kinds of things right. Levels of racial resentment in the population and sexism, a whole bunch of stuff that might be more susceptible to not being able to be fixed with this kind of modeling. So how do you think about that?

Brian Schaffner: Well, first off, I mean, the nice thing about all the things outside from the election outcome is actually, we’ll never know what the actual truth is. So if we’re wrong, you can’t really call us out on it. But the nice thing about election results is we actually know if we’re wrong on those or not. The other trick is… I mean, there’s several tricks, but because we’re an academic survey, we don’t put out the final data till, I don’t know, March of the following year. At that point, we know the outcomes and actually the final CS data, one of the adjustments that’s made is to make sure that the data look like the election outcomes nationally and in each state to make sure that we’re representative on that factor.

And that’s not unique to us. I mean, actually the exit polls have done this over the years. So if you download the exit poll data a few months after the election, it’s going to look wow, super accurate compared to what it looked like on the night of because, well, yeah, they’ve adjusted the data to make sure it looks like the outcome. And for our purposes it makes sense because academics… We’re not trying to predict the outcome. Academics are using our data to try to understand the election.

We’ve done mode studies where we’re trying to understand what other biases there are in our data that maybe we can’t adjust for. And I think the biggest things that we’ve seen is really that it’s about political engagement, that our samples report higher levels of any kind of political activity you might think of. So one of the things I often say to people is I think if you want use the CS to understand people who vote, people who are involved in politics, I think it’s a really good survey. I think if you want to use the CS to understand non-voters or people who are not interested in politics, I would be a little more concerned about that. I think the other challenge we face is that we’re so big. We’re a survey of 60,000 respondents in any given election year, that a lot of people feel at liberty to take a really small segment of that poll and say, well, I have 300 respondents who are, I don’t know, 57 traits, and then try to say something about that group.

But because we’re, again, we’re relying on modeling and adjustments, we are not going to be able to say for sure that our sample of, I don’t know, women and unions who have two children who are between the ages of 60 and 70 is going to be reflective of that population as a whole, even though there’ll be a lot of people in our sample who look like that. And so I think whether we’re right on racial resentment or policy attitudes or things like that, again, we’ll never really know because we will never have population data on that. I think when we compare our results to other surveys, we look a lot like those other surveys on those metrics. But I do think, again, political engagement’s one where I know that we’ve generally been at higher levels than other surveys.

Matt Grossmann: So Michael, your analyses show that neither waiting nor having very large samples really help to remedy these problems that you’ve identified. So walk us through the logic there and why we can’t get out of this with the common two methods.

Michael Bailey: So I mean, right now I have some YouGov data sitting on my computer and I’m analyzing it and it’s super interesting, and I think the academic field knows this better than the broader world is there’s really some strong assumptions. And I mean, the whole, there’s layers that people are answering honestly and all this other stuff. But the thing I focus on and I think is really a thing that we’ve been concerned about lately, and this is going to be true whether you do waiting or whether you do a sampling type thing, is that whenever you get, let’s just say we’re going to do just race and age and we’ve got white people over 50, and that’s a category, and let’s use waiting because waiting’s a little more intuitive and easier to get your head around. For polls to work, whether it is this kind of low response rate, random contact poll or some kind of non-probability poll, for polls to work, the white people who are over 50 that we get have to on average have the same views as white people in the whole population.

And so the missing at random is a kind of terrible word here, but the phrase isn’t a very… Doesn’t flow off your tongue, but that’s the assumption. But it just really boils down to within any subgroup that we’re waiting on that we get the right people or that we get a representative sample within that subgroup. And that’s pretty crazy. Just step back and think about that and especially step back and think about the groups that are really hard to get. Young, non-college people are really hard to get. You’re lucky to get two-tenths of a percent off of a lot of things if you have a random contact. I mean, maybe half a percent, maybe. I mean, just think about who’s going to answer a phone, that phone call they don’t know and talk to someone or even a text. So what we need then is those really unusual young non-college people to represent all young non-college people.

And maybe, maybe not. And in fact, we know, and I mean, we’re buying and seeing, but this is true everywhere. We’re going to get the people who are more politically engaged. They’re also going to be a lot of these full surveys now you’ll get a text and be like, click on this link. And I mean, who would click on a link? That’s terrible. That’s a really bad idea to click on a link from someone you don’t know. But that’s a more trusting people or more outgoing or more curious or more bored. I don’t know what it is, but they’re in the two-tenths of 1% range. And so, one of my favorite, favorite quotes to really bring home this kind of idea, it’s an old quotes, Ken Goldstein is a pollster that’s… Gosh, I forget who he was doing. He was working with a TV network, one of the major TV networks and this is Obama era, and they did a survey, Obama-McCain, I think it was in the state of Iowa, and they came back and they didn’t have enough young people. So that’s a standard thing that we can wade our way out of-

They didn’t have enough young people. So that’s a standard thing that we can weight our way out of. Oh, you don’t have enough young people. We just have a few. We really need a lot. We’re going to add, we’re going to give more weight. So each young person, we’re going to let speak for two people or whatever. That’s 100% how everyone does everything all the time, right? Totally normal.

In Ken’s case, when they did this in that survey, Obama’s numbers went down by a lot and they’re like, “Wait a minute, Obama, Mr. young people love me.” Especially in Iowa actually. It was weird. And then he has this wonderful quote and he’s like, “The problem wasn’t that we had too few young people. The problem was that the young people we had were weird and they’d be being gentle about the weirdness here.

It’s just in that case, in those era, they were still had more landlines and there was just this different kind of thing going on than there is today. But they were just getting a more … The young people they got were actually pretty conservative. And if you’re just flying blind and you’re like, “I just need young people. Let’s weight them up a lot and let’s cross our fingers and hope that they speak for all young people.” Honestly, a lot of times you’re fine, but you might get it wrong.

And Trump is a particular candidate where we have gotten it wrong. And so white working class voters in the Midwest have been the ones we get. The example I give, I had students do an exit poll in 2016 and they were out in Prince William County, and in this particular case, they’re in a fairly white precinct and they would literally, they’re in person. It wasn’t very scientific, it was more just to see the real world a little bit. But they would just see as people come by, the cultural signifiers of who would answer and who wouldn’t.

And a young, they were pretty much all white in that particular precinct. And a guy who’s getting in a truck and the baseball cap and the boots or whatever, they’re not really talking. Whereas the guy who looks more like a barista at Starbucks or whatever, demographically identical in any weighting sense is talking to them. And they literally had cases where they’d approach women and the woman was with … multiple times this happened, the woman was with a man, and then they would say, “Ma’am, we’d like you to,” ’cause they were randomly picking people and the students would say, “Ma’am, we’d like to ask you some questions.” And the husband would say, “She’s not answering.”

So what happens when you weight? If you’re right, you’re great. You make your results better 100%. If you’re wrong and if for example, the white working class voters you got are weird, there’s a small number of them and they’re weird, that’s a problem. And you’re going to make that into a big number by weighting, you’re going to weight them up. So in fact, you’re making things worse.

And it’s this weird thing. In some circumstances, weighting makes things better and in other circumstances, weighting makes things worse. And the thing that I would like to feel to push more is like, well, which circumstance are we in ’cause that’s really the crucial question.

Matt Grossmann: Michael, one of the more counterintuitive things, at least for me, was that we are regularly telling students that the sample size matters a lot. The population size doesn’t matter. It doesn’t matter if you’re trying to survey your local mayor or the nation as a whole and bigger is really better. But you have some simulations to show that those don’t really hold in a universe where we have a lot of response bias. Talk us through that logic.

Michael Bailey: Yeah, and by the way Xiao-Li Meng, a statistician at Harvard has a very nice paper on the paradox of big data or something like that, which is very influential and excellent paper.

But let me just give the example, and by the way, that’s one of my points is we’re in a new world. In random sampling we had this intuition, and by the way, and including me, we spent so much time banging into our students’ heads, sample size matters, population doesn’t ’cause the people are like, “That seems stupid. That’s crazy.” And we’re like, “No, here’s the reason.”

And in random sampling, it’s 100% right, right? It’s about the sample size, not the population size. When you have a non-random sample, and by the way, we know this and we’ve known this for a long time. Well, we know that … Anyway, this is also getting to sample size, just a large sample size and a non-probability sample doesn’t give you the right results.

Literary Digest 1936 had 2 million people and it was terrible, but your question was something different. Your question was about population size and why does that matter? And so I like to give the example. I mean it’s actually, it’s like an analogy and a story, but it’s fairly tight crosswalk to where we are. And so the example is teaching in a class.

And I’m sure people have been either on one side or the other of the dais there. And the professor will say, “Who knows the answer to” and ask a question. And so the people who respond are raising their hand, they’re opting in, they’re the one percenters, right? They’re analogous to a non-probability survey versus compared if a professor said, “I will randomly pick someone,” and then see the answer.

And our intuition is super strong. The kid who raises their hand is much, much more likely to be able to answer the question if it’s a hard enough question than a randomly selected kid. And so there’s a non-probability world versus a probability, a random sample versus non-random sample.

And now just imagine a couple of classrooms. Imagine where I have 100 students in the classroom and I ask, “Hey, who knows the answer to this?” And someone raises their hand. They’re going to know the answer, like a one in a hundred, not 100% of time, but imagine it’s a 20-person classroom or a 5-person classroom. Each of those, I get a sample of one and I get the kid who’s most willing to answer the questions. And as that population gets bigger and bigger, I’m going to like the kid who’s first one to raise their hand is probably more and more likely to answer the question.

And the point of bias is let’s say the actual in the population of the students is only 50% likely. If we were to give a test and everyone answered, 50% of the kids know the answer. Well, in a class of 5, the person raises their hand, maybe 75% likely because there’s only a few kids and they don’t really know. A class of 10, a class of 50, a class of 100, a class of 1000. Suddenly the thousand, 500 kids know the answer in that class of a thousand. But it’s the kid who’s raising their hand is going to really, really be likely to be able to answer that.

Matt Grossmann: So Brian, we’re obviously talking about this now because we’re right before a national election where everyone is repeatedly asking us who’s really going to win? And we have had two presidential elections in a row that have understated Trump’s support, although the midterms in the middle have not necessarily followed that. So what do you make of the polling industry’s responses to that so far? And what do you think will happen if we have a third presidential election in a row in which Trump’s support is underestimated?

Brian Schaffner: The thing I always remember about 2016 is I was on a panel of pollsters a few days after the election and a well-known pollster from New Hampshire was one of the presenters, and he was trying to understand why his poll showed that Clinton was going to win New Hampshire by I think 12 points when in fact I think she barely won it by 12 votes.

And so he’s sort going through all these different explanations and finally gets to the end and says, “Well, so then maybe we thought it was the weighting and we’ve only ever weighted our polls to age, gender, and region.” Which again, I was just sort of shocked. At that moment I was already shocked to hear, both because that’s not a lot to weight on. And also New Hampshire has regions, so who knew?

And then he says, “Yes. So when we added, we noticed that we were getting more college educated respondents than usual.” So after the fact when we added an education weight, we showed Clinton winning by half a percentage point or something. So essentially just weighing on that one factor made such a huge difference in his poll. And the vast majority of state polls in that election did not weight to education.

So everyone after 2016 said, “Well, okay, so we need to weight to education.” And so everyone was sort of like, “Okay, well we fixed that issue. Great.” And then of course 2020 comes along and surprise the issue is not fixed. And so I think at that point, again, the polling industry is still grappling with how to adjust for.

And this is a classic. I think this is a good example of what Michael’s talking about. It’s like you can adjust for education, but if the non-college white people in your sample still aren’t Trumpy enough, then you’re still going to be wrong.

So essentially at that point, I think pollsters up until 2020 were very reticent to weight on anything that was political essentially. It was very much like we only weight to demographics. And so a few people weighted to past vote. I mean YouGov to be the most obvious example, but most people wouldn’t weight to how people had voted in the past. They wouldn’t, certainly wouldn’t weight to partisanship. Maybe in a party registration state they would weight to party registration. But even then, a lot of pollsters in the states that have party registration data weren’t weighting to that. But after 2020, many more pollsters were like, “We have to weight to some sort of measure of some political measure like partisanship.”

And that’s the biggest change we’ve seen I think since 2020, is that whereas very few polls are weighting to partisanship or past vote before then, now many are. And I think that is because pollsters I think are coming around to the idea that is, as Michael said, should be quite obvious at this point that we don’t have random samples. We’re not in the world where we can just weight to age and gender and then call it a day. We really need to focus a lot more on modeling and thinking really carefully about how we’re modeling the data.

And I think this has gotten to some maybe too far extremes where people are talking about like, “Oh, we need to weight on social trust” or whatever. And it’s very much so there’s a concern I think that we’re overfitting to the last election that we’re essentially like that we’re going to overmodel ourselves trying to fix the last election’s problem and that’s going to cause new problems in the future.

At the same time, I do think it’s reasonable at this point in time to be weighting on something like past vote. We know what the vote was in 2020. We know that in an era of polarization, most people don’t forget who they voted for and aren’t ashamed to tell us who they voted for. So in the past 20 years ago, people overreported voting for the winner, but people don’t really do that anymore, I don’t think. Not to any extent that would matter much.

So I think that is a good way to help to some extent fix these issues. Whether it’s going to fix them entirely, I’m not convinced. And whether it’s possible it might actually cause other problems, it may very well. And it doesn’t fix everything because something like 20% of people who vote in this election won’t have voted in the last election. We don’t have no idea what their partisanship is or partisan balance is, but it’s good that the polling industry, I think has become vastly more sophisticated than it was even just eight years ago. And hopefully that’ll be to the betterment of what pollsters are putting out there. But I guess we shall see.

Matt Grossmann: Michael, what do you think of the changes since 2016? On the one hand, we do react to the last election. On the other hand, this sounds like the people who think the electoral college reform will happen next time. If only we get another inversion between the electoral college and the popular vote.

Michael Bailey: So yeah, no, I think that’s a super interesting topic. And it is interesting to see the evolution and as Brian said in 2016, the AAPOR report, the American Association of Public Union Research comes out with the definitive report on what happened and they said, “Hey, look, some people didn’t weight on education.”

But it turned out by 2020 then everybody weighted on education. And by the way, I mean to Brian’s point, it is so crazy to imagine someone not weighting by education and seeing. I mean, you just see it’s just insane. But they weren’t. And then they did. But didn’t fix it. And I mean even digging deeper into the 2020 AAPOR report also showed that wasn’t, it just wasn’t a cure all. It’s just good practice, but it didn’t cure things.

And then where are we now? And I think it’s a really interesting moment. And first I’ll defend the moment ’cause basically there’s where we are is there’s some people who don’t weight on past vote and some people who do and they’re more or less getting the same results. And so that’s a good news.

But let me just play the critic. That’ll be kind of my role here is just to point out some of the things, the challenges with that. So the first thing is we know … Well, all right, so that’s where we are. So one, there’s a set of people who don’t do that. And so New York Times in 2020, they were off, and they’re great. Now this is not, this is just saying like the best. They were off and they were off by a lot in ways that mattered, right? And they were telling us that Biden’s got this in the bag.

I mean, as political scientists, we keep saying again and again and again, none of this stop the steal stuff, that’s all garbage. But it was literally a very, very, very, very, very close election. And the pollsters were misleading us. Not intentionally, but just the poll that.

And so New York Times to my, not that set of pollsters who don’t weight on previous vote, I don’t think they’re doing anything different than 2020. I really don’t. You can take that as good or bad, but I just think I try to dig into what they’re doing and they’re basically doing the same thing. But then there’s this other set, and I mean some people were doing it in 2020 and I didn’t know YouGov was doing that in 2020 as well, but now weighting on the previous vote. And that’s pretty interesting.

But then this slices back, there’s two things to point out about that. So one, we went back at the start, and maybe we’ll talk about this more later as well, is there’s bias towards politically engaged people. That’s pretty clear. And so who of a politically, you just start thinking of your friends and think of a hundred friends if … I don’t have a hundred friends, but if I did, there’s only one of them going to respond, right?

And that’s not the most politically engaged person necessarily, but it’s in the tail of most politically engaged people. And are they moving? Are they going to tell you, “Oh man, I’ve really come around to that Donald Trump guy. I like Biden, but man, Trump is really, he’s got,” and vice versa for, “Oh, I really like Biden, but now,” or whatever the opposite is.

So of the politically engaged people, they’re not moving. And I mean that’s what we’re seeing. I mean, we’re seeing the YouGov data have this different CES, but they’re not moving. Right? And that might be true. I mean it’s a highly polarized environment. It might be true. But it is also possible. And by the way, that’s where 75, 80% of the polling electorate is they’re just stuck at where they were. And so when you do weight on past vote, you just get it’s 2020 again.

And it may well be. I mean, that’s not … That’s kind of where I think we are. But just want us to step back and just say to the extent that there could be problems and so forth, that’s the kind of thing that could be going on.

So I think it’s progress. I think it also helps us think that we won’t, to the extent that there’s going to be problems there, causing problems. Here we are fighting, or at least some people are fighting the last battle. And so now maybe that straight-up just Trump is underestimated by five points. We did it twice in a row. You start weighting on previous vote. It’s hard for me to imagine seeing that again. So that’s the good news.

At the same time, I think we may be missing out on that. We’re getting this very narrow slice of people and we’re really using their behavior that not only people who said they voted for Trump in 2020 or said they voted for Biden in 2020, but then who respond to a poll and we’re really extrapolating out.

And again, it goes back to that, the example I had earlier. If we’re right and those people who are speaking for all 2020 Trump voters and all 2020 Biden voters, amazing. We’ve now calibrated one of the things, ’cause we do tend, and I think it’s pretty true that polls in general tend to get not enough of the Trump 2020 folks. And so we’re boosting them up, even controlling for the education and race and some other things, which in the past we thought education would clean that up. We need to do even more to clean that up.

At the same time, we’re just leaning on this assumption that they’re representative of their demographic broadly defined here as vote choice as well. And so just my thing is I would just love to push on that a little harder. I think it’s reasonable practice. I think it’s reasonable practice to do them both. I’ve heard from some pollsters by the way that moves the results by seven points. So suddenly whether you weighed on previous vote or not, and you’re moving the results by the margin by seven points, I want to know that’s really important. You know what I mean? That’s the decision. And then I want to know if we’re right and I want to see if we can push on that assumption.

Matt Grossmann: So Michael, you said you’re the skeptic in this conversation, but some of us, some of this, we’re probably in the distribution of the most trusting part of the public. There’s a lot of people who’ve given up on this and they think that if you’re not going to do any better, then basically it’s going to be about the same as last time, then are we really adding any value there? So is it possible that we’re already over the edge, there’s not enough responses for these to be adding value? Or how would you push back against that?

Michael Bailey: Yeah, basically, the idea is just should we be doing polling at all? Right? I’m sure Brian has views on this too. So I think it’s just inevitable, right? It’s a democracy. People are going to vote and we need… A, we’re curious. We’re just curious human beings, this is an important election, so I just can’t imagine not looking under that rock, right? That’s just not how humans are wired.

But two is we need to know, we need to get a sense the margin, and I think the margin between Trump and Harris, that’s a tough thing because it’s close. So to really expect super precision maybe is not super reasonable, even though I think people could be doing more. But then we need to know, is it… You know what I mean?

From where I sit, I have trouble seeing, I literally have trouble understanding how other people view the world and make the choices they’re viewing. And I’m sure that’s true on the other side. So we, as social science, we, as a survey field, and then broadly as campaigns and so forth, we must try to systematically understand how people are thinking, right? If the people study American politics have any point on this, obviously study institutions are amazing, super interesting, but at some point we need to try to understand this.

And I think even with all the flaws of polling, we can at least approximately figure some stuff out and we can approximately say voters don’t really care about the environment. I wish they did, they don’t. And we can approximately say voters don’t care about international affairs. I wish they did, they don’t. We can approximately say that immigration is a big deal and that’s pushing a lot of people to one direction. We need them, we need to be smart about understanding them, but we also need to always try to keep improving them.

Matt Grossmann: Brian, I want to give you a chance to answer the critics, but I also want to put it in kind of the context of the variation that we see, which is that presidential actually does pretty well in comparison to lower salience races or primary elections. And one interpretation of that has been that there’s more undecideds or it’s harder. But another has been that we just have a lot of tricks to get us close to the result at the national level. And when those tricks are gone, we see the real state, which is a lot more uncertain.

Brian Schaffner: Yeah. First off, I think we are going to do polling. I think it’s sort of silly to debate whether we should or not because it’s something we are going to do and we’re going to do that as researchers, we’re going to do that just in terms of news outlets, et cetera. I think maybe the thing we should be pushing on more is what Michael’s talking about, which is using polling much more to talk about reasons for why people are voting or the things that matter to them moreso than trying to get a super close election correctly within half a percentage point or something, which is not really what polling is built for anyway.

In terms of how polling does in other races, I used to do more of these pre-election polling at the state level, even did a mayoral race. And for sure, I would stay up at night having nightmares about having to poll ballot questions, for example, or a mayoral race or anything, especially a mayoral race that’s nonpartisan. And again, a lot of that, I don’t know that that’s really about the tricks are better at the national level than people themselves don’t really know. Ballot questions, a lot of people don’t really know how they’re going to vote on ballot questions until literally like the day before. It’s not the same. That’s true of primary elections and stuff like that.

The other thing I’d say is at lower level elections, non-presidential year elections, it’s already hard enough to figure out who’s going to vote in a presidential election. So another source of error that we haven’t even touched on yet is figuring out who’s actually going to vote. It’s even harder in some ways to do that for primary elections or mayoral elections or midterm elections. So that modeling of who’s actually going to cast a vote in the election is also a problem.

I don’t know. So I think yes, we definitely know maybe more about how to model who’s actually going to turn out to vote in a presidential election year and how to get our samples to look good for those types of elections. I just think that the problems are even harder for these other elections at lower levels, off-cycle elections, et cetera. It’s hard to know who’s going to vote. People legitimately aren’t paying as much attention to those elections and therefore are not sure how they’re going to vote until the end.

But is certainly true that there are major polling misses when it comes to primaries and mayoral elections and ballot questions and things like that. I think, I don’t know, to be honest, how much of that is, what you say, tricks, the tricks don’t work at that level, and how much of that’s just legitimately capturing the fact that people themselves aren’t really sure until closer to the election.

Matt Grossmann: You recently found that weighting by education, party, and past presidential vote has increased since 2016, and some of them are helping Trump in the reported results in polls. So what should we be learning from that?

Brian Schaffner: Well, I think one obvious thing to learn is that pollsters are always trying to fix things that have gone wrong in the past. I think the big story after 2016 was that most pollsters were not weighting to education, especially at the state level, for state level polls. So I think most sort of adopted that for 2020. But then in 2020, even though the polls got the right winner, they were still off. So pollsters were trying to evolve again after the 2020 election.

And I think after that, they realized just adding education weights was not doing the trick. So they’ve now, much more than in 2016, started weighting the past vote or partisanship. So one or the other, with the idea being we need to make sure we have enough Republicans in our sample. We can either do that by making sure we have enough Trump voters from the last election in our sample, or we can do that by trying to weight to partisanship. Even though we don’t really know what truth of partisanship is, we can try to estimate that from high quality polls.

So weighting to those things does seem to help Trump on the margin. And whether that means the polls will be more accurate this time, I think very possible that it will fix the issues. Also even possible that maybe it makes the polls a little more favorable to Trump than is true. And it’s also possible it doesn’t fix the issue. So I think we won’t really know until after the election, but I definitely feel more confident about the polls now that most of them are weighting to some measure of basically Republicanism or partisanship.

Matt Grossmann: So you also recently did an analysis of undecided voters, people who are kind of left late in a cycle, which people are looking at now and found that a lot do not turn out at all. And they usually break pretty evenly, but that in the last two elections, they broke toward Trump. So how’d you do that and what should we take from it?

Brian Schaffner: Yeah, so we did that with our cooperative election study data. So we actually interviewed people before and after the election, and I think the key thing is that we interview a lot of people. So in any given cycle these days, we’re interviewing 60,000 adults before and after the election. So that allows us to get a sense of, okay, who’s telling us in October, when we do our pre-election interviews, that they still haven’t made up their mind? And then when we go back to them after the election, we can ask them, “How did you vote?” And then the other key thing is that we match our respondents to the voter file so we can know who actually turned out to vote and who didn’t.

So yeah, what we found is that, first off, people who tell us that they’re undecided, that they’re not sure how they’re going to vote, the vast majority of them actually aren’t going to turn out at all. Even people who say they’re undecided and who are also saying that they’re definitely going to vote, roughly 40% of that group is not going to vote. So even people who are saying, “Yeah, yeah, I’m going to vote, I just don’t know for whom yet,” a lot of them aren’t going to vote.

But then we found in 2020 and 2016, that among those who did end up voting, they made up about 5%. Actually, it was a bigger share in 2016, but in 2020, they were about 5% of the electorate, people who were undecided and ended up voting, and they did break by about 20 points towards Trump. So if you think about that as 5% of the electorate, they’re going by about 20% for Trump, that’s basically means that they added essentially a point to Trump’s margin in 2020.

So what that means for 2024 is still a little unclear. We don’t really know if they will break for Trump in the same way they have in the past. But obviously, the election is close enough that if you did see something similar, a one point swing could actually make a big difference in this election.

Matt Grossmann: So the current response to a lot of these difficulties is for people to say, “Throw it in the average.” That is, one poll might be bad, but we have aggregators and modeling and that should improve things. But obviously, alongside that, we have the concern that polls might get closer to one another at the end of an election with herding.

So either of you have thoughts on to what extent we can rely on the fact that we have multiple polls? I guess I’ll make the positive case, since it seems you are skeptical, that given that there’s a lot of modeling decisions to be made, maybe we just need multiple people making those modeling decisions to get closer.

Michael Bailey: One thing I think this cuts back to one of the themes that I think about is in a random sampling world, you have one random sample of 1,000-person poll, you have another random sample of a 1,000-person poll, and they ask more or less the same questions. Then you have another. Now you have three of them, you have 3,000 people, and your margin error shrunk dramatically, and so that’s amazing, right? In a random sampling world.

And then now when we’re getting in this, it’s all model based, it’s random contact at best, and then 1% response. It could be the Washington Post that just came out today had 10% because they paid people $10 each to respond, so you can get higher response rates and whatever. But it’s just now, it’s just like you’re sampling over not this thing that made sense or you merging in that. Now you’re just sampling over the distribution of choices that pollsters make. And we just don’t know.

And some of them are biased intentionally. Some of them just have just their own idiosyncratic views. There’s a few still don’t weight on education even, there’s still a few of those around. And throwing those in the average, there’s just no theory there. And it’s interesting, and certainly, you want to not overindex on one poll and just forget what everyone else did. But we saw it in 2020, even when you aggregate those polls, they had Biden up by eight or nine points on average depending on who you choose, and he won by four. And then even there, they were not getting at the closest of the race in Wisconsin and other places. So I don’t oppose it, but it’s far from a cure-all that’s, for sure.

Brian Schaffner: Yeah, and I think in 2016 especially, but [inaudible 00:49:03] the models, I think in 2016, there were more of these models. I think, thankfully, a lot of people shut down their models after 2016, but those models also weren’t sophisticated enough to take into account the possibility of herding.

Essentially, you had some models that were sort of like 99% that Clinton’s going to win. And it’s based on this idea that Michael’s talking about, which is like, yeah, we have 20 random samples from Wisconsin all telling us this thing. And whether it’s nefarious or it’s a product of everyone’s making the same modeling choices, whatever, we can’t really treat those as a normally distributed sample of 20 samples of Wisconsin opinion. And therefore, the model is treating that as like, wow, we can be this much more confident. Whereas it’s not clear that another survey that’s sort of modeling in the same way as the previous survey is actually adding the same amount of information as it would if you just had two random samples.

So I think the models we have now are better at adjusting for correlated polling errors and stuff like that. But still, I think there’s a lot of evidence that people don’t really understand probabilities as well as they can understand margins in a poll. That models, therefore, may do more to mislead.

If anything, the good news about this year is actually the models are showing it so close that most people can understand that 50/50 is a coin flip. But when models were getting, like in 2020 or 2016, to 85%, 90%, people were vastly overestimating how certain that was. So it was just compounding the problem that polling was having, that these models were sort of maybe even doing, sort of taking what would already be a misleading set of results and making it all the more misleading.

Michael Bailey: Can I add just one thing? The thing that, so aggregation, I think it’s fine. And certainly, these sites where you can just go and look at the different ones and you don’t like this pollster or you don’t like that pollster, you can do that. It’s a service.

But one of the things I like, and New York Times has done a little bit of this, but I think others do do it and maybe more should do it, is go through and just show us within a poll, here’s things we did and if we did it differently, how much does this move? Right?

And the classic example this year seems to be weighting on previous vote. And I think that would be, that’s almost as if one pollster did it with a method and weighted on a previous vote, another pollster did it with the same method and even the same people as it happened and didn’t weight on that. And then suddenly, now that’s more analogous to trying to sample over decisions by the polling organization. Here, it’s they really are holding everything else constant.

And again, New York Times was not finding a huge difference. Although Trump goes up by quite a bit if they were to… Well, he goes up by a little bit, but it’s enough to matter if they were to weight on previous vote. But I’d like to see other people show, who do weight on previous vote, if they took that away. And again, just anecdotally, one pollster I know in the field who does weight on this says it makes a big deal, and that’s more in the spirit of where the aggregation, I think is value added.

Matt Grossmann: So you both mentioned that subgroup analyses make all of these things more difficult, usually. On the other hand, that is one of the values of pre-election polls is seeing, identifying groups that are moving and has helped to guide journalists to look for stories and to interview people who might not otherwise be interviewed.

So what do you make of that in the wake of 2024 polling? We have a lot of interest, for example, in the African American community, the Arab American community, because of some of those pre-election polling trends. Last time, the polls, at least in aggregate, did successfully anticipate some Hispanic shifts toward the Republicans. On the other hand, they incorrectly predicted some white working class moves toward Biden. So maybe we’re chasing things that aren’t real from the polls. What do you make of the subgroup analyses this year and their role?

Brian Schaffner: I’ve been pretty skeptical, to be honest, about a lot of this stuff. Especially, I know the New York Times, for example, has really been hitting on the, for example, the story about Black voters or Hispanic voters. But their polls aren’t even really consistent necessarily with other polls that are being done right now. And when we’ve also, looking at our own data, in the past several cycles, we’re seeing that if you had just looked at the pre-election-

We’re seeing that if you had just looked at the pre-election polls, you might’ve seen some shifts in previous elections among Black voters that actually don’t materialize either because the people who are saying that they’re going to vote differently are actually not going to vote at all. Or, because Black people are just more likely to be undecided later in the election than they used to be, but they still come home.

So, I think the subgroup stuff is hard. I think New York Times did recently do an oversample of African-Americans, which is a good, because it’s even worse when you’re just doing a sample of 1,200 and then relying on the whatever, 120 or whatever African Americans in your sample are. But you do still maybe run into the problems Michael’s talking about, which is like, yeah, you can get a bigger sample of African Americans, but still, is that a reflective sample of the African American community writ large?

So again, I would be a little skeptical until maybe after the election about some of these things we’re talking about, whether it be young men or African Americans, or even Hispanic voters, whether we’re going to actually see these big shifts materialize.

Michael Bailey: Yeah. I think this is a great question and a great topic for survey and political scientists folks to really think about, because things move, right? It’s so partisan, but just in one election that things could move, right? And I mean, things have moved, and obviously we saw huge movements in white working class, like crazy big movements around 2016. And so, is that happening again? So, it’s a great question. It’s a real question. And there’s some logic behind believing there is movement and some evidence behind that.

But I think this is a classic example where just thinking about the strengths and weaknesses of survey research … And by the way, right now, the way I see it, I see folks, like I think CBS had African American support for Trump at 12 and New York Times is getting up into around 30. That’s a lot.

Brian Schaffner: That’s a big difference.

Michael Bailey: That’s a big difference. That’s doubling or more. Right? And these are serious people. These are serious things. And so, that gives a range of what is plausible.

So, let me just tell one story. It’s a dated story. It’s a little older, but just an example of how I think about things and how maybe this kind of thing could be … We could contribute to things. So, in the background, this all depends on the Black voters, the Black respondents we get, are they speaking on behalf of everybody? Right? And so, one, we translate that into social science speak is like, is the response interest. Right? If the high response interest, meaning the people who respond to polls, are different than the low response interest, that’s when we have a problem. Right? Meaning the people who raise their hands know the answer to the question more than the people who don’t raise their hands, then we can’t use how many kids got it right when they raised their hands as a good proxy for how well this class knows what’s going on.

So, one of the tricks that I do, and it comes out of the book, is we ask a random subset of people, “Hey, it’s a survey or a survey of Americans.” Pretty generic kind of thing. And then just say, “Hey, do you want to talk about politics, sports, health or movies?” And in mine it was around 40% of the people would say politics and 60% of the people would pick one of the other categories. And then, we go through and there’s various things that now it’s getting into a lot of methods and some statistical adjustment models and so forth.

But just like common sensically, we can just look at it and compare those people who chose politics to those people who didn’t. And if there’s a big difference, we don’t know this, but that’s a diagnostic to say, “Look, the kind of people who want to talk about politics are different than the kind of people who don’t want to talk about politics.” And honestly, when we’re only getting a 1% response rate, or we have people self-selected into a non-probability poll, the people that don’t want to talk about politics, we should bear them in mind when we’re trying to extrapolate to the rest of the population. Right? And again, if they’re the same, then we’re like, “Oh, it doesn’t seem to be much going on.”

Anyway, we did a poll with … This was a non-probability poll. I did two of these polls, one in September and one in April, I think. And so, this is when Biden was still the candidate. And we were getting around 200 respondents, Black respondents. So, it’s a small sample, but as these things go not terribly bad, and it was really striking of Black respondents who wanted to talk about politics they were running a good five percentage points warmer, more likely to vote for Trump than the rest. Right?

And then when we put that into the model and the model has assumptions, but we put that into the model, take that raw piece of information. In the raw in both of those, there’s 30% support for Trump among Black respondents. And then do the adjustments and it came down to 12 or 15%, it basically cut it in half. And I’m not sure which is right. And I think during Biden, Biden, there was a very different environment than we have now with Harris. But at that point, that would lead me to say, “Okay, yes…” And by raw, I mean we use typical weights. If we use typical weighting, we are still seeing 30% support. Do the little trick I did and it cuts in half. I’m not sure half was the right answer, but I am sure to say at least I have some diagnostic evidence to believe that 30% is pretty soft.

Matt Grossmann: So, you recommend looking at selection models as opposed to weighting and also different ways of studying non-response. I just wanted to get at whether there’s anything that pollsters really could be doing right now and how likely you think it is to be able to improve?

Michael Bailey: Yeah. I mean, yes. And then, I think, A, just right now today, there’s things they could be doing. And then B, there’s fairly small changes they could be doing. And then C, there’s the bigger stuff. But A, I mean, I’ll just give an example, and it’s in the book, is in 2020, the American National Election Study, which by the way, for listeners who don’t … That’s the best. I mean, CES is a non-probability poll, huge sample size. The American National Election Studies is a probability poll, smaller sample size, but both of them are run by academics and both of them are very, very serious.

And so, the ANES had Biden up by 11. That’s a lot. That’s a lot. That’s in the raw data and then you add weights and he goes up to like 11.5. It’s basically no change. Right? So, weighting didn’t do anything. And so, then the election, Biden wins by 4.4. And what was happening? Well, one of the things that actually Leonie Huddy at SUNY Stony Brook pointed this out to me. And it’s a great example. If you look at just in that survey, just using, and it’s a fairly common question, it’s just one more question, is you ask something about political information. High information, middle, or how much are you interested in? It was an engagement type of question, and, “Very interested,” or, “I don’t care about politics at all.”

Of the people who said the lowest two categories, “I don’t care,” or, “I only care a little,” Trump and Biden were 50/50, like striking. And I have little bar charts and they’re just neck and neck. And then the people who care about it somewhat, Biden opens up a nice eight point lead. And then the people who are very interested in politics, Biden was 60-40. It shocks me even to this day. And even if you control for covariate, and I just do the bivariate or whatever, but you control for covariates, education and all this, the same pattern emerges.

So, that’s just using … That we’re not getting into weighting and all these other … It’s just a common sense like, look, in this ANES survey, they had a 40% response rate, so a nice response rate relative to what we’re used to. But still, it’s like an hour-long survey about politics. Who did we get? Too many people who are really interested in politics or too few of them? I mean, it’s just even a lay person who’s never done any survey is like, “Well, you got too many of these people who are interested in politics.” And Biden’s racking up the lead there.

Somehow we got to get to the voice of those people who don’t care about politics. We got to work that out. And ANES and others, no one was doing that because they’re just … In my view, I think there’s a little more stagnation in the field than there should be. Is it there just like we kind of had random sampling, okay, we’re not there. And the toolkit is weighting. And then by the way, quota sampling and so forth, basically uses the logic of weighting. That’s all we can do. We just find the right weighting variables. But we can’t use political interest as a weighting variable because we don’t know the true distribution of that. And some people now are like, “Well, screw it. Let’s just make one up. Let’s do common sense and make one up.” And they still would’ve got a better result than what they actually got by doing that. But that’s obviously a little sketchy. But at a minimum, just looking at something like that, you saw a big, big, big warning light.

And then B, you could do the thing that I’m doing, and I’m got one on my machine right now trying to analyze it, is where you go and you just ask people, “Do you want to talk about politics or not?” And then there’s an assumption, but at least there’s a diagnostic to the extent you see the differences. And by the way, the thing that we see again and again is on voting turnout. Right? So, of the people who want to talk about politics, they’re like, “I voted,” or, “I will vote.” And we take a little grain of salt that they’re whatever, that they might be lying or exaggerating or something.

But of the people who don’t want to talk about politics, they choose sports, health, or medicine, or whatever the categories are, it’s down by a fair bit. And that’s a sign that probably the people who aren’t even on the survey at all, at a minimum are more like these people who don’t want to talk about politics, and maybe even it keeps trending down.

And so, right now, to the extent that I’m disagree or whatever with the polls, I’m not sure that I don’t see evidence of this Trump effect that we saw in 2020 and 2016 of non-college whites being more supportive than otherwise. At least in the data I have, I haven’t seen that. But I see very, very clear and substantial that we have the people, if you just take it at face value, who says they’re going to vote, and especially in those groups that Brian was talking about, young Hispanics, non-college, they’re not going to vote as much as they say they’re going to vote. They’re just not.

I had a subgroup of, I think it was young, Hispanic, non-college, 70% said they were going to vote. I wish it were. I really, really, really wish that were the right answer, but it’s not. Right? And if you just take that at face value and then you adjust that through your sample, you’re going to get the wrong percentage of them. And I’m not sure which way that cuts, by the way, but you’re just going to get the wrong percentage of them. And non-college is the same thing. And so, I think right now it’s on that turnout. We haven’t talked a ton about that turnout model, which is really hard. It’s really hard. But to the extent that you use people’s answers, we have to keep in mind we’re getting a biased sample if people are more likely to say they’re going to vote than the full population. And that could percolate through the process to lead to some problems.

Matt Grossmann: So, one of the more troubling findings for me, but maybe positive for the state of democracy, is that we might just be overestimating the level of extremism or partisan consistency. So, you find that we’ve been talking about things that where you’re trying to get the right number of people on each side, but we are often … Probably the most studied question of the moment is about polarization. And it seems like your findings suggest there’s a real potential for us to be overestimating that, because we’re repeatedly talking to the interested people on both sides.

So, how should we interpret that? Is it good news that maybe actually we’re not actually that polarized, there’s more movement, inconsistency, and less extremism than we thought?

Michael Bailey: Yeah. This is something I’m super interested in. And so, by the way, just one backdrop is I’m just trying to change our mindset, do diagnostics, and I’m agnostic as to what the diagnostics will find. Right? So, it could be pro-Trump, anti-Trump, whatever.

But then just the thing I find again and again, you know what I mean? And again, I can’t guarantee it’ll always be the case, but it’s a pretty regular effect is that above the Democrats you get, you get the more partisan Democrats, of the Republicans, you get the more Republicans. And by the way, those things can balance out. Right? So, you still get kind of the right percentage overall for how Hillary or Biden or Harris is going to do. But as you go inside of it, it’s more extreme than it should be. So, I just think that’s something to keep in mind. I think it is a polarization, I’m not going to say it’s not. But when I do these things, you’d have the measure of even the weighted data would be people are out here, and then you’d do these adjustments and they’re here. And it doesn’t mean it’s all happy and so forth, but it does change things.

And this is from 2019, but there was two cases where I think these things really matter. And so, here’s getting it wrong within the subsets. And that’s where I think the biases are strongest. Right? And this is more just from experience. I’m not saying it’s always going to be the case, but from experience. And so, for example, among Democrats, we would do this trick and we’d just say … First, we just do it for everyone, “Do you want to talk about politics or other stuff?”

Of Democrats who wanted to talk about politics in 2019, they were super liberal on race. So, we asked a classic battery of four race questions about race, race and public policy. And they were really liberal. In fact, white Democrats were more liberal than Black Democrats on that battery of questions. But then you went just over and you’re like, “What about the white Democrats who didn’t want to talk about politics?” Very different actually. I mean, not like crazy, crazy, totally different, but different. Different, for sure. And certainly less liberal than African American response, for sure. Right? And so, now just thinking about where Democrats are on that issue, right? There’s the Democrats who are engaged were in one place, and the Democrats who are more engaged were in another place, and now we extrapolate out to the rest of the population, for better or for worse there was strong evidence of that. And you kind of hear rumblings of this in the ecosystem of surveys and public opinion, and so forth. That’s the kind of thing I honestly expect to happen fairly commonly.

On the Republicans, it was interesting, and I don’t hear much about this, but we did same thing. And then, here it was taxes. By the way, Republicans weren’t that different on race. The Republicans who didn’t want to talk about politics were a little more liberal, right? But not a lot. So, it’s kind of interesting. But then on tax cuts … This is around 2019, and we’re talking about Trump tax cuts in his first term. The Republicans who wanted to talk about politics, they love it. It’s great. It’s amazing. But even a bigger gap, the Republicans who didn’t want to talk about politics, just not really that into his tax cuts, by a lot, right? And I mean, we kind of know this, the white working class isn’t super jazzed about it. You don’t hear him talking about it. He’s talking about his other stuff more in his rallies. So, he kind of gets it. But now in the data, if you just took an uncritical view of the data, you might overestimate how much Republicans support tax cuts. Right?

And then, I don’t have handy things about immigration and Democrats on immigration and so forth, but my guess would be the engaged Democrats are more liberal on immigration than the less engaged Democrats, and so forth.

Matt Grossmann: There’s a lot more to learn. The Science of Politics is available biweekly from Niskanen Center. I’m your host, Matt Grossmann. If you liked this discussion, here are the episodes you should check out next, all linked on our website. How much are polls misrepresenting Americans? The past and future of polling. Previewing 2024, how voters judge presidents. Interpreting the early results of the 2020 election. And, does the 2022 election show how Democratic campaigns win?

Thanks to Michael Bailey and Brian Schaffner for joining me. Please check out Polling at a Crossroads, and the Cooperative Election Study, and then listen in next time.