Looking At What Went Wrong With The Polls
The public polling industry is not exactly seen in a good light right now. In fact, they are probably at the lowest point in public confidence and trust ever, since modern polling began. Maybe they should take a poll to figure this out, or something.
Snark aside, however, it is indisputable that they've now blown two presidential elections in a row. Not everywhere, to be sure -- some of the state-level polling was indeed accurate. The national polling showed Joe Biden would win with a healthy margin, and he did. But they got this margin wrong -- outside their own vaunted "margin of error," in fact. We won't know by exactly how much until all the final vote counts have been recorded and published, so it'll take a few more weeks to even get all the data that needs sifting through to determine what went so wrong -- again.
Admittedly, a large part of the problem pollsters have had to face is the "black swan" nature of Donald Trump. Trump is not a normal politician. He's not a normal Republican. And he certainly hasn't been a normal president. So using rules honed over decades of predicting how voters see normal politicians didn't work because Trump is so far out of the mainstream that it's impossible to even know how much of his appeal was personality-driven. To put this another way, nobody has any idea if the polling might have been a lot more accurate if Donald Trump hadn't been the candidate and the Republicans had run a normal candidate. We won't even be able to even begin to know this until the results are in for the next presidential election -- or, to be more strictly accurate (because he might run again) -- until we see the results of a presidential election without Donald Trump on the ticket.
Pew Research, a prominent polling organization, has already put out a very preliminary post-mortem document which begins the attempt to answer the question of what went wrong. They admit that they won't really know for a while, until the data is available and they've had time to sift through it and interpret it all. But this document at least begins the self-examination process.
They suggest a handful of possible reasons why everyone was so off in states like Pennsylvania. But the one that caught my eye is the one I've personally been wondering about since Election Day -- how "likely voters" are identified by pollsters.
This election broke all records for turnout. That much is already clear. Two-thirds of eligible voters voted, when that percentage is usually in the 50s, not the high 60s. This means a whole lot of people who usually don't vote actually did so this time around. But how many of them were dismissed by the pollsters as "not a likely voter"? Over ten million of them turned out, which is more than enough to throw off the polling even on a nationwide scale. And, once again, the key motivating factor on both sides was Donald Trump. Millions who had never voted or previously voted infrequently turned out both to vote for Trump and to vote against Trump. Without him on the ticket, it's impossible to know how many of them would have stayed home instead.
Pollsters ask various questions to try to figure out how likely it is that the person they're talking to is actually going to make the effort to get out and vote. Sometimes they ask whether you've voted in the past two elections. Sometimes they ask about what your "enthusiasm level" is about voting. Some just come right out and ask: "How likely is it that you will vote this year -- not at all likely, somewhat likely, or extremely likely?" But whichever voters they deem to be outside their "likely voter" definition get completely ignored in the poll results. Presidential polls start early on by reporting either "all adults" or "registered voters." But then a few months before the election itself, they all switch over to only reporting data coming from "likely voters." To me, this seems an obvious major culprit for what went wrong this time around, in an election with record turnout.
The Pew document only very briefly addresses this issue (the document is only a preliminary attempt to identify polling problems, to be fair). I should mention that their format for the document is to briefly lay out the problem, ask whether such an error would affect other (non-political) polling that they do, and then attempt to suggest a possible way to fix the error. They kind of punted on the last one, as you can see below, so I've retained their internal links (they essentially point to a 2016 document on what was done the last time around, which isn't really an answer at all). Anyway, here's how they put it:
The suggested problem
Election polls, as opposed to issue polling, have an extra hurdle to clear in their attempt to be accurate: They have to predict which respondents are actually going to cast a ballot and then measure the race only among this subset of "likely voters." Under this theory, it's possible that the traditional "likely voter screens" that pollsters use just didn't work as a way to measure Trump voters' enthusiasm to turn out for their candidate. In this case, surveys may have had enough Trump voters in their samples, but not counted enough of them as likely voters.
Is this mainly an election polling problem, or would this be of wider concern to issue pollsters as well?
If the main problem this year was a failure to anticipate the size of Republican turnout, the accuracy of issue polls would be much less affected. It would suggest that survey samples may already adequately represent Americans of all political persuasions but still struggle to properly anticipate who will actually turn out to vote, which we know is quite difficult. Fortunately, the eventual availability of state voter records matched to many election surveys will make it possible to assess the extent to which turnout differences between Trump and Biden supporters explain the errors.
What could we do to fix it?
Back to the mines on reinventing likely voter scales.
Back to the salt mines, indeed. The problem for pollsters is that it is unknown how much Trump's black swan nature changed things both this year and in 2016. Will Trump voters who had never voted before revert to non-voting again after he's gone? Or will they become regular voters? Or will they only turn out when someone sufficiently Trump-like is on the ballot? Nobody knows.
Nobody knows the opposite effect either. How many first-time voters turned only only to defeat Donald Trump? What are they going to do next time around? Will they revert to non-voting, or begin voting in every election? It's hard to say. This is why (as uncomforting as it is to think about) I say that we may not even begin to answer those questions accurately until after the 2024 election.
There's a larger question as well -- is public opinion polling just broken beyond repair? Is it so inaccurate in today's world (where few people answer their phones if they don't recognize the caller) that it is simply no longer relevant because it has become impossible to truly get a large enough statistical sample of the public? That's a much harder thing for the pollsters to contemplate, obviously.
I'll be watching closely to see the results of all the polling industry's self-examination. This process is likely to take a long time -- I don't really expect to see solid results for months, at the very least. But the future of any level of trust for public opinion polling depends on finding some answers at some point. Because as of now, they've pretty much entirely lost their credibility with the public. As pretty much any poll would clearly show right now, if they bothered to ask the question.
-- Chris Weigant
Follow Chris on Twitter: @ChrisWeigant
I never really pay too much attention to polls, anyways. When they sound bad, I prefer to ignore them. When they sound good, I don't really believe them, so ...
Another factor that may be contributing to the polling disasters of recent elections might the existence of a widespread voter sentiment that could be characterized as a desire to have a ballot choice of "None of the above".
When a voter has the impression that there isn't a qualified major-party candidate, his options are 1) vote for the least unqualified major party guy, or 2), go third party.
In 2016, I voted ANTI Trump third party, and this time I voted ANTI Trump for Biden. That type of thing probably represents a pollsters nightmare.
This analysis - likely voters are harder to identify when an entirely new cohort of voters appears - is quite different from several other stories I've been reading that try to figure out why the polling was so far off this year.
One I've seen several times now is the idea that Trump voters have been trained to distrust mainstream media, and they identify poll-takers as part of that group. So they refuse to answer polls on principle, in contrast to Democrats who believe they are serving the public good by civically answering a pollster's inquiries.
The larger question you bring up at the end, of how polling can be conducted in a polity that is completely atomized, is the right one, I think. It connects all the analyses I've seen. People don't answer their phones for strange numbers, and don't talk to anyone they don't know personally. Group social activities, where crowds assemble in actual places, are declining over time as more and more interaction takes place virtually, allowing people to pick and choose who they want to be part of a crowd with. In a society like that, polling based on random sampling of the whole by talking to a representative part is increasingly inaccurate, because there is no whole to find a part of.
Looking forward to more thinking like this as you continue to do the scut work for us of actually reading these things! Thanks for this piece.
i think at this point it's undeniable that opinion polls tend to underestimate trump's turn-out.
Fellow Travelers,
And here I thought it was "Our Chinese Allies were more effective at tipping the scales for Comrade Joe than Putin was for Trump." Heh.
/end sarcasm
I notice that Trump received marginally more % of votes over and above that of his four years of approval ratings, we all knew going in that Trump would be lucky to garner much more support than that which his base provides. In that, there are very little the poles weren't telling us going in. We assumed Biden would win the pop vote, we also felt sure Biden would likely do well in the rust-belt and probably AZ (the McCain mutiny).
Frankly, I think this election was well-enough augured that the outcome was in little doubt, Trump may be a random element, but his ability to divide outweighs his ability to unite, therefore his appeal is predictably easy to understand. It won't ever be more than the sum of his base, plus a few points for people who feel plugging their noses is a small price to pay for whatever return Trump accidentally woos them with.
Aside...
https://www.google.com/imgres?imgurl=https://pbs.twimg.com/media/Em-TvEGXMAY1txV.jpg&imgrefurl=https://www.twigzu.com/hashtag/lmfao&h=600&w=1200&tbnid=nbXRFNFSRo3afM&tbnh=159&tbnw=318&usg=AI4_-kSo0m-CBqNK4jLeN2MTfR21tbNW-Q&vet=1&docid=CDAl3bpTqgM8vM
Only the Trump Campaign can fuck up so staggeringly...
Georiga...
I ask you.
LL&P
Short Fingers says that polls that didn't show him with a lead were voter suppression.
What went wrong? The suppression wasn't very effective in Maine and North Carolina.
Quack Atlas says go have Thanksgiving dinner with Grandma. She's not going to live much longer, so push her off the cliff.
For many people this is their final Thanksgiving, believe it or not.
These Socialist Security blood-suckers have got to go. This 47% taker class must be eliminated one way or another.
BTW - When is Biden going to get this virus under control?
Well, Biden is confused. The transition is moving along smoothly despite Trump shenanigans but, people will die as a result. Good God.
CW brings up an interesting point in his second to last paragraph. A lot of people don't answer their phones if they don't recognize the number. This raises the question, who are the people who are willing to answer their phones? Is it possible that the "random" sample is now inherently biased because only a certain subset of Americans are actually able to be polled? It honestly would not surprise me if the segment of the population willing to answer their phones skewed liberal. I have no idea how you could prove that either way though.
This raises the question, who are the people who are willing to answer their phones?
Yeah, I thought about that, too! But, I'm not sure why one might presume that it would be the liberals who answered their phone not knowing who was calling. Oh, wait ... :)
Liz [11]
I get that you're saying it tongue on cheek, but it's a valid question.
My reasoning is that, at least from my experience, conservatives tend to be a bit more insular and private, at least regarding their personal lives. Liberals, from my experience, tend to be a bit more open and up front about their beliefs. From my experience, conservatives also seem to have a slightly lower tolerance for being bothered by random people, such as spam calls or door-to-door salespeople, while liberals seem at least a bit more willing to humor such people.
Of course, my personal experience is hardly a scientific study, so I could be way off base.
That was far less tongue in cheek that you imagine.
I'm pretty liberal and, very private, so ...
Though, I love to kid the liberals. There should be no doubt about that!
Liz [14]
Of course, there are always exceptions. In fact, we may be talking primarily about exceptions, it's just a matter of whether those exceptions tend to occur more frequently among certain groups of people than others. Ugh, statistics are a such a pain. When I had to take that class in college, I needed to get at least a C, so I was so happy to find out I earned a 69.7%, which would round up to a C.
I think you're quite right about the exceptions and I'm with you on the stats!
Everybody seems to be hung up on how far the actual vote counts were from the polling margin of error. Get over this, the polling margin of error - as we know from TV - is a best case estimate...in real life the variance of individual elections is going to be much higher. Most states are barely sampled by pollsters...if they are sampled at all. The polled population is not the voting population...and getting less representative in a digital age with diffused choices of news "coverage".
Oh, and the Presidential Election is decided by electoral votes, not ballot box votes. We all know this and then we promptly go on to act like we don't know this. Five Thirty Eight predicted Biden would most likely get 348 electoral votes. Biden got 306. The major TV networks did about as well. The British prognosticators closely agreed with their American counterparts.
The Prediction Markets (aka betting shops) seem to have accurately called the race earlier than the conventional prognosticators. This does not surprise me.
All things factored in, CW.com did pretty well this season.
Bleyd [10]
I never answer the phone for unidentified numbers. I would answer for pollsters if I knew that was who was calling. I don't understand why they hide their identities.
LizM [9]
Biden is confused
Well, Fat Donny did warn us.
JFC[19] ... agree 100 percent!
Is this strong desire to "fix what's broken about the polls" of the same spirit as "we really really really want to know the election results the very night that voting ends?
I get it, we like instant gratification down here in the States.
I think that you're right about Trump being something of a "wild card" for two elections now. And Trump will undoubtedly tease a 2024 run to keep himself relevant and as always, the center of attention. There was no reason to believe that Trump thought that he'd win in 2016 so IMO it was always about a "branding exercise."
Trump's post election shenanigans will further incentivize our beloved "deep state" to prosecute him to the full extent of the law, just to get Trump's Twitter machine out of his hands as a result of incarceration.
FPC
[63]
For those who missed my "mea culpa" at the end of last columns Comments Section,
Dear Weigantia,
Upon further review I decided to "yellow card" myself so as to practice some introspection.
While I don't think it quite violated the Geneva Convention, my "pogrom" joke was in extremely poor taste. And I clearly went over the top in my reaction to MyVoice and nypoet22. The use of terms such as "Snowflake," "Adult's table" and the like is unacceptable. I really do hate Political Correctness but that's no excuse.
I was consciously thinking of the Borat character when I threw in the "pogrom" bit. But of course I am not Sasha Baron Cohen and while you'd expect something like this out of Borat you'd not expect it out of non-Borats such as myself.
As such, I want to apologise to my "family" here in Weigantia. For the record I am not unfiltered, but in this case my filters clearly failed me. I see the lesson here and I can (and will) be better than this.
Thank you. We now return you to your regularly scheduled programming
How about some Nov. 17th, 2020 Justice Matters with Glenn Kirschner and his Wife?
How about today's Keith Olbermann?
Hey, is this mic even ON?
On the other hand, Boys & Girls, "Is it really 'edge walking' if you don't go over the edge at least once in a while?
MtnCaddy [23]
Nicely put.