Pollsters want to be right so how did they get the 2020 election so wrong? | Opinion
I used to conduct surveys. Issues with reliability and changing technology have made polling so much harder and less precise as we saw with the 2020 election.
- William Lyons has worked as a political science professor at University of Tennessee and adviser to Knoxville mayors.
For the second straight presidential election year, it was a bad day for the pollsters.
In 2016, there had been a three-point divergence between the late election surveys. This year it was closer to 6% nationally and often higher in key states.
And the divergence was not random. The polls were off in the same direction.
I used to be a member of the pollster, or more properly, the survey research fraternity from academic, business, government and campaign perspectives.
I chose to abandon the latter because I could not justify all the shortcuts and workarounds needed to produce a reliable estimate of the general population.
Like any research tool, survey research’s validity relies on a couple of fundamental assumptions. It's all about identifying and then reaching a representative sample of a group of people you want to generalize to. It's one thing to feel comfortable about a survey of customers or hospital patients. But elections are tougher.
What margin of error means
There is ample theory and evidence to support the notion that one can generalize to a large population, including up to millions in national, state and local electorates, from samples of 1,600 or fewer people.
Some error occurs by chance. But one knows what that error range is.
When pollsters estimate that Biden will get 51% with a margin of error of 4% they are 95% sure that the estimate for all voters could be anywhere from 47% to 55% with 51% being the best prediction.
We could go with 90% certainty of its being between 49% and 53%. But 95% certainty is the custom.
This is the best we can do. It assumes that everyone had an equal chance of being included. If not, we introduce greater error, but we don’t know how to account for it.
Hear more Tennessee Voices:Get the weekly opinion newsletter for insightful and thought provoking columns.
So how do you assure that everybody has an equal chance to be included?
Changing technology made surveying people at home more challenging
Back in the day of landlines the strategy was straightforward. Almost everybody had home phones. And most people picked up the phone when it rang.
Interviewers did not just talk to those who answered because those who answer the phone do not necessarily represent all adults present. They also asked how many adults of voting age were in the household and consulted a random table to ask for a specific respondent.
After all of this, many people did not participate or hung up. All of this led to a valuable statistic: response rate. If it was too low, there were problems. Were those answering the same group as those who answered?
Then came answering machines and call screening. Then came mobile phones and extreme call screening. There is no need to belabor the point. In today’s world it's harder than ever to be confident that those surveyed represent the bigger population.
If not, there is a real likelihood of systematic error in addition to the random error.
Clearly the presidential election surveys suffer from systematic error as well as random error. This produces unintentionally biased results. So what is the nature of this bias?
Trump voters were severely underrepresented in polls
Maybe so-called “shy” Trump voters lie to surveyors. If so, that’s a small part of the problem. It’s more likely in the sampling.
Poll takers have regularly failed to reach Trump voters, who are clearly not as likely to answer their phones and if so, to participate in surveys.
And surveyors may underestimate a Trump supporter’s likelihood of actually voting.
Certainly one could make a case that the package of traits that would tend to lead to a Trump vote included a sense of distrust in media and related institutions and a heightened aversion to engaging a pollster.
The challenge of getting a representative election sample won’t end with Trump’s defeat. His voters remain and the challenges to their participation remain.
Survey professionals want to get it right. Election polling faces a reality check on election day. Credibility matters. It hurts to be wrong.
William Lyons worked as a professor of political science at the University of Tennessee and served for more than 16 years in a number of policy-related roles for Knoxville Mayors Bill Haslam, Daniel Brown, Madeline Rogero and Indya Kincannon.