This morning, the Gallup organization tried to explain why its polls of the 2012 presidential election turned out to be so wrong—Gallup’s final tracking poll indicated a 49 percent to 48 percent win for Mitt Romney, when in fact Barack Obama won by about 52-48 percent. But the problem wasn’t just the final poll. Throughout the race, Gallup’s polls tended to be outliers that favored the Republican candidate. As a point of comparison, here’s how Gallup’s daily tracking poll fared against the Obama data team’s internal tracking poll, which wound up being highly accurate (more on that tomorrow):
Once the election results rolled in and it was clear Gallup had blown it, the venerable organization (it has polled presidential races since 1936) committed to undertake a study to figure out where it had erred. The first results were released at this morning’s press conference (here’s the executive summary). Gallup examined 20 issues and concluded that four were chiefly responsible for its having overestimated Romney’s response. They were:
1) Their “likely voter” model was flawed. It relied more heavily than other polls on someone’s past voting behavior and “thought given to the election,” a variable meant to predict who would and wouldn’t vote that turned out to be not so accurate.
2) They blew the geography. By not interviewing enough people in certain regions of the country, specifically “the Eastern time zone within the Midwestern and Southern regions” and the “Pacific time zone within the Western region” (according to the report), they got a distorted picture of the race.
3) They messed up race and ethnicity. Gallup forced people to answer a series of yes/no questions about their race and ethnicity that didn’t yield an accurate picture, and then assumed that minorities would comprise a smaller share of the vote than they actually did.
4) They used a shortcut that didn’t work. For years, pollsters have relied on “random-digit dialing” (RDD) in order to reach both listed and unlisted numbers. This is expensive but, many pollsters believe, necessary since 45 percent of numbers are unlisted. Some recent academic research has indicated that it might be possible to get an accurate view of the electorate by scrapping RDD and combining (much cheaper) published phone numbers and cell phone users. (Huffington Post‘s polling editor, Mark Blumenthal, has a thorough explainer here.) Gallup attempted this, and it didn’t work.
To me, another point also jumped out from this morning’s proceedings: Gallup said that it hadn’t treated battleground states any differently than other states. Needless to say, the campaigns treated them much differently, running hundreds of millions of dollars of ads in those states that had a marked effect on voter sentiment.
Gallup’s Frank Newport explained that the point of the exercise was to improve performance in future races, and that their methodology was already being tweaked in light of the findings. “We’re in the process of modifying … how we conduct these models,” Newport said. “When the next presidential election rolls around … we think we’ll certainly be in a position to be at the accurate end of the spectrum, rather than in the uncomfortable position where we did not want to be this year, which is at the less accurate end of the spectrum.”
Long before Gallup decided to examine its methods, many others were questioning them. These included the senior members of the Obama campaign. Back in October, Joel Benenson, an Obama pollster, got so fed up with Gallup that he took them to task in this public memo. As David Axelrod, Obama’s chief strategist, pointed out to me today, “All of these were issues that were raised during the campaign, but met with stubborn resistance from the Gallup organization.”
Today’s report suggests that this resistance has begun to subside. Better late than never.
