Picture this. You’re curled up in bed watching the nightly news…“Obama is leading in the polls!” says a pundit decisively. When a commercial comes on, you change the channel. “Romney has a massive lead in the polls!” says another pundit with confidence. Perplexed, you change the channel again. “Obama and Romney are dead even!” proclaims yet another pundit.
What’s going on? Why are poll numbers so different? Is it because the race is very close? Or because there’s something wrong with the way polls are measuring people’s opinions?
Well, let’s take a step back and look at how polling works. First, polling companies call up U.S. residents on their landlines or cell phones. Of these people, only 9% on average, agree to participate in the phone poll.
Why do we care about response rate? Well, who are these 9% of people who are agreeing to do a cell phone survey? And are they really a representative sample of the national population? Maybe. If they aren’t, why is this a problem? Well, if you’re not getting a representative sample of your population of voters, your prediction won’t be correct on Election Day.
Let’s look at a historical example of this very problem. For the 1948 presidential race, the Reader’s Digest predicted that Dewey would beat Truman. What happened? Well, the Reader’s Digest sent their survey to all their subscribers by mail, figuring that they had enough subscribers to make a good prediction. The problem was that most of their subscribers supported Dewey, whereas the rest of the country was much less enthusiastic about Dewey. An online survey might make a similar mistake by assuming that since lots of people use the Internet, its results are fine. In reality, it might only reflect the opinions of rich fashionistas who were emailed the survey link through a Chanel mailing list.
So how does a phone poll work around this problem? First, phone polls have the advantage of being able to call anyone in the 95% of the U.S. population who has a phone. This means that their sampling frame is nearly identical to the population. Phone polling firms also frequently weight their data to correct for any bias in their sample. Each polling firm, however, has its own weighting mechanism, and these can produce variant outcomes.
Another possible solution to this problem? Find a better way to reach more people! Enter, the internet and online surveys. Roughly 80% of households have internet access, and roughly 90% of workplaces do—and these numbers are growing everyday. Moreover, at SurveyMonkey, the average response rate to our online surveys is nearly triple that of the phone poll. So, online surveys can reach an only slightly smaller population of US voters, three times more quickly!
A unique opportunity. Every day, more than a million people in the United States access SurveyMonkey to fill out a survey. This heavy volume allows SurveyMonkey to measure day-to-day shifts in public opinion in a way that phone polls cannot. This means that a daily average will be strongly linked to whatever political events occurred the day before. Additionally, the topics of SurveyMonkey surveys vary widely, from market research to political surveys to party planning to academic experiments. This means that not only do many people take a SurveyMonkey survey every day—many different kinds of people take a SurveyMonkey survey every day! Consequently, we can be confident that we’re not just getting one particular kind of person, but a snapshot of the entire country.
So, what do we have here? This week, we’re unveiling our effort to estimate voting preferences in the upcoming election. In a polling field cluttered with phone polls, we are one of the only online polls to attempt this. We’ve collected nearly a million responses over the past two months and, so far, the data closely mirror the publicly available averages of phone polls. Here are some highlights of our results so far…
- Survey response rate: 30%
- Sample: 810,477 registered, likely voters
- Compared to phone polls: SurveyMonkey’s data is more volatile. This means that the numbers have gone both up and down over the past two months more often than the numbers in phone polls have. This means that SurveyMonkey’s data can show the impact of events like Romney’s RNC speech, Obama’s DNC speech, the leaked 47% video, and the debates on voter preference. This impact is much more difficult to see in the phone polls. To smooth our data and make it a little less sensitive, we use a 7-day average (that is averaging the numbers we get over the past week). This makes our data look very close to the phone poll data. (This makes sense, since phone polls all average their data over the course of anywhere between 2-12 days depending on the poll.)
- SurveyMonkey poll (as of 10/19): Obama, 47.1%, Romney, 45.2%
- Real Clear Politics average (as of 10/19): Obama, 47.1%, Romney, 47.0%
- Check out the all the details of our results so far in the PDF embedded below. Or, click here to access.
- Stay tuned to our blog for up to the day updates on the presidential race…