Wall Street Journal: September 25,
Headline: When Wording Skews Results in Polls
Byline: Carl Bialik
[Professor's note: this article is intended as a supplement to the article "Choice Words" by Richard Morin in the Canon reader.]
Subtle differences in how poll questions are phrased, or in which choices are offered as responses, have a significant effect on polling results. Several recent surveys involving hot-button topics, including Americans' belief about Mr. Obama's religion, illustrate how a problematic poll number can take on a life of its own.
A number of polls regularly track approval ratings for the president and other key figures. Most polling firms, including Gallup, ask simply if the respondent approves or disapproves of job performance. But political polling firm Rasmussen Reports breaks out approval and disapproval into subcategories to allow respondents to qualify their answer to indicate whether they "strongly" or "somewhat" approve or disapprove of the president.
That distinction seems to make the "somewhat" category look like a moderate option, prompting some people who might otherwise register as approvers to disapprove, say polling experts. That makes job ratings for the president look worse under Rasmussen's lens than under other pollsters'.
In fact, the founder of Rasmussen Reports, Scott Rasmussen, recognizes this phenomenon. Last November, he even ran a test to demonstrate this effect. When offered just two options, more people approved of Mr. Obama's performance than disapproved. When offered Rasmussen's standard four options, the disapprovers won out.
Mr. Rasmussen says his method delivers finer-grain readings of the American public. And, he says, "it's healthy to have a large number of firms asking in slightly different ways." Also, like many pollsters, he is loath to change how he asks questions, because it would hurt his ability to make comparisons with earlier polls.
But he says the differences in polling questions sometimes are overlooked by pundits, who seize on the most extreme results. "One of the things that distresses me as a pollster is when a partisan says, this proves what I'm saying, and other polls are wrong," Mr. Rasmussen says.
How survey techniques affect survey results was on display in New York this week, when Republican gubernatorial candidate Carl Paladino did much better against Andrew Cuomo in polls that excluded Rick Lazio, who is running on the Conservative Party line. Another reason for the discrepancy: Mr. Paladino did better when polls were restricted to likely voters.
Even when possible answers don't get many votes, they can sway uncertain respondents. This was highlighted by two recent polls that asked about Mr. Obama's religion—in notably different ways. This summer, Pew Research Center asked, "Do you happen to know what Barack Obama's religion is? Is he Christian, Jewish, Muslim, Buddhist, Hindu, atheist, agnostic, or something else?" In response, 34% said he is Christian, which he is, and 18% said he is Muslim.
Time Magazine asked the question quite differently: "Do you personally believe that Barack Obama is a Muslim or a Christian?" Triggered by that wording, more people said the president is Christian—47%—and Muslim—24%—than in Pew's poll, where 45% of respondents were unsure or refused to answer, compared with just 24% in Time's poll.
In this case, as the number of options rose, fewer respondents chose to give definitive answers. Kenneth Rasinski, who conducts surveys of doctors for the University of Chicago's Department of Medicine, chalks this up to information overload. "A greater complexity of a question could result in greater 'don't know' responses," he says.
Question context also matters. In a March survey, Harris Interactive asked online respondents to rate as true or false 15 statements that have been uttered about Mr. Obama. All were negative or could be perceived as negative, including "he is a racist" and "he is anti-American." Among the most surprising and frequently cited results was that 14% of Americans believed Mr. Obama "may be the antichrist."
Seemingly absurd polling questions give respondents the opportunity to express hostility even if they don't literally believe their answer, says Tom Jensen, director of Public Policy Polling, a nine-year-old Democratic polling firm. "You may not literally think Obama is the antichrist, but [the question] gives you the opportunity to say just about the meanest thing possible you can say about anyone," he says.
The uniform tone of the questions also could have caused some respondents to answer uniformly, in what is known in polling science as flatlining, or responding to all statements identically: More than 40% said all were false, while 4% said all were true.
Howard Schuman, a retired University of Michigan sociologist who has studied polling methods, cautions against putting too much stock in any one survey. "Polls are pretty powerful," Dr. Schuman says. "Whether they should be or not is another question."