Observing Polls Inside-Out ~ #15 By Alan F. Kay, PhD © 2002, all rights reserved Dec. 9, 2002 A movie critic watches movies and rates them. The critic helps by going beyond viewing to include inside information on how movies are made. Similarly, previous columns all critiqued the making of polls, but today we switch to the inside story to tell you what you can learn by taking polls. I always agree to take surveys, whether by telephone, at a shopping mall, on the Internet, or wherever. Sure, I have a professional interest in seeing what I can learn about the polls of others, but you can learn something from this too. When you agree to "please, answer a few questions," your simple "OK" obligates the interviewer to answer a few questions from you — even nosey ones if you like. The interview wants a completion. You are in control. Turnabout is fair play and can be fun even for just a minute. What kinds of questions will teach you something? Well, how about "How long will this take?" "Are you calling internationally, all over the country, this state, or what?" "Where are you calling from?" "What organization are you with?" "Who is paying for this survey?" "Are you paid by completions or by the hour?" "How much?" You'll be surprised by some of the answers. Here's a good one if the interviewer asks a question that confuses you or is on an obscure subject, "Have you found others willing to answer that question — really?" Interviewers read questions, have nothing to do with their creation, and will enjoy agreeing with you. Of course, you can criticize any question you particularly don't like, but it is futile to dispute question wording. The poor interviewer can't defend any aspect of the survey he is reading. After an "agree/disagree", "approve/disapprove" or similar "either-or" question, you can respond with "it depends", or "neither" or "both". Bad polls don't allow such answers. Everything is black or white for them. You can get an idea of how good the poll is, simply by observing whether the interviewer records your answer. On the phone, how can you tell? Simple. Ask! The interviewer wants a completion. You are in control. To further understand how to tell good polls from bad, observe how the interviewer handles "Don't Knows." It takes most people a while to answer any long or difficult question, particularly if they are distracted. Try this one: Instead of a quick "I dunno," take your time. See if the interviewer rushes you through quickly and puts you down for "Don't Know", even before you say it. Good field houses go to considerable lengths to neither push you into one of the choices offered nor into "Don't Know". Those field houses that take the time to do it right get much more accurate results than those that do not. The effect on those who don't do it right is sometimes two to four times larger than the margin of error that most pollsters routinely include in all their survey reports and press releases. Sometimes called "the house effect," this issue is explored in column #14, "Dirty Little Secret of Polling", Nov. 11, 2002. As a sponsor, I also monitor polls. Field houses that conduct telephone polling, without permitting an outsider to speak, allow sponsors, but not others, to listen into interviews under a pretty careful protocol. There I get much more information about what is working (and what is not) in a survey I helped design. Partially from those insights, I learned a lot about how the supposedly foolish and ignorant public knows how to make smart choices of what is needed for governance. Yes, I get a lot of opportunities to take polls. I am on lists of "experts" of various kinds. I get four or five "market" research calls a day, some of which are more tele-marketing sales calls than research and, at election time, maybe 10 political calls which are largely pre-recorded. It takes me about five seconds to realize it's a recording and hang up. Not to mention 40 to 60 postal and e-mail solicitations, growing by 50% a year. Many of the election calls are only one or two questions, like "Are you voting for Smith or Jones on election day." Market research callers often after one or two questions, thank me and hang up. I have little interest in Pepsi vs. Coke questions or patience with brand-oriented surveys. Don't hang up on the poor interviewer. If you don't like polling, tell the interviewer that — with as much detail as you care to include, then hang up. Arianna Huffington, it'll take a second and you'll feel better. One way or another it will eventually help to make polls more useful for the people themselves who need to be heard much more and more accurately than leaders hear the public today. o - o - o - o - o - o - o - o Alan F. Kay is a mathematician, social scientist and pioneer of public-interest polling. He has authored Locating Consensus for Democracy and numerous public policy articles and holds several patents. (see www.publicinterestpolling.com)

Public-interest polling - what's that?    Top    Home    Search    CDI Home