Polling Critic #51 - Galluping On with Sloppy Question Wording

Galluping On with Sloppy Question Wording ~ #51
By Alan F. Kay, PhD
© 2005, (fair use with attribution and copy to authors)
May 29, 2005

Frank Newport, editor in chief of the Gallup Poll and (like David Moore, featured in Column #50) a vice president of the Gallup organization, has conducted surveys with multiple deficiencies in question wording that produce misleading implications. Q1 is the first example

Q1. Just your opinion, do you think that Charles Darwin's theory of evolution is [ROTATED: scientific theory that has been well-supported by evidence, (or) just one of many theories and one that has not been supported by evidence], or don't you know enough about it to say?]

"Just your opinion" that starts off the question, and the alternative choice that ends it, "or don't you know enough to say", both disempower significant groups of respondents. People who feel that they are knowledgeable and want their responses respected hear "Just your opinion" as something else, that Frank Newport doesn't expect them to be able to give a knowledgeable response. Some may disconnect (i.e., hang-up). Mostly they then don't care much for what follows and don't bother to think about what they would otherwise think more carefully about, and finally some may just choose, "Don't know enough to say". When a somewhat similar large group also hears, "don't you know enough to say" at the end, they too opt for that choice because they are intimidated and disempowered.

This phenomenon of surveys have been investigated at great length (See, Locating Consensus for Democracy – a Ten-Year U.S. Experiment, p.48 or Chapter 7, "Down Boy”, pp.61-68 of Spot the Spin, The Fun Way to Keep Democracy Alive and Elections Honest.) Tests of questions with and without the introductory phrase and other tests with and without the ending phrase make a big difference in responses.

Some of this is obvious on its face. If the "Don't know enough to say" option were not offered in Q1, such responses in Q1 would have been zero instead of 29% in 2004 and 25% in 2001. There are some minor exceptions here. If "volunteered" responses were allowed and recorded, these responses would go in what I call the DK bin. That takes care of both choices, "Don't know enough to say," and "Don't know, for whatever reason". The latter in Q1 is only 1%, both times, when it was asked in 2004 and in 2001. The only significant change in responses in almost four years between askings is a four point increase in "Don't know enough to say". This suggests that the groups that want to think for themselves and not be intimidated or disempowered, have been growing.

There is another serious deficiency in Q1. The confusion of what Darwin's theory is and what it means has evolved in a unique way over 150 years. There are at least two prominent and widely known understandings of Darwin's theory. No survey question can produce a very useful result without clarifying that distinction.

Darwin's studies and books made biology a science, introduced the evolution of species as a concept with broad implications, and produced a new understanding of the role of humans among all other life forms. Darwin's concept of "survival of the fittest" has been widely interpreted in plain language several ways. One is "dog eat dog". Another is species below on the food chain are eaten by those above and man is at the top of the food chain. This leads to the belief that the tough, the strong, and the ruthless survive.

That view overlooks much of reality. Those who survive and live to produce surviving descendents are often the fittest in an entirely different way: attracting mates, caring for children, providing for their families, and generally helping (and being helped by) others.

Victorian English elites, needing to control the masses, embraced the brute force view of "survival of the fittest", coining the term "Social Darwinism." Darwin scholars are currently re-appraising this distortion and suggest that Darwin also believed that humanity's success was due to an ability to bond, trust, cooperate, and behave altruistically (see www.thedarwinproject.com).

Between, the two meanings of "Survival of the Fittest" and other aspects of Darwin's seminal work, especially the origin of species and the hundreds of magnificent developments in biology based on Darwin's observations of evolution, any question using Darwin's name needs more context to make the question clear and make significant respondent groups comfortable with answering that question.

Another question, Q2, was asked by Gallup VP Frank Newport, is related to Q1 by subject matter and further illustrates these points:

Q2. Which of the following statements comes closest to your views on the origin and development of human beings – [ROTATE 1-3/3-1: (1) Human Beings have developed over millions of years from less advanced forms of life, but God guided this process, (2) Human beings have developed over millions of years from less advanced forms of life, but God had no part in this process, (3) God created human beings pretty much in their form at one time within the last 10,000 years or so]? Results in the following table:

                       Man developed over millions of years       God created man     Other,
                            with God     but God had no part         in present form            No
                            guiding                 in the process                                          Opinion

2004 Nov 7-10       38                         13                              45                       4
2001 Feb 19-21      37                         12                              45                       5
1999 Aug 24-26      40                             9                              45                       4
1997 Nov 6-9          39                         10                             47                       7
1993 Jun                  35                         11                              44                       7
1982                        38                             9                              44                       9

Even those who are complete believers in evolution might disagree with the characterization of the evolution of human beings from other species as described in both the first and second options and might refuse to accept either choice, perhaps discontinuing or choosing DK (here called by Newport "Other, No Opinion".) DKs in Q2 here are not negligible, ranging from 4% to 9% (as compared to 1% in Q1). Others believing in million-year evolution might have failed to accept either choice because neither "God guided the process" nor "God had no part in the process" reflected their beliefs on how God manifests Himself.

Conversely many in the plurality that opted for the third choice might believe somewhat in evolution while also believing in the Christian God. For example, such people could imagine that some or all early humans were in a sense "created" less than 10,000 years ago when in various ways some were changed by God more-or-less as described by the Bible.