Polling Critic #50 - Galluping to the Wrong Conclusion #50

Galluping to the Wrong Conclusion ~ #50
By Alan F. Kay, PhD
© 2005, (fair use with attribution and copy to authors)
May 18, 2005

David W. Moore, managing editor of the Gallup Poll and a senior analyst at the Gallup Organization, conducted a poll on euthanasia and assisted suicide over the four days of May 2-5, 2005. The poll findings were based on these two questions:

Q1. When a person has a disease that cannot be cured, do you think doctors should be allowed by law to end the patient's life by some painless means if the patient and his family request it? Should Should not DK
75% 24 1

Q2. When a person has a disease that cannot be cured and is living in severe pain, do you think doctors should or should not be allowed by law to assist the patient to commit suicide if the patient requests it? Should Should not DK
58% 39 3

Moore, in the Gallup News Service May 17, 2005, explains the big difference between the responses to these two questions this way, "The apparent conflict in values appears to be a consequence of mentioning, or not mentioning, the word 'suicide'." Wording of the two questions differ, not just by whether they mention 'suicide," but in several ways:

There are three phrases in Q1, not in Q2: A₁: "end the patient's life," B₁:"by some painless means," and C₁: "if the patient and his family request it."

There are also three phrases in Q2, not in Q1: A₂: "and is living in severe pain," B₂: "or should not," C₂: "by law to assist the patient to commit suicide."

Moore's conclusion that the suicide mention in Q2 and not in Q1 is the reason that the "should" response of Q1 drops a very significant 17 points from Q1 to Q2, (even though the sample sizes are rather small, 516 in Q1, and 489 in Q2). It may well be true that this is the case.

However, this is not the only explanation for the 17 point drop. Both B_1, painless means, and C_1, patient and his family, in Q1 definitely bring out personal or humane advantages for the patient over Q2 and might well play a significant role in the 17 point drop.

For Moore to prove his case, he should use a new pair of question wordings, say Q3 and Q4, that differ only by the "suicide" phrase. To make the case even stronger, a further question pair, Q5 and Q6, should be tested which differ only by the presence of the phrases of B₁ and C₁ and, in addition, either both include the suicide concept or both exclude it.

I'll bet my money that a good portion of that 17 point drop is due to those phrase differences. Even if I am wrong, Moore failed to prove his case when he arbitrarily excluded differences between Q1 and Q2, other than the suicide phrase in Q2, from significantly affecting the findings.

Gallup has tested question pairs, identical or very close to Q1 and Q2, annually for years. This historical data shows a drop often less than 17 points. Moore has also examined this year's results further by breaking out several interesting demographic response differences. It is unfortunate so much has been examined that may not rest on solid ground.

Why does this happen? My guess is that the Q1 and Q2 wording seemed very natural and reasonable to the designer(s) and over the years was a standard that never got examined as fully as this column #50 has done. I think it unlikely that anyone felt hands were on the scale when formulating the questions. But . . .