Twitter icon

Facebook icon

About this site
About us
Our beliefs
Is this your first visit?
Contact us
External links

Recommended books

Visitors' essays
Our forum
New essays
Other features
Buy a CD of this site
Vital notes

World religions
Christian def'n
 Shared beliefs
 Handling change
 Bible topics
 Bible inerrancy
 Bible harmony
 Interpret the Bible
 Beliefs & creeds
 Da Vinci code
 Revelation 666
Other religions
Cults and NRMs
Comparing Religions

Non-theistic beliefs

About all religions
Main topics
Basic information
Gods & Goddesses
Handling change
Doubt & security
Confusing terms
End of the World?
True religion?
Seasonal events
Science vs. Religion
More information

Morality & ethics
Absolute truth

Attaining peace
Religious tolerance
Religious freedom
Religious hatred
Religious conflict
Religious violence

"Hot" topics
Very hot topics
Ten Commandments
Abortion access
Assisted suicide
Death penalty

Same-sex marriage

Human rights
Gays in the military
Sex & gender
Stem cells
Other topics

Laws and news
Religious laws
Religious news

Sponsored links




!!!!!!!! Search error!  If the URL ends something like .htm/  or .htm# delete the character(s) after .htm and hit return.

Religious information

How to design opinion polls that lie.
The meaning of statistical terms used in polls.

"There are ... lies, damn lies, and statistics." 1

Sponsored link.


Most results announced by public opinion pollsters in recent years are quite meaningful, understandable and helpful. They are simple attempts to assess the public's opinion on a single topic. They are particularly useful to politicians who often gear their voting decisions to match either the majority opinion of the public, or the majority opinion of the voters within their party, or whichever vote they feel will maximize their chance of re-election.

However, when the topic of a poll is very controversial, or if the stakes are very high, some polls may be designed not for accuracy but to achieve a certain goal. Often the goal is not to estimate public opinion but to convince the public to change their mind, or to convince movers and shakers that public opinion is very different from what it really is. Currently, some polls about abortion access and same-sex marriage appear to have this type of ulterior motive.

Numerous U.S. national polls and polls within states have been conducted on same-sex marriage (SSM) by secular, political, and religious polling groups. Some have been sponsored by lesbian, gay, bisexual, transgender (LGBT) groups; others by religious or secular groups. There is often great variations in the results -- to the point that some wonder if fraud had taken place.

Some cooking of the results may have happened. For example, we have noticed one political polling group that asks their customers what their goals are, and then designs a poll and an explanation of the data to meet those goals. However, we suspect that fraud is generally not the cause of differences among polls. Sometimes, it can be the result of carelessness, ignorance and/or chance.

The following techniques can warp polling results; we should all look out for them:

  • The exact question asked: For example, the question: "Do you believe that marriage should be between one man and one woman" will produce very different results than "Do you believe that same-sex couples should be allowed to marry?" The respondent might answer the former question differently depending upon whether they are answering with regard to themselves only, or their family, or their faith group, or the entire population. Some people may respond differently depending upon whether they view marriage as a religious ritual, a religious sacrament, and/or as a secular ceremony at city hall.

    Changing the phrase "same-sex couples" to "homosexual couples" will probably also bias the result because many people use the term "homosexual" as a snarl word with heavy negative emotional overtones. Inserting a clause like: "change the definition of marriage to..." will probably raise red flags and lower the apparent support for SSM greatly. Similarly, changing the phrase "be allowed to" into "have the right to" may well increase support for SSM, because of the important that Americans place on individual rights.

    It would be fascinating to see a poll that randomly selects a question on SSM from a group of, say, five alternatives, and then reports on how the responses differ among the questions asked. We have never seen such a poll. They probably don't exist, because they would reveal to the public how much the exact wording of the question does matter. That might well lower the credibility of polling in the public's eyes.

  • Questions asked prior to the question on SSM can set up the respondent to answer in a desired way. See the example below.

  • Simply by phoning only land lines, or only cell phones, or some ratio of land lines and cell phones, can result in an entirely different age distribution of respondents, and thus shift the end results.

  • By simply phoning land lines during normal working hours on weekdays will shift the age distribution towards older, retired adults, and thus shift the end results without the bias being too obvious.

  • Lacking data: Some polls are reported without the number of respondents and the margin of error. This makes the data essentially meaningless. See the discussion below.

  • A polling group can simply propose many alternatives to the respondents. For example, ask them which arrangement they would prefer:
    • For same-sex couples to be allowed to marry, or
    • To enter into a civil union with all of the benefits, protections, and responsibilities of marriage except that they can't refer to their relationship as a marriage, or
    • To enter into a domestic partnership with some of the benefits of marriage, or
    • No state recognition of the relationship at all.

Giving many options would be almost certain to result in only a minority of respondents favoring same-sex marriage. The polling group could then accurately, but deceptively, describe the poll as indicating that most adults oppose SSM. What they don't reveal is that each of the options is opposed by most of the respondents.

horizontal rule

An example of bias in the design of a public opinion poll:

Two large polls on the topic of same-sex marriage were conducted in Canada in the early '00's by different polling companies. The results were obtained 3 and 4 years before access to marriage was widened across that country to include all loving, committed same-sex couples:

bullet 2001-JUN: A Canadian Press / Leger Marketing survey found that 65.4% of Canadian adults supported same-sex marriage rights.

bullet 2002-JUN: Strategic Counsel conducted a poll on behalf of the fundamentalist Christian group Focus on the Family Canada. They found that 46% of Canadian adults supported same-sex marriage. 2,3,4

There are suggestions that support in both Canada and the U.S. for same-sex marriage was increasing during that decade by about 1.5 percentage points a year. Why did these two polls show an almost 20 percentage point drop in support over 12 months?

A major component of the reason may be found in the questions that were asked before the question on SSM:

bullet In the Canadian Press survey, Question 2 asked "In your opinion, should homosexuals have the same rights as heterosexuals?" 76% of the subjects responded "Yes." There is no record in their report as to the nature of Question 3. However, Question 4 was worded "In your opinion, should we grant homosexuals the following legal rights...same-sex marriage?" Since more than three out of four subjects already said that they were in favor of equal rights for gays and lesbians, many would probably feel pressured to affirm same-sex marriage. Otherwise, they might feel that they would be judged inconsistent in their views. So this poll probably overestimated support for SSM significantly.

bullet In the Focus on the Family Canada survey, the two questions (#59 and 60) that preceded the question on same-sex marriage (#60) were whether "marriage is an outdated institution" and "Is it alright for married people to have sex with people other that their spouse?" Those questions may have preconditioned the subjects to feel defensively towards marriage just before they were asked about their support for same-sex marriage. This poll probably underestimated support for SSM significantly.

Please note that we are not implying that the bias in these polls was intentional. It may have been the result of incompetence, carelessness, or simply a random effect caused by the way in which the  questions were sorted.

More details on these polls.

horizontal rule

About the accuracy of polls and the importance of the margin of error:

One term associated with polling is the "true value:" This would be the value that one would obtain if every person (or, more likely, every adult) in the population was asked the question. A typical poll only involves 1,000 to 2,000 respondents. Thus, for state polls, it is only practical to take a sample involving a small percentage of the total population. The end result will be an estimate of the true value; the actual true value will be forever unknown.

It is impossible to understand the accuracy of a poll without knowing its margin of error. A typical margin of error value in professionally run polls is ±3 percentage points and involves a sample size of about 1,000 respondents.

Most margins of error are calculated at the 95% confidence level. In the case where the margin of error is ±3 percentage points, this means that one can have 95% confidence that the true value is within 3 percentage points of the reported percentage; anywhere from 3 percentage points higher to 3 percentage points lower than the reported percentage.

Thus if:

  • 52.7% of the respondents in a particular poll are found to be in favor of the legalization of SSM, and
  • the margin of error is ±3.1 percentage points at the 95% confidence level, then

One could say with 95% confidence that the true percentage of adults who favored the legalization of SSM is between 49.6% and 55.8%. 5

The margin of error is mainly a function of the number of respondents who actually contributed their opinion to the poll -- often referred to as "N." In cases where the total population is less than 20,000, the size of the population also becomes an important factor. Fortunately, the smallest state poll would be in Rhode Island which has a population of a little over 1 million. Thus the total population is not important in determining the accuracy of state polls; only the size of the sample is.

American Research Group, has a free Margin of Error Calculator 6 that computes the margin of error for a given population and sample size. They also have a free Sample Size Calculator that computes the required sample size among a given total population in order to achieve a desired margin of error. 7

For a large population like a city, state or province, or country -- even Monaco -- the actual population doesn't have a significant effect on the margin of error and the following relationship between the size of the sample "N" and margin of error for 95% confidence is:

Size of sample "N" Margin of error
±2.17 perc. points
±3.08 perc. points
±4.37 perc. points
±6.19 perc. points

A sample size of about 1000 and a margin of error of ±3.1 percentage points is very commonly found among polling companies.

horizontal rule

Related essay:

horizontal rule

References used:

The following information sources were used to prepare and update the above essay. The hyperlinks are not necessarily still active today.

  1. Variously attributed to Benjamin Disraeli, Alfred Marshall, Mark Twain, etc.
  2. "Canadian Perceptions of Homosexuality," Canadian Press / Leger Marketing, 2001-JUN, at: http://www.legermarketing.com/ This is a PDF file. Survey data: N = 1507. Margin of error is ±2.6%
  3. "U.S. Public opinion polls on same-sex marriage," Marriage Equality New York, at: http://www.marriageequalityny.org/
  4. "Canadian Attitudes on the Family: Complete Report 2002," Focus on the Family Canada at: http://www.fotf.ca/ This is also a PDF file.
  5. "Margin of error," Wikipedia, at: http://en.wikipedia.org/wiki/
  6. "Margin of Error Calculator," American Research Group, at: http://www.americanresearchgroup.com/
  7. Sample Size Calculator," American Research Group, at: http://www.americanresearchgroup.com/

horizontal rule
Site navigation:

Home > Homosexuality > Opinion Polls > here

Home > Religious information< > here

Home > Vital notes > Other important topics > here


horizontal rule

Copyright © 2011 by Ontario Consultants on Religious Tolerance
Originally written: 2011-JUL-02
Latest update: 2011-JUL-02
Author: B.A. Robinson

line.gif (538 bytes)
Sponsored link

Go to the previous page, or go to the "Polls about homosexuality" menu or to the "Other important topics" menu, or to the "Religious information" menu, or choose:

Web ReligiousTolerance.org

Go to home page  We would really appreciate your help

E-mail us about errors, etc.  Purchase a CD of this web site

FreeFind search, lists of new essays...  Having problems printing our essays?

Google Page Translator:

This page translator works on Firefox,
Opera, Chrome, and Safari browsers only

After translating, click on the "show
original" button at the top of this
page to restore page to English.




Sponsored links