What factors affect the accuracy of a survey? What effect does random sampling have on predicting the outcome of an election? How does sample size affect the margin of error of polls?

Election Polls and Surveys
What is the validity of polls?

Insights

Reading or hearing about the results of a public opinion poll has become a common everyday experience for many. Everyone from political candidates to toy manufacturers seems to be trying to find out what people are thinking by polling the general population. The question is, can any of these surveys be trusted?

While many people don't think of surveys as being scientific, the implementation and interpretation of surveys belongs to a branch of mathematics called statistics. Professional statisticians use very strict rules in designing and evaluating these surveys, much like a physicist uses specific techniques to collect data from an experiment.

The two most important aspects of conducting an accurate survey involve random sampling and bias. Random sampling is a technique used to ensure that each member of a specific target population has an equal chance of being selected. There are a number of ways to get a random sample. A simple method would be to put into a hat many pieces of paper on which are written the names of the people in the target population. From this sampling, a certain number of names are literally "pulled out of a hat." This is the same technique used in lottery drawings in which numbered balls are randomly selected. The drawback is obvious: If the population is large, the mechanics of taking the sample become unwieldy. Today, computers can be used to generate a table of random numbers--an electronic version of "shaking up the balls."

The second important factor in surveys is bias, which can creep into the selection process of the person who is to answer the questions, or into the design of the questions that are to be asked. Selection bias usually occurs when a part of the target population is inadvertently excluded. One example would be selecting individuals from a town by drivers' licenses. Since some people don't drive, they would be automatically excluded from the target population.

Bias also becomes a factor by the way in which the questions are written and presented. The wording of the questions and the choices of the answers must be structured so that all questions and answers are of equal value.

Connections

1. What is the relationship between the size of a sample and the margin of error in a survey? Why does increasing the sample size give better results?
2. Most political opinion polls are conducted through telephone surveys of likely voters. Discuss if this kind of selection results in a biased survey. What criteria would you use to select individuals from your town for a survey that didn't have any selection bias?

Vocabulary

margin of error the plus or minus range of variation that the survey answers will have with respect to the true answers, if the entire population were asked the same questions
random sampling the process of selecting individuals from a population so that each member of the population has an equal chance of being included in the survey
selection bias the exclusion, often unintentional, of certain members of a target population because of the way the selection process has been established or conducted
statistics a branch of mathematics dealing with the collection, interpretation, and presentation of numerical data

Resources

Converse, P., and M. Traugott. (1986) Assessing the accuracy of polls and surveys. Science (Nov): 1094-97.

Lipset, S. (1980) Different polls, different results. Public Opinion (Aug/Sept): 19-20.

Mitofsky, W., and M. Plissner. (1980) Reporters guide to published polls. Public Opinion (June/July): 16-18.

Spence, J., et al. (1983) Elementary statistics. Englewood Cliffs, NJ: Prentice-Hall.

Additional source of information:

The Gallup Organization
P.O. Box 628
Princeton, NJ 08542
(609) 924-9600

Community resources:

Market research firm
Political parties

Main Activity

You will discover how bias in questioning affects the outcome of a poll. By asking two sets of questions, you might get two different conclusions.

Materials

• pencil
  
• plain paper
  
• graph paper

1. Prepare two questionnaires that will poll your schoolmates about their favorite foods to have in the school lunchroom. Design one set of questions with a minimal amount of bias (e.g., "On Mondays, which would you rather have: pizza, hamburgers, or fried chicken?"). Design the second set of questions so that they place substantial bias (e.g., "Wouldn't you rather have pizza for lunch instead of a boring hamburger?").

2. Select one grade level to be polled and get permission to poll two classrooms, one with the bias survey and the other with the unbiased one. Conduct the polling so that the two groups of students don't know that they are getting separate questionnaires.

3. Compile the results of the two polls and graph the results.

Questions

1. How did the results of the two surveys differ? In which survey were the answers more predictable?

2. What elements of bias may have entered into the selection of the students in this survey?

3. When professional research firms conduct surveys, they often ask for the same information two or three times, each time worded differently, during the course of the survey. Why might they do this?

Gather together a number of survey questions from magazines and journals. Analyze each of the questions for bias. Try rewriting the questions so that they have little or no bias in them.

Put yourself in the position of a corporate market-research specialist. Your company has just come up with a new fast-food idea but before you spend millions of dollars promoting it, you want to get some idea as to whether or not people are going to eat it. Develop a market research poll that not only describes the product, but targets specific parts of the population.

Recently, there has been a debate on whether news organizations should release the results of election exit polls from the East Coast of the United States while residents of the West Coast have yet to vote. Some people say this influences the election outcome, while others say there is no harm. What's your opinion?

How is probability related to polls and surveys? Place 50 pennies in a shoebox with a lid and shake them up. In theory, 25 of the coins should come up "heads." Count them and see if the numbers are close. Repeat the experiment four times and average the results. Compare the average values to the predicted values. How is the number of trials related to population size in surveys?

Newton's Apple is a production of KTCA Twin Cities Public Television.
Made possible by a grant from 3M.
Educational materials developed with the National Science Teachers Association.