Having Confidence in Surveys
For many years surveys and political polls were reported in the media as absolute facts. The media would say, "such and such candidate was chosen by 49% of the people."
Fortunately the media has been educated so they now include the margin of error, and sometimes they report the size of the sample. Today, the media is more likely to report, "such and such candidate was chosen by 49% of the people -- plus or minus 4 percentage points."
While that's an improvement, it's still not accurate -- the margin of error has a margin of error.
To be fully accurate, survey data should also mention the likelihood that the margin of error is accurate.
Many market researchers use a confidence level of 90% or 95% for a variety of statistical reasons.
Unfortunately, this additional bit of information is almost never done in the media these days.
What got me to thinking about this was a story by by The Associated Press on Yahoo News that did a great job of explaining this:
Results of AP Poll on Candidates
Sun Aug 8,12:39 PM ET
By The Associated Press
The Associated Press-Ipsos poll on the presidential race on based on telephone interviews with 1,001 adults, including 798 registered voters, from all states except Alaska and Hawaii. The interviews were conducted August 3-5 by Ipsos-Public Affairs.
The results were weighted to represent the population by demographic factors such as age, sex, region and education.
No more than one time in 20 should chance variations in the sample cause the results to vary by more than 3 percentage points from the answers that would be obtained if all Americans were polled. The margin of error for registered voters is plus or minus 3.5 percentage points.
There are other sources of potential error in polls, including the wording and order of questions.
Market researchers -- and the media reporting market surveys -- need to help readers understand how much confidence to place in relatively small surveys. When readers have confidence in survey data, they'll have a better appreciation for executives who have to make definitive, multi-million dollar, decisions using valuable, but uncertain, marketing intelligence.
Cliff Allen is the co-author of the book One-to-One Web Marketing; 2nd Ed., published by John Wiley & Sons, and has consulted with companies on strategic marketing for 20 years.