Beware of surveys with small sample sizes, which have large margins of error

 

At the Idaho Freedom Foundation there is an article by Anna Miller on September 2, 2022 titled Idaho’s $330 million increase in K-12 education spending should fund students, not systems. Her fourth paragraph begins by claiming:

 

“Recent polling shows that 75% of parents support school choice policies like education savings accounts.”

 

She links to a web page at edCHOICE that is about Idaho. It has a donut chart that presents that 75% statistic, but it is for a sample consisting of just 50 School Parents (and a total of 98 persons). The claimed 75% is highly questionable, because for a sample of 50 people there only should be even-numbered percentages like 74% or 76% (75% corresponds to 37.5 people). There is a mauve advisory at the top of the page which reads as follows:

 

“The following data reflect rolling 12-month results through the most recent monthly General Population survey. We advise caution on interpreting the polling results for any state or population size having less than 100 respondents.”

 

How far can you trust that percentage? Wikipedia has an article on the Margin of error in surveys, which says that the maximum margin of error (in percent) for a sample size of n at a confidence level of 0.95 is 98/sqrt(n). A typical phone survey uses a sample with n=1,000, where the margin of error is 3.1%. For the School Parents with n=50 the margin of error instead is a much larger 13.9%!

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

How imprecise are these results for Idaho? As shown above, there is an impressive looking vertical bar chart on the web page for If given the option, what type of school would you select in order to obtain the best education for your child? But, when you replot as a horizontal chart, including the plus or minus for the margin of error, the only real difference is between the 37% for Regular public school and the 9% for Charter school.  

 

 

 

 

 

 

 

 

 

 

 

 

If we wanted to compare results for Idaho with those for neighboring states, what margins of error would we find? As shown above via a table, results for Washington (4.7%) are reasonable, but those for both Montana (11.7%) and Wyoming (17.6%) are pathetic.    

 

That web site has another page with trustworthy results for a much larger national sample (n=2,200 and a margin of error of 2.1%). When looking at surveys, don’t forget to check the margin or error. It might be worse than you think!